The Justice Data Lab (JDL) is the realisation of an idea originally proposed by the charity think tank New Philanthropy Capital. In April 2013, the Ministry of Justice (MoJ) established the JDL to improve the evidence base available on successful rehabilitation - giving organisations working with offenders secure access to reoffending metrics. Organisations can then use this information to better assess the impact of their work on reoffending. The JDL was formally established following a two year pilot, and is being used as a template for other data labs across Government.
What a data lab can and cannot do
The purpose of a data lab is to provide people and organisations with meaningful information and analysis that will help them develop and deliver their services in the real world. The data lab is not there to tell them what to do, instead it presents as full and accurate a picture of the evidence available as possible to make sure these organisations can make their own informed decisions about the focus of their work in the future. An organisation’s recognition of these limitations is essential to them understanding and benefiting from the data lab’s work.
The information used in the analysis must be of sufficient quantity and quality to draw conclusions. Data labs, like the JDL, essentially compare two types of people, first, the treatment group – which comprises a user’s clients and, second, a control group made up of people, similar in background, circumstances or experience to the control group, who have not worked with the organisation in question. The smaller the number of people in the treatment group, the more likely it is that any difference between the treatment group and the control group will be down to chance.
Therefore, the JDL requires organisations to provide details of at least 60 people with which they have worked. It will then only proceed with an analysis, if - following further checks – they can form a treatment group of at least 30 people from the original list. Organisations are advised not to exclude people selectively from their records, as this could limit the number in the treatment group and bias the final results.
It’s important to remember that, even with the most careful analysis, the results provided by a data lab should always be seen as evidence, not proof of something. Why? Because, there is a 5% chance of wrongly concluding that there is evidence of an effect when no effect actually exists. This can be higher, if information needed to effectively compare the treatment and control groups is not available. Take for example, an offender’s willingness to change. This could be down to a number of factors, beyond the support they received from the organisation being assessed. As such, JDL reports include caveats, which highlight gaps or information that could not be analysed, but which the JDL believes or knows could have affected the selection of the treatment group.
Data lab results are most valuable when placed within a wider context and considered alongside other information. For example, reoffending statistics form just one of the many ways to judge success in offender rehabilitation. You might also choose employment, mental health or self-worth as further pieces of the picture.
A person’s confidence in a particular result also depends on their assessment of all other relevant evidence they have encountered. This means that, when an organisation needs to justify its rehabilitation programme, it is not enough to declare that a significant result has been found.
They need to look also at all of the other information and analysis available to them, for example evidence about similar treatment programmes, to build as strong a case as possible.
How the Justice Data Lab works
So, if you think you can benefit from working with the JDL, what do you need to do? Once you’ve requested assistance, the JDL will ask you to provide details of the people you have worked with and information about your intervention or programme to better understand how it works to inform the analysis.
The JDL team will then match these details against data available to the MoJ to verify their background and offending history. The team then uses this information to assemble a control group of other similar offenders who are closely matched to the treatment group on a number of personal characteristics: including (but not limited to) age, employment status and criminal history.
This is a crucial step, but difficult for organisations to do without access to the Police National Computer (PNC). It aims to ensure a process in which all differences between the two groups have been minimised (except one, that the treatment group undertook the programme being tested and the control group did not), to allow for as meaningful a comparison as possible.
The JDL will exclude offenders who cannot be matched to the PNC or administrative datasets from the treatment group, as well as those for whom similar control group members cannot be found. The groups are matched as closely as possible, by considering a range of characteristics that affect the probability that an individual would reoffend or enter the programme being tested.
The JDL then examines whether the treatment and control groups are significantly different from one another, primarily using three quantities as proxy measures of success:
- the proportion of each group who committed a proven reoffence during the year following their release from prison or the start of their probation order, normally called the ‘reoffending rate’
- the average number of reoffences committed per person during that year
- the average number of days to first reoffence among those who reoffended within the year
If an organisation is having a positive impact on reoffending, it is expected that the reoffending rate and average number of reoffences will be lower and the average time to first reoffence will be longer.
The differences between the treatment and control groups are assessed using a process called significance testing, which tests the assumption that the programme makes no difference to the measure and that any difference between the groups is down to chance.
If the answer is no, then the assumption is rejected. We can be confident that there is a real difference in the reoffending rates of the treatment and control groups. If the answer is yes, then the assumption is accepted. This does not mean the organisation is having no impact on the reoffending rate of its participants, but that there is insufficient evidence to draw a definite conclusion about that impact.
The results of the JDL’s analysis are published in a report that is freely accessible online. This is designed to be easily understood and used, without omitting important details or caveats. The summarised results from all analyses conducted to date are also published regularly and are grouped into categories such as ‘education’ and ‘accommodation’ to give a broad overview of the success levels of different types of programme. Crucially, information on individual offenders is never revealed publicly and even the organisation that provided the details cannot personally identify those who have reoffended. If any statistics could be used to identify individuals, those statistics would not be released.
The Justice Data Lab so far
The team has already produced almost 130 reports and continues to work with a variety of organisations from the public, private and voluntary sectors to interpret their data and better meet their needs.
A user feedback summary, released in June 2015, revealed that two-thirds of JDL users had made changes to their offender rehabilitation programme as a result of their report. Following the recommendations of this feedback, the team has introduced additional measures such as reoffence severity. It is also exploring the use of information from the Offender Assessment System to form more closely matched control groups, particularly for programmes which treat people with specific characteristics that are not recorded on the PNC such as mental illness or homelessness.
The JDL has analysed reoffence data for treatment groups containing tens to thousands of individuals. Where the treatment group is smaller, the difference between the treatment and control groups is less likely to be significant.
However, nearly a fifth of requests where a significant result was found had smaller treatment groups (between 30 and 100). The team is committed to working with organisations of all sizes, including smaller organisations and those carrying out specific programmes. As demonstrated by user feedback, the results help to direct future rehabilitation efforts, even when they are not statistically significant. Meanwhile, a detailed picture of what works in preventing reoffending is gradually emerging.
The JDL aims to bridge the gap between those who work with offender data and those who work with offenders themselves. The team is keen to spread the word about this free service, and encourages anyone wishing to enquire about it to contact the team by e-mailing justice.datalab@gsi.justice.gov.uk or by calling 0203 334 4770.
Recent Comments