Posts

Magnifying glass over the word analyse to illustrate analysis of TR PbR Figures

TR PbR Figures – Context and Drivers

This blog post summaries a presentation I gave at our recent event: Transforming Rehabilitation: Learning from the PbR results.

 

Overall results  

In this blog, I will discuss the data I presented at our recent event, ‘Transforming Rehabilitation: Learning from the PbR Results’. The event was held shortly after the publication of the first reoffending figures which showed that reoffending in the first cohort had been reduced by 1.9%. My colleague Jack Cattell has already discussed these figures in an excellent blog, so this blog is about the how external drivers across the criminal justice system might be affecting these results.  

By CRC? – PbR Figures, who is in control of reoffending? 

In order to make conclusions from these results, one must be aware that there are many factors that can influence reoffending. When faced with (improved or reduced) performance results – it’s too easy to fall into the trap of thinking that the reasons for this were outside of your control, or that the results were dominated by one particular driver.  

In understanding the reoffending rates by Community Rehabilitation Companies (CRCs), there are many possible drivers across the CJS, and they can be complex to understand. So, could the CRCs have achieved the difference in performance we have seen under TR? Internal factors such as CRC procedures and workflows, and offender profiles will have an effect.  But so too will a myriad of external influences from other agencies of the CJS, for example police positive outcome rates, court conviction rates and timeliness of police & court procedures. Since these factors are outside a CRC’s control, I wanted to investigate and determine whether any might have had an effecting change in the reported reoffending rates.  

National context 

Chart showing - No. of offences thousands

First off, it is important to bear in mind the national context for the period from baseline to first cohort results, which was a time when the total estimated crime fell and police recorded crime remained constant.

 Chart showing proportional change from 2011

Further, positive outcome rates, specifically charges and cautions had been reduced – in general police arrest rates were down and police had been using other disposals (such as community resolutions) to reduce the number of people – particularly young people – in custody.

 

Chart showing days from offence to completion

And individuals were being processed more quickly during the baseline: on average court processes are now taking 10-15 days longer.

Give this context, the national reoffending rate was reduced by 1.8%. That’s a surprisingly resistant figure, given the changes in context nationally. Large changes in this binary rate will not be seen regardless of policy or systematic change in CRCs.

Chart showing reoffending rate

(Of course, the full picture of the reoffending stats has the rate down, but the frequency up – suggesting a shrinking group of more prolific reoffenders. But that is for another post).

Differences in CRCs?

I will now look at these potential drivers of reoffending rates at the CRC level and determine if contexts really were different or helped to drive different performance. To highlight any of these potential differences, I will look at two CRCs who were at opposite ends of the reoffending rate changes.

What is surprising, is that there are no large, obvious differences between those two CRCs when comparing police outcome rates, court effectiveness and court timeliness measures. For example, the police positive outcome rates were reasonably different at the start of the period, but by the end of 2015 had converged to be broadly similar – the best performing CRC had reduced reoffending while police positive outcomes fell by a significant amount.

Chart showing positive outcome rate

Similarly, the lowest performing CRC had raised rates against the baseline despite court timeliness both increasing, and being higher than other CRCs.

So What?

We’ve seen that reoffending rates are complex measures dependent on many factors, from individual, to regional to national level, and the interaction between them. The rates don’t change very much, even though the contexts can differ wildly.

Even comparing the CRCs at either end of performance spectrum there were not huge differences in the police and court factors. This can be seen in different results for CRCs operating in widely similar contexts.

So, my advice is this: don’t fall into the trap of feeling the results are not within your ability to affect. Understanding the wider picture is helpful to contextualise local results. In the following blogs, my colleagues will be focusing on what is in your control, and my colleagues will be identifying what works and using that to inform good practice.

Get involved in the conversation by joining our LinkedIn group.

TR: PbR results - speakers who presented to Community Rehabilitation Companies and other justice sector employees

Community Rehabilitation Companies: PbR Results Event

Transforming Rehabilitation is the UK government’s programme of outsourcing probation services to new community rehabilitation companies. In a radical move, the government is now paying these new companies by the reduction in reoffending results they achieve. GtD is at the forefront of this by providing our cutting-edge social impact analytics to Sodexo Justice Services who manage a number of these new companies.

The first PbR figures were published last month and GtD has been active in informing the debate on their significance. As part of this debate, we recently hosted a sell-out event for senior management and practitioners working in community rehabilitation companies and the justice sector.

An expert panel comprising Prof. Darrick Jolliffe of Greenwich University (above left), Dr Sam King of Leicester University (above right) and GtD’s own Jay Hughes (above centre left), considered the initial findings and what to do next, with Jack Cattell (above centre right) setting out a new vision of how predictive analyses can be used by practitioners to improve performance.

Prof. Darrick Jolliffe – University of Greenwich

If you were unable to attend but would like to learn more about how GtD could support you in evaluating your social impact outcomes or for a free predictive analytic roadmap for your CRC, contact Jack Cattell  The event presentations can also be viewed via the link below:

Transforming Rehabilitation – Learning from the PbR results presentations

Dr Sam King – University of Leicester

 

Jack Cattell – GtD

 

We’ve also set up a LinkedIn group as a forum for shared learning and discussion, for individuals who work or have an interest in, the fields of probation, offender rehabilitation and Transforming Rehabilitation. Click here to request to join – Transforming Rehabilitation

Scales of Justice representing Transforming Rehabilitation

Transforming Rehabilitation: Payment by Results Figures

Last week saw the release of the Transforming Rehabilitation (TR) Payment by Results figures for the October to December 2015 cohort.

The overall result was encouraging, and defy the view that Transforming Rehabilitation’s radical changes to probation, and the ensuing problems, would result in increased reoffending – though it is very important that I point out that this is just the first set of results of many and overall judgement should be reserved for at least a year. The reoffending rate for all CRCs was 45.6% compared to a 2011 baseline rate of 47.5%. I had to make some (conservative) assumptions to estimate the baseline rate but I think it is also safe to say that the difference was statistically significant, suggesting reoffending rates have reduced under TR. Please see the the note at the end of this blog to understand better how I completed the analysis.

 

Transforming Rehabilitation – CRC Performance

The chart below describes each CRC’s reoffending rate in relation to the baseline 2011 rate. The grey line represents the range of reoffending rates that would indicate no change from 2011 (the baseline confidence interval). If the CRC’s rate is outside this range, we are confident in statistically terms to state that the CRC’s performance was either better or worse than the reoffending rate achieved in 2011. The green bars represent the reoffending rates of CRCs that outperformed 2011, the orange bars represent those that performed the same as 2011 and the red bars present those that performed worse than 2011.

Chart to show CRC Performance

Source: Ministry of Justice Final Proven Reoffending Rates TR (Oct to Dec 2015 cohort).

Thirteen of the CRCs beat the baseline rate. The best performing CRC was Cumbria and Lancashire, which beat the baseline rate by 8.2% (49.9% to 41.7%). The nest best was Hampshire and the Isle of Wight which beat the baseline by 5.4% and the third best was Northumbria with a better rate by 4.3%. Two of the CRCs performed worse than the 2011 baseline. Warwickshire and West Mercia recorded a reoffending rate 3% worse than the baseline rate, and South Yorkshire’s rate was 2.8% worse. With most CRCs, however, outperforming the reoffending rate form 2011, the figures are a promising set of results.

 

Transforming Rehabilitation – Comparing CRC performance

Now that the baseline rates have been published, we can better understand how well each area was performing in 2011 and whether a CRC is now being asked to better good or bad performance achieved in that year. The chart below describes the difference between the actual baseline rate and the 2011 baseline’s OGRS score (in other words their expected rate of reoffending). A negative result in the chart means the area performed better in 2011 than the OGRS score expected.

Chart showing the difference between a CRC's actual baseline rate and the 2011 baseline’s OGRS score

Source: Ministry of Justice Final Proven Reoffending Rates TR (Oct to Dec 2015 cohort).

The charts highlights that six of the CRCs are being asked to beat better than expected performance in 2011 (in other words to be better than good). Whereas other CRCs, notably London and Wales, are being asked to outperform potentially poor performance in 2011. It it interesting that South Yorkshire and, Warwickshire & West Mercia – the two areas that recorded poor performance for TR – are being asked to beat good performance from 2011. Merseyside and Cheshire & Greater Manchester, however, are equally being asked to beat good performance from 2011 and were able to do so for the October to December 2015 cohort. The OGRS score does not allow for area effects, which will exist and could explain the differences between the OGRS score and the baseline rate. It not possible now to conclude whether payment by results will be easier in some areas than others, but, going forward, I will monitor the impact of whether a CRC is being asked to perform better than good or poor performance from 2011 on their ability to achieve payment by results bonuses.

 

Notes on analysis

The latest Ministry of Justice bulletin released more data than was previously available and I was able to complete a statistical analysis of the impact of TR. This could only be completed with making conservative assumptions that would make finding a statistically significant result less likely. The following actions were taken:

  • I assumed the spread of offenders across CRCs in 2011 was exactly the same as it was in the October to December 2015 cohort. This would not be the case but any analysis would want to weight the two samples so they represented each other so the impact of this assumption is minimal.
  • The 2011 sample size was assumed to be the same as that of the October to December 2015 cohort. The 2011 sample will be considerably bigger, so this assumption meant the standard error used for the analysis was larger than it should be.
  • A t-test with unequal variances assumed was used to test the difference between the cohort’s and the baseline’s reoffending rate. The t statistic result was 4.6.

Blog originally guest posted on http://www.russellwebster.com 31st October 2017 with additional commentary from Russell Webster.

Predicting the Final CRC Re-Offending Rates

Predicting the Final CRC Reoffending Rates

Predicting the Final CRC Reoffending Rates

On October 26th 2017, the Ministry of Justice will publish the first Transforming Reoffending proven reoffending rates. These will describe the October to December 2015 cohort’s proven reoffending rate and compare this to a 2011 baseline rate (go here for an explanation). If a Community Rehabilitation Company (CRC) does better than the baseline rate they will be paid a bonus. In this blog I am using descriptive statistics to present a prediction of what the final rates will be.

England and Wales Reoffending Rate

Over the last year the Ministry of Justice has published this cohort’s, and each subsequent cohort’s, reoffending rate every 3 months. The national reoffending rate across all CRCs is described in the figure below (these figures are adjusted for differences in likelihood of reoffending across cohorts). Each bar represents a different cohort and gaps exist where the data have not yet been published.

 

Ministry of Justice Proven Reoffending Quarterly Statistics

Source: Ministry of Justice Proven Reoffending Quarterly Statistics

Definition: Months after commencement is the minimum number of months after commencement for an offender in the October 2015 to December 2015 cohort. However in the figures published, the follow up period varies depending on when within those 3 months the offender started.

 

The results are similar across the cohorts. After 8 months, 33% of offenders will have reoffended, and then 39% after 11 months, 42% after 14 months and 43% after 17 months. Most offending will occur within the first 8 months and the subsequent increases are smaller each time.

Predicting the final October to December 2015 rate

In order to predict the final reoffending rate at 18 months I need to estimate the trend. There are different options for doing this, and I will explain in a forthcoming blog how I selected the appropriate method. However, the trend I calculated for the October to December 2015 cohort is presented in the figure below.

 

Predicting the final October to December 2015 rate

Source: Ministry of Justice Proven Reoffending Quarterly Statistics

Definition: Months after commencement is the minimum number of months after commencement for an offender in the October 2015 to December 2015 cohort. However in the figures published, the follow up period varies depending on when within those 3 months the offender started.

 

The blue squares describe the published England and Wales reoffending rates, the red line is the fitted trend, and the red dot describes the predicted reoffending rate after 18 months. The predicted reoffending rate was just under 45% using this method. The selected reoffending trend is a curve to represent the reduction in the rate of increase as time progresses. Even from the limited data released, the curve shows an expected rapid increase in the first months after the start of an order or licence, but over time this increase is not sustained.  This trend is consistent with other research we have conducted using more detailed data.

October to December 2015 CRC results

I extended the same analysis to individual CRCs. When the final results are published, a CRC’s reoffending rate will be compared to a 2011 baseline rate. Since individual baseline rates for each CRC’s baseline have not been published, I cannot anticipate that analysis. However, the baseline OGRS scores (a measure of how likely someone is to reoffend) have been  published and so I was able to compare a CRC’s predicted reoffending rate to the baseline OGRS rate, adjusting for differences in the likelihood of reoffending between the baseline and the October to December 2015 cohorts. The spread in differences between the baseline OGRS rate and the predicted reoffending rate for each CRC are presented in the figure below. I have anonymised each CRC because I believed highlighting relative performance in the public domain with important information missing would not be ethical. If, however, you want to know how your CRC or area is performing please contact me.

 

October to December 2015 CRC results

Source: Ministry of Justice Proven Reoffending Quarterly Statistics

Eight of the 21 CRCS were predicted to beat the baseline OGRS score. The largest difference is 5.1%, followed by two CRCs expected to beat the baseline OGRS score by  4.4%. Twelve of the CRCs were predicted to perform worse that the baseline OGRS. For five of these the difference is less 1%, but three were expected to exceed the baseline OGRS rate by more than 3%.

Next steps

The analysis presented here is based upon a description of the data. The analysis therefore does not allow for the uncertainty in the predicted reoffending rate. A more realistic analysis would present the range of outcomes that are likely to happen. The presented analysis also assumes that the data are independent. In fact a CRC’s current reoffending rate will be dependent upon the rate 3 months ago and the current rate cannot be lower than the previous measure. A statistical model can allow for these issues and I will present that approach in a subsequent blog. This does not mean the presented results are wrong. Instead, it means greater insight and use is possible as we expand the analytical approach.

Crime doesn't pay written on a blackboard

CRC Reoffending Rates: Who’s Striving, Who’s Struggling?

CRC Reoffending Rates: Who’s striving, who’s struggling? Or should we all move to the North West?

Last week the MoJ released the interim Community Rehabilitation Companies (CRCs) proven reoffending rates for cohorts that started orders or licences between October and December 2015. As it is now 12 months since the first payment by results cohort started, I thought it would be interesting and helpful to review the performance of the CRCs – the final results will be available in October 2017. If you don’t know, the CRCs are privately run probation services set up in 2014 to manage less harmful offenders and to reduce reoffending rates. As such, part of their payment from the Ministry of Justice is based upon how well they reduce reoffending compared to a 2011 baseline. If the reoffending rate increases the CRC’s contract could be terminated.

The payment mechanism is complex – see here for the important documents. The basics are that if the CRC’s reoffending rate is lower than the 2011 baseline rate, having adjusted any differences in offenders’ likelihood of reoffending, then the CRC will receive a payment. Being just 0.1% lower would not be good enough to trigger a payment – the CRC’s rate must be statistically significantly lower (I have not found in the public domain information on how significant the reduction must be – do contact me if you know of a source).

Not enough data have been released to estimate which CRC will receive a payment (the 2011 baseline rates have not been published for example) but there are enough published data to assess the relative performance of each CRC. Each CRC’s published reoffending rate cannot be directly compared because the CRCs’ offenders will present with varying likelihoods of reoffending. Fortunately the OGRS4 expected one year reoffending rates were published. With this information, I estimated what each CRC’s proven reoffending rate would be if likelihood of reoffending was exactly the same in company (the adjusted rate – I used the average OGRS4 rate across all CRCs which was 45.7%). The results are in the chart below.

Figure 1: Interim and adjusted reoffending rates in the 21 CRCS (Oct to Dec 2015 Cohort, reoffending rate to December 2016).

Table showing Interim and adjusted reoffending rates in the 21 CRCS
Source: Interim proven reoffending statistics for the Community Rehabilitation Companies and National Probation Service supporting tables January 2017. See table for bases.

There is a large 17.2% spread in interim proven reoffending rate in the MoJ data. This is reduced to a 9.5% spread in the adjusted reoffending rate. The lowest adjusted rate is in Merseyside (34.4%) and the highest is in Warwickshire & West Mercia (43.9%).  There appears to be regional clustering at the top and bottom of the chart. Three of the four CRCs with the lowest adjusted rates are in the North West – Merseyside, Cumbria and Lancashire and Cheshire & Greater Manchester – and the neighbouring areas South Yorkshire, Humberside, Lincolnshire & North Yorkshire, and Durham Tees Valley make up three of the four areas with the highest adjusted rates. This clustering could be due to police practice, the CRCs’ effective work, efficient court processes or in or an unknown factor altogether.

These results do not mean the providers in the North West can expect a payment to come their way. We do not know the baseline to which they will be compared and there is another 6 months of convictions data to be included (the proven reoffending measure includes offences within 12 months that are convicted within 18 months). Also there is likely to be an area effect that is not included in the OGRS4 measure but will be in the 2011 baseline rate. However, the differences do suggest that persons interested in how to reduce reoffending rates should visit the North West to understand if they are doing anything different there.

Table 1: Results in all 21 CRCs

Table Showing Results in all 21 CRCs

Source: Interim proven reoffending statistics for the Community Rehabilitation Companies and National Probation Service supporting tables January 2017.