Magnifying glass over the word analyse to illustrate analysis of TR PbR Figures

TR PbR Figures – Context and Drivers

This blog post summaries a presentation I gave at our recent event: Transforming Rehabilitation: Learning from the PbR results.

 

Overall results  

In this blog, I will discuss the data I presented at our recent event, ‘Transforming Rehabilitation: Learning from the PbR Results’. The event was held shortly after the publication of the first reoffending figures which showed that reoffending in the first cohort had been reduced by 1.9%. My colleague Jack Cattell has already discussed these figures in an excellent blog, so this blog is about the how external drivers across the criminal justice system might be affecting these results.  

By CRC? – PbR Figures, who is in control of reoffending? 

In order to make conclusions from these results, one must be aware that there are many factors that can influence reoffending. When faced with (improved or reduced) performance results – it’s too easy to fall into the trap of thinking that the reasons for this were outside of your control, or that the results were dominated by one particular driver.  

In understanding the reoffending rates by Community Rehabilitation Companies (CRCs), there are many possible drivers across the CJS, and they can be complex to understand. So, could the CRCs have achieved the difference in performance we have seen under TR? Internal factors such as CRC procedures and workflows, and offender profiles will have an effect.  But so too will a myriad of external influences from other agencies of the CJS, for example police positive outcome rates, court conviction rates and timeliness of police & court procedures. Since these factors are outside a CRC’s control, I wanted to investigate and determine whether any might have had an effecting change in the reported reoffending rates.  

National context 

Chart showing - No. of offences thousands

First off, it is important to bear in mind the national context for the period from baseline to first cohort results, which was a time when the total estimated crime fell and police recorded crime remained constant.

 Chart showing proportional change from 2011

Further, positive outcome rates, specifically charges and cautions had been reduced – in general police arrest rates were down and police had been using other disposals (such as community resolutions) to reduce the number of people – particularly young people – in custody.

 

Chart showing days from offence to completion

And individuals were being processed more quickly during the baseline: on average court processes are now taking 10-15 days longer.

Give this context, the national reoffending rate was reduced by 1.8%. That’s a surprisingly resistant figure, given the changes in context nationally. Large changes in this binary rate will not be seen regardless of policy or systematic change in CRCs.

Chart showing reoffending rate

(Of course, the full picture of the reoffending stats has the rate down, but the frequency up – suggesting a shrinking group of more prolific reoffenders. But that is for another post).

Differences in CRCs?

I will now look at these potential drivers of reoffending rates at the CRC level and determine if contexts really were different or helped to drive different performance. To highlight any of these potential differences, I will look at two CRCs who were at opposite ends of the reoffending rate changes.

What is surprising, is that there are no large, obvious differences between those two CRCs when comparing police outcome rates, court effectiveness and court timeliness measures. For example, the police positive outcome rates were reasonably different at the start of the period, but by the end of 2015 had converged to be broadly similar – the best performing CRC had reduced reoffending while police positive outcomes fell by a significant amount.

Chart showing positive outcome rate

Similarly, the lowest performing CRC had raised rates against the baseline despite court timeliness both increasing, and being higher than other CRCs.

So What?

We’ve seen that reoffending rates are complex measures dependent on many factors, from individual, to regional to national level, and the interaction between them. The rates don’t change very much, even though the contexts can differ wildly.

Even comparing the CRCs at either end of performance spectrum there were not huge differences in the police and court factors. This can be seen in different results for CRCs operating in widely similar contexts.

So, my advice is this: don’t fall into the trap of feeling the results are not within your ability to affect. Understanding the wider picture is helpful to contextualise local results. In the following blogs, my colleagues will be focusing on what is in your control, and my colleagues will be identifying what works and using that to inform good practice.

Get involved in the conversation by joining our LinkedIn group.

TR: PbR results - speakers who presented to Community Rehabilitation Companies and other justice sector employees

Community Rehabilitation Companies: PbR Results Event

Transforming Rehabilitation is the UK government’s programme of outsourcing probation services to new community rehabilitation companies. In a radical move, the government is now paying these new companies by the reduction in reoffending results they achieve. GtD is at the forefront of this by providing our cutting-edge social impact analytics to Sodexo Justice Services who manage a number of these new companies.

The first PbR figures were published last month and GtD has been active in informing the debate on their significance. As part of this debate, we recently hosted a sell-out event for senior management and practitioners working in community rehabilitation companies and the justice sector.

An expert panel comprising Prof. Darrick Jolliffe of Greenwich University (above left), Dr Sam King of Leicester University (above right) and GtD’s own Jay Hughes (above centre left), considered the initial findings and what to do next, with Jack Cattell (above centre right) setting out a new vision of how predictive analyses can be used by practitioners to improve performance.

Prof. Darrick Jolliffe – University of Greenwich

If you were unable to attend but would like to learn more about how GtD could support you in evaluating your social impact outcomes or for a free predictive analytic roadmap for your CRC, contact Jack Cattell  The event presentations can also be viewed via the link below:

Transforming Rehabilitation – Learning from the PbR results presentations

Dr Sam King – University of Leicester

 

Jack Cattell – GtD

 

We’ve also set up a LinkedIn group as a forum for shared learning and discussion, for individuals who work or have an interest in, the fields of probation, offender rehabilitation and Transforming Rehabilitation. Click here to request to join – Transforming Rehabilitation

Scales of Justice representing Transforming Rehabilitation

Transforming Rehabilitation: Payment by Results Figures

Last week saw the release of the Transforming Rehabilitation (TR) Payment by Results figures for the October to December 2015 cohort.

The overall result was encouraging, and defy the view that Transforming Rehabilitation’s radical changes to probation, and the ensuing problems, would result in increased reoffending – though it is very important that I point out that this is just the first set of results of many and overall judgement should be reserved for at least a year. The reoffending rate for all CRCs was 45.6% compared to a 2011 baseline rate of 47.5%. I had to make some (conservative) assumptions to estimate the baseline rate but I think it is also safe to say that the difference was statistically significant, suggesting reoffending rates have reduced under TR. Please see the the note at the end of this blog to understand better how I completed the analysis.

 

Transforming Rehabilitation – CRC Performance

The chart below describes each CRC’s reoffending rate in relation to the baseline 2011 rate. The grey line represents the range of reoffending rates that would indicate no change from 2011 (the baseline confidence interval). If the CRC’s rate is outside this range, we are confident in statistically terms to state that the CRC’s performance was either better or worse than the reoffending rate achieved in 2011. The green bars represent the reoffending rates of CRCs that outperformed 2011, the orange bars represent those that performed the same as 2011 and the red bars present those that performed worse than 2011.

Chart to show CRC Performance

Source: Ministry of Justice Final Proven Reoffending Rates TR (Oct to Dec 2015 cohort).

Thirteen of the CRCs beat the baseline rate. The best performing CRC was Cumbria and Lancashire, which beat the baseline rate by 8.2% (49.9% to 41.7%). The nest best was Hampshire and the Isle of Wight which beat the baseline by 5.4% and the third best was Northumbria with a better rate by 4.3%. Two of the CRCs performed worse than the 2011 baseline. Warwickshire and West Mercia recorded a reoffending rate 3% worse than the baseline rate, and South Yorkshire’s rate was 2.8% worse. With most CRCs, however, outperforming the reoffending rate form 2011, the figures are a promising set of results.

 

Transforming Rehabilitation – Comparing CRC performance

Now that the baseline rates have been published, we can better understand how well each area was performing in 2011 and whether a CRC is now being asked to better good or bad performance achieved in that year. The chart below describes the difference between the actual baseline rate and the 2011 baseline’s OGRS score (in other words their expected rate of reoffending). A negative result in the chart means the area performed better in 2011 than the OGRS score expected.

Chart showing the difference between a CRC's actual baseline rate and the 2011 baseline’s OGRS score

Source: Ministry of Justice Final Proven Reoffending Rates TR (Oct to Dec 2015 cohort).

The charts highlights that six of the CRCs are being asked to beat better than expected performance in 2011 (in other words to be better than good). Whereas other CRCs, notably London and Wales, are being asked to outperform potentially poor performance in 2011. It it interesting that South Yorkshire and, Warwickshire & West Mercia – the two areas that recorded poor performance for TR – are being asked to beat good performance from 2011. Merseyside and Cheshire & Greater Manchester, however, are equally being asked to beat good performance from 2011 and were able to do so for the October to December 2015 cohort. The OGRS score does not allow for area effects, which will exist and could explain the differences between the OGRS score and the baseline rate. It not possible now to conclude whether payment by results will be easier in some areas than others, but, going forward, I will monitor the impact of whether a CRC is being asked to perform better than good or poor performance from 2011 on their ability to achieve payment by results bonuses.

 

Notes on analysis

The latest Ministry of Justice bulletin released more data than was previously available and I was able to complete a statistical analysis of the impact of TR. This could only be completed with making conservative assumptions that would make finding a statistically significant result less likely. The following actions were taken:

  • I assumed the spread of offenders across CRCs in 2011 was exactly the same as it was in the October to December 2015 cohort. This would not be the case but any analysis would want to weight the two samples so they represented each other so the impact of this assumption is minimal.
  • The 2011 sample size was assumed to be the same as that of the October to December 2015 cohort. The 2011 sample will be considerably bigger, so this assumption meant the standard error used for the analysis was larger than it should be.
  • A t-test with unequal variances assumed was used to test the difference between the cohort’s and the baseline’s reoffending rate. The t statistic result was 4.6.

Blog originally guest posted on http://www.russellwebster.com 31st October 2017 with additional commentary from Russell Webster.