Posts

Impact Evaluation and Social Impact Bonds

How Rigorous Impact Evaluation Can Improve Social Impact Bonds

Social Impact Bonds

In recent years, Social Impact Bonds are being increasingly used by the British government to deliver public services via outcomes based commissioning. They are also becoming increasingly common in the U.S. By linking payments to good outcomes for society, SIBs are used not only to provide better value for money, but also as a driver of public sector reform. In the words of guidance published by the British government’s Cabinet Office:

“[Social Impact Bonds] are … designed to help reform public service delivery. SIBs improve the social outcomes of publicly funded services by making funding conditional on achieving results. “

While SIBs are not without their critics, their proponents argue that the bonds are a great way to attract private investment to the public sector while focusing all partners on the delivery of the desired social outcomes. This new way of commissioning services also encourages prime contractors to subcontract delivery of some service to the community and voluntary organisations, who bring their own experience, expertise and diversity to the provision of social services.

GtD have completed evaluations that have helped shape social impact bonds, and through our work we have identified five key questions that should be asked by anyone thinking of setting up a SIB or is looking to improve the design of their SIB:

1. Will it work?

Some services delivered by SIBs fail before they start because the planned intervention cannot plausibly achieve the desired outcome. In other words, just because an intervention reduced the number of looked after children entering the criminal justice system doesn’t mean it should work for all young people at risk of offending. That said, if the evidence base around a particular intervention is weak that does not mean one should not proceed – but it should promote a SIB design that includes an evaluation that can state quickly whether the SIB is delivering the hoped for outcomes.

2. Who can benefit from this intervention and who can’t?

We all want to help as many people as possible. However, we can quickly lose sight of who we are seeking to help when we are simply meeting output targets. In other words, if public services are funded by the number of clients they see, then providers could be tempted to increase numbers by accepting referrals of people for whom the service was not intended. So to achieve your outcomes – and receive payments – it’s vital to monitor intelligently the profile of your beneficiaries and ask yourself, “If my targeting were perfect are these the clients I would want to work with to deliver my intended outcomes?”

3. Are we doing what we said we were going to do? And does it work?

Interventions can fail simply because they don’t do what you said you were going to do. If, for example, you are working with young people to raise career prospects and your operating model includes an assessment of need (because the evidence base suggests that assessments increase effectiveness) then it should be no surprise that you did not meet your outcome targets if an assessment was completed with only half of your clients. Identifying your key outputs, monitoring their use and predicting outcomes based on their use can give you much greater confidence in achieving your intended impact.

4. What do we need to learn and how do we learn quickly?

All SIBs we have seen collate a lot of data about their beneficiaries and the service provided but few use those data to their full potential. With predictive analysis, we can monitor who appears to respond best, who is not benefiting and what form of service delivery is the most effective. In other words, is one-to-one work or group work more cost effective? As such you can learn how to define your referral criteria better or learn how to improve your operating model, even within the first months of a SIB.

5. How can we build a counterfactual?

A counterfactual is an estimation of what outcomes would have been achieved without the SIB. Comparing your SIB’s outcomes to the counterfactual can highlight some areas for learning and how to improve over time. Consideration should be given to the counterfactual at the commencement of the SIB. Full advantage can be taken of the publically available data sets to construct the counterfactual: for example, the National Pupil Database, Justice Data Lab or the Health Episode statistics. (Top tip: consent from beneficiaries to use these data sources is generally required).

To discuss how GtD’s impact evaluation can help improve your organisation’s SIB, please contact Jack Cattell, jack.cattell@getthedata.co.uk

Image of data being analysed

There’s no Magic Way of Measuring Impact

Wouldn’t it be great if there was a way of measuring your social impact across multiple projects using a single dependable statistic? Well, I’ve got some bad news, and some good.

I was recently talking to a charity who wanted to know how if they could go about measuring and reporting the overall impact of the organisation on children and families. With multiple strands each aiming to achieve different things, they asked if a single outcome measure – one accurate, reliable number – to sum up the impact of the whole organisation was either possible or desirable.

First, here’s the bad news: it’s very unlikely – I might even be so bold as to say impossible – that any such thing exists. You might think you’ve found one that works but when you put in front of a critic (or a nitpicking critical friend, like me) it will probably get ripped apart in seconds.

Of course, if there is a measure that works across multiple projects, even if not all of them, you should use it, but don’t be tempted to shoehorn other projects into that same framework.

It’s true that measuring impact requires compromise but an arbitrary measure, or one that doesn’t stand up to scrutiny, is the wrong compromise to make.

The Good News

There is, however, a compromise that can work, and that is having the confidence to aggregate upwards knowing your project level data are sound. You might say, for example, that together your projects improved outcomes for 10,000 families, and then give a single example from an individual project that improved service access or well-being to support the claim. In most situations that will be more meaningful than any contrived, supposedly universal measure of impact.

Confidence is the key, though: for this to work you need to find a reliable way of measuring and expressing the success of each individual project, and have ready in reserve information robust enough to hold up to scrutiny.

Measuring Means Data

In conclusion, the underlying solution to the challenge of measuring impact, and communicating it, is a foundation of good project level data. That will also make it easier to improve performance and give you more room to manoeuvre. Placing your faith in a single measure, even if you can decide upon one, could leave you vulnerable in a shifting landscape.

 

Influence through Data

“Yeah, Says Who?” – Influence Through Data

You know you’ve achieved results – the data tells you so – but how do you influence sceptics to believe it?

It can be a rude awakening to take the findings of a study outside your own team or organisation, where trust and mutual support are more or less a given. In front of a wider audience of funding providers or other stakeholders, you will inevitably in my experience find yourself being challenged hard.

This is as it should be – scrutiny is a key part of a healthy system – but, at the same time, it’s always a shame to see an impactful project or programme struggle purely because its operators fail to sell it effectively.

Fortunately, while there are no black-and-white rules, there are some things you can do to improve your chances.

Confidence = Influence

When I present findings I do so with a confidence that comes with experience and from really understanding the underlying mechanics. But if you’re not a specialist and don’t have that experience there are things you can do to make yourself feel more confident and thus inspire greater confidence in your audience.

First, make sure you have thought through and recorded a data management policy. Are you clear how often data should be entered? If information is missing, what will you do to fill the gaps? What are your processes for cleaning and regularising data? Is there information you don’t need to record? A professional, formalised approach to keeping timely and accurate data sends all the right signals about your competence and the underlying foundations of your work.

Secondly, use the data as often as possible, and share the analysis with those who enter your data so that they can understand its purpose, and own it. Demonstrating that your data is valued and has dedicated, accountable managers hugely increases its (and your) credibility.

Thirdly, take the initiative in checking the reliability and validity of your own tools. If you use well-being questionnaires, for example, take the time to check whether they are really measuring what you want to measure in most instances. In other words, try to find fault with your own approach before your stakeholders so that when they find a weak point you have an answer ready that not only reassures them but also underlines the objectivity with which you approach your work.

Own Your Data’s Imperfections

Finally, and this might feel counterintuitive, you should identify the weaknesses in your own data and analysis and be honest about them. All data and analysis has limitations and being clear about those, and the compromises made to work around them demonstrates objectivity which, again, reinforces credibility.

In conclusion, the better you understand your own data and analysis, flaws and all, the more comfortable and confident you will feel when it, in turn, comes under scrutiny.

Social impact webinar

Webinar: Make 2020 the Year of Social Impact

On both sides of the Atlantic, our clients are working hard to think of new ways to deliver vital services to their clients: the homeless, the refugee, offenders.  They are often the people who are most likely to be at the margins and their needs are urgent.  So, now is a time for adapting services to the weeks of lock-down and shelter in place orders. And at GtD we are adapting the ways in which we are delivering our services and helping our clients.

While we wait for the pandemic to subside, the needs of our clients and yours remain pressing, and that will continue long after CoVid19 becomes yesterday’s news. However, I doubt that we will go back to work just as we did before we heard about social distancing and self-quarantining.  When we do go back to work, we will be asked to do things in new ways, to answer new needs and to find new partners to work with.

2020 is GtD’s “Year of Social Impact” when we are encouraging non-profits and charities to think about new ways of demonstrating their social value – definitively. We started out with a roadshow of events in Atlanta and then London, meeting with leaders from the nonprofit sector, listening to their needs and telling them about how our “Measure”, “Learn” and “Prove” packages can assist them.

We were due to take the roadshow to Manchester on April 28th.  Instead, we will be delivering the event as a webinar and inviting people to attend from across the UK and the USA.  This is bound to be a fascinating meeting as we share experiences across the Atlantic and hear more about how GtD’s social impact analytics are helping non-profits on both sides of the ocean.

The webinar is free and we have made it easy to join.  To register, please use this link and learn more about how to attend.

At times like this, we need to keep going as best we can and re-evaluate our priorities. Today, I received some words of wisdom from a valued client and jazz aficionado.  He and I were talking about how best to address the needs of those who have no medical care in rural Georgia. We had to suspend some work and address the new priorities thrown up by lockdown, and so I would like to share his words with you:

“What we’ve been doing lately reminds me of an announcement Louis Armstrong made during an interruption in one of his concerts, “And while they’re fixing the mike, we are going to lay on you our very best version of ‘Tin Roof Blues’”. Getting important things done during times like these can be helpful both to our projects and to our morale”.

Jack and I hope to meet you at the webinar but please contact me on alan.mackie@gtd-us.com if you have any questions. In the meantime, we wish you all the best as you address your new priorities and redouble your efforts to serve your clients.

Prove your social impact

Social Outcomes: The Road to Success is Paved With Good Intentions

I have often said that hardly a week goes by when I don’t encounter an individual or group of people working hard to make the world a better place. Often these people are working with those who are existing on the edge of society: the homeless who are ignored on our streets; refugees who feel that every hand is turned against them; the sick and frail who have no access to local medical provision; or individuals who are unfairly caught up in the “penal-industrial complex”. In every case, I see individuals and groups who have recognized a social need and have a clear intention to address it.

Resourcing & Focusing an Intention

While a good intention is essential to articulating a program’s vision, mission and rationale, intention alone is not enough. All our clients, however, will be able to explain the need for their work and can identify the time, money other resources that are required to make a positive difference.  As Jack and I have written in recent blogs, GtD’s “Measure” and “Learn” services assist our clients to enhance their work. In the case of “Measure” we are helping our clients to ensure that their good intentions are properly resourced, and with “Learn” we are applying sophisticated predictive analyses to make sure intentions are correctly focused for maximum benefit.

Intentions & Results

Some years ago, I came across the following quote from the American economist and Nobel Laurette, Milton Friedman, “One of the great mistakes is to judge policies and programs by their intentions rather than their results”. Like many familiar quotes, we are often unclear about their original context or intended meaning.  Often, however, Friedman’s quote is used against government intervention. In other words, it is used as a counsel of despair that government intervention usually makes matters worse. I don’t know if that is what Friedman intended, but it is generally an absurd conclusion. That said, often GtD’s clients are working hard to complement public services or are providing services to client groups where government bureaucracies would fear to tread.

Proving your Intention

Friedman’s words do, however, support the desirability that well-intentioned programs should deliver good results. I am sure that few people would dispute that, particularly GtD’s clients who are increasingly looking for definitive evidence of the value of their work. This is where our “Prove” service is in demand. Sometimes our clients need the evidence to prove the effectiveness of their work to their funders. More often, however, their quest for evidence is part of their commitment to delivering the most effective service and ensuring that they are benefiting the very people they want to assist.

Our clients know the value of our “Prove” service, whether they are demanding to know the effectiveness of reforms to the youth justice system or requiring evidence of the benefit of innovative approaches to health provision in underserved rural counties. GtD’s “Prove” service not only delivers rigorous impact evaluation. It also provides clearly written reports that communicate key findings to a range of stakeholders, including funders, senior management, practitioners and users.

Turning Good Intentions into Successful Outcomes

An effective program starts with a good intention to help. Let GtD accompany you on your social impact journey through our “Measure”, “Learn” and “Prove” packages. Contact us today to learn more about the range of our services.

 

Offender/Offender Manager Relationships - image of paper people holding hands

Positive Relationships in Offender Management

We are delighted to welcome Prof Darrick Jolliffe, Professor of Criminology at the University of Greenwich as a guest blogger. Prof Jolliffe has worked with GtD on our offender management work and here he discusses the role of relationships in that work.

Positive relationships have the power to change us. This is not just the catchy slogan of the International Coaching Federation, but something that resonates with all of us.  The constructive relationships that we have, or have had with our parents, teachers, friends and colleagues not only define our interactions with them, but our interactions with all others, while defining who we see ourselves to be. Translating this ‘common sense fact’ into the management of those who have committed offences has an extensive history, and some would argue is the bedrock on which the probation service was founded. However, like many things that are common sense, such as we only use 10% of our brains, and carrots provide good eyesight, the suggestion that if ‘offenders’ have a positive relationship with those providing their supervision they will have more positive outcomes, the actual relationship might actually be a bit more complicated. 

I was invited by Get the Data Director Jack Catell to speak their event, Transforming Rehabilitation – Learning from the PbR results, about my experience of attempting to capture the evidence that positive relationships between offenders and offender managers results in definable benefits. This was something that I first worked on over a decade ago, having first been commissioned by the Home Office to develop an Offender Management Feedback Questionnaire, which was a series of questions that asked offenders about their engagement with their named offender manager. I was subsequently commissioned to work on revised versions, and also helped develop a mirror version which asked about how offender managers viewed the relationship that offenders had with probation.

Some findings fit with expectation, for example, female offenders consistently had more positive relationships with offender managers than males, and the longer that an offender had spent on probation the more they felt they had developed positive relationships and acquired useful skills.  However, some findings were very far from expected, for example, those who reported greater engagement with probation were not less likely to reoffend, and even more concerning, or at least confusing, those who reported greater acquisition of skills while on probation were significantly more likely to reoffend.

How do we reconcile these findings with what we know about the power of positive relationships?  Well, I don’t think there is an easy answer here.  I suspect that positive relationships with offender managers are important for offenders, but they may not be important enough to have an impact on a blunt measure such as reoffending over and above the other issues that many of these people are facing in their lives.  We put the finding, that those who reported greater acquisition of skills were more likely to reoffend, down to offender’s overly optimistic view of how easy it would be to continue to stay crime free.  When we looked more closely at the scores of the questionnaires, it was the items which referred specifically to acquiring skills which would reduce later reoffending that those who reoffended tended to endorse more strongly – almost as though they were trying to convince themselves of their ability to stay on the straight and narrow.

Reflecting back on this work, I am really proud of what we did achieve, despite not getting the common sense result we all expected.  I think there would be a lot of mileage in revisiting the offender/offender manager relationship as a potential desistence tool, and if I was going to do this again I would be looking to measure changes in relationships (so administering items more than once) and see how these changes might relate to more short-term outcomes (e.g., offender’s motivation), as well as longer-term outcomes (e.g., reconviction). 

Tape measure

The Data Revolution Will be Measured

“Innovation distinguishes between a leader and follower”, so said the late Steve Jobs. His words came to mind as I prepared to present to Gideon’s Promise’s Summer Session last month.  Gideon’s Promise asserts that a revolution is needed in how the criminal justice system treats indigent defendants. And as befits an organization that sees itself in the vanguard of that revolution, Gideon’s Promise has a radical vision of the impact of its work. For them it is less about increased efficiencies in case management or the effectiveness of sentencing. Rather it is about changing how the criminal justice system and society-at-large regard poor people caught up in the system’s net. To bring this about, Gideon’s Promise provides a community of public defenders who put their clients at the center of their practice. In doing so, they insure their clients’ stories are told and their humanity is revealed to the court. Or in the words of one member of the movement, “we need to think of this person as a human being and not simply a number in an orange jump suit”.

So how to measure a revolution? Over the past 18-months Gideon’s Promise has led an expert collaboration that has included Get the Data and another Atlanta-based organization, Techbridge. Together we have delivered a ground-breaking program of work that has produced innovative measures of both the public defenders’ values and their clients’ experiences of the client-centered approach. In developing those measures, GtD grounded our work on understanding the impacts that Gideon’s Promise strives to achieve and the resources at its disposal. Having articulated this theory of change, we were able to design the appropriate measurement instruments and subject them to cognitive, reliability and validity testing with samples of Gideon-trained public defenders and their clients. The next step was to collaborate with Techbridge to deliver a technological solution that will allow the instruments to be completed on-line and held in a database. This technology will be essential as the measurement work is rolled out across Gideon-affiliated offices. Finally, we designed a “Digital Dashboard” that Techbridge attached to the database and to report data by offices, public defenders and their clients.

“Innovative” is a word that can be overused or cliched, but in this case the program of work undertaken by the collaboration has truly been new and refreshing. Moreover, it provides the means by which Gideon’s Promise can measure its bold and revolutionary vision. The next steps will be to roll-out this program of work and then move to designing measures of the impact on the criminal justice system and ultimately society as a whole.

The revolution is coming, and the revolution will be measured.

Make Social Impact Your Goal! - GtD and Street Soccer Academy

Make Social Impact Your Goal!

I am delighted to be working again with Street Soccer Academy (SSA) to put evidence of their social impact at the heart of their work with ex-offenders. This important work has been made possible by a grant from the Access Impact Foundation whose mission is to make charities and social enterprises more financially resilient and self-reliant, so that they can sustain or increase their impact. Without the generous funding from the Access Impact Foundation we would not able to provide Street Soccer Academy with our powerful data analysis.

Street Soccer Academy is a great client to work with. They use professionally organised sports-based programmes in the rehabilitation and reintegration of people from some of the nation’s hardest to reach groups, including ex-offenders. Our task is to prove that SSA’s pro-social models are affecting the attitudes and thinking of the men and women with whom they work, with particular emphasis on their relationships and roles in society.

In the coming months we will be using our rigorous social impact analytics to contribute to the knowledge of what makes ex-offenders desist from crime. Our previous evaluation of the academy’s prison to community service, produced evidence of SSA’s excellent engagement of ex-offenders into their programme. With the foundation’s funding we will build on that by using our advanced statistical analysis to identify who benefits from the programme, how and in what circumstances. These analyses will assist the academy to identify the most effective practice and allow it to develop its professional programmes. To ensure that it has the right information at the right time we will be building a dashboard to communicate this data to those delivering the programme, their managers and the funder.

Not only will this improve their practice, but it has important implications for government’s Transforming Rehabilitation agenda. That agenda depends on organisations like SSA being commissioned to deliver services through the private community rehabilitation companies. However, the participation of such organisations has been low as they struggle to demonstrate their impact on reoffending pathways and desistance from crime. Access Impact is helping to overcome those obstacles and by funding our work, will enable SSA to attract further funding and make the systemic changes that are essential to support men and women to desist from crime.

To find out more about our analytics services and how we can help your organisation demonstrate your impact definitively, contact us on 020 3371 8950 or email jack.cattell@getthedata.co.uk

GtD Transforming Rehabilitation Event

Transforming Rehabilitation: Learning from the PbR results

Transforming Rehabilitation challenged Community Rehabilitation Companies (CRCs) to reduce reoffending significantly and October will see the publication of the reoffending rates of the first Transforming Rehabilitation cohort.

These results are an important test of the government’s Transforming Rehabilitation agenda and the success of the new CRCs. Some CRCs will be happy with their results, while others will need to improve. The published results also provide opportunities to commission new services for offenders.

Besides the implications for policy and practice, the results also present an opportunity for your organisation to learn how best to reduce reoffending with the tools at your disposal. You might want to increase a CRC’s performance, identify “what works?” and roll that out across all your CRCs, or prove that your specific service will make the difference.

Come join our experts at our TR PbR event, to discuss what needs to be learnt, what is within your control, and how embedding impact analytics in your CRC or service delivery organisation will help you to reduce reoffending.

Transforming Rehabilitation: Learning from the PbR results

Tuesday 28th November

2 – 4.30 pm

The Space Centre

94 Judd St, Kings Cross, London WC1H 9NT

SPEAKERS:

Professor Darrick Joliffe, University of Greenwich

“How to measure the quality of relationships between offenders and probation officers.”

Darrick is Professor of Criminology at the University of Greenwich. Darrick is interested in the broad areas of developmental life-course criminology, programme evaluation, prison research and psychology, individual differences and offending. He has developed tools to measure the satisfaction of offenders and the quality of their relationships with probation officers, and has estimated the association between relationship quality and reoffending. Darrick will also chair a discussion after the presentations.

 

Dr Sam King, University of Leicester

“What is high quality offender management?”

Sam is an expert on offender management and rehabilitation, and has contributed significantly to developments in desistence theory. His talk will discuss the latest evidence on what makes high quality offender management and what can be implemented to reduce reoffending. Sam has published widely on probation work and desistence theory, and has helped CRCs to implement innovative offender management tools to measure offender motivation.

 

Jack Cattell, Get the Data

“Predict your PbR results and continuously learn how to improve them before they happen.”

Jack is an expert on the prediction of reoffending rates, and the analysis of how a probation service can audit and improve its reoffending rate. He has worked for the Ministry of Justice, former probation trusts and Sodexo Justice Service’s six CRCs on research and analysis projects. He heads up GtD’s thought leadership on the use of predictive analysis to improve adult and youth offender management.

Free event – register today! Places are limited.

 

Headed by Jack Cattell, Get the Data is located in the UK and USA. We are an international thought leader on how to reduce reoffending through high quality analyses of offender management data. We have delivered high-profile projects for the Ministry of Justice, HMPPS (formally NOMS), and the police. We are proud to provide our Social Impact Analytics service to Sodexo Justice Service’s six CRCs.

 

Delivering Innovative Social Impact Analytics to Sodexo Justice Get The Data

Delivering Innovative Social Impact Analytics to Sodexo Justice

We are delighted to announce a new contract to deliver our ground breaking social impact analytics to Sodexo Justice, a leading provider of justice services in the UK.

The purpose of our social impact analytics is to provide definitive evidence of an organisation’s impact on society by delivering predictive analyses and impact evaluation.  Under the newly signed contract, we will measure the effectiveness of Sodexo’s six Community Rehabilitation Companies in managing the risk associated with the offenders and delivering interventions that reduce their reoffending.

By understanding “what works?” in changing lives and delivering safer communities, our social impact analytics will also be used by Sodexo Justice to measure the impact of its services.  Sodexo Justice will be paid through a payment by results mechanism that measures its success in reducing reoffending.

Our founding director, Jack Cattell said, “We very much look forward to providing our social impact analytics to Sodexo Justice Services.  Our SIAs will provide offender managers with the information they require to manage resources and deliver high quality interventions to reduce reoffending.”