Posts

Impact Evaluation and Social Impact Bonds

How Rigorous Impact Evaluation Can Improve Social Impact Bonds

Social Impact Bonds

In recent years, Social Impact Bonds are being increasingly used by the British government to deliver public services via outcomes based commissioning. They are also becoming increasingly common in the U.S. By linking payments to good outcomes for society, SIBs are used not only to provide better value for money, but also as a driver of public sector reform. In the words of guidance published by the British government’s Cabinet Office:

“[Social Impact Bonds] are … designed to help reform public service delivery. SIBs improve the social outcomes of publicly funded services by making funding conditional on achieving results. “

While SIBs are not without their critics, their proponents argue that the bonds are a great way to attract private investment to the public sector while focusing all partners on the delivery of the desired social outcomes. This new way of commissioning services also encourages prime contractors to subcontract delivery of some service to the community and voluntary organisations, who bring their own experience, expertise and diversity to the provision of social services.

GtD have completed evaluations that have helped shape social impact bonds, and through our work we have identified five key questions that should be asked by anyone thinking of setting up a SIB or is looking to improve the design of their SIB:

1. Will it work?

Some services delivered by SIBs fail before they start because the planned intervention cannot plausibly achieve the desired outcome. In other words, just because an intervention reduced the number of looked after children entering the criminal justice system doesn’t mean it should work for all young people at risk of offending. That said, if the evidence base around a particular intervention is weak that does not mean one should not proceed – but it should promote a SIB design that includes an evaluation that can state quickly whether the SIB is delivering the hoped for outcomes.

2. Who can benefit from this intervention and who can’t?

We all want to help as many people as possible. However, we can quickly lose sight of who we are seeking to help when we are simply meeting output targets. In other words, if public services are funded by the number of clients they see, then providers could be tempted to increase numbers by accepting referrals of people for whom the service was not intended. So to achieve your outcomes – and receive payments – it’s vital to monitor intelligently the profile of your beneficiaries and ask yourself, “If my targeting were perfect are these the clients I would want to work with to deliver my intended outcomes?”

3. Are we doing what we said we were going to do? And does it work?

Interventions can fail simply because they don’t do what you said you were going to do. If, for example, you are working with young people to raise career prospects and your operating model includes an assessment of need (because the evidence base suggests that assessments increase effectiveness) then it should be no surprise that you did not meet your outcome targets if an assessment was completed with only half of your clients. Identifying your key outputs, monitoring their use and predicting outcomes based on their use can give you much greater confidence in achieving your intended impact.

4. What do we need to learn and how do we learn quickly?

All SIBs we have seen collate a lot of data about their beneficiaries and the service provided but few use those data to their full potential. With predictive analysis, we can monitor who appears to respond best, who is not benefiting and what form of service delivery is the most effective. In other words, is one-to-one work or group work more cost effective? As such you can learn how to define your referral criteria better or learn how to improve your operating model, even within the first months of a SIB.

5. How can we build a counterfactual?

A counterfactual is an estimation of what outcomes would have been achieved without the SIB. Comparing your SIB’s outcomes to the counterfactual can highlight some areas for learning and how to improve over time. Consideration should be given to the counterfactual at the commencement of the SIB. Full advantage can be taken of the publically available data sets to construct the counterfactual: for example, the National Pupil Database, Justice Data Lab or the Health Episode statistics. (Top tip: consent from beneficiaries to use these data sources is generally required).

To discuss how GtD’s impact evaluation can help improve your organisation’s SIB, please contact Jack Cattell, jack.cattell@getthedata.co.uk

Influence through Data

“Yeah, Says Who?” – Influence Through Data

You know you’ve achieved results – the data tells you so – but how do you influence sceptics to believe it?

It can be a rude awakening to take the findings of a study outside your own team or organisation, where trust and mutual support are more or less a given. In front of a wider audience of funding providers or other stakeholders, you will inevitably in my experience find yourself being challenged hard.

This is as it should be – scrutiny is a key part of a healthy system – but, at the same time, it’s always a shame to see an impactful project or programme struggle purely because its operators fail to sell it effectively.

Fortunately, while there are no black-and-white rules, there are some things you can do to improve your chances.

Confidence = Influence

When I present findings I do so with a confidence that comes with experience and from really understanding the underlying mechanics. But if you’re not a specialist and don’t have that experience there are things you can do to make yourself feel more confident and thus inspire greater confidence in your audience.

First, make sure you have thought through and recorded a data management policy. Are you clear how often data should be entered? If information is missing, what will you do to fill the gaps? What are your processes for cleaning and regularising data? Is there information you don’t need to record? A professional, formalised approach to keeping timely and accurate data sends all the right signals about your competence and the underlying foundations of your work.

Secondly, use the data as often as possible, and share the analysis with those who enter your data so that they can understand its purpose, and own it. Demonstrating that your data is valued and has dedicated, accountable managers hugely increases its (and your) credibility.

Thirdly, take the initiative in checking the reliability and validity of your own tools. If you use well-being questionnaires, for example, take the time to check whether they are really measuring what you want to measure in most instances. In other words, try to find fault with your own approach before your stakeholders so that when they find a weak point you have an answer ready that not only reassures them but also underlines the objectivity with which you approach your work.

Own Your Data’s Imperfections

Finally, and this might feel counterintuitive, you should identify the weaknesses in your own data and analysis and be honest about them. All data and analysis has limitations and being clear about those, and the compromises made to work around them demonstrates objectivity which, again, reinforces credibility.

In conclusion, the better you understand your own data and analysis, flaws and all, the more comfortable and confident you will feel when it, in turn, comes under scrutiny.

Tape measure

The Data Revolution Will be Measured

“Innovation distinguishes between a leader and follower”, so said the late Steve Jobs. His words came to mind as I prepared to present to Gideon’s Promise’s Summer Session last month.  Gideon’s Promise asserts that a revolution is needed in how the criminal justice system treats indigent defendants. And as befits an organization that sees itself in the vanguard of that revolution, Gideon’s Promise has a radical vision of the impact of its work. For them it is less about increased efficiencies in case management or the effectiveness of sentencing. Rather it is about changing how the criminal justice system and society-at-large regard poor people caught up in the system’s net. To bring this about, Gideon’s Promise provides a community of public defenders who put their clients at the center of their practice. In doing so, they insure their clients’ stories are told and their humanity is revealed to the court. Or in the words of one member of the movement, “we need to think of this person as a human being and not simply a number in an orange jump suit”.

So how to measure a revolution? Over the past 18-months Gideon’s Promise has led an expert collaboration that has included Get the Data and another Atlanta-based organization, Techbridge. Together we have delivered a ground-breaking program of work that has produced innovative measures of both the public defenders’ values and their clients’ experiences of the client-centered approach. In developing those measures, GtD grounded our work on understanding the impacts that Gideon’s Promise strives to achieve and the resources at its disposal. Having articulated this theory of change, we were able to design the appropriate measurement instruments and subject them to cognitive, reliability and validity testing with samples of Gideon-trained public defenders and their clients. The next step was to collaborate with Techbridge to deliver a technological solution that will allow the instruments to be completed on-line and held in a database. This technology will be essential as the measurement work is rolled out across Gideon-affiliated offices. Finally, we designed a “Digital Dashboard” that Techbridge attached to the database and to report data by offices, public defenders and their clients.

“Innovative” is a word that can be overused or cliched, but in this case the program of work undertaken by the collaboration has truly been new and refreshing. Moreover, it provides the means by which Gideon’s Promise can measure its bold and revolutionary vision. The next steps will be to roll-out this program of work and then move to designing measures of the impact on the criminal justice system and ultimately society as a whole.

The revolution is coming, and the revolution will be measured.

Alan Mackie presenting about Data Driven Policy Development

Using Data to Shed the Cloak of Invisibility

Since time immemorial we have been fascinated by the power of invisibility. Plato used the “Ring of Gyges” to wrestle with the ethical and moral dilemmas it poses. Just think of H.G. Wells’ “Invisible Man”, Tolkien’s One-Ring, or Harry Potter’s Cloak of Invisibility.  Does invisibility free us from moral obligations, and if we possessed this power would we become corrupted or would we enjoy secretly righting wrongs?

Outside the realms of science fiction, “invisibility” is often an every-day problem for our fellow citizens. I am thinking of those whose problems just don’t make the 24/7 news cycle: the homeless woman sleeping in a shop doorway, the ex-offender walking through the prison gate with nowhere to go, the young man who has dropped out of school with no grades. Often we simply ignore those populations, or – to borrow from Harry Potter – throw the “cloak of invisibility” around them. To be out of sight is often to be out of mind.

The ‘invisibility’ of those with developmental disabilities was thrown into sharp relief during my recent visit to the U.S. Virgin Islands. I was invited by the VI Developmental Disabilities Council to present on ‘Data Driven Policy Development’. My audience was policy makers, including those standing for elected office. If I had any doubts about the relevance of my presentation, the audience quickly rose to the occasion. “How could the needs of those with disabilities be met in the absence of data?”, they demanded. Without data, this population was “invisible”, their needs unknown to those in charge of policy. Without data, those who advocated for better services were hampered in their arguments. Without data, practitioners lacked the evidence to seek funding for new services.

The US Virgin Islands are still recovering from the two hurricanes that devasted the territory last year. As they rebuild the fabric of their communities, it was clear to me that data were not a luxury or a ‘nice-to-have”. Rather, my audience recognized the value of data in throwing light on a population that had often been marginalized prior to the hurricanes, and whose needs had since become more acute. While there is much more work to be done in defining data and setting up systems to collect and analyze data, I suggest that their demand for data was a good one and a great call to action.

It might be fun to speculate what we would do if we had our own cloak of invisibility. In the real world, however, we need data to shine a light on social problems. Or to borrow again from Harry Potter, “Lumus Maxima!”

 

Make Social Impact Your Goal! - GtD and Street Soccer Academy

Make Social Impact Your Goal!

I am delighted to be working again with Street Soccer Academy (SSA) to put evidence of their social impact at the heart of their work with ex-offenders. This important work has been made possible by a grant from the Access Impact Foundation whose mission is to make charities and social enterprises more financially resilient and self-reliant, so that they can sustain or increase their impact. Without the generous funding from the Access Impact Foundation we would not able to provide Street Soccer Academy with our powerful data analysis.

Street Soccer Academy is a great client to work with. They use professionally organised sports-based programmes in the rehabilitation and reintegration of people from some of the nation’s hardest to reach groups, including ex-offenders. Our task is to prove that SSA’s pro-social models are affecting the attitudes and thinking of the men and women with whom they work, with particular emphasis on their relationships and roles in society.

In the coming months we will be using our rigorous social impact analytics to contribute to the knowledge of what makes ex-offenders desist from crime. Our previous evaluation of the academy’s prison to community service, produced evidence of SSA’s excellent engagement of ex-offenders into their programme. With the foundation’s funding we will build on that by using our advanced statistical analysis to identify who benefits from the programme, how and in what circumstances. These analyses will assist the academy to identify the most effective practice and allow it to develop its professional programmes. To ensure that it has the right information at the right time we will be building a dashboard to communicate this data to those delivering the programme, their managers and the funder.

Not only will this improve their practice, but it has important implications for government’s Transforming Rehabilitation agenda. That agenda depends on organisations like SSA being commissioned to deliver services through the private community rehabilitation companies. However, the participation of such organisations has been low as they struggle to demonstrate their impact on reoffending pathways and desistance from crime. Access Impact is helping to overcome those obstacles and by funding our work, will enable SSA to attract further funding and make the systemic changes that are essential to support men and women to desist from crime.

To find out more about our analytics services and how we can help your organisation demonstrate your impact definitively, contact us on 020 3371 8950 or email jack.cattell@getthedata.co.uk

Get the Data and Gideon’s Promise Team Up for Innovative Metrics Get The Data

Get the Data and Gideon’s Promise Team Up for Innovative Metrics

Over the past year, GtD has been proud to be work with Gideon’s Promise to develop measures of culture change that will transform the American criminal justice system. We are delighted to welcome Ilham Askia, Executive Director of Gideon’s Promise to guest blog about our partnership.

Get the Data and Gideon’s Promise Team Up for Innovative Metrics Get The Data

Ilham N. Askia
Executive Director
Gideon’s Promise

Gideon’s Promise is a U.S.-based, non-profit whose mission is to transform the criminal justice system by building a movement of public defenders who provide equal justice for marginalized communities. Gideon’s values-based approach uniquely trains public defenders to use a “client-first” method to defense practice. The impact that this approach can have on culture change in public defense will ultimately transform an entire criminal justice system. Since January 2017 we have been working with GtD to create an innovative approach to measure the effects of culture change in public defense systems.

While most experts focus on case outcomes and court processing to determine the effectiveness of a public defense system, Get the Data and Gideon’s Promise have designed a way to measure the effects of a values-based approach to training and supporting public defenders. Gideon’s Promise believes that, in order to have equity in the criminal justice system, the lawyers who represent the accused must be excellent in their profession but also care about the dignity of their clients. For over 80% of the people who are charged with a criminal offense, a public defender has the enormous responsibility to not only defend a charge but also illustrate the humanity of a defendant. If administrators of justice could view the accused more humanely, then the treatment of people warehoused in jails, prisons and courtrooms would be different. Because the criminal justice system does not effectively function to rehabilitate offenders, the dignity of the human spirit is stripped away as soon as an individual is accused of an offense. The culture of the system must change from viewing people as case numbers and files and more like human beings with lives that have value. Public defenders tell the stories of those who are deemed unworthy of support. Public defenders remind the system about the value of human lives.

Get the Data has designed metrics to discern whether the culture of an office, the confidence of a public defender, and environment play into how defendants feel about their representation. These metrics also measure whether the attitudes of Gideon’s Promise public defenders affect the public defender offices where they work, the courtrooms where they practice and the relationships they have with their clients. While no one else is measuring the impact of culture change in the criminal justice system, Get the Data and Gideon’s Promise are on the verge of a transformative approach to repairing a broken criminal justice system by using public defenders as anchors to reform and explaining their importance through quantitative data that measure qualitative relationships.

Although Gideon’s Promise is a national organization, it primarily focuses its work in the southeast region of the U.S. where the highest concentration of inequity exists. There are six public defender sites piloting these metrics. The Defender Value Spectrum Survey (DVSS) and the Client Evaluation Survey (CES) are being completed by a sample of Gideon’s Promise influenced public defender offices and clients who received services from those offices. Our goal is to conclude whether a values-based trained attorney positively correlates with how people view public defenders, clients and how both are treated in the court system. Are there sets of core values to public defense training and mentoring that systemically changes the culture? Is Gideon’s Promise’s curriculum aligning with the goal to transform the culture of public defense? Does a caring lawyer matter? These are all questions this study will answer. Our hope is that we can replicate this evaluative process across the country to not only inform Gideon’s Promise’s programming but also encourage public defense systems to adopt our model while providing data that our culture change model works. Caring, competent and committed lawyers are essential to true criminal justice reform.

This partnership is crucial to capturing a values-based approach to criminal justice reform. Gideon’s Promise is truly grateful to be working with Get the Data on this ground-breaking work.

TR: PbR results - speakers who presented to Community Rehabilitation Companies and other justice sector employees

Community Rehabilitation Companies: PbR Results Event

Transforming Rehabilitation is the UK government’s programme of outsourcing probation services to new community rehabilitation companies. In a radical move, the government is now paying these new companies by the reduction in reoffending results they achieve. GtD is at the forefront of this by providing our cutting-edge social impact analytics to Sodexo Justice Services who manage a number of these new companies.

The first PbR figures were published last month and GtD has been active in informing the debate on their significance. As part of this debate, we recently hosted a sell-out event for senior management and practitioners working in community rehabilitation companies and the justice sector.

An expert panel comprising Prof. Darrick Jolliffe of Greenwich University (above left), Dr Sam King of Leicester University (above right) and GtD’s own Jay Hughes (above centre left), considered the initial findings and what to do next, with Jack Cattell (above centre right) setting out a new vision of how predictive analyses can be used by practitioners to improve performance.

Community Rehabilitation Companies: PbR Results Event Get The Data

Prof. Darrick Jolliffe – University of Greenwich

If you were unable to attend but would like to learn more about how GtD could support you in evaluating your social impact outcomes or for a free predictive analytic roadmap for your CRC, contact Jack Cattell  The event presentations can also be viewed via the link below:

Transforming Rehabilitation – Learning from the PbR results presentations

Community Rehabilitation Companies: PbR Results Event Get The Data

Dr Sam King – University of Leicester

 

Community Rehabilitation Companies: PbR Results Event Get The Data

Jack Cattell – GtD

 

We’ve also set up a LinkedIn group as a forum for shared learning and discussion, for individuals who work or have an interest in, the fields of probation, offender rehabilitation and Transforming Rehabilitation. Click here to request to join – Transforming Rehabilitation

Image to illustrate no data can mean no voice and no idea

No Data? No Voice, No Idea – The Importance of Data

The collection and analysis of data must never be allowed to fall by the wayside – it’s a founding stone, not a ‘nice to have’.

Of course, I would say that, wouldn’t I? But here are six concrete reasons why data is important, drawing on Get the Data’s recent work with the Advice Services Alliance (ASA) as a case study.

  1. Data Is the Great Persuader

There is no more powerful tool for influencing stakeholders than data, as I explained in more detail in this recent blog post on using data to influence stakeholders.

ASA works with its members – various associations who provide advice services – to capture and analyse data from the front-line. This gives weight to conversations with all of those who have an interest in ASA’s direction of travel, reassuring them that strategic decisions are being made in response to changing needs, and reinforcing the professionalism that underlies the work they and their partners do.

 

  1. Data Means Funding

In particular, data is invaluable in drawing in new sources of funding, and persuading potential funding providers. Faced with a choice of projects or programmes in which they might invest, with ever-tighter budgets, funding bodies will regard convincing data as a good reason to choose your work over others. ASA will use data to add weight to its funding applications for this reason.

 

  1. Data Gives You the Power to Lead the Debate

ASA seeks to lead thought, representing its members in national discussions by highlighting issues affecting the people who use advice services. The body of data and analysis to which ASA will refer confers authority and allows the organisation to direct the debate and steer collective thinking around youth issues.

 

  1. Data Defines Good (and Bad) Practice

Using data provided by its members ASA will be able to identify areas for improvement in front line practice and also to pinpoint what is working especially well so that good practice can be shared across the community. It informs training programmes, service improvements and helps determine how resources should be employed for maximum impact.

 

  1. Better data and measurement development 

Good data leads to better data. Use of data means we learn its limitations and how to outcome those. We also learn how to measure the right outcomes, better; particularly in how to measure informal outcomes along said formal attainment.

 

  1. Data Is Cheaper and Easier Than Ever

Cloud-based databases are cheap and easy to implement compared to the cumbersome systems of the past. They make it easier for people to enter data and share it. So there’s really no excuse for failing to collect and analyse data in this day and age.

If you would like to find out more about our cutting-edge approach to data capture and analysis please get in touch.

GtD Transforming Rehabilitation Event

Transforming Rehabilitation: Learning from the PbR results

Transforming Rehabilitation challenged Community Rehabilitation Companies (CRCs) to reduce reoffending significantly and October will see the publication of the reoffending rates of the first Transforming Rehabilitation cohort.

These results are an important test of the government’s Transforming Rehabilitation agenda and the success of the new CRCs. Some CRCs will be happy with their results, while others will need to improve. The published results also provide opportunities to commission new services for offenders.

Besides the implications for policy and practice, the results also present an opportunity for your organisation to learn how best to reduce reoffending with the tools at your disposal. You might want to increase a CRC’s performance, identify “what works?” and roll that out across all your CRCs, or prove that your specific service will make the difference.

Come join our experts at our TR PbR event, to discuss what needs to be learnt, what is within your control, and how embedding impact analytics in your CRC or service delivery organisation will help you to reduce reoffending.

Transforming Rehabilitation: Learning from the PbR results

Tuesday 28th November

2 – 4.30 pm

The Space Centre

94 Judd St, Kings Cross, London WC1H 9NT

SPEAKERS:

Professor Darrick Joliffe, University of Greenwich

“How to measure the quality of relationships between offenders and probation officers.”

Darrick is Professor of Criminology at the University of Greenwich. Darrick is interested in the broad areas of developmental life-course criminology, programme evaluation, prison research and psychology, individual differences and offending. He has developed tools to measure the satisfaction of offenders and the quality of their relationships with probation officers, and has estimated the association between relationship quality and reoffending. Darrick will also chair a discussion after the presentations.

 

Dr Sam King, University of Leicester

“What is high quality offender management?”

Sam is an expert on offender management and rehabilitation, and has contributed significantly to developments in desistence theory. His talk will discuss the latest evidence on what makes high quality offender management and what can be implemented to reduce reoffending. Sam has published widely on probation work and desistence theory, and has helped CRCs to implement innovative offender management tools to measure offender motivation.

 

Jack Cattell, Get the Data

“Predict your PbR results and continuously learn how to improve them before they happen.”

Jack is an expert on the prediction of reoffending rates, and the analysis of how a probation service can audit and improve its reoffending rate. He has worked for the Ministry of Justice, former probation trusts and Sodexo Justice Service’s six CRCs on research and analysis projects. He heads up GtD’s thought leadership on the use of predictive analysis to improve adult and youth offender management.

Free event – register today! Places are limited.

 

Headed by Jack Cattell, Get the Data is located in the UK and USA. We are an international thought leader on how to reduce reoffending through high quality analyses of offender management data. We have delivered high-profile projects for the Ministry of Justice, HMPPS (formally NOMS), and the police. We are proud to provide our Social Impact Analytics service to Sodexo Justice Service’s six CRCs.

 

Databse word cloud

Choosing a Database? Just Keep It Simple

When choosing a database for your project or programme, made to measure software or flashy online tools aren’t necessarily the right choice.

If you’re going to gather and analyse data in a serious way of course you need a database of some sort and, having spent much of my academic and professional career buried in them, I’m very much an advocate. But I too often see evidence of the database itself being regarded as the end rather than the means – as the solution to the challenge at hand rather than a tool for addressing it.

Perhaps that’s because we all so often feel under pressure, either external or self-imposed, to demonstrate what we have delivered in concrete terms. A specially commissioned database, perhaps with a catchy name and fancy interface, is something a project manager can point to and say, ‘I made this.’

The problem is that it takes huge amounts of time, resources and testing to create excellent software from scratch or to implement a powerful online tool, meaning that in practice these products all too often become clunky, creaky and frustrating. And products that don’t work well don’t get used.

As always, the answer is to focus on what you want to achieve. That will help you understand what kind of data you need to collect, who will be collecting and managing it and, therefore, what kind of system you need to house it.

Sometimes, of course, a custom-designed database or powerful online application are absolutely the right choice but, in my view, those occasions are actually very rare. More often than not the humble, rather plain Microsoft Access, or a similar tried and tested generalist professional product, is not only cheaper but also better suited to the task. Remember, these programmes have been worked on over the course of not only years but decades, and have huge amounts of resources behind their development and customer support programmes. Standard software also makes sharing, archiving and moving between systems faster and easier in most cases.

It is also possible to customise Access databases to a fairly high degree, either in-house using the software’s in-built features, using third-party software, or by hiring developers to create a bespoke front end interface without getting bogged down in the complicated underlying machinery.

And cloud data storage has made all this easier and cheaper than ever.

In conclusion, I’d like people to recognise that the real deliverable isn’t necessarily software, and that there’s no shame in off-the-peg. After all, a database is only as good as the information it holds and a clean set of useful, appropriate data is what people should really be proud of.