Impact Evaluation and Social Impact Bonds

How Rigorous Impact Evaluation Can Improve Social Impact Bonds

Social Impact Bonds

In recent years, Social Impact Bonds are being increasingly used by the British government to deliver public services via outcomes based commissioning. They are also becoming increasingly common in the U.S. By linking payments to good outcomes for society, SIBs are used not only to provide better value for money, but also as a driver of public sector reform. In the words of guidance published by the British government’s Cabinet Office:

“[Social Impact Bonds] are … designed to help reform public service delivery. SIBs improve the social outcomes of publicly funded services by making funding conditional on achieving results. “

While SIBs are not without their critics, their proponents argue that the bonds are a great way to attract private investment to the public sector while focusing all partners on the delivery of the desired social outcomes. This new way of commissioning services also encourages prime contractors to subcontract delivery of some service to the community and voluntary organisations, who bring their own experience, expertise and diversity to the provision of social services.

GtD have completed evaluations that have helped shape social impact bonds, and through our work we have identified five key questions that should be asked by anyone thinking of setting up a SIB or is looking to improve the design of their SIB:

1. Will it work?

Some services delivered by SIBs fail before they start because the planned intervention cannot plausibly achieve the desired outcome. In other words, just because an intervention reduced the number of looked after children entering the criminal justice system doesn’t mean it should work for all young people at risk of offending. That said, if the evidence base around a particular intervention is weak that does not mean one should not proceed – but it should promote a SIB design that includes an evaluation that can state quickly whether the SIB is delivering the hoped for outcomes.

2. Who can benefit from this intervention and who can’t?

We all want to help as many people as possible. However, we can quickly lose sight of who we are seeking to help when we are simply meeting output targets. In other words, if public services are funded by the number of clients they see, then providers could be tempted to increase numbers by accepting referrals of people for whom the service was not intended. So to achieve your outcomes – and receive payments – it’s vital to monitor intelligently the profile of your beneficiaries and ask yourself, “If my targeting were perfect are these the clients I would want to work with to deliver my intended outcomes?”

3. Are we doing what we said we were going to do? And does it work?

Interventions can fail simply because they don’t do what you said you were going to do. If, for example, you are working with young people to raise career prospects and your operating model includes an assessment of need (because the evidence base suggests that assessments increase effectiveness) then it should be no surprise that you did not meet your outcome targets if an assessment was completed with only half of your clients. Identifying your key outputs, monitoring their use and predicting outcomes based on their use can give you much greater confidence in achieving your intended impact.

4. What do we need to learn and how do we learn quickly?

All SIBs we have seen collate a lot of data about their beneficiaries and the service provided but few use those data to their full potential. With predictive analysis, we can monitor who appears to respond best, who is not benefiting and what form of service delivery is the most effective. In other words, is one-to-one work or group work more cost effective? As such you can learn how to define your referral criteria better or learn how to improve your operating model, even within the first months of a SIB.

5. How can we build a counterfactual?

A counterfactual is an estimation of what outcomes would have been achieved without the SIB. Comparing your SIB’s outcomes to the counterfactual can highlight some areas for learning and how to improve over time. Consideration should be given to the counterfactual at the commencement of the SIB. Full advantage can be taken of the publically available data sets to construct the counterfactual: for example, the National Pupil Database, Justice Data Lab or the Health Episode statistics. (Top tip: consent from beneficiaries to use these data sources is generally required).

To discuss how GtD’s impact evaluation can help improve your organisation’s SIB, please contact Jack Cattell, jack.cattell@getthedata.co.uk

GtD's Social Impact Packages

Are you ready to embark on your social impact journey?

Since we founded GtD in June 2012, Jack and I have worked with many organizations that are on a journey.  A journey of discovery. A journey to measure their social impact – definitively. 

To assist our clients to navigate their social impact journey, we have created three services: “Measure”, “Learn” and “Prove”.  Each of these services is a package of support that will help you gather data, unlock its meaning and use it demonstrate the effectiveness of your work. 

Are you ready to embark on your social impact journey? Get The Data

Each of our Measure, Learn and Prove services are important stages of one journey, but where you embark and disembark is up to you. You can move between them in any order you like, and travel a long or short distance through each.

So how does each service help you navigate your organization to better outcomes?

Are you ready to embark on your social impact journey? Get The Data

Measure

Our Measure package enables you to solve problems and achieve better outcomes for your service users – download our Measure package guide.

Are you ready to embark on your social impact journey? Get The Data

Learn

Would you like to fine-tune your services, predict performance and optimise resources? Our Learn package provides the insights you need – download our Learn package guide.

Are you ready to embark on your social impact journey? Get The Data

Prove

You know the impact your work has. Our Prove package ensures others can see this as clearly as you do. Find out more by downloading our Prove package guide.

Are you ready to embark on your social impact journey? Get The Data

Plus! If you’re in need of a simple way to collect data? Our new ValiDATA tool could be the ideal solution! Based on Microsoft Access and Excel it’s cost effective and very easy to use. Find out more or contact us to arrange a free demonstration.

Jack and I have been journeying with organizations like yours for seven years and it has been greatly rewarding.  With the launch of our new packages, we are inviting you to come on board and work with us to reach your destination of choice. 

Please contact us for further information about how we can support you on your social impact journey and if you would like to arrange a free social impact review.

All Aboard for a Social Impact Journey Get The Data

All Aboard for a Social Impact Journey

Jack, Jay and I were delighted to host our London social impact seminar last week. We are very grateful to all who took the time to join us and contribute their thoughts and experiences.  As with every good seminar the learning was a two-way process. So, we talked about the social impact journeys we have been travelling on with our clients but also learned from our guests’ experiences of making a social impact.

All Aboard for a Social Impact Journey Get The Data

In introducing the seminar I noted that everyone in the room had a common cause: to improve the lives of the most vulnerable people in society. Most of the attendees represented national and London-based organisations that are concerned with mental health, homelessness and young people.  As always, Jack, Jay and I were struck by the commitment and deep knowledge that our guests brought to their work.  So what, asked one attendee, was our interest in working with these groups? A great question and my answer was straightforward: my colleagues and I simply want to provide our analytical expertise to demonstrate the value of our clients’ work.

Talk about data and analyses can often seem abstract and there is a danger that it might reduce vital work with vulnerable people to a calculus of efficiency and effectiveness. To avoid that, we were able to tell stories of how our work has helped other organizations to measure their impact, learn how to improve their services and demonstrate the success of their programmes.  Sometimes our work is used to demonstrate the value of a policy change or justify funding, but often our work simply helps managers and front-line workers think about their practice and keep track of the various services and activities that they deliver.

A valid concern that arose was how to avoid treating a vulnerable person as a unit of analyses rather than an individual with real needs and wants. This concern included how best to engage vulnerable people in the analyses to ensure that they can provide their voice and experiences.  This is a very current topic in the practice of evaluation.  For GtD’s part, we ensure that our work with vulnerable people is undertaken sensitively and in accordance with ethical standards, and that the processes of data collection and analyses do not further marginalize individuals and groups who have been excluded by institutions and society at large.

All Aboard for a Social Impact Journey Get The Data

 

In concluding the seminar, I reflected that the best social impact journey is one that is taken together.  The success of the journey will require trust and mutual respect between the analysts, workers and clients. I believe that the stories we told last week provide good illustrations of how GtD forge working relationships within which to deliver expert analyses and insight.

A New Year’s Resolution?

A New Year’s Resolution?

New Year’s Eve, and time to take a moment to make a resolution for 2020.  Last year, according to the Statistica website, 71% of Americans resolved to diet and eat better in 2019 with a further 65% intending to exercise more.  It is not unexpected, perhaps, that 8% of those making resolutions failed to keep it within hours of the New Year being rung in, but a whopping 16% claimed to have kept their resolution for two to four months!

New Year’s resolutions are about making changes and adopting better habits. They are often personal in nature, but have you thought of making a New Year’s resolution for your organization? In the course of 2019, have you ever wondered how to measure the difference your organization is making on society?  As we peer into 2020, are you curious to see how your organization can improve the services it delivers to your clients? Is it now time to get the evidence to demonstrate your organization’s impact – definitely?

If the answer to any of those questions is “yes”, then come along to our free seminar on Wednesday, 8th January in London and learn more about “Measuring” your activities; “Learning” how to make a real difference; and, “Proving” your impact definitively.  For more details and how to book your place for this event, please go to http://bit.ly/SocialImpactSeminar.

Resolve to make 2020 the year you and your organization improves its social impact – and be sure that GtD will be with you every step of the way so that this will be one resolution you will keep.

Join us for a social impact journey seminar

Book Your Winter Getaway – Now!

The holiday season seems to be a time for journeying. As the school nativity plays remind us, the first Christmas saw the little donkey carry Mary and Joseph to Bethlehem and the “star of wonder” led the Three Wise Men to the stable. Today, thousands of British and American travelers will drive and fly home to be with family for the holidays.  And, no sooner is the festive period over, than hundreds of us will be heading to the ski slopes and beaches for a dose of much-needed sunlight. Winter is, indeed, a time for journeying.

At this time GtD’s London-based team invite you to join us on a journey. Unlike the Winter Getaways you see advertised in the press and television, we are offering to take you on a journey to understand your organization’s impact on the communities and people it serves.  At GtD we call this your “social impact journey” and we will travel with you to gather data, unlock its meaning and use it to demonstrate the effectiveness of your work.  Each of these is a stopping point on a single journey.

If you are interested in learning more, then you are warmly invited to join a free seminar on Wednesday, 8th January in London. For more details and how to book your place for this event, please go to http://bit.ly/SocialImpactSeminar.

GtD’s “Winter Getaway” doesn’t promise you golden beaches or pristine snow, but it will give you a good start to the new year as you and your organization embark on your social impact journey.  We look forward to seeing you there.

TR: PbR results - speakers who presented to Community Rehabilitation Companies and other justice sector employees

Community Rehabilitation Companies: PbR Results Event

Transforming Rehabilitation is the UK government’s programme of outsourcing probation services to new community rehabilitation companies. In a radical move, the government is now paying these new companies by the reduction in reoffending results they achieve. GtD is at the forefront of this by providing our cutting-edge social impact analytics to Sodexo Justice Services who manage a number of these new companies.

The first PbR figures were published last month and GtD has been active in informing the debate on their significance. As part of this debate, we recently hosted a sell-out event for senior management and practitioners working in community rehabilitation companies and the justice sector.

An expert panel comprising Prof. Darrick Jolliffe of Greenwich University (above left), Dr Sam King of Leicester University (above right) and GtD’s own Jay Hughes (above centre left), considered the initial findings and what to do next, with Jack Cattell (above centre right) setting out a new vision of how predictive analyses can be used by practitioners to improve performance.

Community Rehabilitation Companies: PbR Results Event Get The Data

Prof. Darrick Jolliffe – University of Greenwich

If you were unable to attend but would like to learn more about how GtD could support you in evaluating your social impact outcomes or for a free predictive analytic roadmap for your CRC, contact Jack Cattell  The event presentations can also be viewed via the link below:

Transforming Rehabilitation – Learning from the PbR results presentations

Community Rehabilitation Companies: PbR Results Event Get The Data

Dr Sam King – University of Leicester

 

Community Rehabilitation Companies: PbR Results Event Get The Data

Jack Cattell – GtD

 

We’ve also set up a LinkedIn group as a forum for shared learning and discussion, for individuals who work or have an interest in, the fields of probation, offender rehabilitation and Transforming Rehabilitation. Click here to request to join – Transforming Rehabilitation

Databse word cloud

Choosing a Database? Just Keep It Simple

When choosing a database for your project or programme, made to measure software or flashy online tools aren’t necessarily the right choice.

If you’re going to gather and analyse data in a serious way of course you need a database of some sort and, having spent much of my academic and professional career buried in them, I’m very much an advocate. But I too often see evidence of the database itself being regarded as the end rather than the means – as the solution to the challenge at hand rather than a tool for addressing it.

Perhaps that’s because we all so often feel under pressure, either external or self-imposed, to demonstrate what we have delivered in concrete terms. A specially commissioned database, perhaps with a catchy name and fancy interface, is something a project manager can point to and say, ‘I made this.’

The problem is that it takes huge amounts of time, resources and testing to create excellent software from scratch or to implement a powerful online tool, meaning that in practice these products all too often become clunky, creaky and frustrating. And products that don’t work well don’t get used.

As always, the answer is to focus on what you want to achieve. That will help you understand what kind of data you need to collect, who will be collecting and managing it and, therefore, what kind of system you need to house it.

Sometimes, of course, a custom-designed database or powerful online application are absolutely the right choice but, in my view, those occasions are actually very rare. More often than not the humble, rather plain Microsoft Access, or a similar tried and tested generalist professional product, is not only cheaper but also better suited to the task. Remember, these programmes have been worked on over the course of not only years but decades, and have huge amounts of resources behind their development and customer support programmes. Standard software also makes sharing, archiving and moving between systems faster and easier in most cases.

It is also possible to customise Access databases to a fairly high degree, either in-house using the software’s in-built features, using third-party software, or by hiring developers to create a bespoke front end interface without getting bogged down in the complicated underlying machinery.

And cloud data storage has made all this easier and cheaper than ever.

In conclusion, I’d like people to recognise that the real deliverable isn’t necessarily software, and that there’s no shame in off-the-peg. After all, a database is only as good as the information it holds and a clean set of useful, appropriate data is what people should really be proud of.

We're hiring! Quantitative Researcher / Analyst Get The Data

We’re hiring! Quantitative Researcher / Analyst

Vacancy – Quantitative Researcher / Analyst Senior Quantitative Researcher / Analyst

Get the Data (GtD) is a successful and growing company, both in London and in Atlanta USA. GtD’s exciting social impact analysis approach is helping organisations on both sides of the Atlantic to measure, learn and prove their social impact.

At GtD we offer our employees an opportunity to gain extensive experience within all aspects of social impact analytics (SIAs) whilst working alongside leading social impact analysts and thought leaders in the industry.

Our service offering is unique in the industry. This means you will be gaining invaluable insights into, and experience of, SIAs. We are innovative and forward thinking so you will be helping to advance and evolve our social impact analytics tools and systems knowing that whilst you are doing so, you are ultimately helping other organisations improve the impact they have on society.

We offer our employees a supportive working environment, with training provided, to help you develop and enhance your skills.

We are looking for both newly qualified and more experienced quantitative researchers and analysts. You could be looking for your first role or want to apply your analytical skills to something more rewarding. But you must be passionate about how high quality, well communicated analysis can improve social outcomes. You will be looking forward to influencing senior people in the police, courts, probation services and third sector organisations on both sides of the Atlantic.

In other words, we are most interested in your potential.

Therefore you will be a self-starter, relish the opportunity to help a business grow and want to learn and apply advanced analytical and statistical techniques. You will provide quantitative analysis skills to our evaluation and social impact analysis practice in the areas of criminal justice, education and housing. Specifically you will support an evaluation of an intervention to reduce offending by high harm offenders, a project to predict reoffending rates and an evaluation of a young person’s training initiative. You will also work on police projects with our sister company Crest Analytics.

Quantitative researcher / analyst

You can demonstrate these essential skills:

  • Graduate degree in a subject with a substantial mathematical or statistics component
  • Report writing for non-technical audiences
  • Experience of using Microsoft Excel skills
  • Experience using SPSS syntax or R
  • Experience of applying statistical modelling techniques including regression analysis
  • Team working and ability to update and inform directors and project managers on progress and risks, and how to mitigate those.
  • Work independently to complete project tasks
  • Ability to work independently to improve your quantitative research and analysis skills, and to support the director to develop new products and business opportunities
  • Microsoft Office or equivalent and email
  • Willingness to travel abroad

 

Desirable skills for this opportunity are:

  • Understanding of or experience of the role of research and analysis in either criminal justice policy, careers and training, homelessness or third sector
  • Experience of using databases and writing SQL
  • Masters degree or PhD with a substantial statistical analysis component
  • One or more years’ experience of quantitative research or analysis

 

Senior Quantitative Researcher / Analyst

You can demonstrate these essential skills:

  • Graduate degree in a subject with a substantial mathematical or statistics component
  • Three or more years’ experience of quantitative research or analysis
  • Report writing and presentations for non-technical audiences
  • Good Microsoft Excel skills
  • Good SPSS syntax or R skills
  • Experience of SQL
  • Experience of applying statistical modelling techniques to complex social issues
  • Team working and ability to update and inform directors and project managers on progress and risks, and how to mitigate those.
  • Project management of research or analytical projects
  • Work independently to complete project tasks
  • Ability to work independently to improve your quantitative research and analysis skills, and to support the director to develop new products and business opportunities
  • Microsoft Office or equivalent and email
  • Willingness to travel abroad

 

Desirable skills for this opportunity are:

  • Understanding of or experience of the role of research and analysis in either criminal justice policy, careers and training, homelessness or third sector
  • Masters degree or PhD with a substantial statistical analysis component
  • Experience of web development with C# skills

 

Making an application

You will work from our London office (pro rata 38 hours per week). A competitive salary will be paid dependent upon skills and experience.

If you are interested please send your CV and cover letter with any current salary stated to iqqra.aziz@getthedata.co.uk by 5pm on the 27th October 2017.  Please indicate in your letter whether you are interested in a full-time or part-time position. Please also indicate if there are any dates in late October or early November you cannot make for an interview. If you wish to discuss the role please email jack.cattell@getthedata.co.uk.

 

The word 'victim'

Focusing on Victims of Crime isn’t Just a Nicety

National Crime Victims’ Rights Week in the US runs from 2-8 April 2017 and has prompted me to reflect on the importance of the victim’s voice in delivering effective criminal justice interventions.

NCVRW is led by the Office for Victims of Crime, part of the US Department of Justice, and has taken place every year since 1981. Victims’ rights bodies and law enforcement agencies across America take part.  GtD congratulates all the individuals who work with victims of crime, particularly those who will be honored at this week’s NCVRW award ceremony.

Its continued existence, and the coverage it generates, echoes the ongoing importance of victims of crime in the UK system. Here, victims’ rights have long been a policy priority for central government and a focus for the delivery of services among Police and Crime Commissioners.

Victims are important not least because their very existence indicates that the social contract has broken down. When government takes on responsibility for operating the crime and justice system using taxpayers’ money it does so with an implicit promise to keep citizens safe. Each victim – each shattering experience of crime and the pain felt in its aftermath – represents an individual point of failure, and together they gain a grim weight.

Accordingly, the victims’ lobby can be powerful. Quite rightly, victims’ stories elicit public sympathy and make real the cost of crime, reminding all of us that the damage done is not abstract but measured out in sleepless nights, lasting trauma, and grief. Victims’ satisfaction is therefore central to assessing public confidence in the criminal justice system.

In 2014 Get the Data undertook an evaluation of the Surrey Youth Restorative Intervention (YRI) on behalf of Surrey County Council and Surrey Police. Taking a restorative approach, the Surrey YRI works with both the victim and the offender to address the harm caused and hear how the victim was affected.  Often this concludes with an apology and some form of reparation to the victim.  The idea is that this not only helps victims but also young offenders, making them less likely to reoffend and allowing them to recognise the human cost of their actions. It also avoids criminalising them at a point in their lives when it is not too late to change track.

We found that the Surrey YRI satisfied the victims of crime that were surveyed. On the whole they felt that justice had been done and offenders were held to account. Some individuals also came out of the process expressing greater understanding of the offender and of the lives of young people in their communities. More than one respondent stated that the process made them realise that those who had victimised them were not ‘monsters’.

There’s little to argue with there, then, but such programmes would be hard to justify in today’s economic climate if they also cost a lot more. But, in fact, we found that the YRI cost less to administer per case then a youth caution, and so represented a value-for-money approach to reducing reoffending and satisfying the victim. Putting the needs of victims first, in this case, worked in every sense.

You can read the full text of our report on the Surrey YRI at the Surrey Council website (PDF) and find more information on our evaluation services on the Get the Data website.

The word truth in a maze

Upholding Evidence in the Post-Truth Era

Brexit and the new Trump administration have added “post-truth politics”, “fake news” and “alternative facts” to our political lexicon. Just words?

Well, “words matter”, as the former president reminded us when he launched his own candidacy eight years ago. And these new words matter very much when accompanied with calls to dismiss experts in favour of gut feeling and instinct. Democracy needs to be informed so hearing a senior member of the British Cabinet claim that “people in this country have had enough of experts” was one of the most lamentable claims of the Brexit campaign. All of this is surely at odds with a community of policy researchers and evaluators whose stock in trade is objectivity, fact-finding and balance. So where do we go from here?

Much food for thought comes from the 2016 “Evidence Works” conference. In September 2016, the British based “Alliance for Useful Evidence” collaborated with its American counterpart “Results for All” to convene “Evidence Works 2016: A Global Forum for Government”. Over two days in London, delegates from around the world shared ideas of how governments can use evidence and data in policymaking to improve outcomes for citizens and communities, and their findings are worth sharing here:

  • Government needs diversity of evidence to answer policy questions, and this evidence needs to be timely.
  • To maintain credibility, there is value in keeping some distance between evidence production and government. Independence gives extra authority to evaluation or analysis of policy.
  • Good communication is vital. It’s not easy to pare down a large body of evidence, and technocrats may need to be trained on how to write concise policy briefings.
  • Political leaders and members of the public need to be encouraged to use and demand evidence. Persuading politicians and those controlling the money is not enough. In democracies, we also need to persuade the public to care about evidence.

This was a truly global initiative and there were some notable examples of good practice that we in the West can learn from developing countries. A copy of the report can be found here.

At its best evaluation serves as Socrates’s “gadfly”, challenging assumptions and questioning implicit principles. Public servants – policy makers and evaluators – need to stand their ground, be reasonable and communicate evidence effectively.  If we are indeed entering an era of “post-truth” politics then we need more than ever “to speak truth to power”.