Impact Evaluation and Social Impact Bonds

How Rigorous Impact Evaluation Can Improve Social Impact Bonds

Social Impact Bonds

In recent years, Social Impact Bonds are being increasingly used by the British government to deliver public services via outcomes based commissioning. They are also becoming increasingly common in the U.S. By linking payments to good outcomes for society, SIBs are used not only to provide better value for money, but also as a driver of public sector reform. In the words of guidance published by the British government’s Cabinet Office:

“[Social Impact Bonds] are … designed to help reform public service delivery. SIBs improve the social outcomes of publicly funded services by making funding conditional on achieving results. “

While SIBs are not without their critics, their proponents argue that the bonds are a great way to attract private investment to the public sector while focusing all partners on the delivery of the desired social outcomes. This new way of commissioning services also encourages prime contractors to subcontract delivery of some service to the community and voluntary organisations, who bring their own experience, expertise and diversity to the provision of social services.

GtD have completed evaluations that have helped shape social impact bonds, and through our work we have identified five key questions that should be asked by anyone thinking of setting up a SIB or is looking to improve the design of their SIB:

1. Will it work?

Some services delivered by SIBs fail before they start because the planned intervention cannot plausibly achieve the desired outcome. In other words, just because an intervention reduced the number of looked after children entering the criminal justice system doesn’t mean it should work for all young people at risk of offending. That said, if the evidence base around a particular intervention is weak that does not mean one should not proceed – but it should promote a SIB design that includes an evaluation that can state quickly whether the SIB is delivering the hoped for outcomes.

2. Who can benefit from this intervention and who can’t?

We all want to help as many people as possible. However, we can quickly lose sight of who we are seeking to help when we are simply meeting output targets. In other words, if public services are funded by the number of clients they see, then providers could be tempted to increase numbers by accepting referrals of people for whom the service was not intended. So to achieve your outcomes – and receive payments – it’s vital to monitor intelligently the profile of your beneficiaries and ask yourself, “If my targeting were perfect are these the clients I would want to work with to deliver my intended outcomes?”

3. Are we doing what we said we were going to do? And does it work?

Interventions can fail simply because they don’t do what you said you were going to do. If, for example, you are working with young people to raise career prospects and your operating model includes an assessment of need (because the evidence base suggests that assessments increase effectiveness) then it should be no surprise that you did not meet your outcome targets if an assessment was completed with only half of your clients. Identifying your key outputs, monitoring their use and predicting outcomes based on their use can give you much greater confidence in achieving your intended impact.

4. What do we need to learn and how do we learn quickly?

All SIBs we have seen collate a lot of data about their beneficiaries and the service provided but few use those data to their full potential. With predictive analysis, we can monitor who appears to respond best, who is not benefiting and what form of service delivery is the most effective. In other words, is one-to-one work or group work more cost effective? As such you can learn how to define your referral criteria better or learn how to improve your operating model, even within the first months of a SIB.

5. How can we build a counterfactual?

A counterfactual is an estimation of what outcomes would have been achieved without the SIB. Comparing your SIB’s outcomes to the counterfactual can highlight some areas for learning and how to improve over time. Consideration should be given to the counterfactual at the commencement of the SIB. Full advantage can be taken of the publically available data sets to construct the counterfactual: for example, the National Pupil Database, Justice Data Lab or the Health Episode statistics. (Top tip: consent from beneficiaries to use these data sources is generally required).

To discuss how GtD’s impact evaluation can help improve your organisation’s SIB, please contact Jack Cattell, jack.cattell@getthedata.co.uk

Image of data being analysed

There’s no Magic Way of Measuring Impact

Wouldn’t it be great if there was a way of measuring your social impact across multiple projects using a single dependable statistic? Well, I’ve got some bad news, and some good.

I was recently talking to a charity who wanted to know how if they could go about measuring and reporting the overall impact of the organisation on children and families. With multiple strands each aiming to achieve different things, they asked if a single outcome measure – one accurate, reliable number – to sum up the impact of the whole organisation was either possible or desirable.

First, here’s the bad news: it’s very unlikely – I might even be so bold as to say impossible – that any such thing exists. You might think you’ve found one that works but when you put in front of a critic (or a nitpicking critical friend, like me) it will probably get ripped apart in seconds.

Of course, if there is a measure that works across multiple projects, even if not all of them, you should use it, but don’t be tempted to shoehorn other projects into that same framework.

It’s true that measuring impact requires compromise but an arbitrary measure, or one that doesn’t stand up to scrutiny, is the wrong compromise to make.

The Good News

There is, however, a compromise that can work, and that is having the confidence to aggregate upwards knowing your project level data are sound. You might say, for example, that together your projects improved outcomes for 10,000 families, and then give a single example from an individual project that improved service access or well-being to support the claim. In most situations that will be more meaningful than any contrived, supposedly universal measure of impact.

Confidence is the key, though: for this to work you need to find a reliable way of measuring and expressing the success of each individual project, and have ready in reserve information robust enough to hold up to scrutiny.

Measuring Means Data

In conclusion, the underlying solution to the challenge of measuring impact, and communicating it, is a foundation of good project level data. That will also make it easier to improve performance and give you more room to manoeuvre. Placing your faith in a single measure, even if you can decide upon one, could leave you vulnerable in a shifting landscape.

 

Social impact webinar

Webinar: Make 2020 the Year of Social Impact

On both sides of the Atlantic, our clients are working hard to think of new ways to deliver vital services to their clients: the homeless, the refugee, offenders.  They are often the people who are most likely to be at the margins and their needs are urgent.  So, now is a time for adapting services to the weeks of lock-down and shelter in place orders. And at GtD we are adapting the ways in which we are delivering our services and helping our clients.

While we wait for the pandemic to subside, the needs of our clients and yours remain pressing, and that will continue long after CoVid19 becomes yesterday’s news. However, I doubt that we will go back to work just as we did before we heard about social distancing and self-quarantining.  When we do go back to work, we will be asked to do things in new ways, to answer new needs and to find new partners to work with.

2020 is GtD’s “Year of Social Impact” when we are encouraging non-profits and charities to think about new ways of demonstrating their social value – definitively. We started out with a roadshow of events in Atlanta and then London, meeting with leaders from the nonprofit sector, listening to their needs and telling them about how our “Measure”, “Learn” and “Prove” packages can assist them.

We were due to take the roadshow to Manchester on April 28th.  Instead, we will be delivering the event as a webinar and inviting people to attend from across the UK and the USA.  This is bound to be a fascinating meeting as we share experiences across the Atlantic and hear more about how GtD’s social impact analytics are helping non-profits on both sides of the ocean.

The webinar is free and we have made it easy to join.  To register, please use this link and learn more about how to attend.

At times like this, we need to keep going as best we can and re-evaluate our priorities. Today, I received some words of wisdom from a valued client and jazz aficionado.  He and I were talking about how best to address the needs of those who have no medical care in rural Georgia. We had to suspend some work and address the new priorities thrown up by lockdown, and so I would like to share his words with you:

“What we’ve been doing lately reminds me of an announcement Louis Armstrong made during an interruption in one of his concerts, “And while they’re fixing the mike, we are going to lay on you our very best version of ‘Tin Roof Blues’”. Getting important things done during times like these can be helpful both to our projects and to our morale”.

Jack and I hope to meet you at the webinar but please contact me on alan.mackie@gtd-us.com if you have any questions. In the meantime, we wish you all the best as you address your new priorities and redouble your efforts to serve your clients.

Learn from data

What You Can Learn From Your Data

In a world driven by data, it’s becoming increasingly important for social organisations to prove their impact – not just through what they say, but through cold, hard facts.

While many already collect information on the work they’re doing, there remains a gap.

There’s a limit to what you can understand from the numbers alone, and simply reporting the data can feel far removed from the real-world and the day-to-day work done by the workers in your organisation.

For your information to become truly valuable, you need to start unpicking it. By applying the right methods of data analyses, you should be able to learn from your data and to use the insights you draw out of it to make practical changes within your organisation. This approach to data analysis will allow you to optimise your resources and maximise your results.

More than that, you’ll be able to extract the story behind your successes, and use that to show the world the valuable social impact your organisation is achieving.

Empower your staff

Looking at the data alone, it can be difficult for practitioners to know where to concentrate their efforts, and where changes to their service are most needed. But with the right analysis, the data you already have can be transformed into immensely valuable information to guide practitioners’  decisions, and highlight areas for improvement.

Learning from your data can give your practitioners insights on their individual clients, on how the team is performing, and on the organisation as a whole. It means they’ll know exactly what you need to target, and how to fine-tune your service to achieve the best possible results.

Understand your beneficiaries

No doubt you have worked in your sector long enough to know that your clients are not all the same. By applying the appropriate analytical techniques, you can understand the differences between clients and tailor your services to their varying needs.

Methods like segmentation and cluster analysis allow you to pick out patterns across different groups of people with different characteristics, and identify not only “what works?”, but “what works for whom?”.

Predict future outcomes

Your data can tell you not just what’s happened to date, but the outcomes that your organisation could produce in the future.

This can be done by applying scenario and simulation analysis, to predict the potential results of a variety of situations. These predictions should update dynamically, changing over time as other conditions change.

What’s really valuable about these predictions is that you can use them to optimise your resources, and to expend them in a way that will have maximum effect. That’s important for any organisation to know, but particularly helpful for non-profits and social enterprises whose resources are often limited.

Know your social impact, and prove it

Learning from your data gives you the confidence of knowing – not just hoping – that what you’re doing works. It lets you know that you’re making a real impact in your field, but perhaps more importantly, it offers assurance to the people who are looking for that proof.

Your funders and stakeholders need to have confidence in what your organisation is achieving, and by analysing your data you can give them just that.

To take a  youth service as an example, a potential funder might want to know how much of a difference the organisation is making to youth unemployment in the area. In answer, it could use examples from the analysed data, and effectively demonstrate its improvement in comparison with a control group.

By telling the story of its impact, backed up with concrete evidence and analysis, the youth service would be able to provide a much more convincing argument than it would be, using the raw data alone.

Learn from your data with GtD

Get the Data works with organisations in the UK and the US that want to prove their impact on society. We provide social impact analytics in the form of three main packages: Measure, Learn and Prove.

Our Learn service sits in the middle, making it a perfect next step for organisations that have collected data on their service but need to understand it better in order to use it. “Learn” can also be used in combination with our “Measure” and “Prove”  services, in any order that suits you.

Beginning with a strategic review of your data, we’ll put together an insight strategy and follow up with regular reports to monitor the results – integrating those reports with your own systems so they’re dynamic, useful, and convenient to use.

If you know you could be doing more to learn from your data but you’re not sure where to start, take the first step and contact us today.  We will get you started on your social impact journey.

Use Data to Tell an Effective Story Get The Data

Use Data to Tell an Effective Story

Over the past eight years, GtD has been providing social impact analytics to tell our clients’ stories. Effective stories of what their interventions do, who they work with, and the difference they are making to the people they serve: the homeless, young people, public defenders, refugees and immigrants. But, while we are primarily using numbers rather than words to monitor and evaluate our clients’ work, we need to take care that we are following the golden rule of story-telling.

Golden Rule of Story Telling

The golden rule is that a story should have a beginning, a middle and an end. I’ve just finished Sir Ernest Shackleton’s “South”, the Edwardian explorer’s account of his trans-Antarctic exhibition. The story opens in the months just before the First World War, with the preparations for the expedition, all the charting and planning and the resources being loaded on to his ship “Endurance”. The middle of the story is about the embarkation from Plymouth and the ship sailing into the South Atlantic, and then Shackleton leading his crew across the mountains and glaciers of Antarctica with the constant charting and re-charting of their route to safety. The tale ends with the crew’s rescue and return home. While Shackleton did not achieve his stated objective, “South” provides a compelling account of how a leader at times must respond to adversity by altering course and deploying available resources to the best effect.

Many organizations that work to make a social impact use words to tell the value of their work. This is, often done in the form of case studies that tell their clients’ stories. I recently came across such a case study in the annual report of a London-based homeless charity. It tells a compelling story about the effectiveness of their work with a client referred to as “David”. David came to London looking for a job, but soon became homeless and was found by the charity’s outreach worker sleeping rough. Having set out the “beginning” of David’s story, the case study then provided an account of the outreach worker’s role in helping David find hostel accommodation and registering him with a doctor and the benefits agency. The story ended well for David, as he found employment, a home and was returned to good health. A simple but effective story that informed the reader about the charity and how it benefits its clients.

The Descriptive Power of Numbers

Like an epic adventure or a compelling case study, we can use numbers rather than words to tell an effective story. However, before we do so, we need to get our story straight and give some thought to which data should provide the “beginning”, the “middle” and the “end”. If we get this right, then the story can follow the golden rule. So, in telling David’s story with data, we should start at the beginning and identify all those things that contributed to the delivery of the outreach service. We call these “inputs”, and they might include the number of outreach workers employed, their shifts, and the training they received.

Next, we will want to find the numbers that describe the services that David received. These are known as “outputs” and will include the number of contacts with the worker, the time taken to make referrals to the general practitioner and the benefits agency, and the contacts with the local hostel. Finally, we need to find the numbers that will describe the “outcomes” that were achieved for David, which will include changes to his employment, accommodation and health. Of course, we will be collecting these input, output and outcome data for David and every other client seen by all the outreach workers.  This will become an aggregated database of information that will tell a much broader and compelling story of who the charity helps, how it helps them and what outcomes their clients achieve.

Telling your story with GtD

Telling an effective story is essentially what GtD’s “Measure” package is all about.  It will help your organization monitor its activities and evaluate its outcomes. Contact us today if you want to learn more about how “Measure” will help your organization realize the potential of its data. “Measure” will assist you in deciding what to measure and what data to collect. It will provide you with analyses to solve problems and achieve better outcomes. “Measure” doesn’t just give you data, it gives you data to tell an effective story.

If you’d like to learn more about social impact analytics and how it can benefit your organisation, come along to our free event in Manchester on 28th April. Visit our events page to learn more and book your place.

A New Year’s Resolution?

A New Year’s Resolution?

New Year’s Eve, and time to take a moment to make a resolution for 2020.  Last year, according to the Statistica website, 71% of Americans resolved to diet and eat better in 2019 with a further 65% intending to exercise more.  It is not unexpected, perhaps, that 8% of those making resolutions failed to keep it within hours of the New Year being rung in, but a whopping 16% claimed to have kept their resolution for two to four months!

New Year’s resolutions are about making changes and adopting better habits. They are often personal in nature, but have you thought of making a New Year’s resolution for your organization? In the course of 2019, have you ever wondered how to measure the difference your organization is making on society?  As we peer into 2020, are you curious to see how your organization can improve the services it delivers to your clients? Is it now time to get the evidence to demonstrate your organization’s impact – definitely?

If the answer to any of those questions is “yes”, then come along to our free seminar on Wednesday, 8th January in London and learn more about “Measuring” your activities; “Learning” how to make a real difference; and, “Proving” your impact definitively.  For more details and how to book your place for this event, please go to http://bit.ly/SocialImpactSeminar.

Resolve to make 2020 the year you and your organization improves its social impact – and be sure that GtD will be with you every step of the way so that this will be one resolution you will keep.

Join us for a social impact journey seminar

Book Your Winter Getaway – Now!

The holiday season seems to be a time for journeying. As the school nativity plays remind us, the first Christmas saw the little donkey carry Mary and Joseph to Bethlehem and the “star of wonder” led the Three Wise Men to the stable. Today, thousands of British and American travelers will drive and fly home to be with family for the holidays.  And, no sooner is the festive period over, than hundreds of us will be heading to the ski slopes and beaches for a dose of much-needed sunlight. Winter is, indeed, a time for journeying.

At this time GtD’s London-based team invite you to join us on a journey. Unlike the Winter Getaways you see advertised in the press and television, we are offering to take you on a journey to understand your organization’s impact on the communities and people it serves.  At GtD we call this your “social impact journey” and we will travel with you to gather data, unlock its meaning and use it to demonstrate the effectiveness of your work.  Each of these is a stopping point on a single journey.

If you are interested in learning more, then you are warmly invited to join a free seminar on Wednesday, 8th January in London. For more details and how to book your place for this event, please go to http://bit.ly/SocialImpactSeminar.

GtD’s “Winter Getaway” doesn’t promise you golden beaches or pristine snow, but it will give you a good start to the new year as you and your organization embark on your social impact journey.  We look forward to seeing you there.

Closed eye image to illustrate measurement of qualitative data

More Than Eyes Can See? – Measuring Relationships

As our name implies, Get the Data is concerned with collecting and understanding quantitative empirical data. At the expense reprising scholarly debates about objectivity and truth, it is the Western empirical tradition that informs our research and the work of our clients who are engaged in criminal justice policy and practice. In other words, we want to measure things that can be touched, seen and realized by our senses.

Since the early 90s, much of my work has been the production of empirical data on the operation of the criminal justice system. As a young Home Office researcher, I cut my teeth in the back row of Sunderland magistrates’ court counting the reasons for cracked and ineffective trials. And it continued from there: reviewing police counting rules, measuring youth re-offending, tracking the attrition of DNA evidence from a crime scene to conviction in court.

The development of criminal justice policy demands the rigor of these empirical measures. So, I am proud of my colleagues who provide cutting-edge social impact analytics to a range of clients, from the Ministry of Justice and Sodexo Justice Services to smaller not-for-profits who work with offenders, such as Street Soccer Academy and Best for Pets.

That said, I was struck by two guest blogs that we published earlier this year.  Both Ilham Askia and Prof. Darrick Joliffe wrote about the growing interest in understanding the importance of the relationship between public defenders and offender managers and their respective clients.  Understanding the importance of these relationships is a departure from the measurement of objective outcomes that are recorded in either a police custody suite or pronounced in the court, and I am delighted that GtD is at the forefront of this innovative work.

A few years ago, I read “More Than Eyes Can See”, Rhidian Brook’s account of his nine-month journey in Africa, India, China and the U.S. to detail the Salvation Army’s response to the world-wide AIDs pandemic. By his own account, Brook was initially critical of the efficiency and effectiveness of local efforts to bring relief to those with HIV/AIDS and their families who were often ostracized by the community. Slowly he realized that for many there could be no “good” outcome: there is no cure and palliative care is too expensive for the most vulnerable.  So, in his words he admitted that, “I was slow to spot it really – focused as I was on empirical ways of measuring success – but [it was] the small acts of kindness that were holding things together. They weren’t the added extra, the bonus; they were it … After a while I stopped trying to measure the efficiency of these visits and see them for what they were: self-giving, sweaty acts of love”.

Relationships matter, even if just for their own sake. If the relationship between your organization and its clients is important to you, then please contact either Jack or me. We will be happy to discuss our ground-breaking approaches to measuring those things “that the eye cannot see”.

Make Social Impact Your Goal! - GtD and Street Soccer Academy

Make Social Impact Your Goal!

I am delighted to be working again with Street Soccer Academy (SSA) to put evidence of their social impact at the heart of their work with ex-offenders. This important work has been made possible by a grant from the Access Impact Foundation whose mission is to make charities and social enterprises more financially resilient and self-reliant, so that they can sustain or increase their impact. Without the generous funding from the Access Impact Foundation we would not able to provide Street Soccer Academy with our powerful data analysis.

Street Soccer Academy is a great client to work with. They use professionally organised sports-based programmes in the rehabilitation and reintegration of people from some of the nation’s hardest to reach groups, including ex-offenders. Our task is to prove that SSA’s pro-social models are affecting the attitudes and thinking of the men and women with whom they work, with particular emphasis on their relationships and roles in society.

In the coming months we will be using our rigorous social impact analytics to contribute to the knowledge of what makes ex-offenders desist from crime. Our previous evaluation of the academy’s prison to community service, produced evidence of SSA’s excellent engagement of ex-offenders into their programme. With the foundation’s funding we will build on that by using our advanced statistical analysis to identify who benefits from the programme, how and in what circumstances. These analyses will assist the academy to identify the most effective practice and allow it to develop its professional programmes. To ensure that it has the right information at the right time we will be building a dashboard to communicate this data to those delivering the programme, their managers and the funder.

Not only will this improve their practice, but it has important implications for government’s Transforming Rehabilitation agenda. That agenda depends on organisations like SSA being commissioned to deliver services through the private community rehabilitation companies. However, the participation of such organisations has been low as they struggle to demonstrate their impact on reoffending pathways and desistance from crime. Access Impact is helping to overcome those obstacles and by funding our work, will enable SSA to attract further funding and make the systemic changes that are essential to support men and women to desist from crime.

To find out more about our analytics services and how we can help your organisation demonstrate your impact definitively, contact us on 020 3371 8950 or email jack.cattell@getthedata.co.uk

Delivering Innovative Social Impact Analytics to Sodexo Justice Get The Data

Delivering Innovative Social Impact Analytics to Sodexo Justice

We are delighted to announce a new contract to deliver our ground breaking social impact analytics to Sodexo Justice, a leading provider of justice services in the UK.

The purpose of our social impact analytics is to provide definitive evidence of an organisation’s impact on society by delivering predictive analyses and impact evaluation.  Under the newly signed contract, we will measure the effectiveness of Sodexo’s six Community Rehabilitation Companies in managing the risk associated with the offenders and delivering interventions that reduce their reoffending.

By understanding “what works?” in changing lives and delivering safer communities, our social impact analytics will also be used by Sodexo Justice to measure the impact of its services.  Sodexo Justice will be paid through a payment by results mechanism that measures its success in reducing reoffending.

Our founding director, Jack Cattell said, “We very much look forward to providing our social impact analytics to Sodexo Justice Services.  Our SIAs will provide offender managers with the information they require to manage resources and deliver high quality interventions to reduce reoffending.”