Posts

Non-Profits and the Measurement of What Matters

Tell Me What You Want, What You Really, Really, Want

Non-Profits and the Measurement of What Matters

Hard to believe that it is 25 years since the Spice Girls recorded “Wannabe”, and with it the launch of the “girl power” phenomenon. A catchy tune for all of us “Dad Dancers” out there, but what has it to do with (the more serious) topic of social impact measurement and non-profits?

Well, at the time Posh, Ginger, Sporty, Scary and Baby were storming the charts, I was settling into my job as a government researcher with the British civil service. 1995 was just before the dawn of New Labour’s managerial approach to public services, but already government was demanding greater efficiency and effectiveness. So, while the rest of the world was going all “zig-a-zig-ah”, I was co-authoring my first research paper, the rather ponderous “Managing the Courts Effectively: the reason for adjournments and delays in the magistrates’ courts in England & Wales”.

For the next 20 years, much of the evaluation work that my colleagues and I completed was framed by this managerialism. While working our way through the elusive “what works?” agenda, it became clear that the answers were of interest to government bean counters rather than the general public. So, “what works?” tended to be about government agencies doing things more swiftly, reducing costs to the Exchequer or moderating the politically troublesome issues of youth crime and anti-social behaviour.  While reducing crime in a neighbourhood, and bringing offenders to justice more quickly, remain laudable aims, were they what was “really, really wanted”?

When launching GtD in the US five years ago, we deliberately sought to partner with local and national non-profits. Not surprisingly they see things differently than government, and with that their focus is entirely on the people they serve. I have said it before, but our best non-profit clients know what they are about, and I have remarked that on visiting their offices there is a real energy to their business and commitment to their clients.  That is not to say that there is no tension between a non-profit’s wider goals and the narrower commitments made to a funder, but the best non-profits jealously guard their own ethos and interests.

I saw this first when working with the International Rescue Committee in Atlanta. Its Connect to Success program does vital work in helping young refugees get into employment, training, or education. However, it was clear that for a humanitarian organization the measures of success had to be more than the number of jobs found or college places achieved.  So, our evaluation also measures the wellbeing of the young people and their general satisfaction with their lives in the USA. Similarly, for the public defender organization Gideon’s Promise their metrics of success did not include reductions in convictions or fewer custodial sentences, but something more radical: the quality of the relationship between the public defenders and the clients they represent. Jack and I are currently getting to grips with a new rural health initiative in Georgia. The initiative seeks to tackle the social determinants of health, but it is important that local people should have a say in what good health outcomes look like for their own community.

Of course, non-profits must be diligent in demonstrating their economy, efficiency and effectiveness, but at GtD we also want to measure what our clients believe to be most important to them. This is all part of what we call our “social impact journey”. Before we start measuring the more obvious outcomes, we will take the time to investigate what an organization really wants to achieve, even when it finds that difficult to articulate clearly.  This is all part of what we call the client’s theory of change and it forms the basis for us to find new and exciting ways to measure a client’s true outcomes. If you want to learn more about what your organization can do to measure “what it really, really wants”, why not join our free webinar on October 29th. We will tell you more about GtD’s social impact journey, our packages of “Measure”, “Learn” and “Prove”, and what the next steps are to measure your social impact – definitively.

Theory of Change

Theory of Change – the Starting Point for Impact Measurement

In our work, we like to develop impact measurements from a fully articulated theory of change (ToC). Our clients, however, have approached us to show them how to measure something accurately and rigorously, and sometimes are unsure why we first need to spend the time talking to them about causal mechanisms, context and their long-term goal. And would prefer it if we went straight to the end and told them how to measure wellbeing, employability etc.

If you are thinking of improving your social impact measurement, here are 5 reasons to start with a theory of change:

  1. It identifies what is to be evaluated. Organisations are not always sure what of their many services is in scope for social impact measurement and the ToC helps them to set the boundaries on what they will and will not take responsibility to measure.
  2. It makes complexity understandable. Some charities and social enterprises are complex and deliver many different projects, in different ways, for different funders. A current client of ours has 26 unique projects. It can seem impossible to identify common threads and common measurement requirements, but the ToC helps our clients to see the wood for the trees and understand meaningful impact measurement is possible.
  3. It reminds you of what you’re trying to achieve. During day-to-day operations, it is easy to forget what your organisation is trying to achieve. The ToC brings the long-term aim front and centre and puts the pieces together to show how that is achieved.
  4. It tells a compelling story. The ToC describes how impact measurement data can be used to tell a compelling story for how you contribute to your long-term outcomes. By identifying your short term outcomes – those achieved during an intervention – and your end of programme outcomes, the ToC shows you how to use your data to evidence your impact in a compelling and persuasive way.
  5. It can tell you something you don’t know. In preparing a ToC we identify the evidence base that either confirms or amends your projects’ designs and presents any weaknesses in their current delivery.

The development of a ToC is part of our Measure package. Please click here for more information.

Image of data being analysed

There’s no Magic Way of Measuring Impact

Wouldn’t it be great if there was a way of measuring your social impact across multiple projects using a single dependable statistic? Well, I’ve got some bad news, and some good.

I was recently talking to a charity who wanted to know how if they could go about measuring and reporting the overall impact of the organisation on children and families. With multiple strands each aiming to achieve different things, they asked if a single outcome measure – one accurate, reliable number – to sum up the impact of the whole organisation was either possible or desirable.

First, here’s the bad news: it’s very unlikely – I might even be so bold as to say impossible – that any such thing exists. You might think you’ve found one that works but when you put in front of a critic (or a nitpicking critical friend, like me) it will probably get ripped apart in seconds.

Of course, if there is a measure that works across multiple projects, even if not all of them, you should use it, but don’t be tempted to shoehorn other projects into that same framework.

It’s true that measuring impact requires compromise but an arbitrary measure, or one that doesn’t stand up to scrutiny, is the wrong compromise to make.

The Good News

There is, however, a compromise that can work, and that is having the confidence to aggregate upwards knowing your project level data are sound. You might say, for example, that together your projects improved outcomes for 10,000 families, and then give a single example from an individual project that improved service access or well-being to support the claim. In most situations that will be more meaningful than any contrived, supposedly universal measure of impact.

Confidence is the key, though: for this to work you need to find a reliable way of measuring and expressing the success of each individual project, and have ready in reserve information robust enough to hold up to scrutiny.

Measuring Means Data

In conclusion, the underlying solution to the challenge of measuring impact, and communicating it, is a foundation of good project level data. That will also make it easier to improve performance and give you more room to manoeuvre. Placing your faith in a single measure, even if you can decide upon one, could leave you vulnerable in a shifting landscape.

 

Influence through Data

“Yeah, Says Who?” – Influence Through Data

You know you’ve achieved results – the data tells you so – but how do you influence sceptics to believe it?

It can be a rude awakening to take the findings of a study outside your own team or organisation, where trust and mutual support are more or less a given. In front of a wider audience of funding providers or other stakeholders, you will inevitably in my experience find yourself being challenged hard.

This is as it should be – scrutiny is a key part of a healthy system – but, at the same time, it’s always a shame to see an impactful project or programme struggle purely because its operators fail to sell it effectively.

Fortunately, while there are no black-and-white rules, there are some things you can do to improve your chances.

Confidence = Influence

When I present findings I do so with a confidence that comes with experience and from really understanding the underlying mechanics. But if you’re not a specialist and don’t have that experience there are things you can do to make yourself feel more confident and thus inspire greater confidence in your audience.

First, make sure you have thought through and recorded a data management policy. Are you clear how often data should be entered? If information is missing, what will you do to fill the gaps? What are your processes for cleaning and regularising data? Is there information you don’t need to record? A professional, formalised approach to keeping timely and accurate data sends all the right signals about your competence and the underlying foundations of your work.

Secondly, use the data as often as possible, and share the analysis with those who enter your data so that they can understand its purpose, and own it. Demonstrating that your data is valued and has dedicated, accountable managers hugely increases its (and your) credibility.

Thirdly, take the initiative in checking the reliability and validity of your own tools. If you use well-being questionnaires, for example, take the time to check whether they are really measuring what you want to measure in most instances. In other words, try to find fault with your own approach before your stakeholders so that when they find a weak point you have an answer ready that not only reassures them but also underlines the objectivity with which you approach your work.

Own Your Data’s Imperfections

Finally, and this might feel counterintuitive, you should identify the weaknesses in your own data and analysis and be honest about them. All data and analysis has limitations and being clear about those, and the compromises made to work around them demonstrates objectivity which, again, reinforces credibility.

In conclusion, the better you understand your own data and analysis, flaws and all, the more comfortable and confident you will feel when it, in turn, comes under scrutiny.

Social impact webinar

Webinar: Make 2020 the Year of Social Impact

On both sides of the Atlantic, our clients are working hard to think of new ways to deliver vital services to their clients: the homeless, the refugee, offenders.  They are often the people who are most likely to be at the margins and their needs are urgent.  So, now is a time for adapting services to the weeks of lock-down and shelter in place orders. And at GtD we are adapting the ways in which we are delivering our services and helping our clients.

While we wait for the pandemic to subside, the needs of our clients and yours remain pressing, and that will continue long after CoVid19 becomes yesterday’s news. However, I doubt that we will go back to work just as we did before we heard about social distancing and self-quarantining.  When we do go back to work, we will be asked to do things in new ways, to answer new needs and to find new partners to work with.

2020 is GtD’s “Year of Social Impact” when we are encouraging non-profits and charities to think about new ways of demonstrating their social value – definitively. We started out with a roadshow of events in Atlanta and then London, meeting with leaders from the nonprofit sector, listening to their needs and telling them about how our “Measure”, “Learn” and “Prove” packages can assist them.

We were due to take the roadshow to Manchester on April 28th.  Instead, we will be delivering the event as a webinar and inviting people to attend from across the UK and the USA.  This is bound to be a fascinating meeting as we share experiences across the Atlantic and hear more about how GtD’s social impact analytics are helping non-profits on both sides of the ocean.

The webinar is free and we have made it easy to join.  To register, please use this link and learn more about how to attend.

At times like this, we need to keep going as best we can and re-evaluate our priorities. Today, I received some words of wisdom from a valued client and jazz aficionado.  He and I were talking about how best to address the needs of those who have no medical care in rural Georgia. We had to suspend some work and address the new priorities thrown up by lockdown, and so I would like to share his words with you:

“What we’ve been doing lately reminds me of an announcement Louis Armstrong made during an interruption in one of his concerts, “And while they’re fixing the mike, we are going to lay on you our very best version of ‘Tin Roof Blues’”. Getting important things done during times like these can be helpful both to our projects and to our morale”.

Jack and I hope to meet you at the webinar but please contact me on alan.mackie@gtd-us.com if you have any questions. In the meantime, we wish you all the best as you address your new priorities and redouble your efforts to serve your clients.

Prove your social impact

Social Outcomes: The Road to Success is Paved With Good Intentions

I have often said that hardly a week goes by when I don’t encounter an individual or group of people working hard to make the world a better place. Often these people are working with those who are existing on the edge of society: the homeless who are ignored on our streets; refugees who feel that every hand is turned against them; the sick and frail who have no access to local medical provision; or individuals who are unfairly caught up in the “penal-industrial complex”. In every case, I see individuals and groups who have recognized a social need and have a clear intention to address it.

Resourcing & Focusing an Intention

While a good intention is essential to articulating a program’s vision, mission and rationale, intention alone is not enough. All our clients, however, will be able to explain the need for their work and can identify the time, money other resources that are required to make a positive difference.  As Jack and I have written in recent blogs, GtD’s “Measure” and “Learn” services assist our clients to enhance their work. In the case of “Measure” we are helping our clients to ensure that their good intentions are properly resourced, and with “Learn” we are applying sophisticated predictive analyses to make sure intentions are correctly focused for maximum benefit.

Intentions & Results

Some years ago, I came across the following quote from the American economist and Nobel Laurette, Milton Friedman, “One of the great mistakes is to judge policies and programs by their intentions rather than their results”. Like many familiar quotes, we are often unclear about their original context or intended meaning.  Often, however, Friedman’s quote is used against government intervention. In other words, it is used as a counsel of despair that government intervention usually makes matters worse. I don’t know if that is what Friedman intended, but it is generally an absurd conclusion. That said, often GtD’s clients are working hard to complement public services or are providing services to client groups where government bureaucracies would fear to tread.

Proving your Intention

Friedman’s words do, however, support the desirability that well-intentioned programs should deliver good results. I am sure that few people would dispute that, particularly GtD’s clients who are increasingly looking for definitive evidence of the value of their work. This is where our “Prove” service is in demand. Sometimes our clients need the evidence to prove the effectiveness of their work to their funders. More often, however, their quest for evidence is part of their commitment to delivering the most effective service and ensuring that they are benefiting the very people they want to assist.

Our clients know the value of our “Prove” service, whether they are demanding to know the effectiveness of reforms to the youth justice system or requiring evidence of the benefit of innovative approaches to health provision in underserved rural counties. GtD’s “Prove” service not only delivers rigorous impact evaluation. It also provides clearly written reports that communicate key findings to a range of stakeholders, including funders, senior management, practitioners and users.

Turning Good Intentions into Successful Outcomes

An effective program starts with a good intention to help. Let GtD accompany you on your social impact journey through our “Measure”, “Learn” and “Prove” packages. Contact us today to learn more about the range of our services.

 

Make Social Impact Your Goal! - GtD and Street Soccer Academy

Make Social Impact Your Goal!

I am delighted to be working again with Street Soccer Academy (SSA) to put evidence of their social impact at the heart of their work with ex-offenders. This important work has been made possible by a grant from the Access Impact Foundation whose mission is to make charities and social enterprises more financially resilient and self-reliant, so that they can sustain or increase their impact. Without the generous funding from the Access Impact Foundation we would not able to provide Street Soccer Academy with our powerful data analysis.

Street Soccer Academy is a great client to work with. They use professionally organised sports-based programmes in the rehabilitation and reintegration of people from some of the nation’s hardest to reach groups, including ex-offenders. Our task is to prove that SSA’s pro-social models are affecting the attitudes and thinking of the men and women with whom they work, with particular emphasis on their relationships and roles in society.

In the coming months we will be using our rigorous social impact analytics to contribute to the knowledge of what makes ex-offenders desist from crime. Our previous evaluation of the academy’s prison to community service, produced evidence of SSA’s excellent engagement of ex-offenders into their programme. With the foundation’s funding we will build on that by using our advanced statistical analysis to identify who benefits from the programme, how and in what circumstances. These analyses will assist the academy to identify the most effective practice and allow it to develop its professional programmes. To ensure that it has the right information at the right time we will be building a dashboard to communicate this data to those delivering the programme, their managers and the funder.

Not only will this improve their practice, but it has important implications for government’s Transforming Rehabilitation agenda. That agenda depends on organisations like SSA being commissioned to deliver services through the private community rehabilitation companies. However, the participation of such organisations has been low as they struggle to demonstrate their impact on reoffending pathways and desistance from crime. Access Impact is helping to overcome those obstacles and by funding our work, will enable SSA to attract further funding and make the systemic changes that are essential to support men and women to desist from crime.

To find out more about our analytics services and how we can help your organisation demonstrate your impact definitively, contact us on 020 3371 8950 or email jack.cattell@getthedata.co.uk

Evaluation for accountability courts

Building on Success: Evaluation for Accountability Courts

Across the U.S. Accountability Courts are proving effective in reducing substance misuse and lowering recidivism. Once again, our home state of Georgia is in the vanguard of reform. Next week, GtD will be attending the annual conference of the Council of Accountability Court Judges of Georgia to show how evaluation can be used to build on these successes.   

Accountability courts provide interventions that address the mental health, substance misuse and other health issues that can be associated with an individual’s criminal behaviour. Designed to keep nonviolent offenders out of prison, those who are eligible will agree to completing a plan of action that includes counselling, support and regular drug testing by the court. While sanctions are imposed for those who violate a rule of the program or relapse, the court will be the forum for recognizing and congratulating an individual’s progress.

Once again Georgia is in the vanguard of criminal justice reform. Of the estimated 2,500 accountability courts in the U.S., 93 of them are in Georgia.  With the mounting evidence that they are successful in reducing substance misuse and lowering recidivism, there is also a good economic case to promote accountability courts over the use of jail. But if the evidence is there, what is the role of evaluation?  To answer this question GtD will be attending the annual conference of the Council of Accountability Courts of Georgia.

Building on Success

Although accountability courts can be successful, more can be done to improve, replicate and sustain this innovative approach. So, whether a new accountability court is being set up, an existing program is being extended or a funder needs evidence of impact, then GtD’s social impact analytics can provide definitive data to assist courts measure, learn and prove their impact.

Measuring

If a court is implementing a new program, GtD’s Impact Measurement Service will determine its intended impacts, how to measure them and identify what resources will be required. In providing this service we will collect data and report analyses that will be relevant to judges, court managers and practitioners.

Learning

GtD’s social impact analytics can help accountability courts improve their existing programs. Our Predictive Analysis service will help practitioners identify what is working best for offenders, and will provide information for managers to re-define high-quality interventions and deliver a more effective program.

Proving

Ultimately, accountability courts will want to prove their impact. Our rigorous Impact Evaluation service will provide definitive evidence of reductions in recidivism, lower substance misuse and the wider benefits to individual offenders, the local criminal justice system and community.

If you are attending next week’s conference, come by GtD’s table in the exhibition hall to learn more of the value of our social impact analytics for your court.    

A lightbulb of cogs to illustrate service innovation

Service Innovation – Segment and Conquer

Supermarkets use data to sell us more of the things we want, and even things we don’t yet know we want – a real world example of service innovation through segmentation that we can learn from.

In social policy, we all know that there is no one programme or service that will work equally well for everyone in the target cohort. Even if it is having an impact across the board there will be some people for whom it works better than others and that’s where extra value can be squeezed out.

We might roll our eyes at the buzz-phrase ‘customer segmentation’, and of course there’s a difference between tailoring public services and selling sausages, but both require a similar approach to gathering data, analysing it, and in a sense letting it lead the way.

In the case of Tesco it’s about working out what shoppers want and selling it to them – a far easier job than convincing them to buy things in which they have no interest, a win for both parties. With public services it’s a matter of thinking in broad terms where we want people to end up – or not end up, as the case may be – and then letting what bubbles up from the data determine the most efficient route, and even the specific end point.

For example, working with one client that specialises in tackling youth offending, our data analysis found that though their intervention was effective overall, it was less effective at reducing offending among 12 to 13-year-olds than among young people of 15 and 16. By treating these two segments differently the overall impact of the intervention can be improved and more young people can be set on the right path at the right moment in their lives.

This approach challenges current orthodoxy which would have us determine our theory of change and set out clearly how we will achieve a given outcome before starting work. This can lead people to impose an analysis on the data after the fact, forcing it to fit the predetermined course. It also implies that all service users need more or less the same thing and we know very well that they don’t. The orthodox approach has its place, of course, once data has been collected and analysed, when we can start to make predictions based on prior knowledge.

Equally, it’s not efficient to design a bespoke service for every single end user, but there is a sweet spot in which we can identify sub-groups and thus wring out more value from programmes with relatively little additional time, manpower or funding. I’ll finish with another example: we have been designing approaches to impact management with a number of providers of universal services for young people and adult disability services. These agencies work with different sorts of people, with varying needs, and for whom different outcomes are desirable. Advanced statistical analysis can help us identify groups within that complex body and lead to service innovation which is both tailored and general.

Hallmarks of a Good Evaluation Plan Part 2 – Change & Competence

Hallmarks of a Good Evaluation Plan Part 2 – Change & Competence

People don’t want to fund projects, or organisations, or even people – they want to fund change. And they want to work with professionals who know the territory.

Last week  I introduced the three hallmarks of a good evaluation plan and covered the first of those, “relevance”, in some detail. This week, I’m unpacking the others.

The second hallmark is evidence that evaluation, as planned, will promote learning and change within an organisation.  In our experience at Get the Data, we know that not all organisations are ready for change, so reassuring funding bodies of that willingness at the outset is a good tactical move. You can support this by engaging with changemakers within your organisation – those individuals who, if the evaluation demands change, have the desire and ability to make it happen.

For our part, Get the Data’s cutting edge predictive analyses are helping practitioners to identify what will work best for their clients. Managers are using that information to improve interventions, predict future impact and, in the case of social impact bonds, forecast future income. All of which, of course, goes to demonstrate a focus on improving results through intelligent change.

Knowing Your Stuff

The third and final hallmark of a good evaluation plan is evidence of technical competence which will reassure funding assessors that they are dealing with people who are truly immersed in the field in which they are working.

In practice, that means employing the agreed professional nomenclature of inputs, outputs, outcomes and impacts; and also demonstrating an awareness of the appropriate methods for impact and process evaluation. Though this is partly about sending certain signals (like wearing appropriate clothing to a job interview) it is by no means superficial: it also enables assessors to compare your bid fairly against others, like for like, which is especially important in today’s competitive environment. In effect, it makes their job easier.

Organisations that commission Get the Data are working with some of the most vulnerable people in society. We value their work and are committed to using quantitative methods of evaluation to determine their impact. We are proud that our impact evaluations are not only delivering definitive reports on the impact of their work but also play a decisive role in ensuring vital interventions continue. A rigorous evaluation is a business case, a funding argument and publicity material all in one.

I hope you have found this short introduction to the hallmarks of a good evaluation plan useful.  If you want to learn more about how our social impact analytics can support your application for grant funding then contact me or sign up for a free one-hour Strategic Impact Assessment via our website.