Image of data being analysed

There’s no Magic Way of Measuring Impact

Wouldn’t it be great if there was a way of measuring your social impact across multiple projects using a single dependable statistic? Well, I’ve got some bad news, and some good.

I was recently talking to a charity who wanted to know how if they could go about measuring and reporting the overall impact of the organisation on children and families. With multiple strands each aiming to achieve different things, they asked if a single outcome measure – one accurate, reliable number – to sum up the impact of the whole organisation was either possible or desirable.

First, here’s the bad news: it’s very unlikely – I might even be so bold as to say impossible – that any such thing exists. You might think you’ve found one that works but when you put in front of a critic (or a nitpicking critical friend, like me) it will probably get ripped apart in seconds.

Of course, if there is a measure that works across multiple projects, even if not all of them, you should use it, but don’t be tempted to shoehorn other projects into that same framework.

It’s true that measuring impact requires compromise but an arbitrary measure, or one that doesn’t stand up to scrutiny, is the wrong compromise to make.

The Good News

There is, however, a compromise that can work, and that is having the confidence to aggregate upwards knowing your project level data are sound. You might say, for example, that together your projects improved outcomes for 10,000 families, and then give a single example from an individual project that improved service access or well-being to support the claim. In most situations that will be more meaningful than any contrived, supposedly universal measure of impact.

Confidence is the key, though: for this to work you need to find a reliable way of measuring and expressing the success of each individual project, and have ready in reserve information robust enough to hold up to scrutiny.

Measuring Means Data

In conclusion, the underlying solution to the challenge of measuring impact, and communicating it, is a foundation of good project level data. That will also make it easier to improve performance and give you more room to manoeuvre. Placing your faith in a single measure, even if you can decide upon one, could leave you vulnerable in a shifting landscape.

 

Influence through Data

“Yeah, Says Who?” – Influence Through Data

You know you’ve achieved results – the data tells you so – but how do you influence sceptics to believe it?

It can be a rude awakening to take the findings of a study outside your own team or organisation, where trust and mutual support are more or less a given. In front of a wider audience of funding providers or other stakeholders, you will inevitably in my experience find yourself being challenged hard.

This is as it should be – scrutiny is a key part of a healthy system – but, at the same time, it’s always a shame to see an impactful project or programme struggle purely because its operators fail to sell it effectively.

Fortunately, while there are no black-and-white rules, there are some things you can do to improve your chances.

Confidence = Influence

When I present findings I do so with a confidence that comes with experience and from really understanding the underlying mechanics. But if you’re not a specialist and don’t have that experience there are things you can do to make yourself feel more confident and thus inspire greater confidence in your audience.

First, make sure you have thought through and recorded a data management policy. Are you clear how often data should be entered? If information is missing, what will you do to fill the gaps? What are your processes for cleaning and regularising data? Is there information you don’t need to record? A professional, formalised approach to keeping timely and accurate data sends all the right signals about your competence and the underlying foundations of your work.

Secondly, use the data as often as possible, and share the analysis with those who enter your data so that they can understand its purpose, and own it. Demonstrating that your data is valued and has dedicated, accountable managers hugely increases its (and your) credibility.

Thirdly, take the initiative in checking the reliability and validity of your own tools. If you use well-being questionnaires, for example, take the time to check whether they are really measuring what you want to measure in most instances. In other words, try to find fault with your own approach before your stakeholders so that when they find a weak point you have an answer ready that not only reassures them but also underlines the objectivity with which you approach your work.

Own Your Data’s Imperfections

Finally, and this might feel counterintuitive, you should identify the weaknesses in your own data and analysis and be honest about them. All data and analysis has limitations and being clear about those, and the compromises made to work around them demonstrates objectivity which, again, reinforces credibility.

In conclusion, the better you understand your own data and analysis, flaws and all, the more comfortable and confident you will feel when it, in turn, comes under scrutiny.

Learn from data

What You Can Learn From Your Data

In a world driven by data, it’s becoming increasingly important for social organisations to prove their impact – not just through what they say, but through cold, hard facts.

While many already collect information on the work they’re doing, there remains a gap.

There’s a limit to what you can understand from the numbers alone, and simply reporting the data can feel far removed from the real-world and the day-to-day work done by the workers in your organisation.

For your information to become truly valuable, you need to start unpicking it. By applying the right methods of data analyses, you should be able to learn from your data and to use the insights you draw out of it to make practical changes within your organisation. This approach to data analysis will allow you to optimise your resources and maximise your results.

More than that, you’ll be able to extract the story behind your successes, and use that to show the world the valuable social impact your organisation is achieving.

Empower your staff

Looking at the data alone, it can be difficult for practitioners to know where to concentrate their efforts, and where changes to their service are most needed. But with the right analysis, the data you already have can be transformed into immensely valuable information to guide practitioners’  decisions, and highlight areas for improvement.

Learning from your data can give your practitioners insights on their individual clients, on how the team is performing, and on the organisation as a whole. It means they’ll know exactly what you need to target, and how to fine-tune your service to achieve the best possible results.

Understand your beneficiaries

No doubt you have worked in your sector long enough to know that your clients are not all the same. By applying the appropriate analytical techniques, you can understand the differences between clients and tailor your services to their varying needs.

Methods like segmentation and cluster analysis allow you to pick out patterns across different groups of people with different characteristics, and identify not only “what works?”, but “what works for whom?”.

Predict future outcomes

Your data can tell you not just what’s happened to date, but the outcomes that your organisation could produce in the future.

This can be done by applying scenario and simulation analysis, to predict the potential results of a variety of situations. These predictions should update dynamically, changing over time as other conditions change.

What’s really valuable about these predictions is that you can use them to optimise your resources, and to expend them in a way that will have maximum effect. That’s important for any organisation to know, but particularly helpful for non-profits and social enterprises whose resources are often limited.

Know your social impact, and prove it

Learning from your data gives you the confidence of knowing – not just hoping – that what you’re doing works. It lets you know that you’re making a real impact in your field, but perhaps more importantly, it offers assurance to the people who are looking for that proof.

Your funders and stakeholders need to have confidence in what your organisation is achieving, and by analysing your data you can give them just that.

To take a  youth service as an example, a potential funder might want to know how much of a difference the organisation is making to youth unemployment in the area. In answer, it could use examples from the analysed data, and effectively demonstrate its improvement in comparison with a control group.

By telling the story of its impact, backed up with concrete evidence and analysis, the youth service would be able to provide a much more convincing argument than it would be, using the raw data alone.

Learn from your data with GtD

Get the Data works with organisations in the UK and the US that want to prove their impact on society. We provide social impact analytics in the form of three main packages: Measure, Learn and Prove.

Our Learn service sits in the middle, making it a perfect next step for organisations that have collected data on their service but need to understand it better in order to use it. “Learn” can also be used in combination with our “Measure” and “Prove”  services, in any order that suits you.

Beginning with a strategic review of your data, we’ll put together an insight strategy and follow up with regular reports to monitor the results – integrating those reports with your own systems so they’re dynamic, useful, and convenient to use.

If you know you could be doing more to learn from your data but you’re not sure where to start, take the first step and contact us today.  We will get you started on your social impact journey.

Use Data to Tell an Effective Story Get The Data

Use Data to Tell an Effective Story

Over the past eight years, GtD has been providing social impact analytics to tell our clients’ stories. Effective stories of what their interventions do, who they work with, and the difference they are making to the people they serve: the homeless, young people, public defenders, refugees and immigrants. But, while we are primarily using numbers rather than words to monitor and evaluate our clients’ work, we need to take care that we are following the golden rule of story-telling.

Golden Rule of Story Telling

The golden rule is that a story should have a beginning, a middle and an end. I’ve just finished Sir Ernest Shackleton’s “South”, the Edwardian explorer’s account of his trans-Antarctic exhibition. The story opens in the months just before the First World War, with the preparations for the expedition, all the charting and planning and the resources being loaded on to his ship “Endurance”. The middle of the story is about the embarkation from Plymouth and the ship sailing into the South Atlantic, and then Shackleton leading his crew across the mountains and glaciers of Antarctica with the constant charting and re-charting of their route to safety. The tale ends with the crew’s rescue and return home. While Shackleton did not achieve his stated objective, “South” provides a compelling account of how a leader at times must respond to adversity by altering course and deploying available resources to the best effect.

Many organizations that work to make a social impact use words to tell the value of their work. This is, often done in the form of case studies that tell their clients’ stories. I recently came across such a case study in the annual report of a London-based homeless charity. It tells a compelling story about the effectiveness of their work with a client referred to as “David”. David came to London looking for a job, but soon became homeless and was found by the charity’s outreach worker sleeping rough. Having set out the “beginning” of David’s story, the case study then provided an account of the outreach worker’s role in helping David find hostel accommodation and registering him with a doctor and the benefits agency. The story ended well for David, as he found employment, a home and was returned to good health. A simple but effective story that informed the reader about the charity and how it benefits its clients.

The Descriptive Power of Numbers

Like an epic adventure or a compelling case study, we can use numbers rather than words to tell an effective story. However, before we do so, we need to get our story straight and give some thought to which data should provide the “beginning”, the “middle” and the “end”. If we get this right, then the story can follow the golden rule. So, in telling David’s story with data, we should start at the beginning and identify all those things that contributed to the delivery of the outreach service. We call these “inputs”, and they might include the number of outreach workers employed, their shifts, and the training they received.

Next, we will want to find the numbers that describe the services that David received. These are known as “outputs” and will include the number of contacts with the worker, the time taken to make referrals to the general practitioner and the benefits agency, and the contacts with the local hostel. Finally, we need to find the numbers that will describe the “outcomes” that were achieved for David, which will include changes to his employment, accommodation and health. Of course, we will be collecting these input, output and outcome data for David and every other client seen by all the outreach workers.  This will become an aggregated database of information that will tell a much broader and compelling story of who the charity helps, how it helps them and what outcomes their clients achieve.

Telling your story with GtD

Telling an effective story is essentially what GtD’s “Measure” package is all about.  It will help your organization monitor its activities and evaluate its outcomes. Contact us today if you want to learn more about how “Measure” will help your organization realize the potential of its data. “Measure” will assist you in deciding what to measure and what data to collect. It will provide you with analyses to solve problems and achieve better outcomes. “Measure” doesn’t just give you data, it gives you data to tell an effective story.

If you’d like to learn more about social impact analytics and how it can benefit your organisation, come along to our free event in Manchester on 28th April. Visit our events page to learn more and book your place.

Alan Mackie presenting about Data Driven Policy Development

Using Data to Shed the Cloak of Invisibility

Since time immemorial we have been fascinated by the power of invisibility. Plato used the “Ring of Gyges” to wrestle with the ethical and moral dilemmas it poses. Just think of H.G. Wells’ “Invisible Man”, Tolkien’s One-Ring, or Harry Potter’s Cloak of Invisibility.  Does invisibility free us from moral obligations, and if we possessed this power would we become corrupted or would we enjoy secretly righting wrongs?

Outside the realms of science fiction, “invisibility” is often an every-day problem for our fellow citizens. I am thinking of those whose problems just don’t make the 24/7 news cycle: the homeless woman sleeping in a shop doorway, the ex-offender walking through the prison gate with nowhere to go, the young man who has dropped out of school with no grades. Often we simply ignore those populations, or – to borrow from Harry Potter – throw the “cloak of invisibility” around them. To be out of sight is often to be out of mind.

The ‘invisibility’ of those with developmental disabilities was thrown into sharp relief during my recent visit to the U.S. Virgin Islands. I was invited by the VI Developmental Disabilities Council to present on ‘Data Driven Policy Development’. My audience was policy makers, including those standing for elected office. If I had any doubts about the relevance of my presentation, the audience quickly rose to the occasion. “How could the needs of those with disabilities be met in the absence of data?”, they demanded. Without data, this population was “invisible”, their needs unknown to those in charge of policy. Without data, those who advocated for better services were hampered in their arguments. Without data, practitioners lacked the evidence to seek funding for new services.

The US Virgin Islands are still recovering from the two hurricanes that devasted the territory last year. As they rebuild the fabric of their communities, it was clear to me that data were not a luxury or a ‘nice-to-have”. Rather, my audience recognized the value of data in throwing light on a population that had often been marginalized prior to the hurricanes, and whose needs had since become more acute. While there is much more work to be done in defining data and setting up systems to collect and analyze data, I suggest that their demand for data was a good one and a great call to action.

It might be fun to speculate what we would do if we had our own cloak of invisibility. In the real world, however, we need data to shine a light on social problems. Or to borrow again from Harry Potter, “Lumus Maxima!”

 

Image to illustrate no data can mean no voice and no idea

No Data? No Voice, No Idea – The Importance of Data

The collection and analysis of data must never be allowed to fall by the wayside – it’s a founding stone, not a ‘nice to have’.

Of course, I would say that, wouldn’t I? But here are six concrete reasons why data is important, drawing on Get the Data’s recent work with the Advice Services Alliance (ASA) as a case study.

  1. Data Is the Great Persuader

There is no more powerful tool for influencing stakeholders than data, as I explained in more detail in this recent blog post on using data to influence stakeholders.

ASA works with its members – various associations who provide advice services – to capture and analyse data from the front-line. This gives weight to conversations with all of those who have an interest in ASA’s direction of travel, reassuring them that strategic decisions are being made in response to changing needs, and reinforcing the professionalism that underlies the work they and their partners do.

 

  1. Data Means Funding

In particular, data is invaluable in drawing in new sources of funding, and persuading potential funding providers. Faced with a choice of projects or programmes in which they might invest, with ever-tighter budgets, funding bodies will regard convincing data as a good reason to choose your work over others. ASA will use data to add weight to its funding applications for this reason.

 

  1. Data Gives You the Power to Lead the Debate

ASA seeks to lead thought, representing its members in national discussions by highlighting issues affecting the people who use advice services. The body of data and analysis to which ASA will refer confers authority and allows the organisation to direct the debate and steer collective thinking around youth issues.

 

  1. Data Defines Good (and Bad) Practice

Using data provided by its members ASA will be able to identify areas for improvement in front line practice and also to pinpoint what is working especially well so that good practice can be shared across the community. It informs training programmes, service improvements and helps determine how resources should be employed for maximum impact.

 

  1. Better data and measurement development 

Good data leads to better data. Use of data means we learn its limitations and how to outcome those. We also learn how to measure the right outcomes, better; particularly in how to measure informal outcomes along said formal attainment.

 

  1. Data Is Cheaper and Easier Than Ever

Cloud-based databases are cheap and easy to implement compared to the cumbersome systems of the past. They make it easier for people to enter data and share it. So there’s really no excuse for failing to collect and analyse data in this day and age.

If you would like to find out more about our cutting-edge approach to data capture and analysis please get in touch.

Databse word cloud

Choosing a Database? Just Keep It Simple

When choosing a database for your project or programme, made to measure software or flashy online tools aren’t necessarily the right choice.

If you’re going to gather and analyse data in a serious way of course you need a database of some sort and, having spent much of my academic and professional career buried in them, I’m very much an advocate. But I too often see evidence of the database itself being regarded as the end rather than the means – as the solution to the challenge at hand rather than a tool for addressing it.

Perhaps that’s because we all so often feel under pressure, either external or self-imposed, to demonstrate what we have delivered in concrete terms. A specially commissioned database, perhaps with a catchy name and fancy interface, is something a project manager can point to and say, ‘I made this.’

The problem is that it takes huge amounts of time, resources and testing to create excellent software from scratch or to implement a powerful online tool, meaning that in practice these products all too often become clunky, creaky and frustrating. And products that don’t work well don’t get used.

As always, the answer is to focus on what you want to achieve. That will help you understand what kind of data you need to collect, who will be collecting and managing it and, therefore, what kind of system you need to house it.

Sometimes, of course, a custom-designed database or powerful online application are absolutely the right choice but, in my view, those occasions are actually very rare. More often than not the humble, rather plain Microsoft Access, or a similar tried and tested generalist professional product, is not only cheaper but also better suited to the task. Remember, these programmes have been worked on over the course of not only years but decades, and have huge amounts of resources behind their development and customer support programmes. Standard software also makes sharing, archiving and moving between systems faster and easier in most cases.

It is also possible to customise Access databases to a fairly high degree, either in-house using the software’s in-built features, using third-party software, or by hiring developers to create a bespoke front end interface without getting bogged down in the complicated underlying machinery.

And cloud data storage has made all this easier and cheaper than ever.

In conclusion, I’d like people to recognise that the real deliverable isn’t necessarily software, and that there’s no shame in off-the-peg. After all, a database is only as good as the information it holds and a clean set of useful, appropriate data is what people should really be proud of.

Hallmarks of a Good Evaluation Plan

Hallmarks of a Good Evaluation Plan Part 1 – Introduction & Relevance

When a potential funder glances at your application for a grant will they see reassuring signs of quality or something that immediately makes them wary?

When an antique collector finds what they suspect is a piece of fine English silverware they flip it over and look for a set of hallmarks – simple indicators that certify the metal, identify the maker, the place of production, and the year of manufacture. It can help them distinguish quickly between, say, an item of 17th-century sterling silver produced in London by a famous craftsman, and a mass-produced reproduction with only a thin plating of the real thing.

Similarly, it strikes me that there are three hallmarks of a good evaluation plan. First, it should be relevant. Secondly, it ought to promote adaptive change. And, finally, it must be technically competent. Get this right and you will certainly have a funder’s attention.

What has got me thinking about all this lately is a presentation I’ll be giving at the National Grant Conference which takes place in Atlanta, Georgia, between 25 and 27 July 2017, sponsored by the American Grant Writers’ Association.

My presentation complements the work Get the Data does in the UK where our social impact analytics practice provides organisations, including non-profits with the expertise they need to measure, improve and prove their impact. Our social impact analytics are often used to convince careful funding bodies to fund or invest in programs which ultimately assist the most vulnerable in society.

Demonstrating Relevance

So, going back to the first of those hallmarks mentioned above – what sells an evaluation plan as relevant? You have to know, first, what your organisation needs and what type of evaluation you are looking to conduct. Practitioners, board members and those responsible for awarding funding all think constantly about what impact they are seeking to achieve, how to measure it, and how they can achieve it with the resources at their disposal. You need to convey to them that you understand their priorities and mission and tie your work into theirs.

Of course, that’s easier said than done: stakeholders very often value different impact information so there is rarely a one-size-fits-all solution. This is an area where Get the Data can help. Our impact management services can assist in defining the needs of an organisation, and through smart reporting and analysis systems will ensure individual stakeholders can find the information that matters to them.

In the next post in this series, I will consider how we can help deliver an evaluation plan that promotes adaptive change and is technically competent.

In the meantime, I look forward to hearing from you if you would like to learn more about how our social impact analytics can support your application for grant funding email me to get the conversation started.

Please also visit our website where you can sign up for a free one-hour Strategic Impact Assessment in which we’ll take the time to evaluate your current impact management success and will identify key areas to develop in order to help your organisation maximise your social impact.

 

Images showing analysis, in a light bulb to illustrate project evalution

You Might Be Winning but Not Know It

Have you ever eagerly awaited the results of a project impact study or external evaluation only to be disappointed to be told you had no impact? ‘How can this be?’ you might ask. ‘The users liked it, the staff saw the difference being made, and the funding provider was ecstatic!’ The fact is, if you’re trying to gauge the final success of a project without having analysed your data throughout its life, proving you made a difference is bound to be difficult.

Of course we would all like to know before we invest in a project whether it’s going to work. As that’s practically impossible (sorry) the next best thing is to know as soon as we can whether it is on a path to success or, after the fact, whether it has been successful. But even that, in my view, isn’t always quite the right question: more often we should be asking instead what it has achieved, and for whom.

In most cases – rugby matches and elections aside – success isn’t binary, it’s complex, but good data analysed intelligently can reduce the noise and help to make sense of what is really going on.

A service might in practice work brilliantly for one cohort but have negligible impact on another, skewing anecdotal results. Changes might, for example, boost achievement among girls but do next to nothing for boys, leading to the erroneous conclusion that it has failed outright. Or perhaps across the entire group, attainment is stubbornly unmoving but attendance is improving – a significant success, just not the one anyone expected. Dispassionate, unprejudiced data can reveal that your project is achieving more than you’d hoped for.

Equally, if the goalposts are set in concrete, consistently mining that data can give you the insight you need to learn, improve and change tack to achieve the impact you want while the project is underway. Or, at least, to check that you’re collecting and reviewing the right data – if the answer to any of your questions is a baffled shrug or an anecdote (and it too often is, in my experience) then you have a problem.

I’ll be circling back for a detailed look at some of the case studies hinted at above, as well as several others covering various fields, in later posts in this series.

In the meantime, consider the project that keeps you awake at night – where are its dark corners, and what good news might be lurking there?

The Centre for Youth Impact logo

The Centre for Youth Impact – Conference 16th March

We’re pleased to announce that Jack will be hosting a workshop at The Centre for Youth Impact’s ‘The Measure and the Treasure: Evaluation in personal and social development’ conference on 16th March.

The Centre for Youth Impact is hosting the day-long conference to focus on issues of measurement and personal and social development.

“The day will explore policy, practical and philosophical debates about whether, why and how we should seek to measure the development of social and emotional skills (or non-cognitive skills, soft skills and character, amongst other terms) in young people. We are planning a thought-provoking and engaging day that introduces participants to a range of ideas and activities, and are particularly keen to draw in thinking from outside the ‘traditional’ youth sector.

The question of how to measure and evidence the development of social and emotional skills in young people remains one of the key challenges youth organisations feel they are facing, and many practitioners raise ethical, technical and practical questions about the extent to which this is feasible, desirable and even useful. As such, we want to convene a day that will bring individuals with a wide range of perspectives into the same space to share, explore and progress their thinking, with a focus on practical application.”

Kenton Hall – Communications Officer, The Centre for Youth Impact

To find out more visit http://bit.ly/2mBVOWY