Posts

Asking sensitive questions

Asking Sensitive Questions – Measuring a Sensitive Issue

Our clients work with people to address an important issue in their life and therefore have to engage with sensitive issues in their beneficiaries lives, such as their financial situations, the strength of their relationships with friends and family, and how they have been feeling generally. If this describes your organisation and you want to measure your impact, then it’s a given that you are going to have to ask questions (in say a questionnaire) that may seem prying and beyond your role. We often develop a theory of change with clients that they are happy with, and they believe we have identified the right outcomes to measure, but when we present how to measure these outcomes they can become a little uncomfortable at the change social impact measurement represents. They might be concerned that staff or volunteers will feel uncomfortable asking these questions and not do it correctly or asking such questions will interfere in their relationships with their clients.

These are important concerns and I wanted to share some of my experience on this issue so you can measure your impact confidently.

  1. Are they sensitive?

Often, we perceive some survey questions as sensitive and therefore difficult to ask, but I generally find responders are not as worried. I ask myself “would I mind being asked this question?” and about 90% of the time I find I am OK with it. I could be less sensitive than the next person, but I see no reason why I am terribly different, and it is an important self-check that I perform. It’s always important to pilot any questionnaire and ask for feedback on any of the potentially sensitive questions and how they can be best handled. By being concerned about perceived sensitivity, we can actually exclude the voices of the people we want to help.

  1. Involvement in the development

It might not always be possible to involve staff and volunteers in the development of questions – I haven’t been able to as much as I would like to during the pandemic – but their involvement can allay any fears about questions to be asked. Staff and volunteers can help put the right tone on the questions, so they fit with your organisation’s culture and how it engages with beneficiaries. Their involvement also means they can understand why certain questions are asked and their value.

  1. Staff and volunteers already do it

It might not be in the obvious at first, but staff and volunteers are probably talking to beneficiaries about these sensitive issues already. The reason you need to measure them is that you want to help, and the evidence base suggests you can successfully. If staff and volunteers were not discussing these sensitive issues, then you cannot expect to improve them. Obviously, staff and volunteers have learnt to raise these issues tactfully and this experience means staff and volunteers can help get the tone right.

  1. Train hearts as well as minds

Staff should be trained on how to administer the questionnaire, but they should also be primed on why these questions are being asked, and why they are important and of value to the organisation.

  1. Administration is important

How a questionnaire is administered can reduce any fears about asking a question and prevent respondents from feeling pried upon. Using self-completion questionnaires can reduce any anxiety associated with a question, especially if the questionnaire design prevents staff from reviewing a response in front of your clients. The questionnaire can also be clear to a responder that they do not have to answer every question and incorporating a don’t know response in every question is good practice.

  1. Use tried and tested questions

There are many psychometric scales and national surveys that have high-quality questions, that have been rigorously tested and verified, and are considered ethically sound. Using these questions means you can be assured that you are not causing offence.

  1. Show the value

People are keener to administer and complete questionnaires when they understand their value. That’s why I always recommend that you regularly communicate your results are to staff, volunteers and beneficiaries.

Overall, we must all remember that going from not measuring your impact to doing so is a moment of important change and becoming comfortable with asking new questions and using new measures can take a little time. Once the culture change happens, then organisations can start to use evidence confidently to improve their social impact.

 

 

Getting Data for the Common Good

Getting the Data for the Common Good

The theme for our latest newsletter is ‘Getting Data for the Common Good’. In this edition, we review Jonathan Rapping‘s book Gideon’s Promise: a public defender movement to transform criminal justice, invite you to join us for our webinar, which takes place next week – ‘Measuring Non-Profit Social Impact During the Pandemic’ and talk about the over-representation of BAME young people in custody.

Have a read, and if you’d like any more information about social impact, please get in touch and contact us.

Five Lessons for Impactful Data

Five Lessons for Impactful Data

A client has just told me that our data collection and analysis has helped them secure substantial funding from central government. I wanted to share the lessons I took from that work for impactful quantitative data:

  1. Start simple – the data collection and data analysis were consciously kept very simple to start. This made it easier for staff to collect data and to produce clear messages for internal and external consumption.
  2. Put yourself in their shoes – with the client, we thought long and hard about what different stakeholders would want to know from the data, and the client was able to respond to the Covid-19 pandemic as well.
  3. Be confident  no data are 100% perfect, and there is no need to apologise for one’s data. Instead, be confident by knowing what it can say and be on top of its limitations and how to correct those in the future.  
  4. Visualise the data – not everyone is comfortable with numbers, so after identifying the data story with our client, we took our time to identify the best charts and colours to tell that story.  
  5. Anticipate –the data system was set to so it would be relatively straight forward to collect new data and add new analysis to the current system. This means our client can respond speedily to funder requests.

If you can apply some or all these principles to your data collection and analysis, then you will start making a bigger impact.

Impact Evaluation and Social Impact Bonds

How Rigorous Impact Evaluation Can Improve Social Impact Bonds

Social Impact Bonds

In recent years, Social Impact Bonds are being increasingly used by the British government to deliver public services via outcomes based commissioning. They are also becoming increasingly common in the U.S. By linking payments to good outcomes for society, SIBs are used not only to provide better value for money, but also as a driver of public sector reform. In the words of guidance published by the British government’s Cabinet Office:

“[Social Impact Bonds] are … designed to help reform public service delivery. SIBs improve the social outcomes of publicly funded services by making funding conditional on achieving results. “

While SIBs are not without their critics, their proponents argue that the bonds are a great way to attract private investment to the public sector while focusing all partners on the delivery of the desired social outcomes. This new way of commissioning services also encourages prime contractors to subcontract delivery of some service to the community and voluntary organisations, who bring their own experience, expertise and diversity to the provision of social services.

GtD have completed evaluations that have helped shape social impact bonds, and through our work we have identified five key questions that should be asked by anyone thinking of setting up a SIB or is looking to improve the design of their SIB:

1. Will it work?

Some services delivered by SIBs fail before they start because the planned intervention cannot plausibly achieve the desired outcome. In other words, just because an intervention reduced the number of looked after children entering the criminal justice system doesn’t mean it should work for all young people at risk of offending. That said, if the evidence base around a particular intervention is weak that does not mean one should not proceed – but it should promote a SIB design that includes an evaluation that can state quickly whether the SIB is delivering the hoped for outcomes.

2. Who can benefit from this intervention and who can’t?

We all want to help as many people as possible. However, we can quickly lose sight of who we are seeking to help when we are simply meeting output targets. In other words, if public services are funded by the number of clients they see, then providers could be tempted to increase numbers by accepting referrals of people for whom the service was not intended. So to achieve your outcomes – and receive payments – it’s vital to monitor intelligently the profile of your beneficiaries and ask yourself, “If my targeting were perfect are these the clients I would want to work with to deliver my intended outcomes?”

3. Are we doing what we said we were going to do? And does it work?

Interventions can fail simply because they don’t do what you said you were going to do. If, for example, you are working with young people to raise career prospects and your operating model includes an assessment of need (because the evidence base suggests that assessments increase effectiveness) then it should be no surprise that you did not meet your outcome targets if an assessment was completed with only half of your clients. Identifying your key outputs, monitoring their use and predicting outcomes based on their use can give you much greater confidence in achieving your intended impact.

4. What do we need to learn and how do we learn quickly?

All SIBs we have seen collate a lot of data about their beneficiaries and the service provided but few use those data to their full potential. With predictive analysis, we can monitor who appears to respond best, who is not benefiting and what form of service delivery is the most effective. In other words, is one-to-one work or group work more cost effective? As such you can learn how to define your referral criteria better or learn how to improve your operating model, even within the first months of a SIB.

5. How can we build a counterfactual?

A counterfactual is an estimation of what outcomes would have been achieved without the SIB. Comparing your SIB’s outcomes to the counterfactual can highlight some areas for learning and how to improve over time. Consideration should be given to the counterfactual at the commencement of the SIB. Full advantage can be taken of the publically available data sets to construct the counterfactual: for example, the National Pupil Database, Justice Data Lab or the Health Episode statistics. (Top tip: consent from beneficiaries to use these data sources is generally required).

To discuss how GtD’s impact evaluation can help improve your organisation’s SIB, please contact Jack Cattell, jack.cattell@getthedata.co.uk

Influence through Data

“Yeah, Says Who?” – Influence Through Data

You know you’ve achieved results – the data tells you so – but how do you influence sceptics to believe it?

It can be a rude awakening to take the findings of a study outside your own team or organisation, where trust and mutual support are more or less a given. In front of a wider audience of funding providers or other stakeholders, you will inevitably in my experience find yourself being challenged hard.

This is as it should be – scrutiny is a key part of a healthy system – but, at the same time, it’s always a shame to see an impactful project or programme struggle purely because its operators fail to sell it effectively.

Fortunately, while there are no black-and-white rules, there are some things you can do to improve your chances.

Confidence = Influence

When I present findings I do so with a confidence that comes with experience and from really understanding the underlying mechanics. But if you’re not a specialist and don’t have that experience there are things you can do to make yourself feel more confident and thus inspire greater confidence in your audience.

First, make sure you have thought through and recorded a data management policy. Are you clear how often data should be entered? If information is missing, what will you do to fill the gaps? What are your processes for cleaning and regularising data? Is there information you don’t need to record? A professional, formalised approach to keeping timely and accurate data sends all the right signals about your competence and the underlying foundations of your work.

Secondly, use the data as often as possible, and share the analysis with those who enter your data so that they can understand its purpose, and own it. Demonstrating that your data is valued and has dedicated, accountable managers hugely increases its (and your) credibility.

Thirdly, take the initiative in checking the reliability and validity of your own tools. If you use well-being questionnaires, for example, take the time to check whether they are really measuring what you want to measure in most instances. In other words, try to find fault with your own approach before your stakeholders so that when they find a weak point you have an answer ready that not only reassures them but also underlines the objectivity with which you approach your work.

Own Your Data’s Imperfections

Finally, and this might feel counterintuitive, you should identify the weaknesses in your own data and analysis and be honest about them. All data and analysis has limitations and being clear about those, and the compromises made to work around them demonstrates objectivity which, again, reinforces credibility.

In conclusion, the better you understand your own data and analysis, flaws and all, the more comfortable and confident you will feel when it, in turn, comes under scrutiny.

Tape measure

The Data Revolution Will be Measured

“Innovation distinguishes between a leader and follower”, so said the late Steve Jobs. His words came to mind as I prepared to present to Gideon’s Promise’s Summer Session last month.  Gideon’s Promise asserts that a revolution is needed in how the criminal justice system treats indigent defendants. And as befits an organization that sees itself in the vanguard of that revolution, Gideon’s Promise has a radical vision of the impact of its work. For them it is less about increased efficiencies in case management or the effectiveness of sentencing. Rather it is about changing how the criminal justice system and society-at-large regard poor people caught up in the system’s net. To bring this about, Gideon’s Promise provides a community of public defenders who put their clients at the center of their practice. In doing so, they insure their clients’ stories are told and their humanity is revealed to the court. Or in the words of one member of the movement, “we need to think of this person as a human being and not simply a number in an orange jump suit”.

So how to measure a revolution? Over the past 18-months Gideon’s Promise has led an expert collaboration that has included Get the Data and another Atlanta-based organization, Techbridge. Together we have delivered a ground-breaking program of work that has produced innovative measures of both the public defenders’ values and their clients’ experiences of the client-centered approach. In developing those measures, GtD grounded our work on understanding the impacts that Gideon’s Promise strives to achieve and the resources at its disposal. Having articulated this theory of change, we were able to design the appropriate measurement instruments and subject them to cognitive, reliability and validity testing with samples of Gideon-trained public defenders and their clients. The next step was to collaborate with Techbridge to deliver a technological solution that will allow the instruments to be completed on-line and held in a database. This technology will be essential as the measurement work is rolled out across Gideon-affiliated offices. Finally, we designed a “Digital Dashboard” that Techbridge attached to the database and to report data by offices, public defenders and their clients.

“Innovative” is a word that can be overused or cliched, but in this case the program of work undertaken by the collaboration has truly been new and refreshing. Moreover, it provides the means by which Gideon’s Promise can measure its bold and revolutionary vision. The next steps will be to roll-out this program of work and then move to designing measures of the impact on the criminal justice system and ultimately society as a whole.

The revolution is coming, and the revolution will be measured.

Alan Mackie presenting about Data Driven Policy Development

Using Data to Shed the Cloak of Invisibility

Since time immemorial we have been fascinated by the power of invisibility. Plato used the “Ring of Gyges” to wrestle with the ethical and moral dilemmas it poses. Just think of H.G. Wells’ “Invisible Man”, Tolkien’s One-Ring, or Harry Potter’s Cloak of Invisibility.  Does invisibility free us from moral obligations, and if we possessed this power would we become corrupted or would we enjoy secretly righting wrongs?

Outside the realms of science fiction, “invisibility” is often an every-day problem for our fellow citizens. I am thinking of those whose problems just don’t make the 24/7 news cycle: the homeless woman sleeping in a shop doorway, the ex-offender walking through the prison gate with nowhere to go, the young man who has dropped out of school with no grades. Often we simply ignore those populations, or – to borrow from Harry Potter – throw the “cloak of invisibility” around them. To be out of sight is often to be out of mind.

The ‘invisibility’ of those with developmental disabilities was thrown into sharp relief during my recent visit to the U.S. Virgin Islands. I was invited by the VI Developmental Disabilities Council to present on ‘Data Driven Policy Development’. My audience was policy makers, including those standing for elected office. If I had any doubts about the relevance of my presentation, the audience quickly rose to the occasion. “How could the needs of those with disabilities be met in the absence of data?”, they demanded. Without data, this population was “invisible”, their needs unknown to those in charge of policy. Without data, those who advocated for better services were hampered in their arguments. Without data, practitioners lacked the evidence to seek funding for new services.

The US Virgin Islands are still recovering from the two hurricanes that devasted the territory last year. As they rebuild the fabric of their communities, it was clear to me that data were not a luxury or a ‘nice-to-have”. Rather, my audience recognized the value of data in throwing light on a population that had often been marginalized prior to the hurricanes, and whose needs had since become more acute. While there is much more work to be done in defining data and setting up systems to collect and analyze data, I suggest that their demand for data was a good one and a great call to action.

It might be fun to speculate what we would do if we had our own cloak of invisibility. In the real world, however, we need data to shine a light on social problems. Or to borrow again from Harry Potter, “Lumus Maxima!”

 

Make Social Impact Your Goal! - GtD and Street Soccer Academy

Make Social Impact Your Goal!

I am delighted to be working again with Street Soccer Academy (SSA) to put evidence of their social impact at the heart of their work with ex-offenders. This important work has been made possible by a grant from the Access Impact Foundation whose mission is to make charities and social enterprises more financially resilient and self-reliant, so that they can sustain or increase their impact. Without the generous funding from the Access Impact Foundation we would not able to provide Street Soccer Academy with our powerful data analysis.

Street Soccer Academy is a great client to work with. They use professionally organised sports-based programmes in the rehabilitation and reintegration of people from some of the nation’s hardest to reach groups, including ex-offenders. Our task is to prove that SSA’s pro-social models are affecting the attitudes and thinking of the men and women with whom they work, with particular emphasis on their relationships and roles in society.

In the coming months we will be using our rigorous social impact analytics to contribute to the knowledge of what makes ex-offenders desist from crime. Our previous evaluation of the academy’s prison to community service, produced evidence of SSA’s excellent engagement of ex-offenders into their programme. With the foundation’s funding we will build on that by using our advanced statistical analysis to identify who benefits from the programme, how and in what circumstances. These analyses will assist the academy to identify the most effective practice and allow it to develop its professional programmes. To ensure that it has the right information at the right time we will be building a dashboard to communicate this data to those delivering the programme, their managers and the funder.

Not only will this improve their practice, but it has important implications for government’s Transforming Rehabilitation agenda. That agenda depends on organisations like SSA being commissioned to deliver services through the private community rehabilitation companies. However, the participation of such organisations has been low as they struggle to demonstrate their impact on reoffending pathways and desistance from crime. Access Impact is helping to overcome those obstacles and by funding our work, will enable SSA to attract further funding and make the systemic changes that are essential to support men and women to desist from crime.

To find out more about our analytics services and how we can help your organisation demonstrate your impact definitively, contact us on 020 3371 8950 or email jack.cattell@getthedata.co.uk

Get the Data and Gideon’s Promise Team Up for Innovative Metrics Get The Data

Get the Data and Gideon’s Promise Team Up for Innovative Metrics

Over the past year, GtD has been proud to be work with Gideon’s Promise to develop measures of culture change that will transform the American criminal justice system. We are delighted to welcome Ilham Askia, Executive Director of Gideon’s Promise to guest blog about our partnership.

Get the Data and Gideon’s Promise Team Up for Innovative Metrics Get The Data

Ilham N. Askia
Executive Director
Gideon’s Promise

Gideon’s Promise is a U.S.-based, non-profit whose mission is to transform the criminal justice system by building a movement of public defenders who provide equal justice for marginalized communities. Gideon’s values-based approach uniquely trains public defenders to use a “client-first” method to defense practice. The impact that this approach can have on culture change in public defense will ultimately transform an entire criminal justice system. Since January 2017 we have been working with GtD to create an innovative approach to measure the effects of culture change in public defense systems.

While most experts focus on case outcomes and court processing to determine the effectiveness of a public defense system, Get the Data and Gideon’s Promise have designed a way to measure the effects of a values-based approach to training and supporting public defenders. Gideon’s Promise believes that, in order to have equity in the criminal justice system, the lawyers who represent the accused must be excellent in their profession but also care about the dignity of their clients. For over 80% of the people who are charged with a criminal offense, a public defender has the enormous responsibility to not only defend a charge but also illustrate the humanity of a defendant. If administrators of justice could view the accused more humanely, then the treatment of people warehoused in jails, prisons and courtrooms would be different. Because the criminal justice system does not effectively function to rehabilitate offenders, the dignity of the human spirit is stripped away as soon as an individual is accused of an offense. The culture of the system must change from viewing people as case numbers and files and more like human beings with lives that have value. Public defenders tell the stories of those who are deemed unworthy of support. Public defenders remind the system about the value of human lives.

Get the Data has designed metrics to discern whether the culture of an office, the confidence of a public defender, and environment play into how defendants feel about their representation. These metrics also measure whether the attitudes of Gideon’s Promise public defenders affect the public defender offices where they work, the courtrooms where they practice and the relationships they have with their clients. While no one else is measuring the impact of culture change in the criminal justice system, Get the Data and Gideon’s Promise are on the verge of a transformative approach to repairing a broken criminal justice system by using public defenders as anchors to reform and explaining their importance through quantitative data that measure qualitative relationships.

Although Gideon’s Promise is a national organization, it primarily focuses its work in the southeast region of the U.S. where the highest concentration of inequity exists. There are six public defender sites piloting these metrics. The Defender Value Spectrum Survey (DVSS) and the Client Evaluation Survey (CES) are being completed by a sample of Gideon’s Promise influenced public defender offices and clients who received services from those offices. Our goal is to conclude whether a values-based trained attorney positively correlates with how people view public defenders, clients and how both are treated in the court system. Are there sets of core values to public defense training and mentoring that systemically changes the culture? Is Gideon’s Promise’s curriculum aligning with the goal to transform the culture of public defense? Does a caring lawyer matter? These are all questions this study will answer. Our hope is that we can replicate this evaluative process across the country to not only inform Gideon’s Promise’s programming but also encourage public defense systems to adopt our model while providing data that our culture change model works. Caring, competent and committed lawyers are essential to true criminal justice reform.

This partnership is crucial to capturing a values-based approach to criminal justice reform. Gideon’s Promise is truly grateful to be working with Get the Data on this ground-breaking work.

TR: PbR results - speakers who presented to Community Rehabilitation Companies and other justice sector employees

Community Rehabilitation Companies: PbR Results Event

Transforming Rehabilitation is the UK government’s programme of outsourcing probation services to new community rehabilitation companies. In a radical move, the government is now paying these new companies by the reduction in reoffending results they achieve. GtD is at the forefront of this by providing our cutting-edge social impact analytics to Sodexo Justice Services who manage a number of these new companies.

The first PbR figures were published last month and GtD has been active in informing the debate on their significance. As part of this debate, we recently hosted a sell-out event for senior management and practitioners working in community rehabilitation companies and the justice sector.

An expert panel comprising Prof. Darrick Jolliffe of Greenwich University (above left), Dr Sam King of Leicester University (above right) and GtD’s own Jay Hughes (above centre left), considered the initial findings and what to do next, with Jack Cattell (above centre right) setting out a new vision of how predictive analyses can be used by practitioners to improve performance.

Community Rehabilitation Companies: PbR Results Event Get The Data

Prof. Darrick Jolliffe – University of Greenwich

If you were unable to attend but would like to learn more about how GtD could support you in evaluating your social impact outcomes or for a free predictive analytic roadmap for your CRC, contact Jack Cattell  The event presentations can also be viewed via the link below:

Transforming Rehabilitation – Learning from the PbR results presentations

Community Rehabilitation Companies: PbR Results Event Get The Data

Dr Sam King – University of Leicester

 

Community Rehabilitation Companies: PbR Results Event Get The Data

Jack Cattell – GtD

 

We’ve also set up a LinkedIn group as a forum for shared learning and discussion, for individuals who work or have an interest in, the fields of probation, offender rehabilitation and Transforming Rehabilitation. Click here to request to join – Transforming Rehabilitation