Posts

Kasra Aghajani

Get the Data welcomes Kasra Aghajani to the team

GtD is delighted to introduce our new addition to the Get the Data (GtD) analyst team, Kasra Aghajani. Kasa has a BSc in Mathematics, but recently graduated with an MSc in Data Science from the University of Sussex as he wants to specialise in statistics that are suited to the modern world.  In this month’s blog, Kasra writes about how he applied data science and the latest technology to map English vineyards, and how he now looks forward to contributing his knowledge to GtD. 

When we think of wine, the first images that come to mind might be a Cabernet Sauvignon from Bordeaux or a Chianti from Tuscany, but rarely do we think of England. However, global warming has given the region a more temperate climate, and while devastating for our planet, this has caused the establishment of vineyards. These are being planted at a rapid rate across the South East, and geographical information regarding agricultural practices plays a critical role in ensuring sustainable developmentThe mapping of these vineyards has resulted in a classification problem but since classification is one of the main tasks within machine learning, data science emerged as a solution.  So, using data from the European Space Agency’s Sentinel satellites my Master’s dissertation mapped the increasing number of vineyards within the South of England 

The use of novel data sources to map vineyards and solve viticulture problems has a more general application to our understanding of the world around us.  The analysis of data allows us to understand the world in an objective way and enables us to instigate progressive solutions The increasing availability of new data sources has improved our ability to statistically analyse and understand underlying information, and to create data products that have positive social and economic impacts. While much of the progress we have witnessed, in industry, habeen consumer orientated, I am very impressed by the work GtD is doing in the criminal and youth justice systems. Some of the problems facing justice systems include a failure to assist individuals who have already gone through the system to desist from a criminal lifestyle, resulting in high rates of reconviction and reentry. 

GtD’s work in developing a reoffending predictive algorithm allowed probation to target its limited resources on high-risk individuals. To be able to continue my learning, post academia, by tackling problems within our society is an amazing opportunity for me.   

I believe it is not enough to have only the technical ability of statistical inference and data analysis. One must also be able to effectively communicate a story from the data. To enable this, analysts must first understand the context behind the data and build their knowledge within a client’s domain. From an evaluation point of view, communication skills are very important to demonstrate and describe efficiently certain insights, while justifying methodology and approachSo, from vineyards to probation, I’m excited to bring my skills and perspectives to GtD’s business while being mentored by experienced specialists.  

I look forward to working with you. 

Kasra 

 

 

Asking sensitive questions

Asking Sensitive Questions – Measuring a Sensitive Issue

Our clients work with people to address an important issue in their life and therefore have to engage with sensitive issues in their beneficiaries lives, such as their financial situations, the strength of their relationships with friends and family, and how they have been feeling generally. If this describes your organisation and you want to measure your impact, then it’s a given that you are going to have to ask questions (in say a questionnaire) that may seem prying and beyond your role. We often develop a theory of change with clients that they are happy with, and they believe we have identified the right outcomes to measure, but when we present how to measure these outcomes they can become a little uncomfortable at the change social impact measurement represents. They might be concerned that staff or volunteers will feel uncomfortable asking these questions and not do it correctly or asking such questions will interfere in their relationships with their clients.

These are important concerns and I wanted to share some of my experience on this issue so you can measure your impact confidently.

  1. Are they sensitive?

Often, we perceive some survey questions as sensitive and therefore difficult to ask, but I generally find responders are not as worried. I ask myself “would I mind being asked this question?” and about 90% of the time I find I am OK with it. I could be less sensitive than the next person, but I see no reason why I am terribly different, and it is an important self-check that I perform. It’s always important to pilot any questionnaire and ask for feedback on any of the potentially sensitive questions and how they can be best handled. By being concerned about perceived sensitivity, we can actually exclude the voices of the people we want to help.

  1. Involvement in the development

It might not always be possible to involve staff and volunteers in the development of questions – I haven’t been able to as much as I would like to during the pandemic – but their involvement can allay any fears about questions to be asked. Staff and volunteers can help put the right tone on the questions, so they fit with your organisation’s culture and how it engages with beneficiaries. Their involvement also means they can understand why certain questions are asked and their value.

  1. Staff and volunteers already do it

It might not be in the obvious at first, but staff and volunteers are probably talking to beneficiaries about these sensitive issues already. The reason you need to measure them is that you want to help, and the evidence base suggests you can successfully. If staff and volunteers were not discussing these sensitive issues, then you cannot expect to improve them. Obviously, staff and volunteers have learnt to raise these issues tactfully and this experience means staff and volunteers can help get the tone right.

  1. Train hearts as well as minds

Staff should be trained on how to administer the questionnaire, but they should also be primed on why these questions are being asked, and why they are important and of value to the organisation.

  1. Administration is important

How a questionnaire is administered can reduce any fears about asking a question and prevent respondents from feeling pried upon. Using self-completion questionnaires can reduce any anxiety associated with a question, especially if the questionnaire design prevents staff from reviewing a response in front of your clients. The questionnaire can also be clear to a responder that they do not have to answer every question and incorporating a don’t know response in every question is good practice.

  1. Use tried and tested questions

There are many psychometric scales and national surveys that have high-quality questions, that have been rigorously tested and verified, and are considered ethically sound. Using these questions means you can be assured that you are not causing offence.

  1. Show the value

People are keener to administer and complete questionnaires when they understand their value. That’s why I always recommend that you regularly communicate your results are to staff, volunteers and beneficiaries.

Overall, we must all remember that going from not measuring your impact to doing so is a moment of important change and becoming comfortable with asking new questions and using new measures can take a little time. Once the culture change happens, then organisations can start to use evidence confidently to improve their social impact.

 

 

Getting Data for the Common Good

Getting the Data for the Common Good

The theme for our latest newsletter is ‘Getting Data for the Common Good’. In this edition, we review Jonathan Rapping‘s book Gideon’s Promise: a public defender movement to transform criminal justice, invite you to join us for our webinar, which takes place next week – ‘Measuring Non-Profit Social Impact During the Pandemic’ and talk about the over-representation of BAME young people in custody.

Have a read, and if you’d like any more information about social impact, please get in touch and contact us.

Five Lessons for Impactful Data

Five Lessons for Impactful Data

A client has just told me that our data collection and analysis has helped them secure substantial funding from central government. I wanted to share the lessons I took from that work for impactful quantitative data:

  1. Start simple – the data collection and data analysis were consciously kept very simple to start. This made it easier for staff to collect data and to produce clear messages for internal and external consumption.
  2. Put yourself in their shoes – with the client, we thought long and hard about what different stakeholders would want to know from the data, and the client was able to respond to the Covid-19 pandemic as well.
  3. Be confident  no data are 100% perfect, and there is no need to apologise for one’s data. Instead, be confident by knowing what it can say and be on top of its limitations and how to correct those in the future.  
  4. Visualise the data – not everyone is comfortable with numbers, so after identifying the data story with our client, we took our time to identify the best charts and colours to tell that story.  
  5. Anticipate –the data system was set to so it would be relatively straight forward to collect new data and add new analysis to the current system. This means our client can respond speedily to funder requests.

If you can apply some or all these principles to your data collection and analysis, then you will start making a bigger impact.

Impact Evaluation and Social Impact Bonds

How Rigorous Impact Evaluation Can Improve Social Impact Bonds

Social Impact Bonds

In recent years, Social Impact Bonds are being increasingly used by the British government to deliver public services via outcomes based commissioning. They are also becoming increasingly common in the U.S. By linking payments to good outcomes for society, SIBs are used not only to provide better value for money, but also as a driver of public sector reform. In the words of guidance published by the British government’s Cabinet Office:

“[Social Impact Bonds] are … designed to help reform public service delivery. SIBs improve the social outcomes of publicly funded services by making funding conditional on achieving results. “

While SIBs are not without their critics, their proponents argue that the bonds are a great way to attract private investment to the public sector while focusing all partners on the delivery of the desired social outcomes. This new way of commissioning services also encourages prime contractors to subcontract delivery of some service to the community and voluntary organisations, who bring their own experience, expertise and diversity to the provision of social services.

GtD have completed evaluations that have helped shape social impact bonds, and through our work we have identified five key questions that should be asked by anyone thinking of setting up a SIB or is looking to improve the design of their SIB:

1. Will it work?

Some services delivered by SIBs fail before they start because the planned intervention cannot plausibly achieve the desired outcome. In other words, just because an intervention reduced the number of looked after children entering the criminal justice system doesn’t mean it should work for all young people at risk of offending. That said, if the evidence base around a particular intervention is weak that does not mean one should not proceed – but it should promote a SIB design that includes an evaluation that can state quickly whether the SIB is delivering the hoped for outcomes.

2. Who can benefit from this intervention and who can’t?

We all want to help as many people as possible. However, we can quickly lose sight of who we are seeking to help when we are simply meeting output targets. In other words, if public services are funded by the number of clients they see, then providers could be tempted to increase numbers by accepting referrals of people for whom the service was not intended. So to achieve your outcomes – and receive payments – it’s vital to monitor intelligently the profile of your beneficiaries and ask yourself, “If my targeting were perfect are these the clients I would want to work with to deliver my intended outcomes?”

3. Are we doing what we said we were going to do? And does it work?

Interventions can fail simply because they don’t do what you said you were going to do. If, for example, you are working with young people to raise career prospects and your operating model includes an assessment of need (because the evidence base suggests that assessments increase effectiveness) then it should be no surprise that you did not meet your outcome targets if an assessment was completed with only half of your clients. Identifying your key outputs, monitoring their use and predicting outcomes based on their use can give you much greater confidence in achieving your intended impact.

4. What do we need to learn and how do we learn quickly?

All SIBs we have seen collate a lot of data about their beneficiaries and the service provided but few use those data to their full potential. With predictive analysis, we can monitor who appears to respond best, who is not benefiting and what form of service delivery is the most effective. In other words, is one-to-one work or group work more cost effective? As such you can learn how to define your referral criteria better or learn how to improve your operating model, even within the first months of a SIB.

5. How can we build a counterfactual?

A counterfactual is an estimation of what outcomes would have been achieved without the SIB. Comparing your SIB’s outcomes to the counterfactual can highlight some areas for learning and how to improve over time. Consideration should be given to the counterfactual at the commencement of the SIB. Full advantage can be taken of the publically available data sets to construct the counterfactual: for example, the National Pupil Database, Justice Data Lab or the Health Episode statistics. (Top tip: consent from beneficiaries to use these data sources is generally required).

To discuss how GtD’s impact evaluation can help improve your organisation’s SIB, please contact Jack Cattell, jack.cattell@getthedata.co.uk

Influence through Data

“Yeah, Says Who?” – Influence Through Data

You know you’ve achieved results – the data tells you so – but how do you influence sceptics to believe it?

It can be a rude awakening to take the findings of a study outside your own team or organisation, where trust and mutual support are more or less a given. In front of a wider audience of funding providers or other stakeholders, you will inevitably in my experience find yourself being challenged hard.

This is as it should be – scrutiny is a key part of a healthy system – but, at the same time, it’s always a shame to see an impactful project or programme struggle purely because its operators fail to sell it effectively.

Fortunately, while there are no black-and-white rules, there are some things you can do to improve your chances.

Confidence = Influence

When I present findings I do so with a confidence that comes with experience and from really understanding the underlying mechanics. But if you’re not a specialist and don’t have that experience there are things you can do to make yourself feel more confident and thus inspire greater confidence in your audience.

First, make sure you have thought through and recorded a data management policy. Are you clear how often data should be entered? If information is missing, what will you do to fill the gaps? What are your processes for cleaning and regularising data? Is there information you don’t need to record? A professional, formalised approach to keeping timely and accurate data sends all the right signals about your competence and the underlying foundations of your work.

Secondly, use the data as often as possible, and share the analysis with those who enter your data so that they can understand its purpose, and own it. Demonstrating that your data is valued and has dedicated, accountable managers hugely increases its (and your) credibility.

Thirdly, take the initiative in checking the reliability and validity of your own tools. If you use well-being questionnaires, for example, take the time to check whether they are really measuring what you want to measure in most instances. In other words, try to find fault with your own approach before your stakeholders so that when they find a weak point you have an answer ready that not only reassures them but also underlines the objectivity with which you approach your work.

Own Your Data’s Imperfections

Finally, and this might feel counterintuitive, you should identify the weaknesses in your own data and analysis and be honest about them. All data and analysis has limitations and being clear about those, and the compromises made to work around them demonstrates objectivity which, again, reinforces credibility.

In conclusion, the better you understand your own data and analysis, flaws and all, the more comfortable and confident you will feel when it, in turn, comes under scrutiny.

Tape measure

The Data Revolution Will be Measured

“Innovation distinguishes between a leader and follower”, so said the late Steve Jobs. His words came to mind as I prepared to present to Gideon’s Promise’s Summer Session last month.  Gideon’s Promise asserts that a revolution is needed in how the criminal justice system treats indigent defendants. And as befits an organization that sees itself in the vanguard of that revolution, Gideon’s Promise has a radical vision of the impact of its work. For them it is less about increased efficiencies in case management or the effectiveness of sentencing. Rather it is about changing how the criminal justice system and society-at-large regard poor people caught up in the system’s net. To bring this about, Gideon’s Promise provides a community of public defenders who put their clients at the center of their practice. In doing so, they insure their clients’ stories are told and their humanity is revealed to the court. Or in the words of one member of the movement, “we need to think of this person as a human being and not simply a number in an orange jump suit”.

So how to measure a revolution? Over the past 18-months Gideon’s Promise has led an expert collaboration that has included Get the Data and another Atlanta-based organization, Techbridge. Together we have delivered a ground-breaking program of work that has produced innovative measures of both the public defenders’ values and their clients’ experiences of the client-centered approach. In developing those measures, GtD grounded our work on understanding the impacts that Gideon’s Promise strives to achieve and the resources at its disposal. Having articulated this theory of change, we were able to design the appropriate measurement instruments and subject them to cognitive, reliability and validity testing with samples of Gideon-trained public defenders and their clients. The next step was to collaborate with Techbridge to deliver a technological solution that will allow the instruments to be completed on-line and held in a database. This technology will be essential as the measurement work is rolled out across Gideon-affiliated offices. Finally, we designed a “Digital Dashboard” that Techbridge attached to the database and to report data by offices, public defenders and their clients.

“Innovative” is a word that can be overused or cliched, but in this case the program of work undertaken by the collaboration has truly been new and refreshing. Moreover, it provides the means by which Gideon’s Promise can measure its bold and revolutionary vision. The next steps will be to roll-out this program of work and then move to designing measures of the impact on the criminal justice system and ultimately society as a whole.

The revolution is coming, and the revolution will be measured.

Alan Mackie presenting about Data Driven Policy Development

Using Data to Shed the Cloak of Invisibility

Since time immemorial we have been fascinated by the power of invisibility. Plato used the “Ring of Gyges” to wrestle with the ethical and moral dilemmas it poses. Just think of H.G. Wells’ “Invisible Man”, Tolkien’s One-Ring, or Harry Potter’s Cloak of Invisibility.  Does invisibility free us from moral obligations, and if we possessed this power would we become corrupted or would we enjoy secretly righting wrongs?

Outside the realms of science fiction, “invisibility” is often an every-day problem for our fellow citizens. I am thinking of those whose problems just don’t make the 24/7 news cycle: the homeless woman sleeping in a shop doorway, the ex-offender walking through the prison gate with nowhere to go, the young man who has dropped out of school with no grades. Often we simply ignore those populations, or – to borrow from Harry Potter – throw the “cloak of invisibility” around them. To be out of sight is often to be out of mind.

The ‘invisibility’ of those with developmental disabilities was thrown into sharp relief during my recent visit to the U.S. Virgin Islands. I was invited by the VI Developmental Disabilities Council to present on ‘Data Driven Policy Development’. My audience was policy makers, including those standing for elected office. If I had any doubts about the relevance of my presentation, the audience quickly rose to the occasion. “How could the needs of those with disabilities be met in the absence of data?”, they demanded. Without data, this population was “invisible”, their needs unknown to those in charge of policy. Without data, those who advocated for better services were hampered in their arguments. Without data, practitioners lacked the evidence to seek funding for new services.

The US Virgin Islands are still recovering from the two hurricanes that devasted the territory last year. As they rebuild the fabric of their communities, it was clear to me that data were not a luxury or a ‘nice-to-have”. Rather, my audience recognized the value of data in throwing light on a population that had often been marginalized prior to the hurricanes, and whose needs had since become more acute. While there is much more work to be done in defining data and setting up systems to collect and analyze data, I suggest that their demand for data was a good one and a great call to action.

It might be fun to speculate what we would do if we had our own cloak of invisibility. In the real world, however, we need data to shine a light on social problems. Or to borrow again from Harry Potter, “Lumus Maxima!”

 

Make Social Impact Your Goal! - GtD and Street Soccer Academy

Make Social Impact Your Goal!

I am delighted to be working again with Street Soccer Academy (SSA) to put evidence of their social impact at the heart of their work with ex-offenders. This important work has been made possible by a grant from the Access Impact Foundation whose mission is to make charities and social enterprises more financially resilient and self-reliant, so that they can sustain or increase their impact. Without the generous funding from the Access Impact Foundation we would not able to provide Street Soccer Academy with our powerful data analysis.

Street Soccer Academy is a great client to work with. They use professionally organised sports-based programmes in the rehabilitation and reintegration of people from some of the nation’s hardest to reach groups, including ex-offenders. Our task is to prove that SSA’s pro-social models are affecting the attitudes and thinking of the men and women with whom they work, with particular emphasis on their relationships and roles in society.

In the coming months we will be using our rigorous social impact analytics to contribute to the knowledge of what makes ex-offenders desist from crime. Our previous evaluation of the academy’s prison to community service, produced evidence of SSA’s excellent engagement of ex-offenders into their programme. With the foundation’s funding we will build on that by using our advanced statistical analysis to identify who benefits from the programme, how and in what circumstances. These analyses will assist the academy to identify the most effective practice and allow it to develop its professional programmes. To ensure that it has the right information at the right time we will be building a dashboard to communicate this data to those delivering the programme, their managers and the funder.

Not only will this improve their practice, but it has important implications for government’s Transforming Rehabilitation agenda. That agenda depends on organisations like SSA being commissioned to deliver services through the private community rehabilitation companies. However, the participation of such organisations has been low as they struggle to demonstrate their impact on reoffending pathways and desistance from crime. Access Impact is helping to overcome those obstacles and by funding our work, will enable SSA to attract further funding and make the systemic changes that are essential to support men and women to desist from crime.

To find out more about our analytics services and how we can help your organisation demonstrate your impact definitively, contact us on 020 3371 8950 or email jack.cattell@getthedata.co.uk

Get the Data and Gideon’s Promise Team Up for Innovative Metrics Get The Data

Get the Data and Gideon’s Promise Team Up for Innovative Metrics

Over the past year, GtD has been proud to be work with Gideon’s Promise to develop measures of culture change that will transform the American criminal justice system. We are delighted to welcome Ilham Askia, Executive Director of Gideon’s Promise to guest blog about our partnership.

Get the Data and Gideon’s Promise Team Up for Innovative Metrics Get The Data

Ilham N. Askia
Executive Director
Gideon’s Promise

Gideon’s Promise is a U.S.-based, non-profit whose mission is to transform the criminal justice system by building a movement of public defenders who provide equal justice for marginalized communities. Gideon’s values-based approach uniquely trains public defenders to use a “client-first” method to defense practice. The impact that this approach can have on culture change in public defense will ultimately transform an entire criminal justice system. Since January 2017 we have been working with GtD to create an innovative approach to measure the effects of culture change in public defense systems.

While most experts focus on case outcomes and court processing to determine the effectiveness of a public defense system, Get the Data and Gideon’s Promise have designed a way to measure the effects of a values-based approach to training and supporting public defenders. Gideon’s Promise believes that, in order to have equity in the criminal justice system, the lawyers who represent the accused must be excellent in their profession but also care about the dignity of their clients. For over 80% of the people who are charged with a criminal offense, a public defender has the enormous responsibility to not only defend a charge but also illustrate the humanity of a defendant. If administrators of justice could view the accused more humanely, then the treatment of people warehoused in jails, prisons and courtrooms would be different. Because the criminal justice system does not effectively function to rehabilitate offenders, the dignity of the human spirit is stripped away as soon as an individual is accused of an offense. The culture of the system must change from viewing people as case numbers and files and more like human beings with lives that have value. Public defenders tell the stories of those who are deemed unworthy of support. Public defenders remind the system about the value of human lives.

Get the Data has designed metrics to discern whether the culture of an office, the confidence of a public defender, and environment play into how defendants feel about their representation. These metrics also measure whether the attitudes of Gideon’s Promise public defenders affect the public defender offices where they work, the courtrooms where they practice and the relationships they have with their clients. While no one else is measuring the impact of culture change in the criminal justice system, Get the Data and Gideon’s Promise are on the verge of a transformative approach to repairing a broken criminal justice system by using public defenders as anchors to reform and explaining their importance through quantitative data that measure qualitative relationships.

Although Gideon’s Promise is a national organization, it primarily focuses its work in the southeast region of the U.S. where the highest concentration of inequity exists. There are six public defender sites piloting these metrics. The Defender Value Spectrum Survey (DVSS) and the Client Evaluation Survey (CES) are being completed by a sample of Gideon’s Promise influenced public defender offices and clients who received services from those offices. Our goal is to conclude whether a values-based trained attorney positively correlates with how people view public defenders, clients and how both are treated in the court system. Are there sets of core values to public defense training and mentoring that systemically changes the culture? Is Gideon’s Promise’s curriculum aligning with the goal to transform the culture of public defense? Does a caring lawyer matter? These are all questions this study will answer. Our hope is that we can replicate this evaluative process across the country to not only inform Gideon’s Promise’s programming but also encourage public defense systems to adopt our model while providing data that our culture change model works. Caring, competent and committed lawyers are essential to true criminal justice reform.

This partnership is crucial to capturing a values-based approach to criminal justice reform. Gideon’s Promise is truly grateful to be working with Get the Data on this ground-breaking work.