By Jon Franklin, Chief Economist at PBE

One of the messages we try to emphasise to the charities we work with is that measuring impact should be seen as a journey, rather than a destination. All too often we see charities who want to do a one-off, quick economic evaluation “to help with funding”, but we believe this misses the true value of embedding a measurement approach at the heart of how an organisation makes decisions. 

A great example of this in practice is Artis, a charity that uses the performing arts in primary schools to bring the curriculum to life in a way that helps children to flourish. After applying for support, Artis took part in our Workshop Plus process, which combines a half-day workshop on the basics of economic evaluation with one-to-one support from one of our economists to help embed the learning within the organisation. The upshot was that the team at Artis identified a new measure that they could use, with a random sample of the children they work with, to gauge improvements in socio-emotional outcomes in a way that could support future economic evaluation. 

After gathering data across the 2020/21 academic year, Artis approached PBE to analyse the data and assess the potential economic impact of their work. As our report describes, although the evidence was far from perfect, the early results were promising: 

• The children who demonstrated elevated levels of initial difficulties saw a clear improvement in their socio-emotional outcomes between the beginning and end of the Artis programme, even after accounting for possible natural recovery over time.
• If these improvements in outcomes do not fade out over time, then we estimated the long-term economic benefit per child with elevated needs is around £8,700, from improvements in truancy, exclusion, crime, smoking, adult mental health and earnings, over their lifetime. 
• This is the equivalent of lifetime benefits of up to £2,300 for the average child in the Artis programme. 
• This suggested that for every £1 invested in Artis’s programme, there could be as much as a £32 return in long-term economic benefits. 

However, what has been most impressive is that Artis has not stopped there. Following the early promising results, the charity expanded their data collection to cover a larger sample of students in 2021/22. The results have confirmed many of the findings from the first year, but the increased volume of data helps to strengthen our confidence in the findings. In fact, the scale of improvement could be even bigger than our estimations in the original report, with improvements in scores in 2021/22 nearly twice as big. This could be because the 2020/21 programme delivery was affected by the pandemic and lockdowns. We will have to wait and see how the 2022/23 data looks to see if that higher level of improvement is retained for this year too. 

The latest evidence also starts to address other weaknesses in the original report. In particular, we did not have hard quantitative evidence on how long the effects of the support last. Do they fade out quickly? Or do they provide long-lasting benefits? In 2021/22, Artis captured new information about the number of years that children had participated in the programme previously. Those children that had received support from Artis in previous years have a significantly lower starting level of difficulties than those that have not. This suggests that the effects of Artis’s work is (at least) persisting through the summer holiday and into the start of the following year. Furthermore, we have initial evidence that this effect is stronger for those that have had two or more years of Artis support, compared to those with just one year of previous Artis support. However, the differences are not big enough to be really confident of this as yet. If this initial evidence were proven to be correct then it would support the case that there is a build-up effect over time - with those receiving multiple years of support maintaining better outcomes than those that do not. 

But the real value of more data comes with the ability to analyse impacts in more detail - in a way that can really start to inform the design of services. Artis is now starting to look at how effects vary according to the characteristics of the students: do girls tend to see bigger improvements than boys? Are children from certain ethnicities more likely to make progress than others? In addition, they have also started to roll out the same outcome measures for different types of intervention and programme to start to understand the trade-offs between costs and benefits for shorter online courses, compared to their traditional face-to-face delivery, over a full school year.  

While the data alone does not provide all the answers for why there are differences in impact across groups or programmes - and what decisions Artis should ultimately make - it can provide a powerful new perspective and help to focus further exploration and investigation. Ultimately, if charities can work to embed the use of impact information within decision-making processes then it can play a critical role in driving up the efficiency and effectiveness of what they do over time.