By Economic Associate Sadia Sheikh

We help a wide range of charities measure their impact. For many, economic evaluations are considered a ‘one-off’ exercise with driving fundraising the main goal – this is understandably a key priority for many charities. But one-off analyses, based on a single year of sometimes limited data, more often produce findings that require a range of caveats. Simply put, we can have less confidence in the results of a single snapshot than we can have in a full picture album. 

To help charities more confidently understand the value of their impact, we run Unlocking Impact Workshops where we advise them how to carry out and strengthen data collection to enable economic evaluation. Almost always, the ‘best practice’ advice is for charities to gather data regularly, growing a database over time to establish whether any findings are persistent or just a one-time chance occurrence.

A great case study is Artis, a charity that uses the performing arts to bring the curriculum to life in primary schools to help children flourish. By joining our Workshop, and benefiting from our tailored economist support afterwards, Artis identified a key measure to gauge improvements in socio-emotional outcomes that could support future economic evaluation. After gathering data across the 2020/21 academic year, Artis approached PBE to assess the potential economic impact of their work using their data. As our report describes, the early results were promising: 

• Children who demonstrated elevated levels of initial difficulties saw a clear improvement in their socio-emotional outcomes between the beginning and end of the Artis programme, even after accounting for possible natural recovery over time.

• If these improvements in outcomes do not fade over time, then we estimated the lifetime economic benefit per child with elevated needs is around £8,700, from improvements in truancy, exclusion, crime, smoking, adult mental health and earnings. 

• This is the equivalent of lifetime benefits of up to £2,300 for the average child in the Artis programme. 

• For every £1 invested in Artis’ programme, there could be as much as a £32 return in long-term economic benefits. 

However, the evidence was not conclusive; the analysis was qualified by some caveats. First, the findings were based on a sample of a single year of data – a reasonable challenge was that perhaps a different cohort might yield different results. Second, the study was carried out during the Covid pandemic when Artis’ sessions were moved online for one term, and anxiety amongst children was generally heightened anyway. These factors may have skewed the findings of the study compared to a ‘normal’ year when all teaching is in person.

However, because Artis has been collecting data consistently since 2020/21 and now holds good-quality data on academic years 2021/22 and 2022/23, PBE has been able to repeat the analysis on two new cohorts to address these concerns. This repeat exercise was undertaken first on the 2021/22 data at the end of that academic year.

More recently, the analysis has been repeated on the 2022/23 cohort. The findings of the latest study that examined both the 2021/22 and 2022/23 cohorts were reassuringly consistent with the original study. Pupils with high initial needs and those on Free School Meals made significant improvements consistently across all three cohorts. An updated cost benefit ratio was outside the scope of the latest study. But, assuming service delivery of the programme has not varied significantly, the mental health improvements in these new cohorts are similar to those of the original 2020/21 cohort, so it seems likely that the programme continues to generate significant value for money.

Through their good practice of regular data collection, Artis was successfully able to use the conclusions drawn in the previous study on the impact and cost-effectiveness of the Artis programme with a far greater degree of confidence.

Additionally, collecting data for consecutive academic years allowed Artis to track children’s outcomes over time. Analysis of 2021/22 showed that children who had received support from Artis in previous years had a significantly lower starting level of difficulties than those who had not. This suggests that the effects of Artis are (at least) persisting through the summer holiday and into the start of the following year. Collecting further years of data would help establish whether there is a build-up effect over time in those receiving multiple years of support maintaining better outcomes than their peers.

The real value of more data lies in the ability to analyse impacts in a more nuanced way that can help inform the service design. For example, Artis is now starting to look at how effects vary according to the characteristics of the students: do children of different genders or ethnicities experience different levels of progress? Artis has also used PBE’s work to help secure funding to develop its online services. By gathering data on the experiences of the children they help in this way, they’ll be able to understand the impact that other service delivery models have.

For charities to operate effectively, it is essential their planning and delivery is based on evidence-based analysis wherever that is possible. That takes dedication and investment, but can make a real difference in how charities serve their beneficiaries. Collecting consistent and high-quality data is a first step towards that. Artis is an excellent case study that demonstrates the rewards of this approach.