By Jon Franklin, Chief Economist at PBE

Here at Pro Bono Economics, we believe that data matters. It can help to tell powerful stories and provide insightful evidence that can influence policy priorities and create a better understanding of what works. However, we know that all too often there are marginalised groups that are under-represented and get missed from the data. This creates gaps in evidence that can translate into gaps in support and policy. Children that have been excluded from mainstream schools are one of these under-represented groups and run the risk of being absent in statistics.

The data collected by the #BeeWell programme in Greater Manchester provides an illustration of this challenge. #BeeWell is a programme, co-created with young people, that annually measures the wellbeing of secondary school age pupils and brings together partners from across civil society to deliver meaningful, youth-centred actions as a result.

Overall, the programme has been a huge success – 93% of mainstream schools in Greater Manchester signed up in 2021/22, gathering data on 38,000 young people – providing a fantastic resource to understand and help to improve levels of wellbeing in the region. However, just 41 of these young people were from Pupil Referral Units (PRUs), or Alternative Provision schools (APs). These schools often play an important role in supporting young people that have been permanently excluded, with analysis by FFT suggesting that 40% of their pupils have previously been permanently excluded from mainstream schools.

Follow-up work with PRUs and APs helped #BeeWell to identify several practical challenges which hindered response rates:

• Pupils tend to move in and out of PRUs and APs on a relatively fluid basis. This means that pupil lists for schools (provided by local authorities) quickly become out-of-date, which made it challenging to manage the administration of the survey.
• This problem was exacerbated by a relatively tight two-month survey window within which schools were asked to gather the data. It was challenging for many PRUs and APs to align the consent and data sharing agreements and find resources to administer the surveys around this short window.
• Small pupil numbers and relatively low response rates meant that, even where pupils did respond to the survey, #BeeWell’s disclosure rules (intended to protect the privacy of individual pupils) meant that limited information could be offered back to APs and PRUs. This made the process less valuable for them compared to larger mainstream schools that were provided detailed reports on wellbeing levels among their pupils.

#BeeWell introduced additional support for PRUs and APs for their second round of data collection in autumn 2022. This included dedicated support from the research team at the University of Manchester and subject matter experts to engage the PRUs and APs to re-design the data collection process. The changes included using more up-to-date pupil lists for gathering consent and widening the windows for collecting data, as well as establishing bespoke data sharing agreements for these individual units.

Despite these efforts, it remained difficult to reach young people in these settings. The number of children in PRUs and APs that responded to the survey increased from 41 to just 42. Given engagement with the survey was expected to decline in the second year, this could be viewed as a minor success. However, it still leaves us with relatively small volumes of data about pupils in these settings, which will limit the insight that can be provided.

So, does this mean that we should just accept that we will never have data on this hard-to-reach group? #BeeWell does not believe so and is continuing to develop its approach to engagement in 2023. The programme is considering options to supplement the standard data collection processes – organising in-person engagement and workshops to ensure that the voices of young people attending PRUs and APs are heard. This will provide valuable operational insight that can help to shape the provision of support on a localised basis. However, relying on qualitative evidence runs the risk that we are not able to influence policy on a broader scale, in the same way we can for young people in other settings for whom we have increasingly large amounts of hard, quantitative data.

Providing PRUs and APs with individual pupil-level data could increase their interest in gathering the evidence in the first place, as well as the benefits that arise from it. Crucially, this would require a more fundamental re-design of #BeeWell’s model, posing challenges to data sharing and research ethics processes. With this in mind, an approach like this could require a greater level of involvement from the Department for Education (DfE), which would have pre-established pathways for measuring and sharing data within individual schools. This reinforces the case for a nationwide process for collecting children’s wellbeing data and giving it greater parity with academic outcomes.