Last year Charles River Associates produced a report for us advising The Children’s Society on steps to take to demonstrate the impact of their services. We were sure their advice could be useful for other charities working in similar areas.
The Children's Society and Charles River Associates were keen to share their experiences, and so we held a round-table discussion event for others working with children and families to discuss the challenges involved in understanding impact and value.
Representatives from Action for Children, Catch 22, NCB, NSPCC, Rethink Mental Illness, Save the Children and Scope joined us for an interesting discussion.
Phil Raws, The Children’s Society: Building the Evidence – The Children’s Society’s evaluation journey
Katharina Sailer and Jenny Haydock, CRA: Advice on data collection for The Children’s Society
David Hounsell, The Children’s Society: Practical challenges of estimating net economic benefit
The discussion highlighted a number of important challenges:
- Providing different types of information, and telling different stories, for different audiences. For example, not all funders and commissioners require or want an economic analysis of a service. Organisations are using a variety of methods and commissioning and producing different types of evaluations, but they are not always published.
- Making the case for resources to be allocated to research, especially where interventions are funded solely because people believe in them. Good evaluation is about understanding what really works.
- Identifying a suitable counterfactual - RCTs are often dismissed on ethical grounds, but may, on closer consideration, be suitable. For example, where a service has a waiting list, delayed access to the intervention might be used to create a control group.
- Gaining access to national data sets – this is difficult, and even where raw data is or might be available, charities often lack the resources to make use of this data or extract what is useful. The Health & Social Care Information Centre is a potential new source of information on outcomes for children and young people.
- Ensuring buy-in and compliance from practitioners as data gatherers. Some organisations have found it useful to provide service-specific summaries to practitioners to understand the impact of their services.
- Identifying a monetary value for interventions relating to personal, social or emotional development, and the difficulty in ensuring that the ethos of the work is not compromised when doing so.
We will continue to work with the group to identify whether there are opportunities for collaboration in areas such as data sharing, data access and research, and will share updates on any progress we make.