This article was originally published in Pioneers Post on 7 July. This can be read here.
The benefits of measuring your organisation's social impact are plenty – it can help win public sector tenders, secure social investment and ensures transparency. But Sue Holloway, director of Pro Bono Economics, argues it's time to admit that true measurements can never be achieved.
How often have you heard people talk about measuring impact? The phrase is used all the time, but I would like to argue that measurement implies an accuracy that can never be achieved when we are talking about impact. We can only estimate it.
The first thing you need to do when thinking about impact is to decide on the outcomes that are of interest, both at the beginning and the end of the intervention, and then start to collect data on them.
For example, if you are aiming to reduce homelessness, was everyone you helped actually homeless at the start, and how many were still homeless at the end of the programme? If you are helping people improve their skills and therefore their employability, you need a measure of their skills levels at the start of your work with them, and then track how that changes. If you are improving confidence or selfesteem, you want a measure of distance travelled, and a tool such as the Outcomes Star, which measures and supports progress for service users towards self-reliance or other goals, is one tried and tested way of doing this.
However, what many organisations overlook in this process is to think about what would have happened without their intervention. Economists call this the counterfactual and it is vital if you are to avoid over-claiming – or even underclaiming if things would have got worse without you. This involves identifying some sort of control group. Before and after measures may be suitable if you have strong evidence that there is nothing else going on in the lives of people you are working with which could also be contributing to the change you are seeing.
One project Pro Bono Economics worked on was with the Making Every Adult Matter coalition. The coalition had robust data on the length of time that individuals had been dealing with issues such as homelessness, substance misuse, mental health problems and offending. As service users had been engaging with these services separately for between seven and 11 years on average, and the pilots involved all the services working together to support the individual, it was not unreasonable to assume that any change was a result of this coordinated support.
However, it’s not often that straightforward. Usually you will need to identify a control group – a set of individuals who look as much like the group you are working with as they possibly can, with the only difference being that you are not working with them. Your impact is then the change that you are making over and above what would have happened anyway. Because by definition the counterfactual hasn’t happened, it can only ever be estimated. A measurement (the change in outcomes) minus an estimate (what would have happened without the intervention) is not a measurement, it’s an estimate. The important thing is to be as accurate as you can, and without any counterfactual, you will almost certainly be wildly inaccurate.
So please don’t talk about impact if you haven’t included a counterfactual, and please do talk about impact estimation not impact measurement.