This article was originally published in Civil Society Finance on 1 July. This can be read here.

Many charities struggle to get started with measuring their success. Sue Holloway from Pro Bono Economics looks at the best ways to begin.

Charities have already struggled through spending and public service cuts in recent years, and with the upcoming budget in July, there are certainly more to come. So, it’s more important than ever that charities are able to show the value of their work and the impact of their services.

But it’s not easy, and after more than six years of bringing economists and charities together, here are our tips on how to get started with evaluating impact.

1. Think about what area of your work is most important to measure

Many charities come to us with a general request to measure their impact, but this is usually too big an ask. If, for example, your charity runs several services with different beneficiary groups, you are unlikely to be able to look at all of these. So think about where you most need to understand your impact, where your data might be most comprehensive, or the areas of your work that most need funding that relies on demonstrating your impact.
But be careful that you don’t end up focusing on measurement just to suit funders rather than what is really needed.

2. Consider your theory of change

Think hard about the difference you are making. For example, it is not enough to think that running a programme to improve school attendance is a good thing. Why is it a good thing? Because it increases educational attainment? Does it? Is there evidence that this improved attendance increases attainment? And why is this important? Does it lead to better jobs? Better relationships? Happier families?

A theory of change takes you through your activities and their outcomes, and can help you understand the complexities of these causes and effects. It can also challenge assumptions about the difference you are making. It can help to understand what the impact is that you ought to be looking at and how you might estimate it. There are some very useful tools to get started with this and you can find some of them here.

3. Think about the counterfactual

This is the tricky ‘what would have happened anyway?’ question. So, going back to school attendance, we find that 40 of the 50 children involved improved their attendance - a great result. But how do we know this has anything to do with the charity? What if there were outside factors that influenced those children, such as an exciting art project at school or a parent promised a treat if they attended? They might have started going to school more often anyway. How do you know that it was the charity’s intervention that made the difference? You might be able to claim some of that difference, but perhaps not all of it.

This is where a randomised controlled trial (RCT) can be very useful. An RCT is where participants in the trial are randomly assigned to either a group which is given the intervention or treatment, or one that isn’t. This allows us to see differences that are just due to that intervention (common in medical trials, for example). This is not always possible, sometimes for reasons of cost or ethics. There may be other possibilities, such as comparison with a national average. Are you comparing your intervention with no intervention at all, or with others providing similar services?

4. See what data you already have

A common question we get from charities is what data do they need. Before you start collecting all sorts of new data have a look at what you already have. Is it usable? How much is held electronically and how much on handwritten forms? Are your records completed 100% of the time or only 20% of the time? Is the data relevant to the question being asked? Sometimes impact is really only seen over a long period of time. Are you following up with your beneficiaries, or do they disappear after 3 months of help?

5. Don’t underestimate the work involved

None of this will happen overnight. Our projects can take over a year to complete and require hard work and time from everyone involved. It can take months to dig out data from services across the country and get it in a useable form. Sometimes new data, or unexpected changes will slow down a project.

Whether you are working with volunteers, paid consultants, or taking it on in-house, front-line work will always take priority, so it is important to try and carve out dedicated time for measurement, and clarify expectations about how long it might take.

1st July 2015