Successful projects share six key characteristics:
- Dedicated charity contact
- Sufficient resource on both sides
- Detailed project plan with clear timelines
- Clear and realistic expectations
- Speedy responses
- Keeping PBE informed on progress and problems
Before you get in touch, read our 5 tips for success...
1. Think about what area of your work is most important to measure
Many charities come to us with a general request to measure their impact. As our economists are volunteers there are limits on what they can deliver.
If, for example, your charity has several sites, running several types of intervention with different beneficiary groups, we will not be able to look at all of these. Think about where you most need to understand your impact, where your data might be most comprehensive, or the areas of your work that most need funding that relies on demonstrating your impact.
2. Consider your theory of change
It is important to think hard about the difference you are making. For example, it is not enough to think that running a programme to improve school attendance is a good thing. Why is it a good thing? Because school attendance increases educational attainment? Does it? Is there evidence that improved attendance increases attainment to a certain level? And why is this improved attainment important? Does it lead to better jobs? Better relationships? Happier families?
A theory of change takes you through the activities and outcomes of these activities, and can often help describe the complexities of these causes and effects, challenging some of the assumptions about the difference you are making. Thinking about your theory of change can be a useful tool for many purposes and can help to understand what the impact is that you ought to be measuring and how you might measure it.
A useful website to get you started is www.theoryofchange.org, and a list of other resources can be found here http://www.researchtoaction.org/theory-of-change-useful-resources/
3. Think about the counterfactual
This is the ‘what would have happened anyway?’ question.
Thinking back to our programme to increase school attendance, we find 40 of the 50 children involved increased their attendance. A great result. But how do we know this has anything to do with the charity? What if those children had not been involved in the programme? They might have started going to school more often anyway. Perhaps it was because teaching standards improved, or the next term involved an exciting art project. It might be a bit of all three, in which case we have a problem attributing the results. The charity might be able to claim some of the difference, but perhaps not all of it.
The gold standard evaluation uses a randomised controlled trial, where participants in the trial are randomly assigned to either a group which is given the intervention or treatment, or one that isn’t. This allows us to see differences which are just due to that intervention (common in medical trials, for example). This is not always possible, sometimes for reasons of cost or ethics.
There may be other possibilities – for example comparison with a national average. Are you comparing your intervention with no intervention at all, or with others providing similar services?
4. See what data you have
We know some charities are looking for advice on what they should be collecting, but it may be the case that you can consider this to some extent in advance, and providing as much detail as you can give on what is already available will help to speed up the process.
If you do gather data consider whether it is usable. How much is held electronically and how much is on handwritten forms? Are your records complete 100% of the time or only 20% of the time?
Is the data relevant to the question being asked? If you are arguing that increased school attendance improves educational attainment and employment chances, do you collect information about children’s educational outcomes, or follow up to find out their employment status?
Sometimes impact is really only seen over a long period of time. Are you following up with your beneficiaries, or do they disappear after 3 months of help?
Doing some homework before getting in touch will be useful in our early conversations, and will help an economist get to grips with the issues more quickly.
5. Don’t underestimate the work involved
An average project takes around a year to complete. Though we continue to refine our processes to reduce this time, the reality is that these projects require the hard work and dedication of both the charity and the economists involved.
It can take months to dig out data from services across the country, to clean it up and pass it on in a usable form to the economists. Sometimes new data, or unexpected changes will slow down a project.
More usually, time is scarce because both parties are fitting the project around their usual work, so it can be a few weeks before it comes to the top of the priority list. Even a first meeting may not fit in everyone’s diary for a month.
We ask that charities consider this carefully, and think about how they may manage the work, particularly if there are other competing projects or deadlines, or people moving on from the organisation.