How to measure a social project's impact without fooling yourself

Your organization has been working for years. It feels like there is impact: the people you serve are doing better and there are testimonials that move people.

But when somebody asks for data, all you have is anecdotes. And that is no longer enough: funders, partners and serious volunteers all want evidence.

The four levels

Activities — what your organization does.
Outputs — what it delivers as a result of those activities.
Outcomes — the immediate changes in the people served.
Impact — the lasting changes in their quality of life or in the structure of the community.

Only the fourth is impact. The first three are management indicators.

The mistake made everywhere

Reporting “we ran a thousand workshops” as proof of impact.

That is activity. The right question is what changed in the lives of the people who attended those workshops. If you cannot answer with data, you are not measuring impact: you are counting work done, which is something else and worth far less for evaluating change.

The four things to define before starting

1. A baseline

How the community was before your intervention, with data.

It is the most ignored and the most critical. Without a baseline you cannot know how much changed because of your work, and everything you report at the end is disconnected anecdote. Establishing one looks expensive; not having one is what is expensive.

2. Indicators meeting three conditions

Specific to the change you expect to produce.
Measurable with methods that are feasible and repeatable.
Attributable to your intervention rather than to outside factors.

A vague indicator produces confusing reports and no ability to demonstrate anything.

3. Periodic collection

If you only measure at the end, you find out far too late that something was not working.

Measuring every three or six months lets you adjust as you go, which is where most of the improvement comes from.

4. A final evaluation comparing start and finish

Without that closing comparison, the intermediate data does not tell a complete story.

Combine all three kinds of indicator

Process — workshops run, people served, resources invested.
Outcome — school grades that improved, incomes that rose, behaviours that changed.
Impact — life trajectories altered, poverty cycles broken, communities strengthened.

Only all three together give a complete picture. Reporting only the first is the default, and it says the least.

An example of measurement done properly

A community health project measured, before starting, infant morbidity rates, vaccination coverage, exclusive breastfeeding and malnutrition. It set targets per indicator. It measured every six months for five years.

It could demonstrate reductions of between 40% and 60% in each. On that evidence the model scaled to three more departments with public funding.

Without the initial measurement, that project would have produced the same results and had no way to show them.

Commonly used tools

Theory of change, to design and communicate the logic of the impact.
Social return on investment, to express social value in economic terms.
Qualitative methods — in-depth interviews, focus groups — for what numbers do not capture.

Numbers tell part of it; stories give the numbers meaning. Serious organizations use both.

If you measure nothing today

Start with the minimum that works:

Define one key indicator representing the essential change you are after.
Establish a baseline for your current population, even if you have been working with them for years.
Set up a simple collection routine every six months.

Measuring one thing well is infinitely better than pretending to measure everything and measuring nothing.

Keep reading

At Corag this argument has a minimal public version: received against used with evidence. Two numbers, and the distance between them.