KPI Design: What Can Actually Be Measured?

Most disputes about whether a funded programme succeeded are not really disputes about numbers. They are disputes about what the numbers were supposed to show. One side points to a delivered tool; the other asks whether anyone uses it; a third asks whether usage changed anything. All three can be arguing in good faith from the same report, because the report never said which link in the chain it was measuring. This lesson gives you the vocabulary and the diagram that make that argument tractable before the money is committed rather than after.

• Place a proposed metric on the correct link of a results chain.
• Separate what a team controls from what it can only influence.
• Write the assumption sitting on each arrow.
• Treat the chain as a hypothesis to be tested, not as an argument that it works.

The vocabulary here is not the Institute invention. It comes from the evaluation literature, where the terms in the OECD development-assistance glossary have been the common reference for decades. Adopting standard words has a practical benefit: it stops each proposal inventing its own hierarchy of goals, deliverables, milestones and objectives, in which the same word means something different from one page to the next.

The single most important line in the chain falls between outputs and outcomes. Everything to the left of it is within the team gift. If they receive the funds and do the work, the outputs appear; if the outputs do not appear, that is a delivery question of the kind the delivery-risk material addresses. Everything to the right depends on other people responding. A team can ship an excellent explanatory guide and still see no change in how anyone writes rationales, because the world contains competing demands on attention that the team does not command.

Both directions of confusion cause damage. Holding a team contractually to an outcome - treating "raise participation" as a deliverable - punishes them for the behaviour of third parties and pushes them towards whichever cheap proxy is easiest to move. Accepting an output as though it were an outcome - reporting that six workshops ran and calling that capability-building - retires the interesting question without answering it. The productive position is to fund outputs, measure outcomes honestly with attribution stated carefully, and say plainly which is which.

Impacts sit furthest right and are usually the reason anyone cares. Better-informed representation, healthier treasury stewardship, wider participation: these are the stated purposes of much governance funding, and they are also the things a single programme can least plausibly claim to have caused. That is not a reason to omit them from the chain. It is a reason to state, at the outset, that the programme contributes to them and that nobody will be able to measure that contribution cleanly.

The gap is also fixable, and cheaply. An outcome metric might be: the count of published rationales, from the public on-chain vote records and their linked metadata, among the named cohort of participating representatives, in the three months after their workshop, compared with the three months before. That is not a perfect measure - later lessons will show what it cannot support - but it sits on the right link, it comes from a source nobody has to trust the team about, and it can be computed by a stranger.

Practitioner guidance on theories of change makes a point worth repeating: the value is in the arrows, not the boxes. Between each pair of links sits an assumption, and the programme fails at whichever assumption turns out to be false. Writing them down converts a persuasive narrative into a list of testable statements.

• Inputs → activities: this holds only if the named facilitators are actually available for the committed hours.
• Activities → outputs: this holds only if workshops can be scheduled at times a distributed community can attend.
• Outputs → outcomes: this holds only if the reason representatives publish few rationales is that they lack a method - rather than lacking time, or seeing no audience for the effort.
• Outcomes → impacts: this holds only if delegators read rationales and act on what they read.

Look at the third assumption. The whole programme rests on it, and it is a claim about why people currently behave as they do. If the real constraint is time rather than method, an excellent guide changes nothing, and no amount of delivery competence will rescue it. That is exactly the kind of question a representative can ask before funding and cannot answer afterwards from a completion report. The proportionate response is usually not rejection: it is a small piece of discovery work, or an early outcome check with an agreed decision point.

Label each assumption evidenced, plausible or untested, and be honest that most will be plausible. Plausible is an acceptable basis for funding. Plausible presented as evidenced is not, and the label is the difference.

A proposal includes a full-page diagram with colour-coded links running from a modest tooling budget all the way to "a more decentralised and resilient Cardano". Every arrow is drawn; no assumption is written on any of them. The diagram is doing persuasive work - it makes the leap from a tool to an ecosystem property feel like a series of small steps - while establishing nothing. The correct response is not to dismiss the proposal, but to ask which arrow the team considers most likely to break and what they would observe if it did. A team that has thought seriously about its own work answers that question readily and often improves the proposal in the process.

• Reporting outputs and describing them as outcomes.
• Contracting a team to outcomes as though they were deliverables within its control.
• Leaving the outcome link with no metric while over-instrumenting the output link.
• Drawing arrows without writing the assumption each one carries.
• Marking an assumption evidenced when the evidence is the proposal own confidence.
• Treating a completed diagram as an argument that the programme works.
• Extending the chain to grand impacts and then quietly measuring only the first link.

You can lay out a programme as inputs, activities, outputs, outcomes and impacts using standard vocabulary; place every metric a proposal offers onto the link it truly measures and see immediately which links are unmeasured; separate what the team controls from what it can only influence; and write each arrow assumption as a testable sentence labelled evidenced, plausible or untested.

The chain does not tell you whether any assumption is true, how large an effect to expect, or whether the outcome is worth the input. It cannot rank two coherent programmes against each other. Nothing in this lesson quantifies anything, and no dataset, study or effect size is asserted anywhere in it. Whether a plausible-but-untested chain is a sufficient basis for funding is a judgement each representative makes independently.

Inputs and activities are effort, outputs are products, outcomes and impacts are changes in the world.

Fund outputs, measure outcomes, and never let one be reported as the other.

The assumptions on the arrows are where programmes fail; write them down and label them.

A results chain organises the questions. It never answers them.