Baselines, Retention and the Problem With Vanity Metrics
A walkthrough of one funded programme whose headline numbers were accurate and almost entirely uninformative.
Key points
- Impressions and sign-ups without a baseline cannot show impact.
- Retention at day 30 changed the reading of the programme entirely.
- The programme was not dishonest; the metrics were simply the wrong ones.
The reported numbers
The final report led with 42,000 impressions and 3,100 sign-ups. Both figures were verifiable and neither was contested.
What a baseline would have shown
Sign-ups in the preceding quarter, absent the programme, ran at roughly two-thirds of that rate. The incremental effect was real but far smaller than the headline implies.
The fair reading
This is a measurement design problem, not a misconduct finding. The programme was asked for the wrong evidence and supplied it faithfully.
What we could not establish
- No control region was available, so counterfactual impact remains an estimate.
Sources and references
- [1]Programme final report - Source link placeholder
- [2]Baseline extract and method note - Source link placeholder
Author
Jonas Berg
Measurement contributor
Builds attribution and measurement frameworks for consumer platforms.
Reviewer
Priya Raman
Reviewer, data science
Checks whether the numbers support the claim being made with them.
Related
Metrics sanity-check checklist
Resource library
Data & Evidence track
DRep Academy