Methodology
How we score the evidence
Every Crib on this site is a claim we're willing to defend in a meeting, attached to the original source rather than the blog post that paraphrased it. Here's the rubric.
1. Source tiering
| Tier | What counts | How we weight it |
|---|---|---|
| Academic | Peer-reviewed journals, working papers with published methods (NBER, JMR, JM, MSI). | Highest. Replication and identification strategy matter more than sample size. |
| Primary Research | Original large-scale studies from institutes that publish their method (IPA databank, Ehrenberg-Bass, Baymard, System1). | High, when the method is documented and the sample is disclosed. |
| Independent Analysis | Independent analysts with no product to sell against the finding, using observable data. | Medium-high. Directionally trusted; magnitudes treated as estimates. |
| Vendor Benchmark | Studies published by a company that sells a solution to the problem the study finds. | Lowest. Used only when it's the only dataset that exists, and always labelled. |
2. Scoring rubric
- Impact0–10
- Expected effect on the business outcome if you act on the Crib. 8+ means it changes budget allocation, not just tactics.
- Consensus0–10
- How settled the finding is across independent sources. 9 = replicated repeatedly; 5 = credible but contested; below 4 = emerging.
- Evidence strength0–100
- Method quality: sample, design, transparency and independence. Randomised or quasi-experimental designs score highest; single-vendor surveys score lowest.
- Effort / Volumelow / medium / high
- Effort is what it costs you to act. Volume is how much of your addressable outcome the Crib touches.
3. "Under watch"
A Crib gets flagged Under watch when the underlying system is changing faster than the research can replicate — most of generative engine optimization sits here — or when a new credible study contradicts the current consensus and we haven't yet resolved which holds. Under-watch Cribs are still actionable, but treat the magnitude as unstable and re-check before betting a quarter on them.
4. What we refuse to publish
- Statistics with no traceable primary source, however often they're repeated.
- Vendor case studies presented as generalisable research.
- Correlational findings dressed up in causal language.
- Numbers we can't reproduce from the cited document.
5. Corrections
If a receipt is wrong, misread, or superseded, we change the Crib — the point of the site is to be correct, not consistent. Scores are revised whenever a stronger study lands.