F8. Designing Small Startup Experiments: Metrics and Thresholds
After this lesson you can
- Design a test with one hypothesis, one primary metric and a threshold set in advance
- Explain why small numbers swing by chance and how peeking inflates false positives
- Estimate whether a planned sample is large enough to be informative
- Choose metrics tied to a decision and add a guardrail metric
- Run tests with honest wording and log every result, including failures
Video coming soon
Videos are planned to publish from December 2026.
A lesson on designing small tests that can actually support or weaken a hypothesis. It covers controls, randomization, small-sample noise, false positives, statistical power, metrics that matter versus vanity metrics, and how to run and report tests honestly.
Questions and answers
Questions a learner would ask about this lesson. Think through your own answer before you open each one.
- What goes into a small experiment plan?
Show answer
A hypothesis, variants including a control, randomization, a primary metric, and a threshold with a stopping rule, all written before the test starts. A strong hypothesis names who, what changes and which number, and states the smallest difference worth acting on. A guardrail metric watches for harm somewhere else.
- My sign-ups went from 3 to 5 after a change. Is that a win?
Show answer
Not on its own. The lesson's illustrative calculation shows that if 8 sign-ups fall equally likely in either of two weeks, a split of 5 to 3 or more uneven happens about 73 percent of the time with no real difference. Comparing two different weeks also has no randomization or control, so a larger or longer test is needed.
- Why write the threshold and stopping rule before testing?
Show answer
Deciding while watching noisy results, or stopping when the number looks good, inflates false positives. A strong plan has a threshold written before the test, a fixed sample or end date, and a decision stated for each outcome.
- What if I cannot reach enough people for a proper test?
Show answer
Then the test may be too small to be informative. The lesson suggests testing a bigger, bolder change, using other evidence such as interviews, commitments and pilots, and labeling the result "not informative" rather than "no effect." Founder A's plan, for example, needed about 78 clinics but only about 40 were reachable, so they labeled the test directional only.
- What is a vanity metric?
Show answer
Numbers such as page views, followers and downloads can grow with attention while nothing important changes. Metrics that matter are tied to the hypothesis and a decision, reflect a costly behavior such as repeat use, and are compared against a threshold set in advance.
Practice quiz
- Question 1Why should a threshold and stopping rule be written before a test begins?
- Question 2Which is a vanity metric for Founder C's health-literacy workshops?
- Question 3Founder A's sign-ups rose from three to five after a headline change. What is the best interpretation?
- Question 4A team runs 10 independent comparisons, each with a 5 percent false positive risk. About what is the chance of at least one false positive?
- Question 5What does randomization do in a small experiment?
- Question 6Using the lesson's illustrative table, with a current rate of 4 percent, about how many people per version are needed to detect a rise to 8 percent?
- Question 7What is statistical power?
- Question 8In Founder A's test plan, what were the primary and guardrail metrics?
- Question 9Which is an honest test design?
- Question 10A small test shows no difference. What does the lesson say not to conclude?
Saved in your browser only: no account, no server.
Entrepreneurship for Graduate Students is education, not legal, tax, financial, or immigration advice. Bring questions about your own situation to your university technology transfer office, a licensed professional, or the right campus office.