Running a Survey That Is Not Garbage
Surveys are good at measuring known things and terrible at discovering unknown ones.
Someone in leadership asks you to survey the customer base to find out what to build next. You send fourteen questions to twelve thousand people, get four hundred responses, and the top requested feature is the one that was listed first in your multiple choice. The chart looks authoritative. It contains almost no information, and now it is in a board deck.
Surveys have one genuine strength: measuring how common something is when you already know what that something is. They are close to useless for finding out what you should be working on, because you can only ask about things you already thought of, and respondents can only answer within the frame you gave them. Use the wrong tool and you get confident numbers about the wrong question.
Surveys are good for
- How many of our users do X at least weekly.
- Which of these four problems, all found in interviews, is most common.
- Sizing a segment you have already described qualitatively.
- Tracking one number over time with the same wording.
Surveys are bad for
- What should we build next.
- Would you use this if we built it.
- Why do people churn.
- Anything where you do not already know the plausible answers.
This gives you the ordering rule: interview first, survey second. Interviews tell you what the possible answers are. Surveys tell you how the population splits across them. Running the survey first means inventing the options yourself, which is how you end up measuring your own assumptions with three decimal places.
What is left of a thousand sends
What makes a question bad
- It asks about the future. 'Would you use', 'how likely are you to', 'would you pay'. Same problem as in interviews, except now you cannot follow up.
- It asks about frequency in the abstract. 'How often do you export reports?' produces a construction, not a count. Ask 'how many times did you export a report last week' instead.
- It leads. 'How helpful was the new dashboard?' has already told the respondent the dashboard was helpful. 'What did you use the new dashboard for, if anything?' has not.
- It is double barrelled. 'Was the setup process fast and clear?' cannot be answered by someone for whom it was fast and confusing, and you will never know which half they meant.
- It offers a list you wrote. Any closed list caps the answer at your imagination. If you must use one, put an 'other' box on it and actually read the other box, which is often the most valuable field in the survey.
Worked example
Sitefox, sizing a finding
Interviews with twelve small business owners had surfaced a specific behaviour: several were editing their site on a phone while standing in their shop, because that is when they noticed something wrong. The team wanted to know whether this was a niche or a majority before investing in mobile editing. They sent three questions to active users. Did you edit your site in the last 30 days (yes/no, screening). Thinking about the last time you edited it, what device were you on (phone, tablet, laptop or desktop, cannot remember). In one sentence, where were you and what made you decide to edit right then (free text). Three questions, one screener, all anchored to a specific past event. The free text field was what made the case, because it showed the trigger was usually noticing an error rather than planned work, which changed what they built.
Three questions. That is not a stylistic preference. Response rates fall with every question, and the people who complete a long survey are systematically different from those who abandon it: more engaged, more loyal, more patient, and less like the customers you are probably worried about.
Building a survey that produces something
- Write the decision first
- One sentence: if the answer is A we do this, if B we do that. If no answer changes anything you would do, do not send the survey. This kills roughly half of proposed surveys, which is a feature.
- Screen before you ask
- First question filters to people the decision is about. Opinions from people who have never done the task are noise, and they will answer anyway.
- Anchor every question to a past event
- 'Last time', 'in the past 30 days', 'the most recent'. Same discipline as interviewing. Memory of an event beats self assessment of a pattern.
- Keep one open text field
- Short, specific, and placed last. This is where you find the thing your options missed, and it is the only part of a survey that can surprise you.
- Check who did not answer
- Before you report anything, compare respondents to your overall base on plan, tenure and usage. If they skew, say so in the summary rather than hoping nobody checks.
That last row is where most survey reporting quietly breaks. Four hundred responses out of twelve thousand is not a sample of your customers. It is a sample of the customers who answer surveys, who tend to be the engaged ones, which means churn: The rate at which customers stop paying or stop using the product. risk and quiet dissatisfaction are structurally underrepresented in exactly the study you are using to understand them.
One more practical note. Pilot it on five people before sending, and watch two of them fill it in. You will find a question that means something different to them than to you within about ninety seconds, every single time. Nothing else you can do in fifteen minutes improves a survey more.
Quick check
Why is a survey a poor tool for deciding what your team should build next?
The takeaway
Interview to find the possible answers, then survey to size them, with a decision written down before the first question.
Try this tomorrow
Before you send your next survey, write one sentence stating what you will do differently depending on the result. Then cut the survey to three questions, screen on the first, and watch two colleagues fill it in before it goes out.
Answer the check above, then bank the day.
Where this comes from
- The Mom Test, Rob Fitzpatrick
- Continuous Discovery Habits, Teresa Torres
- The Lean Startup, Eric Ries
Discovery is one of six tracks. These lessons summarise and build on the work above, they do not reproduce it. Buy the books, they are better.