Leading and double-barreled questions: repair what the survey measures
Repair leading and double-barreled questions with twelve fictional examples, balanced options and a fee-comprehension task you can inspect.
A leading question steers people toward a particular answer; a double-barreled question combines two things in one answer. A refill seller asking whether a service is “affordable and easy to use” needs separate price and task-experience measures. Removing “love our convenient refill” also lets the buyer supply their own impression. The twelve illustrative repairs below show how to get a usable answer without supplying it.
A loaded question assumes something you haven't checked. “Why do you buy sustainable products?” assumes that purchase behavior before asking for a reason. Check the behavior first, then ask what influenced it. A question can be loaded and leading at the same time; removing praise doesn't repair an unverified assumption.
Twelve questions, twelve different repairs
Each revision includes a reason you can use in your own instrument. For the fee task, imagine you sell refills to household buyers and need to check whether they understand the complete order price.
A moral opinion won't reveal price understanding
Original question: Do you agree that nobody should have to pay hidden fees?
Order receipt
2 refills × USD 8 = USD 16 before tax
1 delivery charge per order = USD 3 before tax
Total before tax: USD 19
What total would you expect to pay before tax for this order?
Respondents type the amount, say how they worked it out, or choose 'Cannot judge.'
The receipt shows the calculation for you. In fieldwork, show the unit price, quantity and per-order fee, then collect the total before supplying it.
1. “How much do you love our convenient new refill?”
Ask “What is your overall impression of the refill described?” Use very unfavorable / somewhat unfavorable / neither / somewhat favorable / very favorable / cannot judge. The original supplies both a favorable emotion and a benefit before collecting the response.
2. “Is the service affordable and easy to use?”
Split it: “How affordable or unaffordable is the price shown for your circumstances?” and “How easy or difficult was it to complete the task?” A person may find the price manageable while struggling with the interface.
3. “Wouldn’t you agree the premium package is better?”
Show comparable packages and ask which the participant would investigate further, including neither and no preference. Follow with “What led you to that answer?” The original asks for agreement and leaves “better” undefined.
4. “Why do you buy sustainable products?”
First ask about a recent category purchase and what influenced it. Only ask a sustainability follow-up if relevant. The original assumes both the behavior and the reason.
5. “How satisfied are you with delivery and customer support?”
Ask separately about the most recent delivery and, for those who used support, the support interaction. Add a clear not-applicable route. Someone may have received a late order but excellent help resolving it.
6. “Do you agree that nobody should have to pay hidden fees?”
For a fee-disclosure test, use the complete order task below. Moral agreement does not show whether buyers understand the price.
7. “How often do you regularly use the app?”
Ask “On how many days in the past seven days did you use the app?” Use 0 through 7 and do not know. The defined period and count replace an ambiguous qualifier.
8. “Which useful feature should we add next?”
Ask which, if any, of the described features would be useful for the defined task, with none and cannot judge. If prioritization is required, use a specified ranking or choice design. The original assumes every feature merits development.
9. “Are you happy with the speed and accuracy of recommendations?”
Measure response time or perceived speed separately from relevance or correctness. Define the task and evidence available. One favorable response cannot reveal whether fast but irrelevant recommendations are acceptable.
10. “Why did our ad persuade you to buy?”
Check whether the person saw the ad and bought, then ask what influenced the decision without assigning causation. A reported influence is different from an experimentally measured effect.
11. “Do you oppose not offering a discount?”
Show the actual price alternatives and ask for consideration or choice under the specified offer. If asking a policy opinion, use a direct favor/oppose question. The double negative makes response direction harder to interpret.
12. “Compared with unreliable competitors, how trustworthy is our brand?”
Remove the unsupported competitor description. Ask about trust in each defined offer with equivalent information, or ask what creates or reduces trust. A comparative insult changes the stimulus before measurement.
The answer labels are part of the measure
Even with neutral wording, response options can push the answer. “Good / very good / excellent” excludes negative and neutral views. Overlapping frequency categories make it unclear where some respondents belong. “Other,” “none,” “do not know” and “not applicable” serve different purposes and should be added when the question needs them.
Keep ordered scales in order. Randomize unordered alternatives when the design calls for it, and preserve the display order in the data. Pew’s question-writing guidance explains why wording and question sequence affect interpretation.
Unknown, none and not applicable are different answers. A person without the relevant experience needs a route that does not force a fabricated judgment; a person with an unfavorable view needs an actual unfavorable option. Pretest those labels as well as the sentence.
A complete offer makes the neutral question useful
Replace “Do you agree that nobody should have to pay hidden fees?” with a shown offer: two refills at USD 8 each and one USD 3 per-order delivery charge, both before tax. Ask: “What total would you expect to pay before tax for this order?” Record the amount and explanation, with Cannot judge available.
The expected total is USD 19 before tax. If a response counts one refill, inspect quantity; if it applies delivery per refill, inspect fee scope; if the amount is right for the wrong reason, preserve that contradiction. Complete terms and reasoning make the answer useful. This is a hypothetical diagnostic, not a measured improvement.
The moral-agreement original can't supply those price facts. The replacement is useful when buyers see the complete offer and explain their calculation, allowing the team to distinguish a quantity error from a delivery-fee error.
The split questions need different response routes
Splitting “affordable and easy to use” creates two measures. The affordability question needs a complete price offer; task ease needs the relevant attempted task or an explicitly labeled expectation. An unexposed person cannot report an experience they never had.
Ask respondents to restate each question, explain which information they used and describe their chosen answer. Repair the wording, offer, routing or labels causing the mismatch. Save the original item, replacement and actual pretest observations together so another analyst can inspect what changed.
Every repaired item needs a business use
A repaired item should ask one neutral question that the respondent has enough information or experience to answer. Its options need to permit plausible unfavorable, unknown and inapplicable responses. The result should identify a price, product or communication decision the team can make.
Apply the same care to modeled responses and disclose the audience and context assumptions. Use our Likert scales guide for balanced labels and our silicon sampling guide for prompt sensitivity. Discuss the repaired refill questions with Vensia to explore a modeled survey that keeps overall impression, price fit and task ease separate.
Use the question review records to keep the original item, replacement and intended measure together. In a tracker, compare old and new wording under matched conditions or mark a break when the measure changes. The record should say which business answer the revised item can supply and whether its response route supports that answer.
The record preserves what the repair changes
The two supplied records retain the original question, intended measure, failure, replacement, response options and rationale. Pilot notes are blank in both; add actual pretest observations when you have them.
Review record 1
Field | Recorded value |
|---|---|
| How much do you love our convenient new refill? |
| Overall impression |
| Leading benefit and emotion |
| What is your overall impression of the refill described? |
| Very unfavorable to very favorable plus cannot judge |
| Permits unfavorable and neutral reactions |
|
Review record 2
Field | Recorded value |
|---|---|
| Is the service affordable and easy to use? |
| Price fit and task ease |
| Double barrel |
| Ask affordability and task ease separately |
| Separate balanced labels for each question |
| Price and ease can differ |
|
Complete question-review CSV
original,intended_measure,failure,replacement,response_options,rationale,pilot_notes
How much do you love our convenient new refill?,Overall impression,Leading benefit and emotion,What is your overall impression of the refill described?,Very unfavorable to very favorable plus cannot judge,Permits unfavorable and neutral reactions,
Is the service affordable and easy to use?,Price fit and task ease,Double barrel,Ask affordability and task ease separately,Separate balanced labels for each question,Price and ease can differ,
