How Accurate Are Synthetic Users? 14 Tests Against Human Research

Accuracy depends on the research task and on whether an error would change your decision. In the 14 Vensia comparisons below, the leading menu message matched the human result, while one EV scenario reversed the preferred alternative. The tests cover ads, messaging, product choices, pricing and customer experience, using tasks recreated from published human studies.

The model is Vensia, built by our company. The comparisons below put its estimates beside the human evidence, with the original audience, question and measurement scale made clear. Twelve cases report normalized accuracy scores; the other two report dollar differences and matches to interview findings. The paired results show where the agreement was strongest and where a useful overall score concealed a specific miss.

Creative and message testing

1. Ad testing

USA TODAY’s Ad Meter ranks Super Bowl commercials using scores from registered viewers, who rate each ad from one to five. Vensia scored eight ads from the 2025 game to see how closely it could recreate those human ratings.

Vensia reached a reported 96.5% normalized accuracy across the eight tested ads. Its ratings differed from the human scores by about 0.14 points on average. All eight were within half a point; seven were within 0.2.

Scores are out of five. Parentheses show ranks within the eight tested spots, rather than all 57 Ad Meter ads. The human scores come from USA TODAY’s full ranking and winner announcement. The Homes.com sequel ranked 40th overall.

Uber Eats was the sole exception to the relative ordering. Remove it from both lists and the remaining seven ads appear in exactly the same order. Its lower Vensia placement shifts the displayed ranks of Novartis, Mountain Dew and Disney+; those are not three independent ordering errors.

Both groups chose Budweiser as the winner. For a creative team, the comparison supports using the model to shortlist ads, while the 0.46-point Uber Eats gap identifies the spot to investigate. Ad liking alone does not establish which commercial will generate more sales.

2. Message testing

Can a sentence above a menu change what people order? Trial 2 of the World Resources Institute’s menu study tested five environmental messages and a no-message control with 2,902 people. Each person chose lunch and dinner.

Vensia recreated the conditions and counted vegetarian selections across both meals. Every condition was within 0.7 percentage points of the human result, and the complete message ranking matched.

Shares of menu selections across lunch and dinner, rather than shares of participants. The reported normalized accuracy score was 97.8%.

“Small changes, big impact” led in both comparisons. It increased vegetarian selections over the control by 12.8 points in Vensia and 13.0 points in the human trial. A messaging team would have chosen the same line to advance.

3. Claim and label testing

An environmental label can improve a product’s perceived sustainability without producing an equally large change in willingness to buy it.

The Taillie and colleagues experiment randomized 1,013 US parents to see eight products with an eco-score label or a barcode control. They rated sustainability and future purchase intention on 1–5 scales. Vensia recreated those conditions in the US-parent context.

Eco-score minus barcode control, on 1–5 rating scales. Selected effects from the eight-product comparison.

The veggie-pizza effects illustrate the distinction: the sustainability improvement was more than twice the purchase-intention improvement in both sets. Packaging teams can use that separation to assess what a claim accomplishes. These are ratings and stated intentions, rather than observed purchases.

Product decisions and pricing

4. Purchase intent testing

A customer who will try an unfamiliar food once may still decline to buy it regularly. Bryant and colleagues’ 2019 study asked about regular purchase of plant-based meat and “clean meat,” its term for cultivated meat. Vensia recreated the US sample context.

Stated regular-purchase intentions under the study’s 2019 product descriptions.

The high-intent cultivated-meat estimate was within 0.6 percentage points of the human result. The larger miss was outright rejection of plant-based meat: Vensia estimated 6.6 points fewer rejectors. That distinction matters for a product team planning around repeat use rather than trial alone; the comparison does not forecast current sales.

5. Feature prioritization

Which EV benefits deserve the most attention? The 2019 Dutch proEME survey recorded weighted responses from 1,553 participants, who could select several purchase priorities. Vensia recreated the same multi-select factor list.

Selected factors from the 15-answer question. Respondents could choose several answers, so shares do not add to 100%.

The same order appears in both sets, and each displayed share is within 0.7 percentage points. Price and range would lead the brief in either comparison. The results describe priorities in that audience in 2019; a present-day launch needs a current audience and offer.

6. Willingness to pay

A pricing test becomes more informative when buyers can choose nothing. Our recreation of the Van Loo, Caputo and Lusk burger experiment offered four burger types at different prices without brand names, alongside a no-purchase option.

Choice-model estimates relative to buying nothing, compared with the human study’s unbranded MNL control. These are dollars per pound, rather than prices named or paid at checkout.

The average absolute difference was $0.24 per pound across the four products, with a largest gap of $0.39 for farm-raised beef. Both sets put the products in the same order. That gives a pricing team a close starting estimate for this task, while an actual offer test remains necessary before choosing a shelf price.

7. Product trade-off testing

Averages can conceal a reversal that changes the decision. Using Zou, Khaloei and MacKenzie’s EV scenarios, we varied the walk to slow charging and access to fast charging within a 15-minute drive. The selected new- and used-car scenarios specified a 40-year-old man who did not own an EV.

Selected scenarios from a 36-scenario comparison. Human probabilities are approximate reconstructions from the published choice model, rather than raw observed choices or EV sales.

Without fast charging, Vensia put used-car buyers above new-car buyers; the human-study model put them below. The used-car gap was about 18.4 percentage points. A team assessing charging access needs this scenario-level view, even when the broader comparison reports high agreement.

View an example study to inspect the audience, offer and findings before planning your own.

Customer behavior and segmentation

8. Shopping behavior

How do shoppers respond to rising food prices? We compared four reported behaviors from the 2024 IFIC Food and Health Survey. The audience was people who noticed food and beverage price increases; each share combines “always” and “often.”

Human eligible base: 2,728 people who noticed price increases. Human shares are from the IFIC report.

Vensia came within 1.9 points on switching to cheaper products or brands. Its larger miss was trading down from premium products, at 7.8 points below the human result. For a consumer-goods team, those are different responses to price pressure and should remain separate in the research. Both sets measure reported behavior rather than basket transactions.

9. Audience segmentation

Do people with EV experience value the same benefits as people who have never driven one? Vensia recreated the six-benefit question in the 2019 Dutch proEME survey. Each result below is the experienced-versus-never-driven difference, rather than either group’s absolute share.

Weighted human responses from the published survey data: 212 EV-experienced and 1,323 never-driven participants, excluding 18 unsure responses. pp means percentage points.

Vensia matched the price gap and followed the human direction for range, charging stations and running costs, each within two points. The climate-effect gap had the opposite sign. These differences can inform which benefits to emphasize for each audience, but the groups contain different respondents: the comparison cannot establish that driving an EV caused the differences. See the original survey and questionnaire.

10. Brand perception

Knowing a brand and liking it are separate judgments. The BRAND dataset measures both. We tested eight food, beverage and restaurant names from its 2024 US brand-name study, using a 1–7 scale for each judgment.

Brand-name liking, rather than logo liking. Each mean in the human 2024 BRAND data is based on 39–41 ratings.

Vensia placed Cheerios slightly ahead of Chick-fil-A; people placed Chick-fil-A slightly ahead of Cheerios. Budweiser had a wider gap, at 0.66 points. The results help separate a near tie from a more substantial difference, rather than treating every rank change alike.

11. Customer satisfaction

We tested customers’ ratings of their own providers across eight product groups in FCA Financial Lives 2024. Vensia’s averages stayed between 7.57 and 7.91 out of 10. The FCA’s published results vary more sharply.

Ratings from 0 to 10. Each category covers holders of that product; FCA averages exclude “don’t know” answers.

Vensia ranked pension satisfaction highest in its tested set, while the FCA found it comparatively low. Its pension estimate was 1.21 points above the human mean. A financial-services team choosing where to improve customer experience would need to resolve that difference before allocating resources.

Offers and decision clarity

12. Offer testing

Could a simpler reply turn an attractive offer into action? Trial 3 of FCA Occasional Paper 19 sent longstanding savings customers a letter offering a higher-rate account. One version added a prefilled form and prepaid envelope.

The human study recorded action within four weeks: converting or closing the account, or withdrawing at least 95% of its starting balance. Vensia simulated the corresponding responses.

The displayed rates imply 84 additional actions per 1,000 recipients in Vensia, close to the human study’s 87. The model would have led an offer team toward the same simpler response package, with a similar estimate of its benefit. Here the human reference is recorded action; the Vensia result is simulated.

13. Customer interviews

Can synthetic interviews surface the concerns found in human qualitative research? We recreated the early interviews from FSA/Ipsos research into online food ordering, before discussion of hygiene-rating display requirements.

The human report found that participants focused on cost, delivery time, reviews and familiar outlets. Many assumed food was safe without actively checking hygiene until an unfamiliar outlet or bad experience made it salient. The synthetic sample had 40 participants: 20 in England and 20 in Northern Ireland.

Reviewers judged matches between synthetic interviews and published human findings. These percentages describe finding matches, rather than population prevalence or a normalized accuracy score.

Both reviewers found much of the human pattern, but they did not identify exactly the same five matches. The comparison supports using synthetic interviews to develop questions around familiar concerns. It cannot establish how common those concerns are in the wider population.

14. Comprehension testing

Explaining an offer more clearly may help with one benefit while leaving another unchanged. The BIT/Citizens Advice experiment compared a standard home-battery offer with one explained through a plain FAQ.

We focused on understanding cheaper off-peak electricity, solar energy use and “flexibility services,” which let an energy supplier use the battery.

Changes in the share answering correctly, FAQ minus standard offer. The human report gave these changes without testing whether they could be due to chance.

Vensia’s flexibility-services result rose from 21.8% to 37.0% correct, closely matching the human improvement. Its solar result fell from 66.0% to 62.5%, also following the human direction. An offer team would have identified the same strongest gain in comprehension. That is separate from whether the clearer explanation increases purchase.

Validate the decision, then inspect the score

For a team considering synthetic research, the practical question is whether the model is useful for the task it needs to do. A close result on ad liking does not establish accuracy for a different audience's willingness to pay. Choose a human benchmark that resembles the proposed decision, then compare the result in the units that decision uses.

Start with the original study. Record its population, recruitment, question wording, alternatives and outcome measure. Preserve the comparison conditions: a multi-select feature question is different from choosing one favorite, and a comprehension answer is different from a purchase. If you reconstruct a reference result from a published model, identify it as a reconstruction. In the EV trade-off case above, the human figures are approximate probabilities from the published choice model, rather than raw observed shares.

For a new validation, decide what will count as a useful result before examining the reference answers. If the task is to choose a message, inspect the winning option and the size of its advantage. If it is to prioritize a segment, examine whether the model points to the same group. Keep the reference outcomes out of the simulation inputs and retain what the model actually received. With a published study, withholding its answers from a prompt alone does not demonstrate that those answers were absent from the underlying model's training data; the validation record should describe what was controlled and what remains unknown.

Show the miss that could change the recommendation

A summary should include the paired outputs, their differences and the disagreements that matter. The examples above show why those details belong beside the headline:

• In the used-car scenario without fast charging, Vensia estimates EV choice at 53.1% while the reconstructed human estimate is 34.7%. That puts EV choice above versus below the halfway point and reverses the preferred alternative, even though other EV comparisons are closer.
• In the financial-product comparison, a narrow range of synthetic satisfaction scores conceals the pension gap. Vensia's 7.91 and the human 6.7 differ by 1.21 points on the ten-point scale, affecting which product looks most in need of attention.
• In the savings-offer comparison, the useful quantities include the 8.4- and 8.7-percentage-point increases. Readers can see the intervention's estimated size without having to infer it from a normalized score.

Read subgroup differences with the same care. The climate answer in the EV experience comparison has opposite directions in the synthetic and human results. A close overall pattern would not justify carrying that particular segment recommendation forward.

Different measures and study collections stay distinct

Ask how a reported accuracy measure was calculated, which endpoints it includes and how cases were combined. A rank, a percentage-point error, a dollar difference and a qualitative finding match answer different questions. Keep each measure with the task it describes; a combined score needs a justified common definition.

The twelve normalized scores are reported results. Their complete formulas and endpoint-level inputs are not supplied here, so they cannot be independently recalculated from the selected tables alone. Ask Vensia whether the complete formulas and endpoint-level inputs are available; until then, use the paired outcomes and errors shown above for the decisions they directly support.

Finally, take a discrepancy back to the decision. It may call for better audience evidence, a clearer stimulus, a narrower claim or a customer study focused on the unresolved choice. Preserve the original result when revising the setup so the comparison shows what changed. Validation is most useful when it tells the team which recommendation it can act on and which question still needs better evidence.

What these comparisons show

The strongest agreement appeared in practical tasks such as menu-message selection, EV feature priorities, burger pricing and the simplified savings offer. The model chose the same winning menu message, kept the displayed EV priorities in the same order, and estimated burger willingness to pay within an average of $0.24 per pound. In ad testing, Uber Eats was the only departure from the relative order of the eight spots.

Across the 12 cases with reported normalized accuracy scores, the lowest reported score was 95.3% for brand familiarity and liking. The highest reported scores were about 98.3% for EV feature priorities and offer testing, rounded to one decimal place. Each score describes its own task and endpoints, rather than the share of individual people predicted correctly. The dollar and interview-match measures in the other two cases remain separate. The paired results above show what agreement means in the units a team actually uses.

The coverage spans consumer packaged goods, food and restaurants, automotive, financial services and home energy. Use a comparable case to identify what a proposed study must get right: the order of messages, the size of a price gap, the audience difference or the action an offer changes. Where the paired results disagree, resolve that decision before using the modeled result to commit resources.

For a product-concept example, see our illustrative Red Bull protein study. It uses unofficial concepts and simulated consumers; it was not a project commissioned by Red Bull.

If you’re considering Vensia, start with the closest paired case above. Ask which audience, stimulus, outcome and failure check a proposed study would use for your decision. Explore the research you can scope with Vensia.