User research has a new money problem. The companies trying to simulate customers are no longer fringe experiments, and the question is no longer whether an AI-generated participant can produce a plausible interview transcript. The harder question is whether synthetic user research can produce evidence reliable enough to change what a team builds.
On July 30, 2026, behavioral simulation company Simile announced a $200 million Series B at a $2 billion valuation, just five months after emerging from stealth with a $100 million Series A. Simile says its models have already run tens of millions of simulations for large enterprises, with customers and partners including CVS Health, Wealthfront, Deloitte and Gallup.
That valuation does not prove synthetic users work. It does prove that the commercial appetite for replacing some expensive, slow and difficult human research with simulation has become very real.
The timing is unusually useful because the evidence is finally becoming harder to dismiss in either direction. In September and October 2026, Pew Research Center, Nielsen Norman Group and researchers publishing in Nature Human Behaviour released unusually direct comparisons between synthetic respondents and real human data. Meanwhile, Simile has begun publishing its own evaluation methodology, and Gallup is independently validating its approach.
DesignWhine has already explored synthetic users versus real users, reviewed a synthetic-research platform hands-on, and built a Tools Radar guide to synthetic user research tools. This time, the question is narrower and more consequential: when synthetic and human research are given the same problem, do they lead a product team to the same decision?
The $2 Billion Signal
Simile is not simply a chatbot wrapped in persona prompts. The company, founded by Stanford researcher Joon Sung Park and colleagues, is trying to build models specifically for human behavioral simulation. Its stated ambition is startlingly broad: simulate how people, customers and populations respond to situations that have not happened yet.
Its Series B announcement says the company had grown revenue fivefold in five months and had built a foundation model for human behavior. The commercial proposition is obvious. If a company can test product concepts, messages, prices or policy changes against a credible simulated population before paying for recruitment, fieldwork and analysis, research stops being a scarce event and starts behaving more like an always-available computational layer.
The danger is not that synthetic users are always wrong. It is that they may be right often enough to earn authority they do not deserve everywhere.
That is what makes the category more interesting than the early debate around fake personas. A useless simulation is easy to reject. A simulation that gets the headline answer right most of the time, while quietly distorting the distribution underneath it, is much harder to govern.
Plausibility Is Not the Test
Early criticism of synthetic users focused on something easy to notice: their answers often sounded generic, over-helpful or suspiciously articulate. Nielsen Norman Group’s 2024 testing found that synthetic users could produce shallow lists of needs and pain points while struggling to reproduce the messy prioritisation and specific lived context of real people.
That remains relevant, but it is no longer the most useful benchmark. A product team does not ultimately need an AI participant to sound convincingly human. It needs the resulting evidence to support the same decision that sufficiently good human research would support.
This changes the standard. Instead of asking whether a synthetic interview feels real, we should ask whether it preserves the ranking of options, the size of differences between groups, the distribution of preferences, the rare responses that alter the problem definition, and the confidence appropriate to the decision.
That is where the 2026 evidence becomes much more revealing.
When Synthetic Users Get It Right
On October 16, Nielsen Norman Group published one of the clearest UX-specific comparisons so far. Researchers reused a survey question previously answered by 604 human technology and design professionals, then asked 120 synthetic respondents the same underlying question through the Synthetic Users platform. Both datasets produced the same broad conclusion: UX remained the dominant term people used for the field.
The synthetic sample was not numerically identical. NN/g reported that 84% of synthetic respondents used UX-related terminology, compared with roughly 70% of human respondents. For a team asking only which label clearly leads, that difference might not change the decision.
A much larger 2026 Nature study reached a similarly complicated conclusion. Researchers assembled 70 preregistered nationally representative US survey experiments covering 469 experimental effects and 119,330 human participants. LLM-generated predictions were strongly correlated with real treatment effects and performed comparably to pooled human forecasts, although the synthetic predictions systematically overestimated effect sizes.
Stanford’s earlier work makes the positive case even stronger, with an important condition attached. Researchers recruited 1,052 real people for two-hour qualitative interviews, built generative agents from those interview transcripts, and found that the agents reproduced participants’ General Social Survey responses about 85% as accurately as the humans reproduced their own answers two weeks later.
That last detail matters enormously. The best-performing synthetic person in this example was not conjured from age, occupation and a few personality adjectives. It was grounded in two hours of actual human testimony from the individual it was trying to simulate.
When Correct Becomes Misleading
The problem appears when teams confuse a correct headline with a faithful population. In the NN/g study, synthetic and human respondents agreed on the winner, but the synthetic sample produced far less variation and pulled different professional roles toward the same dominant terminology. The decision survived. The diversity of the underlying evidence did not.
Pew Research Center found a more serious version of that problem in September. Its researchers created AI “digital twins” using detailed profiles of real American Trends Panel members, then asked those synthetic respondents the same questions humans had answered across three 2026 survey waves. The average absolute error across the full sample was 12.4 percentage points, with larger errors for several subgroups, including 16.1 points for Republicans and Republican-leaning adults and 15.1 points for Black adults.
A new Nature Human Behaviour study pushed the test across 74 countries and 67,201 matched human respondents. Synthetic responses recovered the direction of many associations involving gender, age and cognitive ability, but captured only 48.1% of the tested cultural associations. Performance was often weaker in less economically developed and less digitally connected countries.
Another 2026 benchmark, SSDataBench, evaluated 15 LLMs against real longitudinal and cross-sectional social-science data. The recurring failure was not random nonsense. It was compression: models tended to turn complicated populations into cleaner, more typical patterns than real humans actually produced.
Synthetic research can reproduce the headline while erasing the tail.
For product teams, the tail is often where the expensive insight lives. The customer who uses a workaround nobody anticipated, the subgroup that reacts differently, the edge case that reveals an accessibility failure, or the participant who rejects the premise of the concept can matter more than a beautifully stable average.
| Head-to-Head Evidence | What Synthetic Respondents Got Right | What They Missed |
|---|---|---|
| NN/g, 2026 | Same overall conclusion on UX terminology | Much less variation; stronger pull toward the dominant answer |
| Nature, 2026 | Strong correlation with many experimental treatment effects | Systematically overestimated effect sizes |
| Pew, 2026 | Could reproduce some broad patterns | Large absolute errors and uneven subgroup fidelity |
| Nature Human Behaviour, 2026 | Recovered many demographic-direction patterns | Weak cultural fidelity across countries |
| Stanford, 2024-25 | High individual-response fidelity | Required rich two-hour human interviews to ground each agent |
The Failure Mode DesignWhine Already Saw
DesignWhine encountered a smaller but instructive version of this problem while testing Articos. We asked the platform to analyse DesignWhine itself and found parts of the workflow genuinely useful. It could rapidly construct personas, conduct simulated interviews, cluster themes and turn an unstructured research question into something a product team could interrogate.
Then one synthetic participant confidently praised DesignWhine’s supposedly well-structured pricing tiers. There was no pricing section and there were no reader subscription tiers to evaluate. The finding was not absurd enough to announce itself as a hallucination. It was exactly the kind of plausible observation a busy team might accept because it sounded like a normal research insight.
This is why the synthetic-user problem is not identical to ordinary model hallucination. A research system can generate dozens of sensible observations and one false one, then present all of them with the same typography, confidence and methodological theatre. The interface makes the weak signal look like evidence.
Our earlier article argued that the most plausible future was hybrid: synthetic research before, between and alongside human research. The newer evidence makes that position more precise. The real dividing line is not human versus synthetic. It is calibrated evidence versus uncalibrated simulation.
Why Simile Changes the Debate
It would be a mistake to evaluate every synthetic-research company as though it were asking a general-purpose LLM to role-play “a 34-year-old marketing manager.” Simile is explicitly trying to move beyond that model. It says its simulation system learns from interviews, surveys, transactions, app usage and experiments, then predicts distributions of behavior or preference for new situations.
On October 28, Simile published a detailed description of an evaluation suite containing more than 10,000 scenarios, each grounded in the attitudes or behavior of hundreds or thousands of real people. The company says evaluation cases are held out from training, results are assessed with measures appropriate to the decision, and performance is tested across audiences and research objectives rather than reduced to one universal accuracy number.
That is a materially better standard than showing a few convincing synthetic interview transcripts. Simile also trains a confidence model to estimate how much error a new simulation may contain, using roughly 8,600 held-out questions in the evaluation described in its technical post.
But the details also demonstrate why independent validation still matters. Simile’s current “decision-quality” threshold was calibrated using thousands of internal judgments, and the company itself says it expects that definition to evolve with external raters and more diverse evaluation sets. A model that predicts its own likely error is useful, but it is not the same thing as an independent guarantee that the simulation is correct.
Gallup is providing one of the most important external checks. Since fall 2025, approximately 1,000 Gallup Panel members have completed in-depth interviews that were used to create agents. Gallup says early general-population estimates for questions close to the interview material were sufficiently close to human estimates to justify continued testing, while explicitly investigating how accuracy changes across topics, subgroups, question formats and time.
That is encouraging, but it is also a reminder of how narrow a useful claim should be. “This simulation is validated for this type of question, against this kind of human reference, for this audience, within this error range” is a much stronger statement than “synthetic users work.”
A Reliability Ladder for Product Teams
The evidence suggests that synthetic user research should not be governed by a binary rule. Product teams need a ladder that reflects what the research is being asked to do and the cost of being wrong.
| Research Task | Synthetic Value | Human Validation |
|---|---|---|
| Pressure-test an interview guide | High | Usually later, when the real study runs |
| Generate hypotheses or obvious objections | High | Required before treating hypotheses as findings |
| Early concept triage | Moderate to high when the method is calibrated | Required before major product or launch commitments |
| Estimate segment differences | Moderate to low | Strongly required because subgroup errors can be large |
| Cross-cultural or minority-group inference | Low without direct validation | Essential |
| Behavioral usability, accessibility or contextual discovery | Low as a substitute for people | Essential |
| High-stakes pricing, policy, health or financial decisions | Potentially useful as a pre-test | Non-negotiable |
This is also how Tools Radar should judge the category. The important product questions are no longer just whether a platform offers personas, interviews and automatic reports. The differentiators are what human evidence grounds the simulation, whether the vendor publishes held-out validation, how subgroup performance is measured, whether uncertainty is surfaced, and whether the software makes it difficult to confuse generated quotes with human testimony.
That makes some of the category’s least glamorous features the most important. Provenance, calibration, confidence intervals, benchmark coverage and explicit boundaries may matter more than how lifelike the simulated participant feels in a chat window.
The Real Research Test
So, can synthetic users survive a real research test? Increasingly, yes, for some tests. They can reproduce broad rankings, forecast many experimental effects, approximate some population estimates and become surprisingly faithful when grounded in deep information about the specific people they simulate.
But the failures are not disappearing as quickly as the interfaces are improving. Synthetic respondents still compress variation, distort subgroup differences, struggle across cultural contexts, overstate effects and occasionally invent evidence with unnerving fluency. Worse, a polished research report can hide those errors better than a messy human study ever could.
That is why Simile’s $2 billion valuation should not be read as proof that real user research is becoming obsolete. It is a wager that simulation can become a useful measurement instrument for human behavior. The companies that win this category will be the ones that can show, repeatedly and independently, exactly where that instrument is calibrated and where it is not.
A synthetic user should be judged less like a participant and more like a measurement instrument.
That distinction may be the most useful way to move the debate forward. Researchers do not need to decide whether an AI respondent is “real enough.” They need to know what it measures, how much error it introduces, which populations it misrepresents, and whether the decision would change if humans were asked instead.
If synthetic user research can answer those questions, it has a serious future. If it cannot, the industry may simply be automating the production of confidence.
What are synthetic users in UX research?
Synthetic users are AI-generated or AI-simulated respondents designed to stand in for people during research tasks such as interviews, surveys, concept tests or usability-related evaluations. They may be generated from demographic prompts, first-party data, interviews, behavioral records or combinations of those sources.
Can synthetic users replace real user research?
Current evidence does not support treating synthetic users as a general replacement for real research. They can be useful for hypothesis generation, early concept triage and some calibrated prediction tasks, but human validation remains important when behavior, context, subgroup differences, cultural variation or high-stakes decisions matter.
How accurate are synthetic respondents?
Accuracy varies dramatically by task, population, model and the data used to ground the simulation. Some studies find synthetic respondents reproduce broad conclusions or experimental effects, while others show large errors, reduced variation and weak subgroup or cross-cultural fidelity. There is no single accuracy figure that applies to synthetic user research as a category.
What is synthetic user research best used for?
The strongest current use cases are preparing research, generating hypotheses, pressure-testing questions, exploring obvious objections and cheaply comparing early directions before spending on human studies. The more consequential the decision, the more important matched human validation becomes.
What is Simile and why is it important to this debate?
Simile is a behavioral simulation company founded by researchers connected to Stanford’s generative-agent work. It raised $200 million at a $2 billion valuation in July 2026 and is building models trained on multiple forms of human evidence. Its emphasis on held-out evaluation and question-level confidence makes it an important test of whether synthetic research can become a calibrated decision tool rather than a persona generator.
Would you trust synthetic users to shortlist an idea if every major launch decision still required human validation? Where would you draw the line?