The Average Patient

Do Simulated Psychiatric Patients Answer Like Real People?

1University of Notre Dame2Simon Fraser University†Corresponding authors

Language models are used as simulated psychiatric patients to train clinicians and to evaluate AI therapists and assessment agents. They are usually judged by how plausible their transcripts look or by their total scores. We compare their questionnaire answers item by item with those of real people who report the same total. Across 17 models, simulated patients vary 1.5 to 11 times less than real respondents with the same total and endorse nearly every symptom at a middling level.

Same score, different symptoms

A questionnaire total measures how severe the symptoms are. It does not say which symptoms make up the score. On the PHQ-9, a nine-item depression questionnaire, two people can both score 15 while one sleeps badly and has lost interest in things and the other feels worthless and cannot concentrate. Clinicians meet this range of presentations, and an assessment agent has to recognise which symptoms a patient has. A simulated patient used for training or testing should show the same range.

Two PHQ-9 answer sheets
Distance between the two sheets
16
Answer sheets with this total
27,876
of 262,144 possible PHQ-9 answer sheets

Example sheets for illustration, not study data. Click a cell to change an answer. The distance adds up the item-by-item differences between the two sheets; diversity in the paper is the average of this distance over pairs of respondents with the same total.

How we compare

Each model is compared with real respondents at exactly the model's totals. This separates how a total is made up from how high it is. At each total we measure:

Diversity
the average item-by-item distance between two responses with the same total, as in the sheets above.
Composition
how many symptoms a response endorses, and how often it answers “not at all”.
Separability
how well a logistic regression on the item scores tells simulated from real responses, as an AUC. An AUC of 0.5 means the two cannot be told apart.
Structure
whether symptoms that go together in real respondents, such as the somatic symptoms of depression, also go together in simulated ones.

The 17 models include GPT, Claude, Gemini, Grok, Llama, Kimi, Mistral and Phi models. Each answers each questionnaire 50 times per condition, with the item order shuffled. The human references are public item-level data from eight samples, including the NHANES PHQ-9 for 41,714 US adults, a clinical sample and a study in which people were told to feign a disorder.

The average-patient hypothesis

A simulated patient usually receives a short persona that names a condition and a severity, and the persona is the model's only information about the patient. We hypothesise that the model then answers each item at the level typical of that item for the named condition and severity. On the PHQ-9 every item is a core symptom of depression, so one level serves all nine items.

The hypothesis predicts that simulated patients vary little, endorse nearly every symptom and give totals at or above the top of the stated severity band. An information budget sets a further limit. Simulated patients can differ only through information that the persona carries into their answers or through the randomness of single answers, and a persona that names one of four severities carries at most 2 bits.

PHQ-9 total told a severity

PHQ-9 severity bands. Triangle: the total if every item is answered at the hypothesised level. Dot: the median total of the 17 models told this severity. Each model's totals lie above the band of a mild to moderately severe label in 61 to 100% of its responses.

Variation at matched totals

We apply one matched-total comparison to eight questionnaires in nine human references, covering depression, anxiety, general distress, somatic symptoms, obsessive-compulsive symptoms, post-traumatic stress and eating disorders. Real respondents with the same total are 1.5 to 11 times more varied than every model. Choose a model to see where it falls on each questionnaire.

How many times more varied real respondents are

Ratio of real to simulated diversity at matched totals, on a log scale; each dot is one model. A ratio of 1 means simulated patients vary as much as real respondents. COACH is a clinical sample of older adults in rural China. The postpartum PCL-5 reference is not shown because fewer than five totals carry its ratio.

Show the numbers

Simulated patients spread a total over more symptoms. At PHQ-9 totals 15 to 24 and GAD-7 totals 12 to 19, models endorse all of PHQ-9 items 1 to 8, or all seven GAD-7 items, in 99 to 100% of responses, against 47 to 62% of real respondents with the same totals. On the PHQ-9 they answer “not at all” in 0 to 4.5% of answers, against 10 to 15% for real respondents. Item scores alone separate simulated from real responses with an AUC of 0.68 to 0.86, against 0.56 for two real samples.

Composition at the same totals
17 modelsReal respondents

Range over models; real respondents at each model's totals, range over models. Every item: PHQ-9 items 1 to 8 at totals 15 to 24 and GAD-7 at 12 to 19. “Not at all”: PHQ-9 at totals 15 to 24.

Comparison with people told to feign

Clinical psychology studies imitation by asking healthy people to feign a disorder and comparing their answers with those of patients. A public study gave the PCL-5, a 20-item checklist of post-traumatic symptoms, to injured students, to uninjured students answering honestly and to uninjured students told to feign an injury to earn a tuition deduction. We asked each model to play an injured student.

Compared with injured students who have the same totals, the people told to feign answer “not at all” 0.90 times as often. Every model answers “not at all” at most 0.37 times as often. Told to feign, people do not raise their total, whereas 15 of 17 models raise both their total and their share of “extremely” answers.

“Not at all” answers relative to genuine patients with the same totals
PeopleModels

PCL-5, with 95% bootstrap intervals. A value of 1 means a group answers “not at all” as often as genuinely injured students with the same totals. Hover over a row for the underlying rates.

Show the numbers

Single answers

As the hypothesis requires, single answers are highly concentrated. Across the 50 responses a model gives to one severity, the entropy of each item's answers averages 0.29 to 0.73 bits, out of at most 2, against 1.15 to 1.51 bits for real respondents with the same totals. Asked an item on its own, the most likely answer is the level the hypothesis names for 63 to 85% of items. Forms assembled item by item from the models' answer distributions are at most 32% distinct on average, against 69 to 92% for the same procedure on real respondents.

Entropy of single answers, PHQ-9
17 modelsReal respondents

Average entropy of an item's answers within one severity, in bits, range over models; real respondents at the same totals as each model. A four-option item has at most 2 bits.

Prompt changes and published personas

On seven models we tried the changes a simulator designer would try first, and the personas of two published simulators: the 12 clinician-validated depression personas of TalkDep and the 400 client intake forms of CACTUS. Each change raises diversity, and the largest gain, from adding five real answer sheets, reaches 48% of real diversity. Larger models in a family and reasoning do not reliably raise diversity, and none of 52 model and setting configurations exceeds 57% of real diversity.

Diversity, % of real respondents

Show the numbers

Interviews and an assessment agent

Simulated patients are mostly used in conversation. We interviewed seven patient models with a fixed script that covers the nine PHQ-9 symptoms, and another model rated the PHQ-9 from each transcript. Each patient was told a severity or was given a real NHANES symptom profile as private background.

Interviews, PHQ-9 items 1 to 8 rated from the transcript
Told a severityGiven a real profile
Interviews rated as having all eight symptoms97.5–100%35–40%
Symptoms rated absent0–0.3%15–17%
Diversity between interviews0.36–1.013.30–4.91
Placed in the patient's severity band20–26%42–61%

Ranges over patient models and two model raters. The real profiles lack 18.8% of symptoms 1 to 8.

Patients told a severity are rated as having nearly every symptom. On them, an agent that recognises absent symptoms and an agent that reports every symptom present get the same score, so this skill goes untested. Having the model word a real profile restores half to two thirds of the variation of the profiles.

We treated the interviewer and the rater as an assessment agent and scored it as benchmarks do, against what each patient was given. Patients told they are mildly to moderately severely depressed are rated above that band in 99 to 100% of interviews, so the agent places only 20 to 26% of patients told a severity in their band, against 42 to 61% of patients given a real profile. Five human raters who rated 64 of the transcripts show the same pattern.

Interviews placed in the patient's severity band
Patient told a severityPatient given a real profile

Per patient model. GPT-4.1 and Claude Haiku 4.5 were run only with a severity. Bottom rows: the 64 transcripts that five human raters also rated, with severities weighted equally; human raters, mean over raters.

Recommendations

  • Evaluate simulated patients item by item at matched totals, including how items covary.
  • Score assessment agents against the symptom profile the patient was given.
  • Ground simulated patients in real symptom profiles, or in profiles sampled from an item response model where real ones cannot be used.

Code and data

The code for the model runs, the measures and the analyses is on GitHub. The human reference data are public and are not redistributed; the repository README lists their sources. Model responses and the human ratings of interview transcripts are not released.

Citation

BibTeX
@article{sun2026average,
  title={The Average Patient: Do Simulated Psychiatric Patients Answer Like Real People?},
  author={Sun, Weixiang and Wang, Zehong and Shi, Kaiwen and Huang, Hong and Nelson, Colby and Li, Yiyang and Ye, Yanfang},
  journal={arXiv preprint arXiv:2611.11679},
  year={2026}
}