Editorial illustration for Larger Model Variants Reproduce and Amplify Human Cross-Scale Correlations
Large Language Models Mirror Human Personality Traits
Larger Model Variants Reproduce and Amplify Human Cross-Scale Correlations
A team testing 14 language models, ranging from half a billion to 14 billion parameters and spanning the Llama 3 and Qwen 2.5 families, wanted to know something specific: when a model reads one person's answers to a psychological scale, does it actually form an internal sense of who that person is, or is it just pattern-matching its way to plausible-looking output on the next six scales?
The setup borrowed real data. 272 university students in Taiwan filled out personality questionnaires, and researchers fed each model a single scale's worth of responses from one of those students, then asked the model to predict how that same person answered on six other scales. That's the cross-persona paradigm at the center of this work, base and instruct versions of each model included, so the comparison covers both raw pretraining and fine-tuned behavior.
The stakes are narrower than they sound. Getting the predicted numbers right doesn't settle whether a model is tracking a construct like conscientiousness or extraversion the way a psychologist would define it, or whether it's exploiting surface correlations in the training data. Ridge probing techniques, the tool used to check what's happening inside these models, come with their own risk of manufacturing signal where none exists.
Given responses on one psychological scale from real individuals (N = 272 Taiwanese university students), models predict responses on six other scales. At the behavioral level, larger variants reproduce and amplify human cross-scale correlations; amplification persists after correcting the seven-scale human correlations for measurement attenuation.
Why this matters
The 14-model comparison here matters because it separates two claims that get conflated constantly: a model producing psychometrically plausible outputs, and a model actually encoding the construct behind those outputs. Finding that bigger Llama 3 and Qwen 2.5 variants don't just reproduce human cross-scale correlations from that 272-person Taiwanese sample, but amplify them even after correcting for measurement attenuation, tells us something uncomfortable about scale. It's not making models more human, it's making them more confidently wrong in a specific, structured way.
For anyone building on these models for survey simulation, synthetic personas, or psychological assessment tools, that's a real warning sign: the correlational structure looks "cleaner" than real humans produce, which should read as a red flag, not a feature. The contrastive direction analysis probing whether this shows up internally, not just behaviorally, is the part worth watching closely. If internal representations don't match the amplified outputs, that's evidence models are pattern-matching statistical regularities in training data rather than representing the construct at all.
Either way, treat "model agrees with psychometric theory" as a result to interrogate, not trust.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives EarthRecord Benchmark Maps Four Decades of Land Change Across 56 Sites
From Archives Anthropic turns Claude into a full workspace with plugins and spreadsheets
Common Questions Answered
What was the main research question when testing the 14 language models across Llama 3 and Qwen 2.5 families?
The researchers wanted to determine whether language models actually form an internal sense of who a person is when reading their psychological scale responses, or if they are simply pattern-matching to produce plausible-looking outputs. This distinction is crucial for understanding whether models truly encode psychological constructs or merely reproduce statistical patterns from training data.
How did the study use real data from the 272 Taiwanese university students to test model behavior?
The researchers had 272 university students in Taiwan complete personality questionnaires and psychological scales. They then used the models' responses on one psychological scale to predict how those same individuals would respond on six other scales, measuring whether the models could accurately infer cross-scale correlations.
What did the researchers discover about how larger model variants handle human cross-scale correlations?
Larger variants of Llama 3 and Qwen 2.5 not only reproduced human cross-scale correlations from the sample but actually amplified them, even after correcting for measurement attenuation. This amplification suggests that scale in language models may introduce concerning distortions in how psychological constructs are represented and predicted.
Why is the distinction between producing psychometrically plausible outputs and encoding actual psychological constructs important?
This distinction matters because it separates whether models are simply generating statistically plausible responses versus actually understanding and representing the underlying psychological constructs. The study reveals that larger models can amplify correlations beyond what humans naturally exhibit, suggesting they may be encoding something different from genuine psychological understanding.
Further Reading
- Tracing Psychometric Inference in Large Language Models - arXiv
- From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilers - arXiv
- Can Open-Weight Large Language Models (LLMs) Reproduce Human Cross-Instrument Correlations? - arXiv
- Language Models Embody and Amplify Human Cognitive Biases - arXiv
- Comparing the Framing Effect in Humans and LLMs - ACL Anthology
