Large Language Models as Computational Instruments for Social Science Research

Classical traditional surveys

Beloved standard tool of the social sciences

Often considered the gold standard, “ground truth” (especially when working with large representative samples of a population)

But, classical survey methodologies increasingly suffer from problems

First line of the 2024 Book “Polling at a Crossroads: Rethinking Modern Survey Research”: Survey research is in a state of crisis

Last example: US presidential elections 2024

Biggest issue is non-response

Alternatives?

Synthetic Surveys

Britain’s mood, measured weekly

One example of an easily accessible, representative survey (UK) in the affective domain

https://yougov.co.uk/topics/politics/trackers/britains-mood-measured-weekly

Results

We can recreate dynamics with that approach of longitudinal adaptors

Not equally well for all constructs

Remember, our approach is just self-supervised next token prediction (no labels present as for example with the supervised text classification method of TweetNLP)

Our approach is very flexible, we can in principle ask any question and get survey-like responses for each week

Why does that work?

Wrap-up

I don’t think we should be replacing survey research

Also with complementary synthetic methods we will need classical approaches for example to learn about the sampling frame

But we should be making use of the text that people are producing (and potentially other modalities too)

It’s first steps for now and we strongly have to validate what we are doing

Huge potential: Low costs, scalability, unobtrusive observation, high temporal resolution, …

Bridging the gap between “qualitative” data and quantitative insights

Next: LLMs for Digital Twinning

LLMs increasingly deployed as autonomous agents

Research gap: alignment with actual human decision-making

Why Game Theory?

Analytical solutions (Nash equilibria) as benchmarks

Rich empirical data from human experiments

Simple, well-defined tasks

Real-world relevance

Goal: Replication of human experimental data with LLMs, systematically validated → novel predictions

Methods

Models: Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct

Original Experiment: 500+ humans, 121 games (Human behavioral phenotypes across games: All deviate from Nash equilibrium)

Payoff Structure:

	C	D
C	(10,10)	(S,T)
D	(T,S)	(5,5)

S ∈ [0,10], T ∈ [5,15] → Extended through our simulations to [0,20]

Progressive Answer Extraction

Simple: Direct answer → (mostly) random patterns
Double: Long answer + extraction → some structure
Multi-step: Guided reasoning → clear patterns
Logical Verifier: + validation → high algorithmic fidelity

“Thinking step-by-step” improves coherence

Logical verifier acts as an “LLM attention check”, specifically checking on consistency in the Harmony Game region

Quantitative Model Comparison

	Human		Nash
	MSD	r	MSD	r
Llama	0.031	0.89	0.089	0.77
Mistral	0.091	0.70	0.182	0.60
Qwen	0.065	0.79	0.036	0.93
Nash	0.096	0.78	-	-

Llama replicates humans (better than Nash); Qwen follows Nash; Mistral intermediate

Observations

Human vs. Llama similarities:

High cooperation when S ≥ T
Low cooperation when T > R
Binary-like patterns

Llama (and human) vs Nash:

No mixed equilibria
Discrete choices
Emulating (human) psychological heuristics?

Average cooperation: Llama 40.2%, Human 48.0% vs. Nash prediction of 50%

Novel Game Predictions

Extended 121 → 441 games

Llama patterns beyond human-tested space:

S ≥ T diagonal holds
T > R reduces cooperation
Instability near (0,0)

Pre-registered experiment for future validation¹

Key Contributions

Population-level replication without personas
Open-source models (reproducible)
Logical verification as quality control
Outperforms Nash at predicting humans (Training creates behavioral imitators)
Generates testable hypotheses

Limitations

Edge case instability
Potential memorization concerns
Black-box mechanisms
Requires human validation

:::

Solution: Pre-registered experiments decide on the predictive validity of our approach

Conclusions and Implications

With the right protocol, we can use LLMs to replicate human patterns and to capture deviations from rationality

Complementary tool for the social and social and behavioral sciences

Rapid experimental space exploration

Generate hypotheses → validate with humans

AI-assisted scientific discovery

Pre-Print: Palatsi, A. C., Martin-Gutierrez, S., Cardenal, A. S., & Pellert, M. (2025). Large language models replicate and predict human cooperation across experiments in game theory (arXiv:2511.04500). arXiv. https://doi.org/10.48550/arXiv.2511.04500

Next: AI Psychometrics

The Promise and the Problem

Psychometrics depends on human rater judgments to measure latent constructs (values, personality, attitudes)

Questionnaire development is iterative and costly: large candidate item sets, extensive piloting, respondent fatigue, low motivation

Growing interest in using NLP and embeddings to complement psychometric workflows

But: Existing embedding approaches face a critical limitation: they cannot produce negative correlations between items

Hello everyone, thank you for watching this presentation of our Findings paper at EACL 2026. Psychometrics, the field concerned with measuring human attributes like personality, skills, or values, traditionally requires collecting ratings from large samples of human respondents on carefully designed questionnaire items. This process is expensive, time-consuming, and plagued by issues such as respondent fatigue and low motivation. There is growing interest in using NLP technologies to complement these workflows. Several recent studies have proposed using embeddings of questionnaire items, but they all face a critical limitation: the resulting similarity matrices cannot produce negative correlations between items. This is a serious problem for many psychometric applications, and it is the problem we address in this paper.

Test Case: Schwartz’s Value Theory

Values are trans-situational goals that guide behavior

The Revised Portrait Value Questionnaire (PVQ-RR): 57 items, 19 fine-grained value dimensions

Values form a circumplex structure: motivationally compatible values correlate positively and are close, opposing values correlate negatively and are far apart

Cross-culturally validated with human data from 49 countries (Schwartz & Cieciuch, 2022)

Why is this hard? The circular structure depends critically on negative correlations between opposing value dimensions, which is exactly what embeddings struggle with

Prior Work and the Missing Negatives

Cutler & Condon (2023): Template-based approach with BERT, limited to personality descriptors

Wulff & Mata (2025): Domain-specific finetuning on 200,000 item pairs, dropped sign of correlation entirely

Hommel & Arslan (2025): Re-annotated NLI corpus to handle negation, then finetuned

The root cause: High-dimensional embeddings encode many shared abstract linguistic features that artificially inflate similarity between all items, even semantically opposing ones

SQuID: Our Methods Contribution

Survey and Questionnaire Item Embeddings Differentials

For each item embedding \(\mathbf{y}_i\), subtract the mean embedding over all items: \[\mathbf{y}_i - \bar{\mathbf{y}}, \quad \text{where } \bar{\mathbf{y}} = \frac{1}{57}\sum_{i=1}^{57}\mathbf{y}_i\]

Removes shared linguistic features that inflate similarity

Inspired by Word2Vec arithmetic (Mikolov et al., 2013) and community embeddings (Waller & Anderson, 2021)

Works with any model, any embedding dimension, requires no finetuning, no re-annotation, applied purely post-hoc

Our methodological contribution is called SQuID, which stands for Survey and Questionnaire Item Embeddings Differentials. The idea is straightforward: we compute an average embedding over all items of the questionnaire, what we call a questionnaire embedding, and subtract it from each individual item embedding. This removes the shared abstract linguistic features that inflate similarity and obscure the signal of negative correlations. The method is inspired by early examples of vector arithmetic demonstrated with Word2Vec and the more recent idea of community embeddings by Waller and Anderson. Crucially, SQuID requires no domain-specific finetuning, no training data re-annotation, and works with any embedding model and any embedding dimensionality. It is applied purely post-hoc to already computed embeddings.

The Effect of SQuID

By subtracting the questionnaire embedding, negative correlations clearly appear without any domain-specific adaptation

Workflow Overview

From the similarity matrix onward, the analysis pipeline is identical for embeddings and human rater judgment data

Internal Consistency: Cronbach’s Alpha

	Average alpha
Linq-Embed-Mistral	0.77
Human data (49 countries)	0.70
gemini	0.64
jina / kalm	0.67
mpnet-personality	0.57
Random baseline	-0.05

Linq-Embed-Mistral beats human data on average Cronbach’s alpha and wins 14 out of 19 individual dimensions

MTEB benchmark ranking serves as a good guide for model selection in this task

Dimension-Dimension Correlations

171 dimension pairs: r = 0.74 [0.66, 0.80], simple linear model explains 55% of variance, no bias

MDS: Circular Value Structure Emerges

Facets of the same construct cluster together, opposing values are placed on opposite sides

Generalizability Beyond Value Theory

SQuID tested on three personality questionnaires: IPIP, BFI-2, HEXACO

Questionnaire	Avg. relative gain in range
IPIP	261.5%
BFI-2	207.2%
HEXACO	147.9%

Individual model gains range from 21% (mpnet, already domain-finetuned) to 724% (kalm)

Even domain-finetuned models benefit from SQuID. Consistent improvements suggest broad applicability.

Discussion and Future Directions

What we showed: SQuID enables embeddings to recover psychometric structures on par with human data, with no finetuning and broad generalizability

Exciting future direction: Systematic assessment of cultural biases in LLMs by comparing embedding MDS configurations per country with human data from 49 countries

Open challenges: Reverse-keyed items, hierarchical and network-based psychometric models, multilingual evaluation, clinical inventories

Our vision: Not merely another step in the validation pipeline, but a new way to investigate the full spectrum of human behavior and experience across all natural languages

To summarize, we introduced SQuID, a simple yet effective method that enables neural embeddings to recover latent value dimensions from psychometric questionnaire items on par with human data. The approach requires no finetuning and generalizes across different psychometric instruments. For future work, we see a particularly exciting opportunity to use SQuID for systematic assessment of cultural biases in language models. Schwartz and Cieciuch validated their theory across 49 countries, and we can now compare embedding-derived configurations at the country level to quantitatively assess an LLM’s closeness or distance to specific cultural value orientations. Open challenges include handling reverse-keyed items, testing on different psychometric model structures, and multilingual evaluation. We view our approach not merely as another step in the validation pipeline but as opening fundamentally new ways to investigate the full and diverse spectrum of human experience and behavior across all natural languages.

The end

Max Pellert

max.pellert@bsc.es

Barcelona Supercomputing Center

Code and data: github.com/maxpel/embeddings_values