Llama replicates humans (better than Nash); Qwen follows Nash; Mistral intermediate

The behavioral result raises a mechanistic question: why does Llama track human cooperation while Qwen converges to Nash?

Attention-based payoff salience analysis (ongoing work, currently added to the Science Advances revision): we extract attention weights over payoff matrix entries across transformer layers

Preliminary finding: Llama attention concentrates on R(eward) and T(emptation) in a consistent pattern throughout its layers; Qwen distributes attention more uniformly across the full payoff structure

The behavioral difference between models seems to be traceable to how they internally represent and weight payoff-relevant information

The black box critique is not unanswerable: LLMs are not entirely opaque

Extended from 121 → 441 games, reaching regions of the payoff space the human experiments we build on haven’t covered

Llama extended experimental grid reveal a much simpler picture: the S ≥ T diagonal, reduced cooperation when T > R

Publicly pre-registered experiments for future validation (https://aspredicted.org/fe6z2k.pdf)

Simulation generates hypotheses, pre-registration ensures transparent testing

Training creates behavioral imitators: models that outperform Nash at predicting what humans will do

Credits: First author Andrea Cera Palatsi – internship master thesis with me at BSC, now starting her PhD at the highly competitive CSH Digital Innovation School in Vienna

Classical psychometrics: instruments designed to measure latent human constructs (values, personality, attitudes, decision-making styles)

The same instruments can be applied to large language models

Two complementary modes:

• Behavioral: present stimuli, record responses (e.g. game theory, surveys)
• Representational: probe the internal geometry of model embeddings

Both modes are needed and both feed directly into REVEAL, the observatory framework I will describe shortly

Why does this matter? Deployed LLMs shape recommendation, moderation, and increasingly governance; we need rigorous tools to audit them.

Psychometric instruments like the Schwartz PVQ-RR (57 items, 19 value dimensions, validated across 49 countries) depend on costly human rating studies – and opposing value dimensions require negative correlations that raw embeddings of the text of items cannot produce due to similarity inflation

The fix: subtract the mean embedding over all questionnaire items

\[\mathbf{y}_i - \bar{\mathbf{y}}, \quad \text{where } \bar{\mathbf{y}} = \frac{1}{57}\sum_{i=1}^{57}\mathbf{y}_i\]

Removes shared linguistic features. Works post-hoc, any model, no finetuning.

The mini class later covers the embedding mechanics behind this in more detail – here I want to show you what it recovers.

Example items from the PVQ-RR:

“It is important to her to take advantage of every opportunity to have fun.” → Hedonism

“It is important to her to follow her family’s customs or the customs of a religion.” → Tradition

Facets of the same construct cluster together, opposing values are placed on opposite sides (reconstruction from embeddings is close to human data)

Cultural bias auditing: systematically compare internal model representation against per country against human data from 49 countries

Longitudinal monitoring: track value drift across model versions

Explanatory framework for the science of emergent misalignment: Betley et al., 2026 in Nature

Instrument design: use embedding geometry to pre-screen questionnaire items before costly human piloting

Open challenges: reverse-keyed items, hierarchical and network-based psychometric models, multilingual evaluation, clinical inventories

Aim 1 · Measure: Apply SQuID and the PVQ-RR to derive value configurations from LLM embeddings, validated against the theory and benchmarked against human data from 49 countries

Aim 2 · Compare: Do value representations converge across models (Platonic Representations Hypothesis) or diverge? Convergence implies fundamental limits to cultural differentiation through training data alone with direct implications for AI governance

Aim 3 · Intervene: Use linear probing to locate where value-relevant information is encoded, then steer specific value dimensions and measure downstream behavioral effects, for example on hiring, medical advice, and content moderation decisions

Broader vision: An European AI Behavioral Observatory providing open psychometric workflows, regulatory toolkits, systematic auditing of deployed AI systems against culturally grounded value norms

Horizon Europe AI4SOCIALPLUS – successfully funded

Use Case 4: AI-Assisted Social Simulation for Actionable Hypothesis Generation

LLM agents grounded in empirical findings and social media data via RAG architecture, for designing surveys, experiments, and policies

FWF Principal Investigator Project – Marie Curie Seal of Excellence proposal building on REVEAL ready as basis; concrete funding runway with externally validated research agenda

DEMOS-AI – Spanish national funding (Proyectos de Generación de Conocimiento), decoding moral and sociopolitical biases in LLMs, co-PI with Paula Szewach at BSC

Emma Fraxanet – worked together across her PhD on signed networks, online polarization, and the FAULTANA pipeline (PNAS Nexus); now PostDoc in my team at BSC

Andrea Palatsi – master’s student at the BSC, first author on the game theory paper; starting PhD at the CSH Digital Innovation School in Vienna

Paula Szewach – co-PI on DEMOS-AI, BSC

COMPASS group – COMplex Political And Social Simulation, the research group of 5 people I lead at the Barcelona Supercomputing Center, bringing together computational social science, political science, and complexity science

I am Associate Faculty at the Complexity Science Hub Vienna, which is a natural bridge to the Viennese research ecosystem

Computational social science has mastered people on networks; the next frontier is AI agents embedded in social systems.

Studying those agents behaviorally, at scale, with rigorous methods, is a network and social data science problem

Link 1: LLM populations as social systems; cooperation and defection propagating through interaction structures; extending from dyadic games to games in groups to study new kinds of coordination games

Link 2: temporal and sequential dynamics in LLM behavior; societal-scale implications of internal representations; synthetic surveys as a complementary methodology to extract signals from large-scale unstructured (textual) data

Link 3: ethics and accountability of AI auditing; whose values are the reference? what does that mean for using LLMs as instruments for tasks like annotation?

Link 4: AI behavioral measurement as accountability infrastructure for regulators and civil society (the EU AI Act creates demand for exactly this)

In a nutshell, I can offer to extend together what you already do into novel domains that I think will become unavoidable.

What I would build here: A group on measurement approaches to AI, social data science and computational social science; researchers working on behavioral benchmarking of LLMs, value representation across cultures and model versions, and AI-assisted social simulation; direct collaboration with the DNDS faculty on network science, NLP, data science and governance

On teaching: I have taught data analysis, NLP, statistics, and visualization in Mannheim and Konstanz, and am ready to contribute directly to the department’s core curriculum from day one (in person, in Vienna)