Summary: A new study introduces an automated cognitive mapping framework that pairs the raw computational power of Large Language Models (LLMs) with precise behavioral choice mathematics. By utilizing an LLM to read, interpret, and categorize thousands of free-text personal justifications written by human participants during high-stakes gambling tasks, the team successfully proved that human self-insights are a highly reliable and mathematically valid source of data, showing that the core logic people rely on shifts systematically depending on the immediate shape of the problem.
Key Facts
- The Gambling Text Experiment: During simulated gambling rounds involving shifting risk parameters, participants were blocked from simply clicking a choice; they were required to actively describe their internal thought processes and subjective justifications in their own words after every single round.
- The Algorithmic Codebook: Drawing on decades of established behavioral finance and decision-making theories, the researchers built an extensive taxonomy of possible human rationales, ranging from hyper-focused optimization of the best possible outcome (“maximax heuristics”) to intense avoidance of maximum devastation (“minimax loss aversion”).
- LLMs as Scalable Qualitative Auditors: Rather than relying on human research assistants to manually read and tag thousands of individual journal sheets, the team deployed fine-tuned LLMs. The models functioned as high-speed qualitative auditors, reading the free-text data at scale and instantly tagging the exact psychological reasons driving each entry.
- Mathematical Choice Validation: To verify that the LLM’s text classifications weren’t hallucinated or arbitrary, the researchers cross-referenced the model’s text tags with objective mathematical modeling of the participants’ actual physical choices. The choice equations validated the text profiles with remarkable precision, confirming that what people said they were doing aligned perfectly with how they acted.
- Dynamic Strategic Shifting: The unified data clearly demonstrated that human decision-making strategies are not permanent personality traits. Instead, individuals are highly adaptive: they dynamically and systematically shift their reasoning profiles from round to round based on how a problem is presented.
- A New Toolkit for Public Policy: Dr. Fuławka stresses that this automated analytical framework opens up vast horizons for studying human behavior inside complex, real-world ecosystems. By allowing researchers to parse massive amounts of free-text public feedback, policymakers can better understand how communities interpret and simplify complicated trade-offs in public health, economic planning, and technological adaptation.
Source: TUD
“Our understanding of human behavior, including decision making, can be deepened by asking people to elaborate on their thought processes,” says lead author Dr. Kamil Fuławka, researcher at SynoSys. “However, the systematic analysis of such free-text data requires scalable and rigorous analytical frameworks — an endeavor that can now be supported by LLMs”
In the experiment, participants took part in gambling and had to explain each decision in their own words. To analyze these explanations, the researchers drew on existing theories and models of decision making to develop a large set of possible decision reasons, such as focusing on the best possible outcome or avoiding a big loss. Large language models (LLMs) identified which of these reasons appeared in participants’ free-text explanations, while mathematical modeling of people’s choices provided validation.
The combination of verbal reports, LLMs, and rigorous mathematical modeling clearly demonstrated that people’s own insights are a valuable source of data. It also showed that the reasons people rely on are not fixed, but shift systematically with the structure of the decision problem.
“Many important decisions — from financial planning and medical choices to social dilemmas, technology use, and public policy — involve complex trade-offs that cannot be fully understood from observing choices alone,” says Kamil Fuławka, emphasizing the relevance of the study’s findings: “In such settings, people’s own explanations may be especially valuable for revealing how they simplify complex problems, focus on particular pieces of information, and adaptively use simple decision strategies.” The framework presented in the study shows how LLMs can help researchers analyze these explanations at scale, opening new opportunities to study human decision making in more realistic and complex environments.”
The framework presented in the study also demonstrates how LLMs can help researchers analyze these explanations on a large scale, thereby opening up innovative possibilities for studying human decision-making processes in more realistic and complex environments.
Key Questions Answered:
A: Looking only at a person’s final choice is like trying to understand a complex murder mystery by only looking at the very last page of the book. Traditional behavioral economics tracks what button someone clicks or what product they buy, but it remains completely blind to the hidden thought processes, doubts, and mental shortcuts that led to that action. While researchers have always wanted to read people’s written explanations to understand their true motives, manual reading is a massive bottleneck. By using an LLM as a high-speed, automated reader, scientists can now analyze thousands of detailed personal journals instantly, giving them an unprecedented look inside the human mind at a massive scale.
A: This was the most brilliant and vital layer of the SynoSys experiment. To ensure the LLM was accurately mapping human psychology rather than generating “hallucinations,” the team used objective choice mathematics as a strict validation shield. They took the psychological reasons the AI discovered in a participant’s text, like “trying to avoid a catastrophic loss”, and plugged that exact rationale into a separate mathematical algorithm tracking the participant’s actual physical gambling behavior. The math and the text aligned perfectly: the choices people made matched up precisely with the written motivations the AI extracted from their journals, proving the LLM’s scale is highly accurate and scientifically sound.
A: Most monumental choices in life, like planning for retirement, choosing a cancer treatment, or adapting to new technology, involve incredibly messy trade-offs that don’t fit into a simple multiple-choice question. Historically, governments and banks have struggled to analyze mass public surveys because coding open-ended feedback is too slow. This new framework allows public policy analysts to deploy LLMs to immediately read and quantify thousands of complex, written responses from real citizens. This reveals exactly how the public simplifies complicated problems, what specific pieces of information they focus on, and how to design safer, clearer, and more supportive programs that match real human behavior.
Editorial Notes:
- This article was edited by a Neuroscience News editor.
- Journal paper reviewed in full.
- Additional context added by our staff.
About this AI and decision-making research news
Author: Benjamin Griebe
Source: TUD
Contact: Benjamin Griebe – TUD
Image: The image is credited to Neuroscience News
Original Research: Open access.
“Large language models accurately identify decision reasons in verbal reports” by Dirk U. Wulff, Kamil Fuławka, Ralph Hertwig. PNAS
DOI:10.1073/pnas.2526798123
Abstract
Large language models accurately identify decision reasons in verbal reports
Understanding the reasons behind human choices under risk is a central goal of decision scientists, but traditional methods relying on behavioral data are limited by strict invariance assumptions. We introduce a scalable analytical framework using large language models (LLMs) to analyze verbal reports and identify articulated reasons for choice between monetary lotteries.
A validated LLM accurately identified predefined decision reasons in participants’ free-text reports, aligning with their actual choices in 95% of trials. Our analysis reveals that the reasons behind people’s decisions vary systematically and are driven more by the structure of the choice problem than by individual differences.
Crucially, reasons identified from verbal reports yield more parsimonious and informative representations of decision processes compared to those inferred from choices alone; furthermore, problem-specific reason profiles achieve out-of-sample prediction accuracy that is competitive with established computational models.
This work demonstrates that verbal reports are a rich data source and our analytical framework can unlock their potential, delivering results that challenge the field’s foundational invariance assumptions and pave the way for more context-sensitive and interpretable models of human decision making.
