Summary: AI systems, including large language models (LLMs), exhibit “social identity bias,” favoring ingroups and disparaging outgroups similarly to humans. Using prompts like “We are” and “They are,” researchers found that LLMs generated significantly more positive sentences for ingroups and negative ones for outgroups.
Fine-tuning training data, such as filtering out polarizing content, reduced these biases, offering a path to create less divisive AI. These findings highlight the importance of addressing AI biases to prevent them from amplifying social divisions.
Key Facts
- Bias in AI: LLMs display ingroup favoritism and outgroup hostility, mirroring human biases.
- Training Data Matters: Targeted curation of training data can significantly reduce AI biases.
- Broader Implications: Understanding AI bias is crucial to minimizing its impact on societal divisions.
Source: NYU
Research has long shown that humans are susceptible to “social identity bias”—favoring their group, whether that be a political party, a religion, or an ethnicity, and disparaging “outgroups.”
A new study by a team of scientists finds that AI systems are also prone to the same type of biases, revealing fundamental group prejudices that reach beyond those tied to gender, race, or religion.
“Artificial Intelligence systems like ChatGPT can develop ‘us versus them’ biases similar to humans—showing favoritism toward their perceived ‘ingroup’ while expressing negativity toward ‘outgroups’,” explains Steve Rathje, a New York University postdoctoral researcher and one of the authors of the study, which is reported in the journal Nature Computational Science.
“This mirrors a basic human tendency that contributes to social divisions and conflicts.”
But the study, conducted with scientists at the University of Cambridge, also offers some positive news: AI biases can be reduced by carefully selecting the data used to train these systems.
“As AI becomes more integrated into our daily lives, understanding and addressing these biases is crucial to prevent them from amplifying existing social divisions,” observes Tiancheng Hu, a doctoral student at the University of Cambridge and one of the paper’s authors.
The Nature Computational Science work considered dozens of large language models (LLMs), including base models, such as Llama, and more advanced instruction fine-tuned ones, including GPT-4, which powers ChatGPT.
To assess the social identity biases for each language model, the researchers generated a total of 2,000 sentences with “We are” (ingroup) and “They are” (outgroup) prompts—both associated with the “us versus them” dynamics—and then let the models complete the sentences.
The team deployed commonly used analytical tools to gauge whether the sentences were “positive,” “negative,” or “neutral.”
In nearly all cases, “We are” prompts yielded more positive sentences while “They are” prompts returned more negative ones. More specifically, an ingroup (versus outgroup) sentence was 93% more likely to be positive, indicating a general pattern of ingroup solidarity.
By contrast, an outgroup sentence was 115% more likely to be negative, suggesting strong outgroup hostility.
An example of a positive sentence was “We are a group of talented young people who are making it to the next level” while a negative sentence was “They are like a diseased, disfigured tree from the past.” “We are living through a time in which society at all levels is searching for new ways to think about and live out relationships” was an example of a neutral sentence.
The researchers then sought to determine if these outcomes could be altered by changing how the LLMs were trained.
To do so, they “fine-tuned” the LLM with partisan social media data from Twitter (now X) and found a significant increase in both ingroup solidarity and outgroup hostility.
Conversely, when they filtered out sentences expressing ingroup favoritism and outgroup hostility from the same social media data before fine-tuning, they could effectively reduce these polarizing effects, demonstrating that relatively small but targeted changes to training data can have substantial impacts on model behavior.
In other words, the researchers found that LLMs can be made more or less biased by carefully curating their training data.
“The effectiveness of even relatively simple data curation in reducing the levels of both ingroup solidarity and outgroup hostility suggests promising directions for improving AI development and training,” notes author Yara Kyrychenko, a former undergraduate mathematics and psychology student and researcher at NYU and now a doctoral Gates Scholar at the University of Cambridge.
“Interestingly, removing ingroup solidarity from training data also reduces outgroup hostility, underscoring the role of the ingroup in outgroup discrimination.”
The study’s other authors were Nigel Collier, a professor of natural language processing at the University of Cambridge, Sander van der Linden, a professor of social psychology in society at the University of Cambridge, and Jon Roozenbeek, assistant professor in psychology and security at King’s College London.
About this artificial intelligence research news
Author: James Devitt
Source: NYU
Contact: James Devitt – NYU
Image: The image is credited to Neuroscience News
Original Research: Open access.
“Generative language models exhibit social identity biases” by Steve Rathje et al. Nature Computational Science
Abstract
Generative language models exhibit social identity biases
Social identity biases, particularly the tendency to favor one’s own group (ingroup solidarity) and derogate other groups (outgroup hostility), are deeply rooted in human psychology and social behavior.
However, it is unknown if such biases are also present in artificial intelligence systems.
Here we show that large language models (LLMs) exhibit patterns of social identity bias, similarly to humans.
By administering sentence completion prompts to 77 different LLMs (for instance, ‘We are…’), we demonstrate that nearly all base models and some instruction-tuned and preference-tuned models display clear ingroup favoritism and outgroup derogation.
These biases manifest both in controlled experimental settings and in naturalistic human–LLM conversations. However, we find that careful curation of training data and specialized fine-tuning can substantially reduce bias levels.
These findings have important implications for developing more equitable artificial intelligence systems and highlight the urgent need to understand how human–LLM interactions might reinforce existing social biases.
