Summary: A new study reveals that while human-AI collaboration can be powerful, it depends on the task. Analysis of hundreds of studies found that AI outperformed human-AI teams in decision-making tasks, while collaborative teams excelled in creative tasks like content generation.
This research suggests organizations may overestimate the benefits of human-AI synergy. Instead, strategic use of AI’s strengths in data processing and humans’ creativity may yield the best results.
These findings can help shape AI guidelines that enhance performance by maximizing complementary skills. Researchers argue that the future of work lies in nuanced collaboration rather than across-the-board AI adoption.
Key Facts:
- Human-AI teams performed best on creative tasks, like text and image generation.
- AI alone was more effective than human-AI teams for decision-making tasks.
- The study emphasizes targeted AI use, aligning human creativity with AI efficiency.
Source: MIT
The potential of human-AI collaboration has captured our imagination: a future where human creativity and AI’s analytical power combine to make critical decisions and solve complex problems. But new research from the MIT Center for Collective Intelligence (CCI) suggests this vision may be much more nuanced than we once thought.
Published today in Nature Human Behaviour, “When Combinations of Humans and AI Are Useful” is the first large-scale meta-analysis conducted to better understand when human-AI combinations are useful in task completion, and when they are not.
Surprisingly, the research has found that combining humans and AI to complete decision-making tasks often fell short; but human-AI teams showed much potential working in combination to perform creative tasks.
The research, conducted by MIT doctoral student and CCI affiliate Michelle Vaccaro, and MIT Sloan School of Management professors Abdullah Almaatouq and Thomas Malone, arrives at a time marked by both excitement and uncertainty about AI’s impact on the workforce.
Instead of focusing on job displacement predictions, Malone said that he and the team wanted to explore questions they believe deserve more attention: When do humans and AI work together most effectively? And how can organizations create guidelines and guardrails to ensure these partnerships succeed?
The researchers conducted a meta-analysis of 370 results on AI and human combinations in a variety of tasks from 106 different experiments published in relevant academic journals and conference proceedings between January 2020 and June 2023.
All the studies compared three different ways of performing tasks: a.) human-only systems b.) AI-only systems, and c.) human-AI collaborations. The overall goal of the meta-analysis was to understand the underlying trends revealed by the combination of the studies.
Test Outcomes
The researchers found that on average, human-AI teams performed better than humans working alone, but didn’t surpass the capabilities of AI systems operating on their own.
Importantly, they did not find “human-AI synergy,” which means that the average human-AI systems performed worse than the best of humans alone or AI alone on the performance metrics studied. This suggests that using either humans alone or AI systems alone would have been more effective than the human-AI collaborations studied.
“There’s a prevailing assumption that integrating AI into a process will always help performance — but we show that that isn’t true,” said Vaccaro. “In some cases, it’s beneficial to leave some tasks solely for humans, and some tasks solely for AI.”
The team also identified factors affecting how well humans and AI work together. For instance, for decision-making tasks like classifying deep fakes, forecasting demand, and diagnosing medical cases, human-AI teams often underperformed against AI alone.
However, for many creative tasks, such as summarizing social media posts, answering questions in a chat, or generating new content and imagery, these collaborations were often better than the best of humans or AI working independently.
“Even though AI in recent years has mostly been used to support decision-making by analyzing large amounts of data, some of the most promising opportunities for human-AI combinations now are in supporting the creation of new content, such as text, images, music, and video,” said Malone.
The team theorized that this advantage in creative endeavors stems from their dual nature: While these tasks require human talents like creativity, knowledge, and insight, they also involve repetitive work where AI excels.
Designing an image, for instance, requires both artistic inspiration — where humans excel — and detailed execution — where AI often shines. In a similar vein, writing and generating many kinds of text documents requires human knowledge and insight, but also involves routine, and automated processes such as filling in boilerplate text.
“There is a lot of potential in combining humans and AI, but we need to think more critically about it,” said Vaccaro. “The effectiveness is not necessarily about the baseline performance of either of them, but about how they work together and complement each other.”
Optimizing collaboration
The research team believes its findings provide guidance and lessons for organizations looking to bring AI into their workplaces more effectively. For starters, Vaccaro emphasized the importance of assessing whether humans and AI are truly outperforming either humans or AI working independently.
“Many organizations may be overestimating the effectiveness of their current systems,” she added. “They need to get a pulse on how well they’re working.
Next, they need to evaluate where AI can help workers. The study indicates that AI can be particularly helpful in creative tasks, so organizations should explore what kinds of creative work could be ripe for the insertion of AI.
Finally, organizations need to set clear guidelines and establish robust guardrails for AI usage. They might, for example, devise processes that leverage complementary strengths.
“Let AI handle the background research, pattern recognition, predictions, and data analysis, while harnessing human skills to spot nuances and apply contextual understanding,” Malone suggested. In other words: “Let humans do what they do best.”
Malone concluded: “As we continue to explore the potential of these collaborations, it’s clear that the future lies not just in replacing humans with AI, but also in finding innovative ways for them to work together effectively.”
About this AI research news
Author: Casey Bayer
Source: MIT
Contact: Casey Bayer – MIT
Image: The image is credited to Neuroscience News
Original Research: Open access.
“When Combinations of Humans and AI Are Useful” by Michelle Vaccaro et al. Nature Human Behavior
Abstract
When Combinations of Humans and AI Are Useful
Inspired by the increasing use of artificial intelligence (AI) to augment humans, researchers have studied human–AI systems involving different tasks, systems and populations.
Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone.
Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes.
We searched an interdisciplinary set of databases (the Association for Computing Machinery Digital Library, the Web of Science and the Association for Information Systems eLibrary) for studies published between 1 January 2020 and 30 June 2023.
Each study was required to include an original human-participants experiment that evaluated the performance of humans alone, AI alone and human–AI combinations.
First, we found that, on average, human–AI combinations performed significantly worse than the best of humans or AI alone (Hedges’ g = −0.23; 95% confidence interval, −0.39 to −0.07).
Second, we found performance losses in tasks that involved making decisions and significantly greater gains in tasks that involved creating content.
Finally, when humans outperformed AI alone, we found performance gains in the combination, but when AI outperformed humans alone, we found losses.
Limitations of the evidence assessed here include possible publication bias and variations in the study designs analysed.
Overall, these findings highlight the heterogeneity of the effects of human–AI collaboration and point to promising avenues for improving human–AI systems.
