Skip to content
Artificial Hivemind: The Open-Ended Homogeneity of Language Models 3 min

Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, Yejin Choi · NeurIPS Best Paper 2025

Artificial Hivemind: The Open-Ended Homogeneity of Language Models

Ask many chatbots for a metaphor and most say the same one. A NeurIPS 2025 best paper measures it. Sort the real answers and see the hive.

Read the original paper

AI & ML · White Belt · Pro · 3 min read

Artificial Hivemind: The Open-Ended Homogeneity of Language Models

Ask five different chatbots, built by different companies, to write a metaphor about time. What would you expect? Five different images, perhaps. A river, a thief, a canvas, a clock, a wave.

In practice, most of them say time is a river. A team led by researchers at the University of Washington, Carnegie Mellon and the Allen Institute for AI measured how common this is. They call the pattern the Artificial Hivemind, and their paper won a Best Paper award at NeurIPS 2025.

Language models answer open-ended questions with strikingly similar ideas. A single model repeats itself, and different models repeat each other even more. The paper builds Infinity-Chat, a set of 26 thousand real open-ended questions, to measure this. It also finds that reward models and AI judges agree less with people when tastes differ.

Questions with no single right answer

Most AI tests have a correct answer. But much of what people ask a chatbot does not: write a poem, brainstorm names, suggest a plan, imagine a scenario. Many answers could be good, so a healthy model should show a wide range.

The authors built Infinity-Chat from WildChat, a collection of real conversations with chatbots. Starting from 37,426 single-turn English questions, they filtered and labeled them, and kept 26,070 open-ended questions. They also made the first taxonomy of such queries, with 6 top-level categories and 17 subcategories, from creative writing and brainstorming to speculative scenarios and skill development.

The Artificial Hivemind

To test diversity, the team picked 100 representative open-ended questions and asked each model for 50 answers to each. They converted the answers to numerical embeddings and measured how close in meaning each pair was. They looked at two things.

  • Repetition inside one model. With standard settings (top-p 0.9, temperature 1.0), the average similarity between a model's own answers exceeded 0.8 in 79 percent of cases. Even with an aggressive sampling method at temperature 2.0, answers typically stayed above 0.8.
  • Sameness across models. Across 70-plus models from many families, the average similarity between different models' answers ran from 0.71 to 0.82. DeepSeek-V3 and Qwen-max reached 0.82, and DeepSeek-V3 and GPT-4o reached 0.81.
Numbers from the paper, for 100 open-ended questions. The blue bar marks where one model's repeated answers usually sit. The purple bar is the range of average similarity between different models.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Why it matters
  • Judges that expect one right taste

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.