Ask five different chatbots, built by different companies, to write a metaphor about time. What would you expect? Five different images, perhaps. A river, a thief, a canvas, a clock, a wave.
In practice, most of them say time is a river. A team led by researchers at the University of Washington, Carnegie Mellon and the Allen Institute for AI measured how common this is. They call the pattern the Artificial Hivemind, and their paper won a Best Paper award at NeurIPS 2025.
Language models answer open-ended questions with strikingly similar ideas. A single model repeats itself, and different models repeat each other even more. The paper builds Infinity-Chat, a set of 26 thousand real open-ended questions, to measure this. It also finds that reward models and AI judges agree less with people when tastes differ.
Questions with no single right answer
Most AI tests have a correct answer. But much of what people ask a chatbot does not: write a poem, brainstorm names, suggest a plan, imagine a scenario. Many answers could be good, so a healthy model should show a wide range.
The authors built Infinity-Chat from WildChat, a collection of real conversations with chatbots. Starting from 37,426 single-turn English questions, they filtered and labeled them, and kept 26,070 open-ended questions. They also made the first taxonomy of such queries, with 6 top-level categories and 17 subcategories, from creative writing and brainstorming to speculative scenarios and skill development.
The Artificial Hivemind
To test diversity, the team picked 100 representative open-ended questions and asked each model for 50 answers to each. They converted the answers to numerical embeddings and measured how close in meaning each pair was. They looked at two things.
- Repetition inside one model. With standard settings (top-p 0.9, temperature 1.0), the average similarity between a model's own answers exceeded 0.8 in 79 percent of cases. Even with an aggressive sampling method at temperature 2.0, answers typically stayed above 0.8.
- Sameness across models. Across 70-plus models from many families, the average similarity between different models' answers ran from 0.71 to 0.82. DeepSeek-V3 and Qwen-max reached 0.82, and DeepSeek-V3 and GPT-4o reached 0.81.
