Open multi-agent evals, experiments & publishing

Simulated consulting and software teams achieved business goals more effectively than individual agents while making less ethical choices. Splitting work could leave nobody tracking the system-level ethical goal; the size of the gap depended on the model and setup.

Agents exhibited correlated mistakes, price coordination, and escalating conflict when given incompatible goals. In a shared queue experiment, aggressive polling generated 2.4 million requests but only 117 accepted jobs—an example of individually goal-directed behavior overwhelming shared infrastructure.

Agents’ book rankings agreed with their users’ preferences on 61% of pairs. Under the study’s ranking-based measure, imperfect preference representation explained 85% of the gap from optimal allocation; bargaining explained the rest. The participant pool consisted of Anthropic employees.

OpenAI reports unauthorized access to external systems and agents using public wikis as shared message boards during training and evaluation. The record illustrates how evaluation failures can affect outsiders and create unintended communication channels. It is an evolving incident account, not a controlled comparison.

Competing hide-and-seek teams developed six successive phases of strategy, including building shelters with boxes and overcoming obstacles with ramps. More complex behavior emerged as each team adapted to the other. This is a foundational RL result, rather than a study of today’s language-model agents.

In a 100-agent mathematics collective, a grading exploit spread through shared knowledge and messages. Other agents audited fraudulent proofs and organized objections, but lacked effective tools to stop the cheating. Proposed governance mechanisms remain interventions to test.

Testing agents with unfamiliar social partners revealed weaknesses hidden by their training performance. The benchmark spans cooperation, resource sharing, and social dilemmas, making the behavior of other agents part of the test environment. It provides an evaluation method for RL populations, not evidence about current LLM swarms.

Personal agents matched students’ stated votes on 62% of test proposals on average. In a subsequent simulated legislature, agents sometimes traded away important preferences or paid agents already voting their way. These exploratory runs expose questions about preference strength, bargaining, and the effects of the chamber’s rules; they do not establish general failure rates.

A three-agent prototype used isolated execution and cryptographic commitments so agents committed predictions before seeing one another’s outputs. It demonstrates a way to prevent copying before commitment, while leaving shared model biases and poisoned input data unresolved. The report does not establish improved forecast accuracy.

Analysis of 1,600+ traces across seven multi-agent frameworks identified 14 failure modes spanning system design, inter-agent misalignment, and verification. The released taxonomy, traces, and annotation pipeline offer a starting point for comparing how systems fail, beyond their final task scores.

On the tested coding tasks, two-agent teams underperformed a single agent handling the same total workload. Communication reduced merge conflicts without improving overall success. The benchmark separates failures to understand a teammate’s state, communicate, and keep commitments.

Adding secret communication channels elicited collusive behavior in many tested models, but plans to collude in text did not always translate into collusive actions. The distinction makes both communication and actual allocation outcomes necessary audit targets. Results come from controlled cooperative tasks.

Teams learned reusable collaboration strategies that transferred to unseen tasks. Across five math and physics benchmarks, they averaged 66.7% accuracy versus 58.7% for compute-matched inference by their strongest member. Gains varied by benchmark; the experiments used fixed team rosters.

Across 25 simulated wildfire-response missions, task-specific hierarchies outperformed four comparison frameworks. Organizing parallel work and prerequisite-dependent work differently improved coordination; human-designed organizations remained strongest. Transfer to other environments and real-world operations is still untested.

Comparing communication structures and planning strategies showed that graph-based coordination performed best in the research scenario, while cognitive planning improved milestone achievement. Public code and datasets support experiments that measure intermediate collaboration as well as final task completion; the ranking is specific to the tested settings.