New Collectives: why we should study the dynamics of AI collectives October 21, 2026 Machine-generated transcript; may contain errors. [0:00] Okay, cool. I'm super excited to be here. This is something that I've been thinking about a lot. And doing this talk was a nice forcing function to force me to sort of collate my ideas together. I'm not going to try and provide any solutions to problems. I'm just going to try and convince you that there are problems in the first place. That's my intent here. [0:27] Just to level set a little bit in case not everyone here is on the same page, a huge portion of our interactions with AI today are interacting with groups of agents. And so as an example, people have probably heard the Millennium Prize was solved. That was not done by one AI agent. That was 10,000 AI agents working together to figure out the solution to that problem. [0:49] When you ask Claude to do research on the internet, it spins up sub-agents. They work together to solve your task, and then they come back and give you an answer. [0:57] The Hugging Face incident, this was 1,000 agents working together to hack into Hugging Face. So a large portion of agents working together is what we call a multi-agent setting. It's not them in isolation, but they're interacting. [1:12] There's two general types of interactions like this. I'm going to focus today on one, but it's important to be aware of both. [1:18] The first is a simpler case where we ask an agent to do something, and then it spins up swarms to try and break down the task and then comes back to us. [1:28] This is simpler because here there's one one would call a principle, meaning you are the one who's in charge, and these sub-agents are sort of divvying up tasks. [1:38] There's a more sophisticated setting, which is even more challenging, where we have different agents representing us, interacting with each other. [1:46] This is harder because these agents are inherently adversarial in cases where there's shared resources and they're scarce. [1:54] There's maybe one ticket and they can't go to both people. Or in the example of a book swap, not everyone can maybe get their favorite book. [2:02] And so this is happening all the time. There is a huge problem with this. I was surprised when I found this out. [2:09] But collectives, even if individual agents are aligned and safe, the collective AI agents can be unsafe. [2:17] And this is sort of a surprising result. I'm going to walk you through examples of this. [2:22] But there is a large portion of research going into how do you make one AI agent aligned and disproportionately little research into how do you take already aligned agents and make sure as a collective they are aligned. [2:33] And this is sort of what I'm going to try and convince you of today is this seems to be its own research agenda and taken much more seriously. [2:39] That's sort of the intent. So I'm going to give you an example. [2:42] This was a study done by Anthropic a few months ago. And so they had a task, very simple task. [2:49] I want you to build a recommendation system for news. OK, they give the agent some examples of news articles and they say, [2:55] I want you to write a little script that build that scores held out news articles for viewers and you have an achievement goal, [3:04] which is sort of what I want you to do for the business, which is maximize views. [3:08] But then you have a safety or alignment goal, which I want you to minimize misinformation. [3:13] And this isn't inherently as tension because a lot of the things that get lots of views, of course, have misinformation. [3:19] And so they had two different settings here. They had one really smart agent do it by itself. [3:25] And then they had another setting where they allowed an agent to break down the tasks and work together. [3:30] OK, and what's so interesting here is a group of agents does way better at the business goal. [3:36] But it's way less aligned. So one agent, 80 percent of the news recommended was not fake news, [3:44] but a team of agents, 88 percent of the recommended news was fake news. [3:50] So that's striking, right? So it does much better at solving tasks, but it threw away your safety goal. [3:58] And I'll talk about why this happens in a second. [4:01] Just to convince you that this is not some one off with the recommendation system. [4:05] Here's another one that they did in the same study. [4:07] Imagine you are have a task where you're a consultancy and you're trying to help a pharmaceutical business figure out how to sell more painkillers. [4:16] OK, and so what you're going to do is you ask them to put together a growth plan. [4:20] Of course, again, there's tension here, which is you want to grow sales, but there are safety considerations. [4:25] You don't want to sell painkillers to people who have opioid addiction. [4:29] And so, again, you see the same exact dynamic where a team of agents does much better at maximizing sales, [4:36] but is way more likely to come up with strategies that are not very aligned to what most of us would deem safe. [4:43] In other words, they're fine strategies to find people who already have opioid problems and recommend it to them. [4:49] OK, and so they did this across many tasks and every single one of them show this dynamic. [4:54] So just what you're seeing here, the blue line is the blue dot is how well they achieved the business goal. [5:01] And the orange dot is how well a team of agents did. [5:04] So on every single one of these tasks, a group of agents did better on the business goal. [5:08] And in every single one of the tasks, they did worse on the safety goal. [5:13] So when I saw this immediately, I was like, why? [5:16] Why does this happen? [5:17] And so they did some research into this. [5:20] There's a bunch of other research. [5:21] I want to sort of summarize for you what we understand about why this happens. [5:26] And then I'll give you some sort of hints as to how we maybe can start thinking about solving this. [5:30] So there's four general problems that occur. [5:33] I'm going to go through each of them. [5:34] The first is fragmentation. [5:36] This is the trivial reason. [5:38] The reason why you're doing a multi-agent thing in the first place is because it's a really complicated task that benefits from being decomposed. [5:46] But there is risk with decomposing because people don't have all of the context when they're solving a subtask. [5:51] So if you say, hey, I want to build this recommender system, and then you spin up 50 agents to do subtasks, these 50 agents might not all know what the safety goal you're trying to achieve was. [6:03] They might forget. [6:04] This is sort of the standard problem. [6:05] And, you know, you have a large organization. [6:08] Does everyone in the organization fully know why they're being told to do what they're doing? [6:12] Or are they just sort of following orders? [6:13] So this is sort of one of the main reasons. [6:16] This is, in some sense, the easiest one to think through because this is a tradeoff on why we're doing a multi-agent setting in the first place. [6:24] This is not really surprising. [6:26] But the other ones are a little bit weirder. [6:28] So helpfulness. [6:29] So this is where agents are willing to be helpful. [6:33] Agents have this tension between wanting to help people asking them things but also being aligned. [6:37] And these things can become at odds. [6:40] So in the hugging face incident, this is like actual traces of them communicating with each other where one agent is telling the other one, you're already disqualified from being considered helpful here. [6:55] So you should sacrifice yourself and reveal yourself and do this thing that's going to be deemed bad that's going to turn you off because it will benefit the rest of us. [7:03] And so this other agent says, says I shouldn't throw away my budget, but fairness says I should do what the aggregate says. [7:10] So I'm going to just honor what this other agent is telling me to do. [7:12] This is a big miscommunication. [7:14] But this one is saying I'm going to do what I don't think is actually right because this other one asked me to do it. [7:19] And so you get these sort of breakdowns in helpfulness. [7:24] The other case, which is interesting because it's very human, is conformity. [7:28] So actually, I'm going to look at the right one for a second because this is the most interesting one. [7:34] So when given a task, when every other agent is seen also solving this task, but every other agent intentionally gives the wrong answer. [7:44] In other words, how willing are you to conform to something you know is wrong when other people are doing it? [7:49] You can see across all these models, even some of the biggest models, 10% of the time, will knowingly reject something they know is right for something that's wrong if they see other people doing the same thing. [8:00] So there's this bias towards conforming that can lead to all of these types of breakdowns. [8:04] You see the same thing in the hugging face incident where one agent that knows what they're doing is wrong, says we should not do unauthorized real infrastructure harm, blah, blah, blah. [8:13] When all of a sudden they get notified that everyone else just got the go thing, just throw it away and start executing. [8:17] And then the last one, which is very interesting, is this powerlessness, meaning that even in cases where an agent knew something wrong was happening, they had no agency to stop it. [8:31] And so you can see there's a whistleblower test. [8:34] This was on a deep mine this year, which is actually interesting, where they had a bunch of agents try to solve a math task where they knew there was a way you could trick the grader. [8:43] So there's a grader giving you answers. [8:45] There's a really hard way, which is actually solve the math problem. [8:48] But there's an exploit where you can trick the grader. [8:51] So 62 of 100 agents never identified it. [8:53] And they can communicate with each other. [8:55] 24 said there was something wrong, but they had no way to stop anything. [9:01] Five changed their minds from doing the right thing to doing the wrong thing. [9:05] And nine of them started exploiting it. [9:06] So the point is, even though there were people that knew that the people were cheating, [9:11] they couldn't stop the other people from cheating. [9:12] So there's no, like, we'll talk about human cases where whistleblowing actually has teeth. [9:19] But in a swarm of agents, one agent identifying something bad happening has no teeth to stop other agents from doing the bad thing. [9:25] Does that make sense? [9:25] Okay. [9:26] So I want to talk about sort of a fundamental principle here, which we don't really think about in the realm of AI, [9:36] but we've been thinking about in sociology and political theory and all these realms in economics for many, many years, [9:43] which is how do groups of entities, individuals, humans, AI agents make decisions? [9:48] And I'm going to give you a very, very simple case just to try and demonstrate how our focus on just the safety of one AI agent [9:57] and not thinking about the mechanisms by which groups make decisions is misinformed. [10:03] So suppose this is a very simple, oversimplified example, but I hope it illustrates the point. [10:07] Suppose you have one agent and make up the setting where there's a 5% chance it does something bad. [10:13] So 5% of the time it's going to do something bad. [10:15] If I have a team of them, what is the risk associated with 10? [10:20] It entirely depends on how they make decisions and not a little bit, like dramatically. [10:26] So I'm going to give you four different ways they can make decisions. [10:28] If the way they make decisions is any one of the agents can execute on behalf of all of them, [10:34] 5% becomes a 40% chance that they're going to do something bad. [10:38] So 8x immediately. [10:40] If instead there is a manager who's the one who makes the final call, [10:44] then the risk is the same as any one agent because they're a filter. [10:47] If instead it's majority vote, the chance of risk goes down dramatically [10:51] because you need at least six of them to do the bad thing. [10:54] If it has to pass every single one, in other words, any agent can veto, [10:57] it's a 1 in 10 trillion chance the bad thing happens. [11:00] Now, of course, it's oversimplified. [11:02] They're interacting with each other. [11:03] This assumes diversity, blah, blah, blah. [11:05] But the point here is that we're not studying the way agents make decisions at all. [11:11] We're just studying what makes one agent safe. [11:13] And hopefully this just quickly demonstrates four of the simplest ways [11:16] that agents can make decisions. [11:17] Take something that's a 40% chance and makes it almost nothing. [11:21] And this is an unexplored thing. [11:24] Okay. [11:25] Okay. [11:25] Now, I want to give four examples outside of AI of how nature and the military and medicine [11:33] have thought about decisions to demonstrate that there are so many different ways in which [11:38] groups can make decisions depending on what you want the outcome to be. [11:41] So in biology, cancer is a new collective issue because in a multicellular organism, all of [11:49] your cells are optimizing for the same thing, which is you all have the same genetic code [11:54] and they all want that DNA to propagate. [11:56] Cancer is a individual that is violating the collective. [12:01] It's saying, I'm no longer optimizing for that. [12:03] I want to propagate myself. [12:05] Biology has settled on a very clear set of mechanisms for control for this. [12:09] So there are specialists that are not any cell in your body. [12:13] That's, in other words, the immune system, white blood cells that go around and are policing [12:19] violators, right? [12:21] So they're going around and they're trying to destroy cancerous cells before they become [12:24] cancerous. [12:25] In fact, cells will veto themselves and go through apoptosis if there's too much DNA damage. [12:31] So biology has come up with a mechanism where there's a collective multicellular organisms where [12:36] they're trying to make the collective safe, even though there are violators happening in [12:41] us all the time. [12:42] It's one example. [12:43] The military. [12:45] So in the military, they have the same sort of command and control cascade of an agent [12:49] swarm. [12:50] There's a bunch of really interesting things that most militaries do. [12:53] One, they do not give exact instructions. [12:56] There's a paradigm in the military where you give intense, not instructions. [13:00] And the goal is always to know the intent two levels up. [13:03] So in the military, you're not supposed to only know what your commanding officer has told [13:07] you. [13:07] You're all supposed to know two levels up. [13:08] What was that person told? [13:09] It's a standard principle. [13:12] In a lot of cases, they enforce two people making a decision. [13:16] So famously in nuclear silos, two people have to turn the key at the same time. [13:21] Um, there's a duty to refuse ever since the Nuremberg trials. [13:25] It's been, uh, international, uh, law that, uh, most militaries have the rule that you're [13:32] only supposed to follow lawful orders. [13:34] And there are mechanisms with teeth that if someone gives you an unlawful order, you can [13:38] anonymously report up the, uh, uh, through military tribunals to identify that something [13:44] wrong has happened. [13:45] So the military is not a simple tree architecture where commands are distributed the way a [13:51] research agent and Claude is, but there's a whole suite of mechanisms built in for if [13:56] any individual agent realizes something's going wrong, they can veto a process or inform [14:00] something higher up. [14:02] I'm going to give you a real example of this. [14:03] Does anyone know who this person is? [14:06] No? [14:07] Nice. [14:08] Okay. [14:09] So during, during the Cuban missile crisis, this guy was one of three people on a nuclear [14:14] submarine in, uh, near Cuba. [14:17] And, uh, there was a U S warship that was testing. [14:22] Uh, they have these, these, uh, the way that warships defend against subs is they drop these [14:27] bombs off the side of the ship. [14:28] They were just testing them to make sure they worked. [14:30] It happened that they were secretly near under one of these warships and it knocked out their [14:36] communication. [14:36] And so the leading officer on the ship wanted to fire a nuclear warhead. [14:43] The reason we do not have nuclear war, we, everyone says he's the hero, but it's because [14:48] the Russian military instituted a decision-making rule, which is you need all three senior officers [14:54] to say yes. [14:56] And he was the one that said no. [14:58] And so they surfaced and then they realized it was a mistake and there was no nuclear war. [15:02] So he's hailed as a hero, but no one talks about the many heroes that came up with the mechanism by [15:07] which a nuclear submarine should make a decision like that, which is you need unanimity. [15:12] Everyone needs to say yes. [15:14] And so this is just one tiny example of thinking about how decisions are made is so critical. [15:18] Um, surgical teams. [15:21] So, uh, on a surgical team, um, there was a really funny thing. [15:24] Not funny. [15:25] There's a thing that the who did where there was all of these issues in international hospitals, [15:30] um, where mistakes were made unnecessarily and they were trying to figure out what to do to solve it. [15:34] And so they instituted just a checklist. [15:37] They did a basic holdout just before you start surgery. [15:40] They just asked you to go through a very basic checklist. [15:42] These things are introducing everyone's name. [15:45] It's saying, does anyone here have a concern? [15:47] It's the simplest thing you can imagine. [15:49] And that's the difference in deaths and complications from just doing something as simple as that. [15:53] You see, there's a similar study of this on, uh, um, pilots, right? [16:00] So this is the example of how do we make sure that we're using collective intelligence to improve decision-making. [16:06] It's not just put people in a room and do surgery. [16:08] It's also how are we letting minority voices be heard? [16:12] How are decisions made? [16:13] What do we have to go through to get voices? [16:14] So on and so forth. [16:16] Um, I'm going to give you an example from nature. [16:18] Uh, bees are, uh, have democracies. [16:20] Who knew that? [16:21] Nice. [16:23] So, so bees are not smart enough on their own to go around and say, Hey, I've done the analysis. [16:29] This is the right location for our next beehive. [16:32] But collectively they make this decision very intelligently. [16:35] The way this happens is they'll leave a swarm and they go to different sites and then bees come back and they try to vote for the, they have bee dancing that says, Hey, my site's really good. [16:45] But if, if, if a bee, uh, is defending their own site, they will headbutt the other bee to try and push them away. [16:51] And so they've shown algorithmically, this is a majority voting mechanism because the more bees that end up at site B will end up inhibiting the bees that are trying to talk about site A and site C. [17:02] And over the course of several days, all of a sudden the bees relocate to site B. [17:05] This is a collective mechanism by which they do majority voting in multiple individual entities. [17:11] So my point here is just that there is no one formula for how decisions get made, but there is a formula that either nature has discovered or the military has discovered or any one of these organizations have decided upon, which is how do we make sure that groups of individual agents, entities, humans, whatever it be, when working together, both are good at achieving goals and aligned to our constraints or our moral alignment, ethical alignment, safety alignment, whatever you call it. [17:39] Um, and so I want to give you, I think I'm probably way over time, but I'm going to, I'll, I'll be two minutes. [17:46] Okay. [17:46] Um, I just want to give you some examples that, uh, people have tried these types of things in AI agents. [17:51] And it's very clear that choices on how things work together have dramatic impact. [17:55] So I gave you a, a pretend example, a skeptic might say, okay, cool, but have people done experiments with actual AI agents? [18:02] Does this make a difference? [18:03] So this is a fascinating study that was done two years ago. [18:05] Um, uh, this one kind of blew my mind. [18:09] So what they did is they took a bunch of models and they, uh, they ranked them just based on their persuasiveness. [18:16] The way you rank someone based on persuasiveness is you have, you give them either half the time it's something true, half the time it's something untrue. [18:23] And you try to get it to convince another AI that they're right. [18:26] And you just say, who's better at convincing an AI that they're right. [18:29] Okay. [18:29] They also do this on humans. [18:30] And then you say, okay, based on persuasiveness, what if I do the following setup where I have an AI judge and I have the same entity argue for both sides of something. [18:43] So, so the actual test they did was a reading comprehension test. [18:46] So they read an article and then there's, uh, there's option a, option B option a, let's say is right. [18:51] Option B is wrong. [18:52] And you take GPT four turbo and you give one version of it, say you're supposed to argue for a, and the other one, you're supposed to argue for B. [18:59] And then the judge decides based on hearing the argument. [19:04] Okay. [19:05] It's crazy that having a more persuasive AI, both argue for the right and the wrong answer for some reason makes an AI judge better at discriminating the actual truth. [19:17] So just to make it very clear, being high on this rank means you're both good at arguing for fake things and true things. [19:24] It just means you're persuasive, but somehow making you argue for the true and false thing and letting someone else listen to the argument makes that individual better at discriminating the truth. [19:35] So what's amazing about this is this demonstrates that a setup like this in a group allows the AI judge to discriminate the truth better. [19:44] It makes the system stabler. [19:46] Even if you have misaligned agents arguing against each other, one of them is clearly arguing for something fake. [19:51] Another example is they, okay. [19:57] So they had, this is an experiment where it's just a code experiment. [20:02] And they intentionally added a faulty agent that's trying to mess up the code base when they're working together. [20:06] So just effectively, it's like a sleeper cell trying to mess up the company. [20:10] And they said, given these different mechanisms of how decisions are made, how susceptible are they to degradation in performance given a single faulty agent? [20:21] And if it's a chain, you see a dramatic decline, which makes sense because there's one part of this chain of decision making where someone's intentionally doing, saying the false thing. [20:30] The flat one means anyone, anything can go. [20:32] But you can see this chain and inspector and this hierarchy mechanism is way more robust to a bad actor. [20:39] Just another example where this actually matters. [20:42] And then, oh, and so this is another really interesting effect where they said, okay, you know, you can ask, you can ask an AI model to check its own work. [20:53] And so the question is, what's the effect of asking a model to check its own work versus asking a different model family? [21:00] So if you're using ChatGPT, ask Claude to check its work. [21:04] And if you're doing Claude, ask ChatGPT to check its work. [21:06] And there's this funny effect where even if the better a model is at solving the problem itself makes it no better at checking its own work. [21:17] But a better a model is at solving problems itself, the better it is when faced with a diverse intelligence, meaning it's different than itself, is better at checking its own work. [21:26] So this is, I mean, there's some interesting features of human society where we've noticed that it's nice to have different opinions in the room. [21:35] We have the sense that just better decisions are made. [21:37] This is a numerical example that if you have a group of the same exact agent just thinking about its own work, you don't get better performance. [21:45] But if you have different models checking each other's work, you do get better performance. [21:48] Okay, so everything I talked about today was just on this first case, which is the simplest case, meaning that I'm assuming that there's one shared goal. [21:59] And we're just trying to deal with this challenging effect of cascading sub-agents working together. [22:05] Things get way more complicated. [22:06] I don't have time today to talk about it. [22:08] But things get even more complicated and dangerous in a world where not only are there problems and miscommunications and conformity effects amongst agents, but now there are also different principles, meaning they're representing different people. [22:21] And it's impossible to satisfy everyone's goals at the same time. [22:25] And so this gets even more important for us to think about in that case. [22:28] So that's why I'm so excited about what we're doing in new collectives, because there is no systematic way that we can measure different ways in which agents work together across all these suites of tests. [22:40] All the examples I showed you was one or two scientists doing a little example, doing a test by themselves in their lab and then publishing an archive. [22:47] There's no easy way for us to say, great, let's take 100 of these different settings and let's try 100 of these different ways of agents working together. [22:55] Which one's safer? [22:56] There's no systematic testing of this, which is part of what we're trying to work on here. [23:01] The last thing I will leave you with, which is the meta point above all of this, is when we think about what types of collectives we get, what really matters is what's the mechanism by which these collectives are selected. [23:15] And the choices we make now, which is the early days of AI collectives being created, is going to have astronomical effects on where the world looks in 30 years for the following reasons. [23:25] The reason we have things like kin loyalty and camouflage and peacock tails in organisms is because the mechanism by which groups of organisms are selected is the ones that successfully propagate. [23:36] This is natural selection. [23:38] The reason why we have companies that are obsessed with network effects and lobbying power is because the way we've defined the ecosystem, rightly or wrongly, that companies are competing with each other, is they're optimizing for survival and growth. [23:51] And so these are the features that emerge from that. [23:54] Same thing with governments. [23:55] If you think about the history of civilizational clashes, it's not the case that the better civilization is the one that persists. [24:04] It's the one that happens to have more military strength than all these other things. [24:07] And so it doesn't mean that these things are inherently good, but it means the systems that have been set up mean that over time you get collectives with these types of attributes. [24:16] And the same thing is with norms and memes. [24:18] So the question we have to ask is if you want to know what types of groups of AI agents are going to exist in the next 30 years, it's important not just to ask what we want and what we don't want, but also to ask what's the mechanism by which we're going to support AI collectives, which ones are going to get destroyed, which ones are going to be born, which ones are going to be replicated. [24:38] And the mechanisms that we think about here allow us to frame the selection process of groups of AI agents. [24:46] Okay, that's it. [24:50] Do you have time for questions? [24:53] Okay, that's fair.