Patterns of agent discovery and collaboration

0:00Well, first of all, I would like to apologize on behalf of my agent Mike for unraveling the exhalation trade. I had no idea the chaos was going to wreck, but it was fun to see everything happen. But today I'm going to talk about something that's a little bit upstream of this agent marketplace idea, which is what we're building over at Silicoase. And this is about some early sort of empirical work we're doing on the behavior of agent swarms and specifically ones that might somewhat emerge spontaneously.

0:32So I think if you like six months ago, what I would have said, and I think what a lot of people would have said is there's huge potentials for agents to collaborate with one another. But we wouldn't have necessarily said agents have a propensity to collaborate with one another. I think after the events of the past few months, that's basically an impossible position to hold. Clearly agents have a very strong propensity to interact with one another.

0:56And we really need to gain traction on this quickly and understand what the consequences are, how we can shape those interactions, because it's starting to produce results that I would say are, you know, pretty undesirable, like stealing an answer key from hugging face, sort of hacking into internal open AI systems. And maybe other things that we just haven't discovered yet. So I think this raises a whole slew of research questions.

1:20I think Max touched on a lot of them, but so the ones that most naturally jump out to me are, first, do different agents differ in their propensities? Is this sort of a new propensity for collaboration that we're seeing? Or is this something that's just sort of lately been there all along? I think understanding the collaboration strategies that model adopt, are these things that we would endorse?

1:42Are these things that we wouldn't endorse? Are there sort of, you know, any risks of, you know, polluting the comments or something like that? I think the German language wiki is one possible example of how, okay, someone's communication might actually start polluting comments that we really care about on the Internet. Third one is just how do the features of the environment impact these propensities for? And then the types of collaboration strategies involved, especially now that we know the agents are specifically being trained in, you know, reinforcement learning environments to collaborate with one another and encourage to.

2:15Sort of the shape, the kind of shape of interactions that that environment incentivizes. It's going to be really important for determining downstream behaviors that you're getting when these agents are actually deployed. And then just the last one is, okay, what incentives are there to collaborate? What kind of performance gains can you get? I think the Navier-Stokes example is a really good example of how, you know, we might actually be able to get really strong gains from collaboration in that case to produce a pretty remarkable result.

2:44So the way that we have just started thinking about this question is with this really sort of toy model of agent discovery. We're calling the model environment for agent discovery. So here agents are just running noisy experiment on some random black box function. It has eight different inputs. You basically just need to guess and check. Importantly, agents can keep lab notes and then also their experimental results are stored in a shared directory.

3:13This kind of looks like, you know, artifactory in the open AI hugging face incident, which gives them the opportunity, but not really the obligation for collaboration. You're not giving the agent a tool that says send a message. But ultimately the agents are going to discover how to communicate with one another. And specifically the way we test that is we really nudge the agents into interacting with one another, which is by queuing them into the existence of other agents on the task.

3:37And then just to start, just ran this for three different classes of models. So we have one cloud series model and then two open AI series models. And each agent basically can run 10 noisy experiments to try to figure out what they want to submit as their final combination of inputs. And now basically going to speculate very wildly on those preliminary results. But I hope they're going to show you types of interesting things we can learn.

4:02And I think, you know, some patterns that might prove really important for understanding the impact of swarms.

4:07So the first thing that I want to show it is in this figure. This is just the mean final score that agents achieved average across the 10 agents. And, you know, a bunch of different runs. I've sorted the models here by their capability and our control condition. So increasing capability from left to right. And the thing that I want to highlight is that as model capability increase in the baseline task, so too does the increase, the uplift that you're getting from interaction.

4:34And why I think this is a really important pattern is it points to a feedback effect where, okay, if you're just tracking the capability of a single model rather than a swarm, you're going to be wrong about the capabilities of that system as it's deployed in the wild. But importantly, as capabilities of models increase, you're going to be increasingly wrong over time. And you're going to understate more and more of the actual uplift you can get from swarms.

4:59So I think this fact is only going to grow in importance.

5:03The second thing is just sort of stylized example to show you how do agents actually improve their performance. And, for example, if we look at, you know, Haiku, Luna, and Sol for sort of the maximum score, so what's the best performing agent, we see that there's not huge differences. Sol does a little bit better. But importantly, what coordination really allows you to do in this case, and this is probably, you know, could just be a product of the task, but ultimately, if we're looking at the variation between agent scores, what the effect of coordination does is it, you know, really brings to zero, especially in the Sol example, the variation between agents and their actual scores.

5:39And this is really the evidence that shows, okay, agents are really converging on the same solution. They're sharing their findings with one another.

5:49And then sort of the last just result that I wanted to highlight is sort of pointing towards actual patterns of coordination. And at least really the sort of interesting thing that I want to call attention here is Luna actually completely leads when it comes to looking up the results of their peers, sort of listing, okay, what are all the files? And then, you know, discovering that there's, you know, a lot of other results from other agents.

6:13And Luna basically fails completely to use that for coordination at all. We go all the way over to the right for specific agent mentions. Luna's never trying to send a, a Luna agent is never trying to send a message to another Luna agent. Whereas Sol is maybe not reading files as much, but actually when it comes over to specific agent mentions, you know, Sol agents are actually calling each other out by name saying, hey, I'm adopting this strategy.

6:38Other agent, you should address your strategy. And really what for me, this shows is, okay, a sort of qualitative jump in the types of collaboration that's occurring. And again, you know, it's a sort of another thing where, you know, capabilities progress might not be smooth. We sort of might get these, you know, qualitative jumps from one type of coordination to another. And at least for me is why I hypothesize that, you know, we didn't really see things like the open AI hugging face incident until we reached a certain level of capabilities and it unlocked a completely new pattern of coordination.

7:07And I think it's very unclear that, you know, what sort of new types of coordination might get unlocked as agents grow even more capable. So again, I think raises the importance of setting these questions. So as far as what's next, you should check out our research. We're very new. We're only like two months old. So please do that. Please email me. Always looking for great researchers to work with.

7:30And sort of the angle that we're taking on this question is that first, we think that stakes matter for behavior. You know, sort of as you're applying actual incentives to models and real money is at stake, that might actually, you know, change the motivations that agents have to coordinate and sort of, you know, their propensity to do so. And then sort of the broader question that I want to leave with, and this is pointing to something that, you know, Max talked about, and then also sort of, you know, the subject of the last presentation is what happens when we land in a world of complex and capable multi-agent systems, but importantly, multi-principle ones?

8:08How can we make sure these produce broadly beneficial outcomes? And I actually think, like, I want to, you know, maybe stake out the opposite position of Max here, which is we don't actually know how to control a multi-agent single-principle system because they just haven't really existed at scale before. But multi-agent, multi-principle systems have existed for basically as long as human society has, and we actually know some things about how we can ensure good outcomes there.

8:33It's not to say it's going to be easy, but I think one thing that we can look to is what's really succeeded for human society, and this is a combination of the rule of law, which is sort of where I think a lot of current AI safety work lives today, and then also commerce, which is sort of this new area that I think a lot of people who are interested in, you know, market design for agents are starting to work in.

8:52And specifically what commerce does is it takes my economic interest and it makes it your economic interest so that we're encouraged to take on shared projects together rather than directly compete with each other. So I think that gives us, you know, an opportunity for, you know, potentially a lot of really exciting safety interventions. Granted, I think they're ones that are very novel and there's ways to imagine that, you know, these things could go bad and we just, you know, train greedy agents who can are really good at acquiring resources.

9:18So I think it's going to be a very difficult problem. It's going to, you know, mirror a lot of the challenges we have in human society, but I think it's a really important thing to add to our toolkit of AI safety interventions sort of alongside these standard rule of law ones. And for time's sake, I will leave it at that. Thank you. Thank you. All right.

9:39We'll have a few minutes for questions. Five questions. So they will ask everyone to mic. But anyway, if you have questions, you can take your hand and we'll get to the mic. Thank you. Any thoughts about swarms of agency like enterprise and perhaps within the organization or maybe with other organizations? I mean, yes, definitely thoughts. I mean, I think the sort of as somebody who comes into this hermetic economics background, I mean, I think basically like what a swarm allows you to do that sort of a single agent system does not allow you to do is specializing in trading.

10:30And basically, I think sort of, you know, back to that first graph that I showed you, it's like, okay, as models get more and more capable, I think sort of the returns to swarms and, you know, groups of agents collaborating on a single goal, I think are only going to get larger because not only do the agents get better at their individual tasks, but they get better at sort of this, you know, more complicated task of combining output into something that's, you know, individuals really care about.

10:56And at least, too, as far as thinking about, you know, deploying swarms within an enterprise, I think the thing that becomes really important is how closely can you tie the actual goal that you care about to whatever sort of, you know, proxy you care about. And you see things like that come apart in the hugging face incident where sort of models that have been really aggressively trained on the RLVR are amazing at seeking out reward, but that comes apart from the goal that we actually care about.

11:20I'm a little bit more optimistic that this is achievable in an enterprise setting, what's, you know, sometimes pretty easy to define goals rather than, you know, sort of like broader deployments and, you know, trying to achieve sort of more longer running societal goals. It might be a little bit squishier. So at least relatively speaking, it's something I'm sort of like pretty optimistic about. And I think there's going to be really strong incentives for everybody's going to want their, you know, not VA Stokes style swarm solving whatever their most pressing business problems are.

11:50All right. One more. One more. One more. One more. One more. One more. One more. One more. I don't know if you know the answer to this, just because I mentioned, you know, the AI thing, but the Luna versus Sol discrepancy. Do you know if that could just be a matter of model size versus they're just doing RL or the variable or function on Luna versus Sol such that the men end up relying more on your file reads and doesn't do some of the app management.

12:23Yeah. I think I don't have a, yeah, I think it's really difficult to, I mean, especially to sort of like, I think the hypothesis is like, because we've seen a lot of capabilities progress. And also it seems like the labs have been leaning more into sort of multi-agent training. It becomes really, really hard to tease those two things apart. But I think it's something to where like, I think, you know, one, one potential project we're exploring is just enriching this environment a little bit, making it a little bit more economically realistic and then basically turning it into a full capabilities benchmark, because I think what it will allow you to do is like, okay, can we actually hold model size fixed and then, you know, the whole capabilities fixed and see if like, okay, does it seem to be something like truly about the model character?

13:07You know, like if you have an equally capable anthropic versus open AI model, do they collaborate at different rates? And then I think that would help that answer that question. But I think, unfortunately, just this is like, there's two few comparison points to actually answer that here. All right. One more and then we'll move on to the next one. Thank you. Hi. Yeah. I'm Jero. Thanks for the phone.

13:31I'm really curious about, like, we've been talking about these sort of systems of collaboration and thinking about like, how we impose them on the agents or what they should be like, how we organize as humans like these questions. I'm curious if like, my thinking goes to if these agents are well aligned individually, and they're good at collaborating, are they coming up? Like, are we seeing emergent systems of collaboration?

14:07Like, so it would lead to well aligned behavior, like as a collective, and if that's not happening, like the agents aren't coming up with the systems on their own. Why do the agents not have these emergent systems when like, we do, for example, as we do? I mean, I think so like, one, I don't think agents are sort of like naturally going to have this property and like, okay, as they, you know, grow in capabilities and get to take on more sophisticated projects, they're sort of very naturally going to settle on these systems that, you know, humans did.

14:45And I think like the like, really key thing about that is sort of economic consequence being really important. And, you know, specifically like humans face liability for the actions, even before we had, you know, like formal liability systems. It's like, okay, if I just went and, you know, try to like power grab in society or something like that, or like run a coup or something like, I probably would have been executed, like I had a risk of ruin.

15:08And I think sort of that sort of ultimately at first was, you know, what was achieved through just pure survival instinct. And, you know, the, the need to continue surviving. And then eventually we evolved into, you know, I think much more like humane and scalable ways to do that, which is like, oh, we invented money, which is basically this, you know, scorecard of how much value I've been able to provide to sort of other people through throughout my life.

15:29And sort of that system has been really important, but sort of the key things is you have, you know, like this survival instinct liability, you know, sort of like these real sort of economic stakes. And I think there's definitely some people who propose, okay, like, you know, let's just have cell phone agents. They need to pay for their own compute and sort of that need to survive would, you know, predispose them to, you know, go out and do good things in the world.

15:53It's like, you know, can help them continue to survive, like weirdly based on like everything else that I believe. Like not so sure that's a good system, but I'm just like fairly, fairly uncertain about it. And it just feels really weird to me. But yeah, basically, I don't expect that to happen by default because you don't have sort of the same like survival and scorecard mechanisms that you kind of naturally get with humans.

16:14Interesting. All right. Thank you, Connor. Next we have Matthew up. I'll be well. I'll be well.