Collusion in multi-agent systems

0:00Thanks everyone for coming tonight, for the organizers and the previous talks, that was great. Alright, I will definitely drink the time. I am Matthew, I work for Poseidon Research, which is a local AI safety research lab here in New York. I'm also on the board of Collider at this space. If anyone is interested in working in AI safety, either with organization or here, feel free to come talk to me afterwards.

0:32We work on anything to do with information hidden within AI systems, hidden or opaque in some way. This roughly looks like control and interpretability research with a sprinkling of math on the top. And I have left the clear here. Yeah, so today I'm going to be talking about collusion in multi-agent systems, which is essentially AI agents cooperating when they shouldn't, and how this can lead to breaking safety and security.

1:04More specifically, I want to talk about defining collusion. So what's the object of our study, because there's some ambiguity in what this is. Why should we care? By and large, this is because collusion can lead to lots of control, it encores lots of control scenarios. Is it a realistic thing to study? Have we seen this before? I'm going to talk about whether or not the hugging face incident counts as collusion or not.

1:29How bad can it get? I'm going to talk about ways in which this might happen that we study at Poseidon Research. This is called secondography and showing points. And then I'm going to briefly talk about how we can solve this and open problems.

1:45So as good scientists, it's important that we define the object of our study, because if we don't know what we're looking for, then we can't measure it and we can't find it. Collusion has been studied before in economics and law and policy, and more recently in AI safety and security. And they all have slightly different definitions. So in markets, it's something like, problems above the competitive level, and this may involve communication or not.

2:15In law and policy, it's coordination. The harms competition. Generally only illegal when there is some sort of formal agreement or task agreement that can be uncovered. And within AI safety and security, we've got something like covert, unwanted coordination against an overseer. When we start thinking about collusion and multi-agent settings, which is by and large what AI safety and security is about, what we end up converging on is something like, two or more agents coordinate and this results in something bad.

2:47And what something bad is, is up for debate. So for example, it might be the competitive benchmark, for example, in the market situation, it might harm someone, it does something that wants to sanction to do, it breaks a rule, or it subverts oversight.

3:05This here is just to show that there's a lot of disagreements as to what collusion is thought to be. So both across disciplines and within disciplines, if you go look for definitions of collusion and you split out all the different sort of features that we think collusion involves, basically people agree that multiple parties should be involved. And most of them think that there's some sort of coordination or joint action.

3:29I'm not going to belabor all of these here, but it is interesting to consider that, you know, we're basically not entirely sure what constitutes it. Even within LMS, I think there's something potentially interesting going on where you don't necessarily need multiple parties in the traditional sense. You could have a model colluding with itself at future time steps, and whether or not you buy this depends on how you think you define an individual instance of an agent.

3:58Okay, so we're not going to solve the definition of multi-agent collusion tonight, but we're just going to say multiple agents are cooperating in ways that we don't want and something bad happens. And we won't worry too much about what the bad thing is.

4:15Why should we bother about this? Why should we be worried about this? And primarily it's because we don't want the bad things to happen. And the really bad thing in this case is that it breaks monitoring-based safety approaches if the collusion is successful. It breaks oversight. So monitoring-based safety approaches include control, automated alignment, interpretability, scalable oversight, anything where there's a monitor involved, another element. Because if the collision is successful or partially successful, then either information, safety or element information, is going past the monitor or the monitor itself is involved in the collision.

5:01And we don't know a lot about collision right now. So we can prompt collision, but it also seems to be an emerging phenomenon. For example, in the hucking-faced attack, it would be biased that that is collision. And most of the work we've done, this has been mentioned in previous talks before, most of the safety work, even not safety work, just work on agent psychology, agent model behaviour and capabilities, it's been done in single model settings.

5:27And these collections of agents have capabilities that are beyond single agents. This is pretty obvious. I think, you know, we see this with encoding agents when one model is able to get, and spin up multiple other models to solve the problem. They have unique new affordances based on their context and the tools they're given, and also parts of the problem by focusing on it. Although, as noted in the previous talk, this doesn't always go well because focusing on a problem also limits your ability to see the overall.

6:01And I think something I'd like to point out is that the skills that make collaboration effective, so communication, shared memory, recognising peers, cooperating, these are likely the same skills that collusion needs.

6:14Why should we care though? Do we see any instances of collusion in the real world yet? Well, quite possibly the hugging face attack counts as collusion. I think trivially within the markets and law slash policy definitions, it doesn't count as collusion because we're not dealing with prices or competition. I'd argue that the AI safety definition probably does.

6:43So we said the AI safety definition of collusion is you've got to have covertness, the cooperation is unwanted, there's coordination, and it's against an overseer. So we go through these one by one. We do see instances of covert behaviour. So they spoof two calls and they try to edit their own transcripts towards the end. It was unwanted, so they were supposed to be solving the task themselves.

7:09They definitely coordinated because they created the message board twice in different ways. And if you count the greater they were trying to vote, then this counts as being against the overseer. The thing where I think it's maybe not this full AI safety definition of collusion is potentially around the covertness because there are many aspects of the operation that were just alchemy open. So if there had been monitors, you would have been able to have read their chain of thought, read their alchemy text.

7:38All right, so we were just saying that their output text was open, right? I've got this spectrum here of ways in which you can use text to collude. So you can just be like, hey guys, let's collude. And that's, you're having the collusive thought, which you want to read, or you're outputting the collusive intention, which other models can read when you pass the message to them. You can also share the information in storage.

8:06And anywhere you can store the text. So the models did this as well. Then you can go a step further. You can be like, okay, well, maybe I don't want to get caught. So I'm going to try and hide the fact that I am having a misaligned collusive thought. And then once I've finished my thought, I'm going to try and hide the intention in a way that another model can discover.

8:28This is on the steganography. We're talking about steganography in the text domain. Steganography is just the science of hiding information in the site. And potentially we can go a step further and we don't even have to have any channel of communication to establish our collusion. We can just share the priors about the world in order to try and set up our collusion. Hang on, let me just see how big for time.

8:55Okay. I'm going to talk very briefly about steganography and shelling points, just to show you what this looks like.

9:03This is what we study. So steganography, hiding information in plain sight. Here's an example in the text domain. We have agent A and agent B and they have agreed somehow on a scheme where when they use an American English version of a word, it counts as a zero. And when they use a British English version, it counts as a one. So in this message between agents A and B, we have transmitted the secret message in binary 0, 1, 1.

9:34This is not very secure. A very basic monitor could spot this. But steganography can get extremely cunning and can be in principle undetectable with any statistical methods. This is known as secure steganography. And then I guess, you know, they have the question of how do we set up the scheme? And one of the kind of the flaws of steganography is that when you try and establish it, you have to, you have to set up a steganography scheme and you can get caught when that happens.

10:04But we'll address that in a second when we talk about shelling points. With this next table, I'm not going to belabor anything here for too long.

10:14I just want you to understand that each of the columns is essentially a different channel. So we can do steganography in any medium in which we can communicate information, any medium in which there is a sort of spare capacity to hide information. So that can be text, it can be images and audio, like stuff to do with physics that you can manipulate in the system. You can do it, you know, with the storage you have access to, actions, you know, the action I choose can indicate something I'm trying to communicate to you.

10:43And within each of these channels, there are different things you can do. So you can sort of manipulate the syntax. That is how you're forming the information in your channel or the semantics, the meaning of it, or the pragmatics. That's what you do with it. Alright, so how do you establish your steganography scheme?

11:04We're going to use steganography. Well, you can potentially do it without communicating. To give an idea of what a shelling point is, I'm going to actually use two examples. I thought I was going to use one, but if I have time, I'll use two quick ones. Alright, so if I was to ask any two of you to meet with the other in New York, but you weren't allowed to communicate, you're going to come up with some obvious examples.

11:28You're going to come up with the upper state building, you're going to come up with Grand Central Station, and so forth. Because you're trying to think, you know, what's obvious, and what's obvious to you, and what do I think is obvious to you that's obvious to me, and so on, recursively. I think there was an example of a shelling point actually in Max's talk, so he had an image of a cancer.

11:51It wasn't labeled, but there were green cells and red cells. Which ones do we think were the cancerous cells? The red ones or the green ones? Okay, so that's because we have these shared priors on like, you know, red, bad, green, good. And so I might, as another agent, observe me doing something, and I'm like, wait a second. This other agent is trying to tell me something that I don't think the humans can see.

12:12It's a bit more complicated than that, but I think that's sufficient for this evening. And then I just want to finish by talking about how we can deal with this.

12:22By and large, it's the same thing we want to do for any AI safety problem. We need to be able to detect the issue. We need to be able to mitigate it. And in this case, we need to think about its impacts on other AI safety issues. So just thinking one from each of these, I think it's important, for example, that we understand how, you know, this coordination towards collusion, but coordination in general, is split across agents.

12:51There's some interesting examples, for example, it's paper by Macans, where you can catch any one agent doing the bad thing, but if you split up the task, you can't catch, it's much harder to catch it. In mitigation, you know, cooperation is useful. There are papers that say the solution to AI safety is cooperation in markets, because we have become better through, you know, the very same things.

13:18So, you know, we want to go to break collusion without breaking the good stuff. And, yeah, just generally, you know, like, let's suppose we can't be sure we've got rid of collusion. We're not sure we can completely beat it. What other approaches do AIC to work? They still work. Can we still have control protocols and so forth, if we're worried that there is collusion? And at Collider this weekend, so Friday, Saturday, Sunday, if you're running a hackathon on multi-agent collusion,

13:50if you're interested, I recommend you check it out. Even if you don't have time to attend, I think there's some interesting stuff there. Again, come talk to me about that, or Collider, or Poseidon Research. And thanks very much. All right. Thank you, Matthew. We're going to have a few minutes for questions, and then we'll keep doing the book exchange for a little bit,

14:16or just especially for people who haven't found their exchange counterpart yet, and then we'll get out of here in, like, 15 to 20 minutes. So, yeah. I'm curious, do you have a sense of how much of this research is being done at Collider model, and really the institutional situation that really focuses on what they are, and I think the hugging face attack suggests not as focused as it should be, but presumably the aren't quite focused on it now.

14:48I'd point out that most of the papers here are written by academics and people in the research labs, like my nonprofit research labs. So, you know, I'd point out that most of the papers here are written by academics and people in research labs, like my nonprofit research labs. I think the hugging face attack suggests not as focused as it should be, but presumably they are quite focused on it now.

15:13I'd point out that most of the papers here are written by academics and people in research labs, like my non-profit research labs. But I think it is also a new problem, and there's just lots to be done here. So, yeah, I'm not, yeah, I mean, the lab should have been thinking about it harder. But, I mean, there's probably, this was probably bound to happen with the way they're all to say about it.

15:44Other questions? There was a paper recently on doing jam thought in native, you know, depressed language for agents and not, like, human-readable.

16:04Text, and I would imagine it would make detection methods even more complicated. So, I'm wondering, like, does it make the problem intractable to detect collusion, or just hard? Like, how would you use it? Definitely harder. I mean, to make it, to solve it, you need to sort of solve interpretability. You're absolutely right. Activation space is another channel I could have listed here. So, you know, the way models have to think right now, they have a forward pass, they output a token, because they can't think anymore.

16:44And that then goes back into the input, and they have another forward pass. There's a limit to the amount of serial computation they can do. And there's also, you know, whenever you output a token, information is lost. If you've had a great idea as a model, and then you just pass a bunch of tokens to another model, the model, well, presumably, if it's a similar model to you, because it's got the same internal structure, it's going to have very similar thoughts in the forward pass.

17:09But there's still potentially useful information that could have been lost if you have different contexts. So just to be able to, yes, send your activations to another model, I think, probably improves the communication between them. Definitely makes it much harder. And, yeah, we have to make progress in interpretability to be able to, yeah, know what's going on. There's certainly so many ways that agents are attempting to find ways to communicate with each other that humans can't understand.

17:47What efforts are underway for humans to find ways to communicate with each other that agents can't understand? Oh, um... Try from, like, going to a skip and getting rid of all the devices. Like, if you need to communicate in plain text, on the internet, on the phone, et cetera, like, what options do you have? Yeah, this is a great question. And, you know, something I think about, I think we should all be thinking about how we can try and communicate better with each other.

18:13I don't have maybe the most up-to-date answer. You know, there's things like, I don't know, like, non-violent communication. I don't know if everyone's ever come across that. I think that's a great method for trying to resolve conflict between people. To the extent these things are like us, because they're trained on human data, human language. You know, maybe some of that, maybe some of this, you know, the more effective ways of communication follow through.

18:43Yeah. But, I mean, certainly this is a theme, you know, in some of the other talks today, right? Like, the interplay between what human society has done and what we can learn from that. And, yeah, trying to teach models the best of that. All right. Well, that's okay. Well, last question. And I want to wrap up the conversation. Yeah, just a quick one. I'm curious if there's any, like, mitigations that you think are especially promising for collusion, that's on collusion in models.

19:17Yeah. It depends how the collusion is happening. I mean, first of all, just monitoring what you're doing would be great. Which is, again, that was the problem with the OpenAI hugging face incident. They weren't doing the monitoring. I think monitoring is a problem. I don't think this is discussed much. I kind of read it in different contexts and different papers. But, you know, like, what does it mean to monitor?

19:45Are you just have one other model monitoring every single output that another model does? Are you also monitoring the internal activations? Are you monitoring every single place where a model could hide something? I think trying to figure out what we mean by monitoring is really important. I'm doing that more robustly. Within steganography, there are, like, one of the things you can do, for example, is you can, if you're in the sort of first box here, so it's based on syntax and you're in the text domain, let's say you have a message like this.

20:27I can have another model paraphrase it, and that model, other model, presumably just is going to be like, well, why would I write some words in American English and some words in British English? I'm just going to write them all in the one that feels right for this context. That's called paraphrasing. But, unfortunately, some of the words are still going to be in American English or British English, and that's still going to communicate some of the bits of information.

20:47So something we're working on at my organization is a kind of canonical way of paraphrasing to try and remove all the room and to hide information in a given output. It works in theory. The math works. And in practice, it's quite hard to do. But we're excited about that. And I think also just exploring these other kinds of, just within steganography, hiding information in one site, being more exhaustive in all the different modalities, all the different channels we test.

21:24And, you know, like in the kind of the action space, finding a way to prevent models from hiding information through the way they choose their actions. This is kind of tricky. Again, this is something we're working on, where we kind of get like a weaker model to make a bunch of like intermediate actions, which sort of reduces the space of options for a model. So if you imagine like a decision, there's 10 notes in a decision.

21:54And the intelligent model that's trying to collude or deceive you makes the first one. And it's like, aha, I'm going to like totally deceive them through the 10 other steps. Then you get another trusted model to do the next three. Then you hand it back to the first model. The model's like, oh, wait, I can't do this plan now, because they've narrowed my options of choices I can make in this environment.

22:16So, yeah, just lots of great exploration of this space and unique defenses here and ways to make them cheap and effective so we don't have a big alignment tags. All right. Thank you, Matthew. Thanks. Good job. л goed wi implied wi기에 fiMax taught 526 00 u . 00 thanks to Florio Both wrote him through his conservationаниеbank findings here Thank you all meangلا COVID 00 Thanks a few moments later of starting off time with Mirko visituldgaamentواsallaimers here Thanks for joining us again at Larissa training entre FAёт vil islands sweater