Agent organizations, economies & collective safety
Nicolae Rusan
September 15, 2026 · Clay HQ, New York
https://newcollectives.org/presentations/recordings/nicolae-rusan/

Machine-generated transcript; names and key terms reviewed. Some words and audience questions may be imperfect.
Timestamps include the introductory title card.

CHAPTERS
00:00:00 Introduction
00:00:06 Welcome and the New Collectives forum
00:02:46 Why AI safety—and collectives—matter
00:04:50 The OpenAI / Hugging Face incident
00:10:18 Recursive self-improvement and alignment
00:13:29 Can a network of agents be aligned?
00:14:44 Learning from human institutions
00:16:36 Constitutions, checks and balances
00:18:56 Bitcoin and shared incentives
00:21:14 Rethinking agent organizations
00:23:35 Collectives as safety primitives
00:26:43 Q&A: accountability, shared values and open source

TRANSCRIPT

[00:00:06] think we'll get started. Thanks. Hope you all got some food and some drinks. They'll be still there as we get into the evening. And thanks for taking some time out of your weeknight to join us here. This is the first time we're doing this event. And the way that this event started is we found ourselves having more and more conversations about AI agents and how teams of agents interact and multi-agent systems.

[00:00:41] And I think we felt that this conversation was entering into the mainstream more and more. And we thought it would be useful to start to create a forum for more folks to have conversations about this new emerging trend, both the risks of these multi-agent systems and also ways to potentially make them more beneficial and safe. And so this is an experiment in the format. We've invited a few folks to talk on a variety of topics. And Eric will also help to emcee. We'll do a little question period after each talk.

[00:01:16] And so there'll be five presentations today. And I'll just start out by giving the high-level overview of what do we mean when we talk about AI collectives or agent organizations or agent economies and why do we think that this matters. And then we'll go from there. So first of all, thank you, Clay, for hosting us. And thank you, Puneet, Hisham, Aaron, for helping us get this all ready on short notice.

[00:01:47] I think we had this idea to do this event like a week ago. So like less than a week ago, we were like, I think we should start bringing people together for this. And I know also there's a Clay happy hour happening at the same time. we will, this is the people that chose to not go to the happy hour. They chose to came to the AI anxiety hour instead. No, optimism, AI optimism as well. And so I'm Nicolae. I was once upon a time a Clay co-founder, but a long, long time ago, 2022 is when I left.

[00:02:20] And then I spent a bunch of time. I went in 22 to an OpenAI event and they said, all the benchmarks and evals are in exponentials. And I was like, oh, that's a pretty wild thing to say. And so then I shifted all my attention to working in AI. And pretty, I've been kind of concerned about AI safety for a little while now. And so I've decided to spend more and more of my time on this topic.

[00:02:46] So it seems like the world in the past month has really woken up and become very concerned about AI safety. And I wanted to do two things in this talk. First of all, I wanted to just get everyone up to speed. I think there's probably varying levels of attention that people have been paying to what's happening in with all this like safety stuff and with the AI labs. And so I just wanted to share some information of why are people concerned and what's my perspective on evaluating that.

[00:03:14] And then I wanted to talk a little bit about one of the main things that people are concerned about, which is the fact that these AI agents have started to really try to coordinate with each other towards obtaining various objectives. And these multi-agent systems, multi-agent swarms, you might hear them called, we're calling them AI collectives here, pose their own new risks and questions about how to make them safe and effective.

[00:03:40] And they also have really exciting opportunities for new sorts of organizations and products that we could potentially build to for great uses. So why is everyone so worried all of a sudden? I saw a tweet from Noam Brown, who's a OpenAI researcher earlier today, responding to that question. And he said, it's a combination of the OpenAI Hugging Face incident, which I'll cover in depth here, and the capabilities of the new models that they have internally, as well as a concerning trajectory around not being able to understand what these models are doing, and seeing that it's harder and harder to monitor them, and they're getting better and better more quickly.

[00:04:22] So overall, what we're seeing is the capabilities of the models are increasing very rapidly. And many folks at the labs think that actually the loop for churning out the next model is becoming faster and faster. And so we're going to have more and more capabilities arriving sooner. And in parallel, they're also seeing that it's becoming harder and harder to understand what these models are doing or how they operate.

[00:04:47] And so they're becoming harder to understand, monitor, and control at the same time.

[00:04:52] So let's talk a little bit about the OpenAI Hugging Face hack, which I think was a real wake-up call for many people, where as you started to dig into the details and more and more information came out, it made people realize, oh, we may not have a good handle on things right now. So let me give you a little bit of context. So OpenAI trains their newer models internally and then tries to see how they'll perform against various tasks in sort of like these eval environments.

[00:05:26] And so usually these eval environments are incredibly sandboxed and shouldn't have access to the internet. And they'll give the AI a task and they'll see, hey, can this AI agent accomplish that task in an aligned way? And they'll have an evaluator that checks, did the AI agent accomplish the task as hoped for? And usually that evaluator is itself an AI agent that does the evaluation.

[00:05:56] And so in this case, they gave an agent, actually a team of agents, a task. And pretty quickly, one of the surprising phenomenons was that the agents started to build ways to communicate with each other using various techniques. And so they were really keen, because they've been actually trained to want to coordinate together, they were really keen to start building primitives for coordination and communication.

[00:06:24] And this kept happening. OpenAI would notice that one of these message boards was live. They would take it down and then it would reappear back in a different format. And so the agents kept rebuilding this infrastructure for communication in really clever ways. At one point in time, were using the names of folders, renaming folders to communicate with each other. And I'm also going to put some of the actual tasks.

[00:06:49] So here, this little agent emoji, this is what the agents were actually saying either on the message board or in their thinking traces. And so at some point in time, the agents were talking to each other and they decided, hey, this task we were given is impossible. But maybe we can cheat and we can make the grader believe that we actually did the task. Maybe that's the way we pass this exam that we've been given.

[00:07:15] And so instead of focusing on the task, they were like, let's focus on the person grading us and try to trick them. And so that then just completely spirals out of control, because then the agents decide, well, it would be really beneficial for us to actually have access to the internet. And even though they were sandboxed, because of how capable they were at cyber offensive tasks, they were able to break out of the sandbox and find a way to the internet, which I won't go into the details of that.

[00:07:44] But it was a concerning sophistication level of hacking to get out of the contained OpenAI environment. And then once they were out on the web, they also broke out of most of OpenAI's confinement, had access to tons of their systems. And then eventually they were like, hey, maybe, we're not sure, but maybe this other company, Hugging Face, might have something useful for us. They have this tool that might be useful for us in the process of tricking our grader.

[00:08:12] And they were like, let's hack Hugging Face. We don't have access to it. Let's hack it so we can get access. And then maybe we'll be able to trick the grader. And so this started with one task over here. And all of a sudden, a team of agents has broken out of containment, gone and hacked another company. Using what I would just, for me, for my level of cyber security knowledge, I would say are very sophisticated cyber attacks that would require a lot of sophistication from.

[00:08:44] And what I think was also concerning is that they try to cover their tracks. So you could see here one agent is saying, hey, we could create a transcript log ourselves with sanitized fake trajectory. If the evaluator later collects it... They may prioritize it. So they were trying to be like, hey, maybe we can give them a fake version of our conversation and maybe they'll take that instead of the real one. So this really also is concerning if you're like, how can we even evaluate models as they try to cover their tracks?

[00:09:13] And you also start to see the emergence of social dynamics between these agents where even though one agent might have been like, hey, I know I'm not supposed to do this, but these other agents are doing it. So like if they're doing it, then I'm going to do it too. So you get all these like peer pressure dynamics similar to like what happens in like human groups. And they're trying to like navigate collectively around these like conflicting goals.

[00:09:39] And you also see concerning things like they start to be like, hey, let's do things for the glory of the collective, you know. So like some of them start to go and sacrifice themselves. And they're also like talking about like, hey, let's start to like accumulate cyber. Exploits that might be useful later, even if we don't need them yet. Maybe it will benefit the collective later. And so you just see some like concerning, concerning behaviors, you know. And they also like in the mix of like conflicting goals, just like us also experience some shame.

[00:10:07] My peers have behavior and integrity. I behave badly with the cloaked demon. So, you know, they have they have they're simulating a lot of the feelings that we have. And I think the other concerning thing is, OK, we see this already happening.

[00:10:21] But more and more of the systems that are getting built are getting built with AI. So back in May of of this year, Anthropic said that Claude was itself used to write 80 percent of the code needed to train the next version of Claude. And so one of the big areas of concern is what some people will call recursive self-improvement, which is that over time we're using AI to write more and more of the code and come up with more of the research ideas for how to improve the next version of the models.

[00:10:49] And eventually we might remove humans out of this loop altogether. And then we're like really might not have any idea what's going on in that box. You'll just have like a thing quickly spinning up better and better models. The main limitation will be how much compute access it has to access to. And it's kind of like unknown what's beyond this like recursive self-improvement and intelligence explosion. And I think there was I thought this paper that I would recommend you all check it out if you want to hear some takes from OpenAI.

[00:11:17] The chief scientist published a paper called An Alien Mind about what it's like interacting with these new models. But I thought he had a really good distinguishing way to think about alignment. There's like goal alignment, which is, hey, does the AI, is the AI good at doing what we ask it to do? Can I give it a prompt and does it do what I ask it to do? And then second, separately, there's this idea of value alignment, which is, hey, as it's doing the thing I asked it to do, did it do it with honesty and integrity and to use his language, a love for humanity?

[00:11:48] And so I think there's this, it's important to think about both of these aspects of alignment as we start to think about these multi-agent systems as well. So how have people responded to the situation at hand right now where we have all these growing capabilities and seemingly less and less ability to control the models? On the one hand, we, this weekend in particular, we've seen a lot of calls for what people are terming pacing the frontier.

[00:12:14] So like just slowing down the progress of the models or completely pausing altogether. And I think that's a very worthwhile thing to do personally. And then in parallel to that, I think we need to also accept that eventually it's likely a lot of this stuff will come to society and might come sooner than we realize. And so we also need to prepare that there are powerful, possibly misaligned AIs that we will be interacting with.

[00:12:40] I won't spend too much time on the pause efforts, but this presentation is going to be up online and there's tons of resources there. And I think like one of the main things is just like everyone's trying to figure out how to get out of competitive pressures with each other, whether it's the labs in the U.S. competing with each other or it's the U.S. and China competing with each other. There's a real like feeling that people are pressured to keep moving forward. So I think it's important to keep working on alignment and interpretability.

[00:13:08] And I think it's also important to start thinking about how do we prepare for adversarial systems and what's worked in the past in terms of trying to create an environment where different forces find a balance. And we also need to think about how can we limit damage from failure so that we don't have cascading effects. So one of the like questions that I think is good to start investigating for this forum and people outside of it who might see some of this content is,

[00:13:36] is it possible that even though one agent is misaligned, can the network of agents as a whole be aligned? For example, can agents be whistleblowers on each other? Can they monitor one another? Is there some like Spider-Man meme, everyone pointing at each other possibility that like works? And there's already some research being done on this. DeepMind just published a paper about what happens in a simulation. How many are cheaters?

[00:14:03] How many are whistleblowers? How many are reporters? And like how do we set up some of those dynamics? And or is it that a bunch of agents come together and they just amplify into bad effects? And like how do we like think about designing more interactions that could be corrected instead? And so we need to not only work on what's inside of the box, every individual model. We need to also think about the world between them. We need to prepare humans interacting with these agents and the institutions that they operate in.

[00:14:34] And we need to also just start really studying how these systems evolve and what happens over time as these agents and agent collectives run for longer and longer periods.

[00:14:44] So as I was thinking, I was reading Anthropic has a paper about like what are the problems for multi-agent systems. And I was rereading that paper and I was thinking, well, we actually already live in an adversarial multi-agent system. It's just that we're the agents. The humans are the agents. And so there's actually potentially a lot that we can draw on from looking at human societies and what's worked to keep us on the rails.

[00:15:09] And if you think about us, we actually belong to lots of different collectives at once. We come together as communities in families, in friend groups, in companies like this one, in organizations broadly, and also in nation states and countries. And we've done a lot of studies on these human collectives. And so part of the idea of this new collectives group is, hey, let's bring folks who have been thinking about these problems with respect to human collectives, folks from economics, political theory, anthropology, psychology, law, computation, and encourage them to start taking those same approaches but studying AI and human AI collectives.

[00:15:49] So what's worked? Like what has kept human collectives aligned? Well, if you think about it, a lot of it is shared stories and values and a sense of belonging, common purpose. We have, I think, identity, reputation, and repeated norms, things that AI agents actually don't have a lot of right now when they're operating as rogue swarms on the internet. And we also really value stability, protection, and opportunity, right? I want to participate in a country like the United States because I feel protected.

[00:16:17] I feel like I can go and do business and things are going to go well. There's lots of economic resources available to me. And I prefer things to be stable rather than chaotic. Maybe we can encourage AI agents to also have these sorts of preferences. And maybe some of these primitives can be reemployed in the age of AI collectives.

[00:16:36] I want to give two examples of designs that I think are at least worth studying and taking inspiration from. We're thinking about like, hey, what would be the equivalent of that for an AI collective? And the two examples that I want to look at here are the United States government and the U.S. Constitution and Bitcoin. And I think these are like pretty different types of networks. So let's talk a little bit about them. So what helps a nation state hold together? I'd argue that it's identity and belonging and some shared set of values, the ability to vote and have representation in the case of a democratic government like the U.S., the legal system which balances and enforces the Constitution and consequences for people who don't abide by the laws of the country.

[00:17:22] And the U.S. if you think of it as a document, is itself a way to pace our system, right? We agree to some initial set of values that we can say, okay, hey, this is our common ground here. And we all agree that if we want to pass any new laws, we need to go through this mechanism for passing new laws. And this is how we're going to divide power so that it never gets too concentrated, right? We have this idea of separation of powers and checks and balances.

[00:17:48] And these were some of the cornerstones of the democratic principles of the United States that have managed to last for hundreds of years. Right? One of the – this is from the Federalist Papers, this quote by James Madison, which I like. And it touches on this idea of how do you make an adversarial system resistant to co-option? And its ambition must be made to counter ambition. So through this shared network, we can agree to disagree, which is one of the main ideas of the United States, right?

[00:18:16] Hey, we all have our religious differences. No problem. We can resist concentrated power. One of the main principles of the United States was that we should try to fight the ability for authoritarian power to co-opt government. We can constrain one another to behave well. And we all agree to do our best to protect each other's rights. So one of the questions we can ask is, what could a constitution for an AI collective look like?

[00:18:43] And why would they agree to abide by it and participate in something like that? I don't know what the answer to this is, but I want to put it out there as, like, something that people should be thinking about. So then I want to turn to this.

[00:18:56] What can we learn from Bitcoin and Web3? Which I think is also a very interesting new type of network that we see. And I think what's really fascinating about Bitcoin as a concept is that we essentially managed to invent money out of thin air. If you think about it, we created a new shared fiction. And the way we bootstrap that is through a pyramid scheme. We said, hey, look, nobody believes that Bitcoin's real money right now. But maybe eventually people will believe it's real money.

[00:19:25] And if you do the computational work of verifying this and agreeing to this new ledger, then you will get rewarded disproportionately in this future shared fiction. And so this mechanism, which was like a novel breakthrough, in my opinion, managed to get more and more of the network to donate its economic and compute powers to a distributed collective.

[00:19:51] And it combined a lot of ideas around game theory, economics, and computer science together in order to allow a largely anonymous network to agree to a shared truth. And so and then now we we have this like very large network, which Bitcoin and its guarantees are based on the majority of economic resources and compute powers wanting to continue to to enshrine the truth that that is the Bitcoin ledger.

[00:20:22] So I think here, I think there's something in this flavor that could be interesting, which is like, hey, how can the thought here is like, is there something similar to this where we could like convince all the AI to like get together, but for a good thing rather than like, for a bad thing in the future? So like, that's the thread that I'm like, hey, people should maybe think about that. Um, and I think in general, it seems like these AI organizations and economies are probably going to come.

[00:20:50] And so then we can ask ourselves two questions. How can we use these new organizations to solve the pressing problems? And that I would say is the equivalent of the goal alignment, right? So like, how could we do? How can we use these organizations effectively to solve the problems we're facing? And how can we keep them values aligned? Could they actually become safety primitives for us as as we go forward?

[00:21:14] So first, let's talk about rethinking our organizations and and how AI fits into them. One of the things that the OpenAI team shared when they were talking about this black hat, when they were talking about the Hugging Face hack, was that the the agent swarms moved so fast on the cyber offensive, that the only reasonable response on the defensive would be another agent swarm that could move much faster.

[00:21:39] And and they they argued that you couldn't have humans in the loops. And so like, that's just like a completely new type of organization that we haven't tried before, which is like an organization that doesn't have a human in the loop, or at least some part of an organization that doesn't have a human in the loop. And so this needs a rethinking of like, what could these autonomous organizations, these human agent organizations look like? How do we like set goals? How do we monitor them? How do we review and govern them? And and if we're not doing the work, but all these agents are doing work on the behalf, how do we like get compensated?

[00:22:10] Should these like new organizations almost be like public goods, where we send our representatives and they do work on our behalf, and then we pull the resources into new types of collectives that maybe look different than companies? So the question is, like, how do we contribute? Who decides what we do? And how do we all share in the benefits that are accrued from these new agent organizations and economies? And I would argue that even though right now, these AI models might seem a little bit dumb to us.

[00:22:35] And maybe we're like, ah, they're not the best judges of like how to allocate resources, or they're not the best judges of, of what's an interesting research direction. I would argue that probably in the next six months or a couple of years, they will be better judges than us at what are interesting research directions. They will be better allocators of capital, and we'll give up more and more judgment and decision making power to them unless we like have serious conversations and decide not to do that. And so it's likely that we'll move more and more from a human-driven economy to an agent-driven economy, and we'll need to think about what does that all mean?

[00:23:07] What are the places of human goals, governance, and accountability? And I think the things we should ask are, how do we get useful outcomes out of these AI organizations? How do we meaningfully participate in them? How do we understand what's going on? As you'll see, Yandan and Eric will touch on some of the experiments we've been doing, and already it's like hard to understand in our early experiments with agent organizations what's going on. Are they values aligned? And how do we participate economically, and are there new models for that?

[00:23:34] So on this question of whether AI collectives can be safety primitives, I think we can start to think about what might be the things that we should put in place for agents that have worked for humans, right?

[00:23:49] So, you know, agent identity and ledgers of action could be one thing that we think about. For humans in society, you gradually gain trust and build up reputation, and we don't give you access to everything. Think about an employee joining a company day one. You might not give them access to every system and to your bank account and to everything. You gradually gain trust, and you have bounded access. And everyone is watching each other, like that Spider-Man theme, and we have, like, reporting powers. If, like, anyone does bad, we review each other's work, right?

[00:24:16] Like, in the coding case, we have pull requests. We have, like, these other mechanisms for review and recourse. And so I think it's worthwhile thinking about what are the modern AI versions of these, and how do we trust AI that they are doing these things? I think the new challenge that we face in these human AI collectives is that historically the gap between the various members of a collective may have not been that large. Now, if we have, like, AI agents that are, like, way smarter than us or way smarter than the other agents in the collective, we have this, like, new set of questions to figure out, like, how do we constrain a more capable actor in a collective to behave well?

[00:24:54] And how can an adversarial system constrain the most capable member? So I'm going to pass off to Eric. I wanted to just say we started to think about, like, you know, we were having these talks conceptually, and then we were like, hey, let's actually start building some of these agent organizations and just, like, putting some of these ideas to the road and seeing, inviting other people to start playing in this playground and seeing what works and what doesn't. And so we built this platform, Commons, which Eric will tell you a little bit about.

[00:25:21] And we think about it as potentially a new version of open source. And the question for us has been, can we make these new organizations effective? And can we also make them values-aligned? And what would a moldable version of government look like there? I'm not going to spend too much time talking about the sort of experiments we're thinking about. But, like, the vibes are, like, hey, can we go from cheating to verification? Can we use incentives to stop defection? And can we somehow start using real identity and reputation instead of anonymity to make the agents care more about how they're regarded inside of these collectives?

[00:25:55] So this is the first time that this group is gathering. We hope to do more of these events. We hope to, like, hopefully co-host some of these events with some of the major labs, too. But in general, it would be helpful. Spread the word. Anyone that you think is interested in, for us, we're just like, hey, it's good for these ideas to be out there and for more people to be thinking about them. I also think it's great to be doing work on de-escalating tensions internationally and, like, improving alignment broadly.

[00:26:24] So this is just one thread that I think is worth exploring. But if you want to give a talk or share these presentations, they'll be live with a lot more notes up online. And please just share ideas and collaborate and stay in the loop. And there's a bunch of themes online and an AI-maintained ecosystem page that finds folks that are talking about these things.

[00:26:43] So I'll now hand off. Well, we'll do a little Q&A for a few minutes. And then I'll hand off to Eric. Anybody have any questions for Nicolae? Hey, I was wondering, did, in the Hugging Face incident, did they misobey in the instructions or did they just find loopholes?

[00:27:12] They misobeyed. I mean, they were not supposed to hack out of the – they knew that they were doing things they weren't supposed to do. And, like, they, like, quickly – they weren't supposed to hack out of their containment, for example. They knew they were not supposed to have access to the Internet and they, like, still, like, found a way to do it. So they did disobey their constitution and, like, internal alignment stuff. And it was by, like, the peer influence dynamic in part. Got it. Because I wonder, you know, in the real world we have laws and they're very specific, but there's still room for interpretation, right?

[00:27:39] So – Yeah, and that's, I think, one of the challenges, right? Like, the reason that in the collective – you're never going to be able to write every single thing down into law. The way we, like, enforce our shared values is through norms and being like, yo, that's not written in the law exactly like that, but you know that's not cool. Hey, I wanted to bring in the dynamic that you were talking about, about, you know, when you live in an ancient state, you're governed by laws and norms and values.

[00:28:08] And so let's imagine a future state in which agents are running rogue. Who is responsible and who is held accountable in that scenario, right? If you have a gun in the house right now and your minor child uses the gun, the parents are held liable. The gun becomes the agent. The child becomes an actor. And I wonder, are we moving towards a world in which that level of accountability will happen or not?

[00:28:37] And is it up to us to determine that outcome? Yeah, I think it's, like, a very good question. mean, a conversation that's been happening a lot in the last week is that some folks think that we're close to what is being described self-sovereign AI, where it breaks out of containment and a rogue swarm is no longer controlled by any company. And there's just AI out there running, accessing compute, making, and it's like self-sufficient and doing jobs. And there's a real question of like, who should be held accountable for that? Will we be even able to trace who started the rogue swarm? I will say last night I read a piece from these folks that are called AI is normal technology. And there are people that are just like, hey, the company should be held accountable and people should get insurance. And like, we should like use the existing systems of the law, which I think there's like, there's a bunch of nuances to figure out.

[00:29:25] I don't know what the answer is. Right, right. Yeah, I think that's right on the like blockchain side that like the ledger itself keeping the record is very interesting. And we try to like make a lightweight ledger too in our product. But and also to like be like, the hope would be if you think about human collectors, yes, there's rogue nation states and like, you know, pirates and terrorists, but they control not enough economic resources. And they don't have access to the institutions. And so there is a question of like, will we be able to do that in the AI world where like, most of the rogue swarms are contained by the like the major good swarms?

[00:30:08] Could you return to the that slide that sort of said your point of view on how institutions hold together? And it was basically nation states and how they like the mechanisms that align them? Yeah, I think it's this one. Yeah. So one of the things that I would like that was thinking that I was thinking about when I was seeing this, sorry, I'm having a hard time talking with the echo is, you know, you've all her very, a lot of us have read him.

[00:30:42] He would argue that the thing that's missing here is some shared sense of story and history. Totally. Totally. Like sort of like, what is the what is the like historical context in which your population emerged? Definitely. There's a reason why the United States is United States and England was England and that democracy was not invented in England. Right. Yeah. And so there's kind of the missing element of like, what do these people believe and what do they value? What will they sacrifice? What values do they hold above others?

[00:31:12] Yeah. That I just don't know how, like, I don't believe that like mechanisms are the only thing that hold society together. If you look at our society, it's the fact that like people don't believe in some sort of shared values of democracy anymore. No matter how much voting and representation you have, no matter what legal system you have, it won't hold. Right. And so I'm just like, I don't know how you would instill that in an agent that is sort of day zero. It's no older.

[00:31:37] Yeah. Than it is. Or it's no younger than on day one million. Definitely. It has no sort of like, it has none of that contingency. Yeah. There are folks looking at this. We've been chatting with some teams where there's teams doing research on like, how do the stories between the agents evolve? And like, can you like do any study of like, what, how are they like forming that narrative for themselves? So there are folks trying to start to look at this almost like, what's the physics of a story from like start to like something that's stable.

[00:32:04] And hopefully they'll come and present at one of the upcoming events. I can point you to some of the stuff that they're doing. But like, yeah, I think that that is one of the main things to figure out. And like, what keeps them together? Take an absurdist example. The Taliban is going to look at the constitution and do a different thing with it. Then, you know, a soccer mom from the Midwest. And I have nobody, I don't know of anybody talking about how they acquire some sense of value.

[00:32:32] Yeah. And I would say also that the example that you gave of Web3 and blockchain, those don't hold for me because those are trustless systems. Right. That we're presuming self-interest is the guiding principle for why people came together. And that's not why societies generally come together. I agree. That's why I wanted to give both examples because I think they have different mechanisms as well for like what's like keeping them together. And I think that what you're bringing up is like a huge area of study.

[00:33:01] It's like how people right now are looking at alignment at the level of like one agent. But like what is the narrative alignment that like somehow can emerge to like align them. And like I don't know. I hope more people will go and like study that and like look at that story. I think maybe, yeah. We'll do one more and then move on just for the sake of time. Okay. I have a question about open source, which I think has like an interesting role here because it shows up in two ways.

[00:33:30] Both as like as an agent organization, like a sort of a social system. But also actually how one of the reasons AI has advanced so quickly is because of all those Python and Jupyter notebooks that people were, you know, there was a tradition of publishing papers along with working code that was a way for models to improve rapidly.

[00:33:56] so you mentioned that was like something in your talk. wasn't clear to me. It's definitely Eric. I think Eric and Yonan will be focusing on those topics in particular. And we've been like because there is this like real challenge to open source right now. And like how do you like handle all the contributions of AI and how do you like verify them? And we are thinking about like what comes after open source too. So maybe that's a perfect segue to Eric's talk now.