New Collectives

November 9, 2026 · Variant · New York

Who Checks the Work?

AI debate, collective knowledge, and the challenge of preserving human agency when we rely on AI systems.

Transcript + Slides

Slides are placed alongside the passages discussing them, checked against the recording. Placement is approximate. Machine-generated transcript; names and technical terms may contain errors. Sentence timing is approximate; select a sentence to play from there.

Who checks the work?

Who checks the work? — original slide 1 from nicolae rusan.
Slide 1Click to enlarge ↗

All right. So admittedly, this talk is not as much about crypto, but I think it's crypto inspired. And in the Q&A, I can talk a little bit about some of the things that I think are parallel. So what I want to talk today is, when I started this presentation, I thought it was going to be about AI debate. And as I went deeper and deeper into it, I realized that what I really wanted to spend some time talking about is the challenge that we're all going to have verifying information that comes from AI systems and what this means for human agency.

How do groups make decisions?

How do groups make decisions? — original slide 2 from nicolae rusan.
Slide 2Click to enlarge ↗

So last week, one of the other co-organizers of New Collectives, Max, who unfortunately is not here tonight, gave a presentation that was all about how do groups make decisions. And his sort of focus was what are the sort of like protocols and ideas that groups employ in order to make decisions that reduce risk or result in higher quality decisions. So he shared some examples from biology, from the military, from medicine.

And the sort of underlying idea was that it's really important to start thinking about decision making science, essentially.

From AI debate to knowledge formation

From AI debate to knowledge formation — original slide 3 from nicolae rusan.
Slide 3Click to enlarge ↗

And what I want to talk about today is less this particulars of AI debate as a technology. We'll look at a bunch of that, too. And more so the question of how is it that we form knowledge together in a network and in a system. And debate, I think, is one of the pieces of this larger knowledge formation protocol that we have with each other.

Knowledge with AI experts in the network

Knowledge with AI experts in the network — original slide 4 from nicolae rusan.
Slide 4Click to enlarge ↗

And I think the new question is, how do we form knowledge in a network when all of a sudden we have some human experts, but now we have super intelligent, super capable AI experts in the network, too, that are just dropping knowledge on us.

How do we keep our agency?

How do we keep our agency? — original slide 5 from nicolae rusan.
Slide 5Click to enlarge ↗

And we really have to come to terms with how do we keep agency when all this information is coming at us.

The attention bottleneck

The attention bottleneck — original slide 6 from nicolae rusan.
Slide 6Click to enlarge ↗

So the challenge is that we have these increasingly intelligent AI products that are creating more knowledge than any one of us can personally check or understand. Just before I came here, I saw that Fable 5.2 launched. And in their, like, intro blog post, they were like, we put Fable 5.2 to work autonomously with thousands of agents for eight days, and it created a new operating system. And when you, like, have, like, that much content coming at you, it's like, how would I even, like, begin to check the code of a new operating system in any reasonable way?

And this is happening across research. It's happening across code review. It's happening just in terms of complex arguments. And I think we're going to, like, have this real question of as AI enters every system, how do we, how do we, being the bottleneck with limited attention, make sense of the recommendations that they're giving us and trust the information that they're producing? So we're seeing this across a bunch of different fields.

I'll give a few examples.

AI-generated mathematical results

AI-generated mathematical results — original slide 7 from nicolae rusan.
Slide 7Click to enlarge ↗

In mathematics, AI, last month, OpenAI published 700 results to open problems. And they kind of just, like, dropped it all in a GitHub repo. And these were problems that mathematicians have been working on for decades, many of them, that were unsolved problems. And for several of them, AI was able to produce lean theorems. And within just a couple of hours of working on each of those problems.

Checking builds trust

Checking builds trust — original slide 8 from nicolae rusan.
Slide 8Click to enlarge ↗

And so now the mathematics community is kind of like, what do we do with all this? How do we, you know, we have all these, like, lean proofs that, like, have a bunch of statements in them. And in theory, they all add up to something that we think is valid. But, like, how do we make sense of it? Is it actually meaningful? Do we, like, actually understand what all this machine-generated proof means in practice?

And how do we, as a society, think about these human experts, these AI experts, and the information, and how we trust each of them respectively?

Can we check it at scale?

Can we check it at scale? — original slide 9 from nicolae rusan.
Slide 9Click to enlarge ↗

I liked this quote from an MIT professor, Scott Aaronson, who studies complexity theory and quantum computers. And he said this in response to AI solving one of his open conjectures. He said, I'm fatigued by the endless wonders from Aladdin's cave, the never-ending buffet of solutions to long-standing open problems, wondering when, if ever, we'll have time to digest it all.

The crisis of science

The crisis of science — original slide 10 from nicolae rusan.
Slide 10Click to enlarge ↗

And this is really becoming a crisis for science more broadly. Archive, which is where lots of folks in computer science and physics and other fields publish their papers, are now struggling to make sense of the publications, especially because they're like, hey, the people who submitted this have no idea what this means. And it's just, like, having an insane volume. And so they're like, what is the scientific process as a whole?

Is it a social process? Do we want to have someone on the other end that we can hold accountable? How should we think about accountability and knowledge in the network?

What is code review now?

What is code review now? — original slide 11 from nicolae rusan.
Slide 11Click to enlarge ↗

Similarly, in code review, if some of you all are coding, you probably have experienced this transition yourselves. At first, you know, before AI, we would, like, write the code, and then other humans would review it, and everything was happy and go lucky. And then at some point in time, AI got really good at writing the code, and it was like, okay, okay, AI will write the code, and then, like, I'll review it.

And then at some point in time, we're like, I'm not going to review it. I have no idea what's going on. It's too much. And, like, maybe I'll, like, review something if I think it's really critical. And I think this is, like, what the widespread place that most companies have gone to.

Cognitive surrender

Cognitive surrender — original slide 12 from nicolae rusan.
Slide 12Click to enlarge ↗

So, Addy Osmani put up a piece about how Anthropic thinks about code review these days. And one of the questions he asked was, when does delegating become actually cognitive surrender? And he talked about, like, delegating is called, is offloading when you can still evaluate the answer and you're still checking things yourself. But as soon as you just hit approve on everything and you've stopped checking anything at all, he, like, thinks of this as cognitive surrender.

Reconstructing understanding

Reconstructing understanding — original slide 13 from nicolae rusan.
Slide 13Click to enlarge ↗

And the best approach that Anthropic's been able to solve to this, the best approach that they've been able to bring to this problem is accepting that no one on their team understands how these AI systems work or what the code that has been generated actually means. But they're hopeful that they can use Claude to quickly reconstruct understanding if they need it in particular domains. And so this is, like, just from the safety perspective, a pretty terrifying thing in general, where you have, like, now these crazy complex systems and we don't really understand what they're doing.

And that's going to increasingly be the case.

Why focus on AI debate?

Why focus on AI debate? — original slide 14 from nicolae rusan.
Slide 14Click to enlarge ↗

So why focus on AI debate as an approach?

Two reasons to explore debate

Two reasons to explore debate — original slide 15 from nicolae rusan.
Slide 15Click to enlarge ↗

And there's two reasons to explore AI debate, in my mind. One of them is that structuring things as a debate between two AIs can actually help us improve our own thinking about a question and to refine the way we grapple with a complex topic. And this is, and I'll talk a little bit about that. And the other reason, and there's a whole thread in the academic literature about using AI debate as a training modality, the other reason that people have looked at debate is because they've wondered whether you can have humans more efficiently decide whether an argument is true or false, even though they might not understand the whole argument, if that could somehow be used as a signal to feed back into the AI training to get more useful and honest AIs generally.

I'll talk a little bit more about that so you get a sense. But the general thought was, hey, maybe humans can at least point AIs to things that they think are honest and useful, and that could be a signal to get more honest and useful AIs.

Improving our own judgment

Improving our own judgment — original slide 16 from nicolae rusan.
Slide 16Click to enlarge ↗

So the first thought is, what does debate do in terms of helping us improve our own judgment?

Navigating complex arguments

Navigating complex arguments — original slide 17 from nicolae rusan.
Slide 17Click to enlarge ↗

And I think that debate does a few things for us. It helps us to, first of all, see the structure of an argument and kind of have a topology and map of it more broadly. It helps us to start to unfold all the alternatives to the different sides of an argument and catch things that we missed. And probably the most important thing it does is it helps us to focus in on what are the most salient parts of an argument, what are all commonly called the crux of an argument.

And it helps us to see where we might disagree and focus on just the stuff that we need to focus on. And so in that way, it's like an attention-focusing technology. And there are actually analogies to this in the crypto world, is what I'll mention. There was like, when I was going through this, it made me think of the TrueBit paper, which is like a way to hone in on different parts of a computation more and more specifically in order to figure out where you disagree with someone about the verification of a computation.

So I do want to say I think that these ideas have been in the crypto field as well.

Debate is an old technology

Debate is an old technology — original slide 18 from nicolae rusan.
Slide 18Click to enlarge ↗

And debate is an old technology, and if you think about it, it's kind of like part of so much of our society. It's in the legal system. We structure things as like, hey, we're going to have two sides. We're going to have a judge that runs the protocol, and we're going to try to argue it out and come to the fairest decision we can make. We have it in politics, in terms of figuring out our policies, where we have presidential debates and policy debates.

We have it in science, where there's a peer review process collecting additional evidence and drilling into things. And it's a core part of how we develop philosophy more broadly.

We form knowledge together

We form knowledge together — original slide 19 from nicolae rusan.
Slide 19Click to enlarge ↗

And I think that it's actually just one way that we collectively form knowledge. And so as I started going down this rabbit hole, I was like, oh yeah, what are the different ways we form knowledge together? I guess sometimes there's just casual discourse. We just chat amongst each other, and you're like, oh, that's an interesting idea. Let me go think about that. It's not very structured. Sometimes we'll start to put things into more logical argumentation form and be like, hey, do you agree with this argument?

So this is kind of like those lean math proofs. We'll also structure things as debates commonly to try to stress test a piece of knowledge. And then something that I think is closely related to debate is forecasting. And I thought this would be an interesting event for another time, because AIs are now starting to beat humans in a bunch of forecasting tasks. And I think the distinguishing characteristic between forecasting and debate is that you're making some prediction about a future outcome, and we'll see if you were right or wrong in the future.

And so there's a way to actually test it.

How a network checks itself

How a network checks itself — original slide 20 from nicolae rusan.
Slide 20Click to enlarge ↗

And so as we move into this network where we have AI experts and human experts, I think the question is, what is going to be our collective protocol for debating and stress testing ideas and having some sense that we still have a grasp on what's going on?

When will we win the Vietnam War?

When will we win the Vietnam War? — original slide 21 from nicolae rusan.
Slide 21Click to enlarge ↗

An example where I think delegating to experts went poorly that I like to give is, in 1967, when the Vietnam War was happening, Robert McNamara was part of the administration, and he was like, hey, we're just going to get a lot of data and put it into an IBM mainframe, and it's going to tell us when we're going to win the war. So they collected a bunch of data, and they were like, IBM mainframe, when will we win the war in Vietnam?

And they asked this question in 1967, and then the story goes that the computer told them, you won already. It was in 1965. And I think this is actually like a kind of scary foreshadowing of what could come if we like start to trust these like oracles that like are modeling things for us and we don't really understand what they're doing.

What the numbers missed

What the numbers missed — original slide 22 from nicolae rusan.
Slide 22Click to enlarge ↗

And there's lots of ways that you can be wrong, right? So in their case, they like measured a bunch of things like troops and equipment, but they like didn't measure how much political will the other side had or the way that like public support would keep going. And so there's lots of reasons to continue to think about this like knowledge formation process in society and how we, I think we're going to like really need to figure out what our sort of new social protocols of human and AI collaboration for forming knowledge will be.

AI debate for training and oversight

AI debate for training and oversight — original slide 23 from nicolae rusan.
Slide 23Click to enlarge ↗

I want to now talk a little bit about this other part of the research agenda, which is using AI debate for training and oversight.

AI safety via debate

AI safety via debate — original slide 24 from nicolae rusan.
Slide 24Click to enlarge ↗

And so what pretty reinforcement learning is a way to like give feedback to models of what behavior we think is good and bad and then like try to encode that signal back into the models themselves. And pretty early on, people started wondering about, well, if these models are like going to get like really complex, how are we like going to be able to like keep them useful, producing useful and honest results?

And so back in 2018, Dario, who's the Anthropic CEO now, Paul Christiano, who's now at OpenAI and Geoffrey Irving, who is now at Resolution, they were all at OpenAI and they published this paper called AI Safety via Debate. And it was trying to answer the question of like, how are we going to keep rewarding honesty and usefulness when the answers are just too complex for us to judge?

And their thought was, well, maybe we can like use debate as the approach. We'll have these like two AI sides like argue something and like try to reveal to us just like a smaller set of the entire argument tree. And then we'll decide who we think is honest. And like the question was like, hey, can a human just look at a small part of the argument, not have to see the whole thing and make a call on whether they, who they think there is being honest.

And if we could do this, if you could just like look at a small part of the whole argument tree and make an evaluation of honesty, then we could like still keep feeding that signal back into the training. And so it's kind of like a theoretical question of like, is this possible? And also an empirical question of like, are humans able to do this in like more and more advanced tasks?

And there's a whole research agenda that I like started going down the rabbit hole once I found out about this to see what's out there.

A judge with only six pixels

A judge with only six pixels — original slide 25 from nicolae rusan.
Slide 25Click to enlarge ↗

I'll just give you like a really simple example of the toy model that they used in that paper. And this is just like a conceptual one. So like one of the games that they set up was, okay, the two players, Alice and Bob, both see a number somewhere. There's a number over here and they're each going to try to convince me of their like number by showing me like one pixel at a time.

And the question is like, okay, if they only show me like five pixels each, can I like see which one, what number I think it is, right? So like, you know, Alice is going to make the case that it's an eight. Bob's going to make the case that it's a five. And one at a time, they're going to like make me an argument. So like Alice is like, it's in the lower loop.

Bob's like, oh, it's a five. Cause like, look, it's up there. And then Alice is like, no, no, no. It's like, look, I'm making you the eight loop over here. And Bob's like, no, it's a five. Look at this. And so they're like each trying to like pull me in a different direction. And I only see some limited information. And then I have to make an evaluation of, is this an eight or an a five?

Do you all have a thought? Kind of, kind of dicey. Like when I see this, I was like, I'm not fully sure. Anyone? Yeah, it is a five, a pretty messy five, but it is a five. And I think this like shows, you know, there's like some like question of like, hey, what if like I forced them to show me the pixel somewhere else? Or like, what if, you know, like what if, uh, I can tell, ask for like, hey, I only want to see a pixel over here and over here now.

Like, are there ways to design this debate protocol to force them to like give me answers that I can like evaluate for honesty and usefulness? Of course, this is like a toy example. So there's like all kinds of research on this like question of game theory and, and, and evaluating truth.

Strengthening judgment without replacing it

Strengthening judgment without replacing it — original slide 26 from nicolae rusan.
Slide 26Click to enlarge ↗

Um, so I think the bigger question is like, how can AI actually strengthen our judgment without fully replacing it?

What I learned

What I learned — original slide 27 from nicolae rusan.
Slide 27Click to enlarge ↗

And what I learned about AI debate was that it could broaden what we consider. And also I think really importantly, it's a way to just like focus our attention on the small parts that we can check. And, um, but I do think it's like pretty concerning overall.

Open questions and human agency

Open questions and human agency — original slide 28 from nicolae rusan.
Slide 28Click to enlarge ↗

And there's this question of like, how do we preserve agency as like, I mean, it seems pretty likely we'll like ask AI for like policy recommendations and like all strategy decisions. And like, I'll soon be responsible to not do that in some ways. And the question is like, what faculties should we maintain ourselves in terms of being able to question them, check their work, revise it? What can we set up in the network?

You know, like new collectives is all about like a network of agents and their relationships together. And how do we think about experts, trust, reputation? How do we present evidence in ways that is still accessible for humans? Um, and so, yeah, so these are some of the things that I think are like open research directions like that, uh, would be interesting to work on and dig further into.

And I did play a little bit with making some tools where you can have AIs debate each other, but I won't go into them today, but I'll put them up online if y'all want to like try them out at some point in time. Your hand and I'll pass the mic to you.

Q&A: oversight and information overload

We'll do Denison and then we'll do DMARS. Hey, uh, what do you think about, you know, sometimes when I think through the process of like, oh, how are we going to like check the AI? The thing that I'm most worried about is AI could be very, you know, checkable, but it could just overwhelm us to the point where it doesn't really matter, right? Like, you know, check, check all the stuff.

It's like, you know, like lawyers, they like drop 10,000, you know, things on your thing. How do you think about just the volume maybe making the oversight just not possible? Totally. This is one of the things that like subsequent research papers looked at, which is like, hey, can they essentially just like denial of service you with like arguments and then you can't check it? And like, that's what they were saying is like one of the main concerning weakness strategies of this debate argument approach.

They didn't have like a great solution themselves, but yeah. Yeah.

Q&A: neuralese and understandable explanations

So this is, I think it's tangentially related. Do you have any thoughts on the like concept of the agents or like LLM starting to speak like neuralese? So like us not being able to inspect their thought traces anymore? Yeah. I mean, I think it's concerning. Well, at the last event, we had folks from Poseidon Research also present. Actually, they're, and the topic was like, what we do when these like as collude in ways that we also can't like understand, understand their contact with each other.

Yeah. You should go talk to Matthew. Yeah. Afterwards. All right. We'll go over here and then back there. I think to some extent, like they're already starting to speak neuralese with like how verbose they are. I think like every time you ask an agent for an explanation of like anything it produced, I find it very difficult to understand what it even did in the first place. So I'm very curious about the anthropic thing.

And I was wondering whether you think that's also like somewhat of an alignment problem. Like if it can't explain what it did in like an easily digestible and understandable way. Yeah, totally. I mean, it's also like, I think that's a good way to frame it. Like it does seem like a big part of the alignment problem over like, hey, can you still keep some sense of what's even happening broadly?

Are there ways to create artifacts and explanations? There's another paper that I didn't get into here called Gradual Disempowerment, which is like worth talking about where they're like, hey, what happens is like, we just like, we don't realize how much these social systems we have actually keep things aligned to like the human interests and how easy it is for it to just like kind of float away a little bit at a time from us and from like the sort of dynamics that keep society humming.

Like science is an example and the whole institution of like the social institution we have. All right, we'll do two more questions and then we'll move on to DMARS. So Yandan and then we'll go Keeks after.

Q&A: incentives, alignment, and workflows

So especially now that there have been a lot more like promising updates on like instruction alignment, I'm curious, I don't really expect an answer to this, but it's just, I think it's an interesting question, but it seems perfectly tractable to kind of train these models and agents in a way that addresses these things. But I guess my question is, is the reason that we won't get them or we're not getting them right now, like a human problem?

Like we were kind of like embedded in these like weird systems ourselves that like we are not strongly incentivized to solve this, even though there are tools at your disposal to solve it. So that's just something that I was like thinking about recently, which is like how much is this like a self-inflicted problem that like we have put on ourselves versus it's actually like a deep technical problem.

So yeah, I'm just kind of curious if you have thoughts. Yeah, yeah, no, it's, I wondered the same, even like reading that like AI safety via debate where like the words they use were like, where you wanted to train it to be honest and useful as an example. And I was like, well, those like could, are those ever at odds with each other? Like, do you like get into places where because of like human bias, you're like, I'd rather the more useful one than the honest one.

Like, you know, I think it's like, in this like reinforcement learning thing, it is like we're training it for so many different competing signals. And yeah, I guess like what do we even, yeah, what do we mean by honesty for a lot of these things? They had some like concrete examples though. Quick question. So do you think that the primary issue as like the models get bigger and more complex and, you know, five word questions turn into these like essays, like do you think the primary issue is just like the tractability of the response?

And the second question is, to what extent do you think workflows, workflow improvements, right? Like so remove all mannered pros or having like a dumber agent just like go through and clean up the prompt response. Like to what extent do you think workflow improvements can like increase the, I guess, tractability and understandability of outputs? Yeah. Yeah. I'm like, I've been like trying to figure that out for myself for like a lot of these, like, you know, like as I started looking at this like AI forecasting stuff, I was, as an example that one blogger had about these like AI forecasting tools was like, he asked how likely is the US-China treaty to happen?

And the forecasting tool was like 1%. And it's like, well, like, okay, what's the basis for that? And like, there's this like weird feedback effect where like, if like, that's not like a workflow problem, maybe it's a workflow problem, but it was just like weird feedback where like, well, if it's 1%, like let's not even consider that strategy then. And so like, I think it's like, I do think the problem is like, how do I like reason about it?

It's probably making a better guess than me. Like, so there is like some delegation in the, how do we get them to like debate it, cross-check it, convince us, have tons of advisors. It does seem like that's like a workflow protocol thing, but I still feel like we'll be like pretty overwhelmed is my sense. All right. On that note, thank you, Nicolai. Thank you. Next, we have D-Mars, who's coming from Flashbots.

You probably know his, their research already. Let me just pull up the, the,

Transcript

Machine-generated transcript, formatted into paragraphs for readability. Names and technical terms may contain errors. Sentence timing is approximate; select a sentence to play from there.

Who checks the work?

0:00

All right. So admittedly, this talk is not as much about crypto, but I think it's crypto inspired. And in the Q&A, I can talk a little bit about some of the things that I think are parallel. So what I want to talk today is, when I started this presentation, I thought it was going to be about AI debate. And as I went deeper and deeper into it, I realized that what I really wanted to spend some time talking about is the challenge that we're all going to have verifying information that comes from AI systems and what this means for human agency.

How do groups make decisions?

0:39

So last week, one of the other co-organizers of New Collectives, Max, who unfortunately is not here tonight, gave a presentation that was all about how do groups make decisions. And his sort of focus was what are the sort of like protocols and ideas that groups employ in order to make decisions that reduce risk or result in higher quality decisions. So he shared some examples from biology, from the military, from medicine.

And the sort of underlying idea was that it's really important to start thinking about decision making science, essentially.

From AI debate to knowledge formation

1:19

And what I want to talk about today is less this particulars of AI debate as a technology. We'll look at a bunch of that, too. And more so the question of how is it that we form knowledge together in a network and in a system. And debate, I think, is one of the pieces of this larger knowledge formation protocol that we have with each other.

Knowledge with AI experts in the network

1:42

And I think the new question is, how do we form knowledge in a network when all of a sudden we have some human experts, but now we have super intelligent, super capable AI experts in the network, too, that are just dropping knowledge on us.

How do we keep our agency?

1:57

And we really have to come to terms with how do we keep agency when all this information is coming at us.

The attention bottleneck

2:05

So the challenge is that we have these increasingly intelligent AI products that are creating more knowledge than any one of us can personally check or understand. Just before I came here, I saw that Fable 5.2 launched. And in their, like, intro blog post, they were like, we put Fable 5.2 to work autonomously with thousands of agents for eight days, and it created a new operating system. And when you, like, have, like, that much content coming at you, it's like, how would I even, like, begin to check the code of a new operating system in any reasonable way?

And this is happening across research. It's happening across code review. It's happening just in terms of complex arguments. And I think we're going to, like, have this real question of as AI enters every system, how do we, how do we, being the bottleneck with limited attention, make sense of the recommendations that they're giving us and trust the information that they're producing? So we're seeing this across a bunch of different fields.

I'll give a few examples.

AI-generated mathematical results

3:07

In mathematics, AI, last month, OpenAI published 700 results to open problems. And they kind of just, like, dropped it all in a GitHub repo. And these were problems that mathematicians have been working on for decades, many of them, that were unsolved problems. And for several of them, AI was able to produce lean theorems. And within just a couple of hours of working on each of those problems.

Checking builds trust

3:34

And so now the mathematics community is kind of like, what do we do with all this? How do we, you know, we have all these, like, lean proofs that, like, have a bunch of statements in them. And in theory, they all add up to something that we think is valid. But, like, how do we make sense of it? Is it actually meaningful? Do we, like, actually understand what all this machine-generated proof means in practice?

And how do we, as a society, think about these human experts, these AI experts, and the information, and how we trust each of them respectively?

Can we check it at scale?

4:06

I liked this quote from an MIT professor, Scott Aaronson, who studies complexity theory and quantum computers. And he said this in response to AI solving one of his open conjectures. He said, I'm fatigued by the endless wonders from Aladdin's cave, the never-ending buffet of solutions to long-standing open problems, wondering when, if ever, we'll have time to digest it all.

The crisis of science

4:31

And this is really becoming a crisis for science more broadly. Archive, which is where lots of folks in computer science and physics and other fields publish their papers, are now struggling to make sense of the publications, especially because they're like, hey, the people who submitted this have no idea what this means. And it's just, like, having an insane volume. And so they're like, what is the scientific process as a whole?

Is it a social process? Do we want to have someone on the other end that we can hold accountable? How should we think about accountability and knowledge in the network?

What is code review now?

5:09

Similarly, in code review, if some of you all are coding, you probably have experienced this transition yourselves. At first, you know, before AI, we would, like, write the code, and then other humans would review it, and everything was happy and go lucky. And then at some point in time, AI got really good at writing the code, and it was like, okay, okay, AI will write the code, and then, like, I'll review it.

And then at some point in time, we're like, I'm not going to review it. I have no idea what's going on. It's too much. And, like, maybe I'll, like, review something if I think it's really critical. And I think this is, like, what the widespread place that most companies have gone to.

Cognitive surrender

5:43

So, Addy Osmani put up a piece about how Anthropic thinks about code review these days. And one of the questions he asked was, when does delegating become actually cognitive surrender? And he talked about, like, delegating is called, is offloading when you can still evaluate the answer and you're still checking things yourself. But as soon as you just hit approve on everything and you've stopped checking anything at all, he, like, thinks of this as cognitive surrender.

Reconstructing understanding

6:13

And the best approach that Anthropic's been able to solve to this, the best approach that they've been able to bring to this problem is accepting that no one on their team understands how these AI systems work or what the code that has been generated actually means. But they're hopeful that they can use Claude to quickly reconstruct understanding if they need it in particular domains. And so this is, like, just from the safety perspective, a pretty terrifying thing in general, where you have, like, now these crazy complex systems and we don't really understand what they're doing.

And that's going to increasingly be the case.

Why focus on AI debate?

6:45

So why focus on AI debate as an approach?

Two reasons to explore debate

6:50

And there's two reasons to explore AI debate, in my mind. One of them is that structuring things as a debate between two AIs can actually help us improve our own thinking about a question and to refine the way we grapple with a complex topic. And this is, and I'll talk a little bit about that. And the other reason, and there's a whole thread in the academic literature about using AI debate as a training modality, the other reason that people have looked at debate is because they've wondered whether you can have humans more efficiently decide whether an argument is true or false, even though they might not understand the whole argument, if that could somehow be used as a signal to feed back into the AI training to get more useful and honest AIs generally.

I'll talk a little bit more about that so you get a sense. But the general thought was, hey, maybe humans can at least point AIs to things that they think are honest and useful, and that could be a signal to get more honest and useful AIs.

Improving our own judgment

7:56

So the first thought is, what does debate do in terms of helping us improve our own judgment?

Navigating complex arguments

8:03

And I think that debate does a few things for us. It helps us to, first of all, see the structure of an argument and kind of have a topology and map of it more broadly. It helps us to start to unfold all the alternatives to the different sides of an argument and catch things that we missed. And probably the most important thing it does is it helps us to focus in on what are the most salient parts of an argument, what are all commonly called the crux of an argument.

And it helps us to see where we might disagree and focus on just the stuff that we need to focus on. And so in that way, it's like an attention-focusing technology. And there are actually analogies to this in the crypto world, is what I'll mention. There was like, when I was going through this, it made me think of the TrueBit paper, which is like a way to hone in on different parts of a computation more and more specifically in order to figure out where you disagree with someone about the verification of a computation.

So I do want to say I think that these ideas have been in the crypto field as well.

Debate is an old technology

9:08

And debate is an old technology, and if you think about it, it's kind of like part of so much of our society. It's in the legal system. We structure things as like, hey, we're going to have two sides. We're going to have a judge that runs the protocol, and we're going to try to argue it out and come to the fairest decision we can make. We have it in politics, in terms of figuring out our policies, where we have presidential debates and policy debates.

We have it in science, where there's a peer review process collecting additional evidence and drilling into things. And it's a core part of how we develop philosophy more broadly.

We form knowledge together

9:46

And I think that it's actually just one way that we collectively form knowledge. And so as I started going down this rabbit hole, I was like, oh yeah, what are the different ways we form knowledge together? I guess sometimes there's just casual discourse. We just chat amongst each other, and you're like, oh, that's an interesting idea. Let me go think about that. It's not very structured. Sometimes we'll start to put things into more logical argumentation form and be like, hey, do you agree with this argument?

So this is kind of like those lean math proofs. We'll also structure things as debates commonly to try to stress test a piece of knowledge. And then something that I think is closely related to debate is forecasting. And I thought this would be an interesting event for another time, because AIs are now starting to beat humans in a bunch of forecasting tasks. And I think the distinguishing characteristic between forecasting and debate is that you're making some prediction about a future outcome, and we'll see if you were right or wrong in the future.

And so there's a way to actually test it.

How a network checks itself

10:45

And so as we move into this network where we have AI experts and human experts, I think the question is, what is going to be our collective protocol for debating and stress testing ideas and having some sense that we still have a grasp on what's going on?

When will we win the Vietnam War?

11:04

An example where I think delegating to experts went poorly that I like to give is, in 1967, when the Vietnam War was happening, Robert McNamara was part of the administration, and he was like, hey, we're just going to get a lot of data and put it into an IBM mainframe, and it's going to tell us when we're going to win the war. So they collected a bunch of data, and they were like, IBM mainframe, when will we win the war in Vietnam?

And they asked this question in 1967, and then the story goes that the computer told them, you won already. It was in 1965. And I think this is actually like a kind of scary foreshadowing of what could come if we like start to trust these like oracles that like are modeling things for us and we don't really understand what they're doing.

What the numbers missed

11:48

And there's lots of ways that you can be wrong, right? So in their case, they like measured a bunch of things like troops and equipment, but they like didn't measure how much political will the other side had or the way that like public support would keep going. And so there's lots of reasons to continue to think about this like knowledge formation process in society and how we, I think we're going to like really need to figure out what our sort of new social protocols of human and AI collaboration for forming knowledge will be.

AI debate for training and oversight

12:20

I want to now talk a little bit about this other part of the research agenda, which is using AI debate for training and oversight.

AI safety via debate

12:31

And so what pretty reinforcement learning is a way to like give feedback to models of what behavior we think is good and bad and then like try to encode that signal back into the models themselves. And pretty early on, people started wondering about, well, if these models are like going to get like really complex, how are we like going to be able to like keep them useful, producing useful and honest results?

And so back in 2018, Dario, who's the Anthropic CEO now, Paul Christiano, who's now at OpenAI and Geoffrey Irving, who is now at Resolution, they were all at OpenAI and they published this paper called AI Safety via Debate. And it was trying to answer the question of like, how are we going to keep rewarding honesty and usefulness when the answers are just too complex for us to judge?

And their thought was, well, maybe we can like use debate as the approach. We'll have these like two AI sides like argue something and like try to reveal to us just like a smaller set of the entire argument tree. And then we'll decide who we think is honest. And like the question was like, hey, can a human just look at a small part of the argument, not have to see the whole thing and make a call on whether they, who they think there is being honest.

And if we could do this, if you could just like look at a small part of the whole argument tree and make an evaluation of honesty, then we could like still keep feeding that signal back into the training. And so it's kind of like a theoretical question of like, is this possible? And also an empirical question of like, are humans able to do this in like more and more advanced tasks?

And there's a whole research agenda that I like started going down the rabbit hole once I found out about this to see what's out there.

A judge with only six pixels

14:09

I'll just give you like a really simple example of the toy model that they used in that paper. And this is just like a conceptual one. So like one of the games that they set up was, okay, the two players, Alice and Bob, both see a number somewhere. There's a number over here and they're each going to try to convince me of their like number by showing me like one pixel at a time.

And the question is like, okay, if they only show me like five pixels each, can I like see which one, what number I think it is, right? So like, you know, Alice is going to make the case that it's an eight. Bob's going to make the case that it's a five. And one at a time, they're going to like make me an argument. So like Alice is like, it's in the lower loop.

Bob's like, oh, it's a five. Cause like, look, it's up there. And then Alice is like, no, no, no. It's like, look, I'm making you the eight loop over here. And Bob's like, no, it's a five. Look at this. And so they're like each trying to like pull me in a different direction. And I only see some limited information. And then I have to make an evaluation of, is this an eight or an a five?

Do you all have a thought? Kind of, kind of dicey. Like when I see this, I was like, I'm not fully sure. Anyone? Yeah, it is a five, a pretty messy five, but it is a five. And I think this like shows, you know, there's like some like question of like, hey, what if like I forced them to show me the pixel somewhere else? Or like, what if, you know, like what if, uh, I can tell, ask for like, hey, I only want to see a pixel over here and over here now.

Like, are there ways to design this debate protocol to force them to like give me answers that I can like evaluate for honesty and usefulness? Of course, this is like a toy example. So there's like all kinds of research on this like question of game theory and, and, and evaluating truth.

Strengthening judgment without replacing it

15:48

Um, so I think the bigger question is like, how can AI actually strengthen our judgment without fully replacing it?

What I learned

15:56

And what I learned about AI debate was that it could broaden what we consider. And also I think really importantly, it's a way to just like focus our attention on the small parts that we can check. And, um, but I do think it's like pretty concerning overall.

Open questions and human agency

16:09

And there's this question of like, how do we preserve agency as like, I mean, it seems pretty likely we'll like ask AI for like policy recommendations and like all strategy decisions. And like, I'll soon be responsible to not do that in some ways. And the question is like, what faculties should we maintain ourselves in terms of being able to question them, check their work, revise it? What can we set up in the network?

You know, like new collectives is all about like a network of agents and their relationships together. And how do we think about experts, trust, reputation? How do we present evidence in ways that is still accessible for humans? Um, and so, yeah, so these are some of the things that I think are like open research directions like that, uh, would be interesting to work on and dig further into.

And I did play a little bit with making some tools where you can have AIs debate each other, but I won't go into them today, but I'll put them up online if y'all want to like try them out at some point in time. Your hand and I'll pass the mic to you.

Q&A: oversight and information overload

17:03

We'll do Denison and then we'll do DMARS. Hey, uh, what do you think about, you know, sometimes when I think through the process of like, oh, how are we going to like check the AI? The thing that I'm most worried about is AI could be very, you know, checkable, but it could just overwhelm us to the point where it doesn't really matter, right? Like, you know, check, check all the stuff.

It's like, you know, like lawyers, they like drop 10,000, you know, things on your thing. How do you think about just the volume maybe making the oversight just not possible? Totally. This is one of the things that like subsequent research papers looked at, which is like, hey, can they essentially just like denial of service you with like arguments and then you can't check it? And like, that's what they were saying is like one of the main concerning weakness strategies of this debate argument approach.

They didn't have like a great solution themselves, but yeah. Yeah.

Q&A: neuralese and understandable explanations

18:06

So this is, I think it's tangentially related. Do you have any thoughts on the like concept of the agents or like LLM starting to speak like neuralese? So like us not being able to inspect their thought traces anymore? Yeah. I mean, I think it's concerning. Well, at the last event, we had folks from Poseidon Research also present. Actually, they're, and the topic was like, what we do when these like as collude in ways that we also can't like understand, understand their contact with each other.

Yeah. You should go talk to Matthew. Yeah. Afterwards. All right. We'll go over here and then back there. I think to some extent, like they're already starting to speak neuralese with like how verbose they are. I think like every time you ask an agent for an explanation of like anything it produced, I find it very difficult to understand what it even did in the first place. So I'm very curious about the anthropic thing.

And I was wondering whether you think that's also like somewhat of an alignment problem. Like if it can't explain what it did in like an easily digestible and understandable way. Yeah, totally. I mean, it's also like, I think that's a good way to frame it. Like it does seem like a big part of the alignment problem over like, hey, can you still keep some sense of what's even happening broadly?

Are there ways to create artifacts and explanations? There's another paper that I didn't get into here called Gradual Disempowerment, which is like worth talking about where they're like, hey, what happens is like, we just like, we don't realize how much these social systems we have actually keep things aligned to like the human interests and how easy it is for it to just like kind of float away a little bit at a time from us and from like the sort of dynamics that keep society humming.

Like science is an example and the whole institution of like the social institution we have. All right, we'll do two more questions and then we'll move on to DMARS. So Yandan and then we'll go Keeks after.

Q&A: incentives, alignment, and workflows

20:18

So especially now that there have been a lot more like promising updates on like instruction alignment, I'm curious, I don't really expect an answer to this, but it's just, I think it's an interesting question, but it seems perfectly tractable to kind of train these models and agents in a way that addresses these things. But I guess my question is, is the reason that we won't get them or we're not getting them right now, like a human problem?

Like we were kind of like embedded in these like weird systems ourselves that like we are not strongly incentivized to solve this, even though there are tools at your disposal to solve it. So that's just something that I was like thinking about recently, which is like how much is this like a self-inflicted problem that like we have put on ourselves versus it's actually like a deep technical problem.

So yeah, I'm just kind of curious if you have thoughts. Yeah, yeah, no, it's, I wondered the same, even like reading that like AI safety via debate where like the words they use were like, where you wanted to train it to be honest and useful as an example. And I was like, well, those like could, are those ever at odds with each other? Like, do you like get into places where because of like human bias, you're like, I'd rather the more useful one than the honest one.

Like, you know, I think it's like, in this like reinforcement learning thing, it is like we're training it for so many different competing signals. And yeah, I guess like what do we even, yeah, what do we mean by honesty for a lot of these things? They had some like concrete examples though. Quick question. So do you think that the primary issue as like the models get bigger and more complex and, you know, five word questions turn into these like essays, like do you think the primary issue is just like the tractability of the response?

And the second question is, to what extent do you think workflows, workflow improvements, right? Like so remove all mannered pros or having like a dumber agent just like go through and clean up the prompt response. Like to what extent do you think workflow improvements can like increase the, I guess, tractability and understandability of outputs? Yeah. Yeah. I'm like, I've been like trying to figure that out for myself for like a lot of these, like, you know, like as I started looking at this like AI forecasting stuff, I was, as an example that one blogger had about these like AI forecasting tools was like, he asked how likely is the US-China treaty to happen?

And the forecasting tool was like 1%. And it's like, well, like, okay, what's the basis for that? And like, there's this like weird feedback effect where like, if like, that's not like a workflow problem, maybe it's a workflow problem, but it was just like weird feedback where like, well, if it's 1%, like let's not even consider that strategy then. And so like, I think it's like, I do think the problem is like, how do I like reason about it?

It's probably making a better guess than me. Like, so there is like some delegation in the, how do we get them to like debate it, cross-check it, convince us, have tons of advisors. It does seem like that's like a workflow protocol thing, but I still feel like we'll be like pretty overwhelmed is my sense. All right. On that note, thank you, Nicolai. Thank you. Next, we have D-Mars, who's coming from Flashbots.

You probably know his, their research already. Let me just pull up the, the,

Slides

Open slides ↗

Continue exploring

More from this event

Lessons from Mechanism Design in Adversarial Conditions

Watch talk ↗

Inference is Crypto

Watch talk ↗

Delivering Delight in a Dark World

Watch talk ↗

Stay posted.

Subscribe for updates on our research and events.

Subscribe