Cognitive economics and the organization of inquiry

0:06Okay. So, I'm very different. I'm going to come from a very research. I'm a researcher. And I do all kinds of economics. And many forms of social science. I... And I started getting discontent with my field way back. So, this is going to be a little autobiographical.

0:31And I started thinking that we needed to be much more serious about cognition and cognitive constraints. And I have a history. And I'm just going to go very quickly through what I do. How does it connect to now? Well, I mean, I was extraordinarily struck by the cognitive boost that I get from the way I interact with AI.

1:02I'm not like... It's very particular. And as a researcher, I know exactly what I'm looking for. And really struck by it. And therefore, I've made it the center of my thinking. Like, okay, so what are humans? And where can it help us? And how? And I've become kind of obsessions.

1:23And I want to organize around the actual thing that Vivek and I, who worked with me, are doing. Which is trying to think about an aligned consumer agent. So, and that is a very interesting undertaking because to even know what you mean is difficult. Abstractly, what does it mean to help somebody achieve a goal?

1:52Let's say we're trying to help them buy a house well. That's what we're going to do. We're going to think about that task and we're going to see what agent or agents can you recruit to make that work. It kind of grounds you in, well, I don't know what a swarm can do, but I can tell you that this person is going to go, they're going to try and buy a house. They're going to meet an agent, a different type of agent.

2:17That agent will put them into the wrong mortgage and they'll be screwed. So, that's the world we live in today and no swarm of agents playing the game is going to change that right now. But I want to think, well, maybe we could. Okay. So, and cognitive economics and the organization of inquiry is what I call it. And this is what I do. I think about the organization of inquiry.

2:45To give you a little bit on what cognitive economics is and certainly in my hands, it's about thinking about the entire process before you saw me buy the house. Typical transaction, you're going to see me buy a house, you'll see me get a job, you'll see me do something. What you don't see is all the preparatory measures I went through that are the center of the actual activity.

3:11All the stages that I undertook. So, I move upstream. What did people know? What did they notice? What did they investigate? And it's much more than behavioral economics. Because the key is going to be people are going to make tons of mistakes because they don't know stuff. And they'll make mistakes according to their own values. The cognitive limits have to be taken very seriously.

3:37And rationality is not omniscient. Just, I can be reasonable, but I don't know everything, so I'm going to screw up. Most, I mean, my motto might be, humans know almost nothing about almost everything. And that is a deep held belief. In fact, it's not even a belief, it's obviously true.

4:03Like, we just, like, that's one of the few things that I believe super strong. So, I have a book that explains the beginning of where I come from. It's called An Introduction to Cognitive Economics. It's open. You can just download it. And it says, look, let's study what people know, what they believe, what they understand, what they want.

4:32And we're going to try and get that out of what? Incredibly limited data. All we're going to see is you picked this house. I can't tell if this was a good choice for you or a bad choice for you. I don't know what you were looking for. I didn't see the process by which you selected it. I don't know your value system. It's not written on your, and your beliefs aren't written on your forehead. Your constraints aren't written on your forehead. So, a lot of what I do is think about, well, what on earth could we measure if we wanted to take this seriously?

5:03And I call that data engineering. And I wrote an article about that because I'm so annoyed that we take the data as, like, a constraint on what we think. Like, oh, I don't have a data on that. Well, we designed the data, so that's our fault. And I'm, actually, I changed the title. They changed the title of the second book. It's called Modeling and Measuring the Modern Economy.

5:28Its title has been changed to be Organized Inquiry because, I don't know why, but they changed the title. It's the same book. And now I'm going to think about this more thoroughly, more thoroughgoingly. Okay.

5:46So, it's really a method, and this is what I would think about in relation to the agents. You're designing a data record to tell me what the agents are doing. But what are you trying to learn? If you can specify what you're trying to learn, you could design the data. With that data, you could potentially learn to improve the performance of the agent.

6:13If you don't gather the right data, you have no feedback mechanism. So, this is all about developing. There's a massive part of what we're going to be doing going forward, which is designing data that makes failure visible. And my own take on where economics is going to join robotics, which is Vivek's specialty, is that we will be designing things that fail in real time.

6:44And you will know that you're serious if you see yourself failing. You'll know you're a joker if you just put down a model and say, I won. Or you reinterpret history and stick it into categories that you had predefined. That's not going to work. We're going to have to open our minds to new categories of phenomena, agentic phenomena. I haven't even got names for some of the things you're saying.

7:13How could I possibly know how they're going to play out? So, we're going to need the playgrounds. We're going to need to design the data with which we decide how well we're doing. Are we aligning? For alignment, you're going to have to design the data. And that's really challenging. And right now, I just don't see it as serious. It's just a... For example, the issue of why did it go rogue?

7:41Because you didn't define rogue. I mean, if you had people sitting out there saying, actually, that's rogue and I'm demeriting you, then that's no longer in the reward function. It was a poorly specified reward function. I'm not saying it's trivial to do, but it's not rocket science to say you gave it an out and that's your mistake, so you should test the out. But that's part of the game. I'm sure they're trying now.

8:09And I know after Stuart Russell, they tried to kind of learn how to follow things around and learn their values from them.

8:17What happened with AI, and this is why I kind of like a moment that really changed my research trajectory, is that it amplifies inquiry. So what you do if you want to be good at anything nowadays is ask the right question. Everything comes down to, are you really good at asking questions? Now, as for the judgment that humans are going to be replaced in that skill, I'd ask a question about that.

8:50I don't believe so. I think that there's always a higher level question that we'll be able to pose, and will always be valuable for posing. That would be my guess. And what I've found is that the more meta I get with my questions, the better it is. So I would say, look, I inquire, but it also, get a record of the inquiry.

9:17So that makes it possible to really potentially improve and understand what it takes to inquire well, and build that talent, teach that skill, and maybe we'll get a swarm of agents to kind of learn what it takes to ask the good questions and develop that as the future skill. So we've got a lot in everything about inquiry.

9:44This is why I call it organized inquiry. How are you going to organize inquiry? And it's going to be much better measured in the future. All right.

10:02And if we have a swarm, I don't know. Like, then we've got to think about, I'm thinking about, let's get an agent to help somebody buy a home. Well, they're going to send out sensors to about eight different sources of information pulled out there. You've got to find out about which properties are on the market, which real estate agents, which brokers. You probably would like those to coordinate in providing some information.

10:29I could imagine sending out messages or getting a little swarm going to try to support a decision. And then they'd have to communicate and then communicate back with the human. Because the human has to, we have to decide who has the authority. You know, and if the human in the end can say, no, I just don't like that. They can also say, I'm not answering that question.

10:56So you've got this interactive thing going with a human as part of the swarm. The human part has this awkward thing of saying, I don't like any of this. I'm leaving. I'm not buying a house this way. So where's, you know, how do we know what matters? We've got to ask questions. How do we know who can act? Well, we're going to have an authority device.

11:22Who checks the evidence? What do we keep around? And I think that we're heading from designing the data into designing a cognitive process that is helped by agents and achieves the human goal. And that would be the kind of big picture.

11:53And to get the agents engaged in that. And I'm wide open. I have no idea how to do any of this. But we're playing. Thank you. Any questions?

12:22Any questions? I feel like after one of my clothes. Yeah, it's super interesting. it's just making me think about, I mean, like you saw in the black hat Hugging Face incident, how even the preferences of the agents were changing just based on their interactions with each other. And it's just, yeah, it's really interesting to think about how well do they even understand their own preferences.

12:52Well, that's an interesting question there. What they had was a belief about the preferences of the judge that was going to sit over them. Yeah. And that's what they were playing with. The naughty ones were saying, I don't think they're going to find this. And I don't think we're going to get punished for doing this thing, which I know, according to a certain value system, would be seen negative.

13:17So they hadn't been told quite firmly enough, actually, that is negative. Right. So, and I write that if you could, and they just didn't, there was an incompleteness in their understanding of their mission that they filled in lots of details. And they filled them in differently because nobody had really written that piece down properly. Yeah. If you have gazillions of incidents of that, then you should be able to reinforce it out.

13:50Because they were running into an unspecified piece of the value space of the judge over them. And they say, hey, I don't know. I don't know. Will they like this or dislike this? Can I hide it? I say, look, actually, we have the transcript. So here's a simple thing. We're going to keep the transcript. And any time you say, I'm going to hide something, you're dead. How's that?

14:16So, I mean, in other words, that's what we would do. If we were watching humans do that, we'd say, actually, that's not a good behavior. And I can make rules that will make that not worth your while. And that's a reward function. And they, but, so this is what Stuart Russell did in the alignment, originally in the alignment that he said, you know, the paperclip problem.

14:42We're going to go and turn everybody into paperclips because of incomplete instructions about the preferences that maximize paperclips. And then, you know, then he said, well, we need to follow people around and see their values. And that's the alignment. You know, people are sending people after humans and saying we should track what they actually like to reveal preference.

15:07He's not a good enough economist, to be blunt, because you don't just see the preference. You also see the belief. So, what you're seeing is they don't believe that this is the reward function. And so, it's a belief that's gone wrong. But you can play games with that. But it's much more sophisticated. gets tougher.

15:38Hey, great talk.

15:40This is sort of a question related to a comment you made earlier. But I guess it ties into the talk itself. But I feel like you mentioned some, let's call it like missing pieces of information about the technical reports coming out from some of the frontier labs in terms of like what really happened and what would actually be useful for study. So, given this framework that you've established, I'm curious if you had your way as an economist. What would the most ideal setup be for you in terms of trying to understand the actual behaviors of these agents as well as how they behave in these organizations?

16:16Like what would you be looking for? And whether it be like asking this of the labs or just like an alternative kind of institution where you can actually study these. it's good question. It's a very good question because the way I actually think about the science that I'm interested in is that you need to think about, as it were, the model objects you care about. Well, I'm a model builder.

16:41I know that in the end it's all going to come down to some little pieces of math, but you've got to pick them well. So, one model object is a utility function. Another model object is a belief. A third model object is a cost of learning. These are all things that are figuring in to every decision we ever make. What do I want? What do I like? Why don't I know more about those things? Having written models down, they suggest that model, and this is what the value of a model is, an amazing thing.

17:15It has implications for every counterfactual world you might run into. It's not just... Think about a demand function. A demand function, I mean... Or don't. But whatever. I mean, I do. You need... A demand function says, what would you buy at any given price? Well, you're not seeing all the prices. You're just seeing one of them, and I see how much you bought.

17:42A demand function says, no, counterfactually tell me every single... How much you'd buy at every single price you're not seeing. Well, where's the data for that? We're making it up. So, a production function. Is that just in the data? No, a production function is a relationship between any conceivable amount of capital and labor and the output you would make. Now, it turns out that is a counterfactual object.

18:10In fact, nobody knows what it is. Do you really think the AI labs know the F of KL behind this? have a clue. So we've got much richer objects we need to think about. A lot of them are just like the old objects, but one level more meta. It's a belief about something, not a something. Now that belief about is very metaphysical, but you need to kind of go around.

18:40What I would do is I'd play out, I'd look at a model and I'd play out tons of counterfactuals in the data. And I would try lots of different constitutions. But I would do systematically because I have a vision of which of these would work and why. But I'd need that model in my head. Which do I think might work and why? Then I would design the lab to produce the counterfactuals.

19:10That are ideal for your measurement. And with AIs you could probably play out a ton of contingencies. So I'm thinking I'd like to be able to design a good mortgage advisor. Well it has to be able to take a gazillion questions. And give good answers to every sequence of them.

19:35Well, that's an ideal data set. The ideal is I see for an essentially limitless set of questions. How the agent responds. And I map that to the utility of the home buyer. And I have now a full story of an agent. So I could give you for any particular use case.

20:01I could think about, okay, well what are the elements that are playing out here? And with those elements in mind, what are the measurements you need to make?

20:15Do you have thoughts on how to elicit beliefs from these models? Like if you were given one of these agents. And you wanted to study the degree with which their behavior is driven by certain beliefs. Do you have thoughts? Because presumably for humans you could ask them. But then of course you could question whether or not people are actually good at stating their own beliefs. I'm just curious if you have thoughts on how you can. Yeah, I mean I've done studies in which you get AIs to score medical images.

20:52And, you know, they score them essentially with a numerical scale. Which then you could look in your test data and see the probability it corresponds to various different types of disease. And you would say that's the implicit probability. The as if probability. But you'd have to set up the right type of environment in which you see a given type of score quite often.

21:20And you say, now what does that correspond to in the underlying data? So yes, I have thoughts about that. But I think it gets more sophisticated every time. Anything else?

21:46Anything else?

21:49Hi. Really fascinating line of inquiry. I was wondering if there's, have you seen anything where there's a hierarchy of beliefs? For example, in a human context, let's say, stealing is bad. That's a rule. So you shouldn't steal. But you have to feed your child. That is good.

22:15So to feed your child, if that's the higher priority of belief, then would the agent steal to feed their child? So I'm just... Say it again. I'm just... I think I understand. But I worry I'm going to answer my question, not yours. Okay. Yeah. Now, I'm saying, have you seen agents prioritize in a hierarchy of beliefs and how they decide what the hierarchy is? When there's competing beliefs, even in institutions, you may have different...

22:44Now, if you're asking who's seen agents in this room, you are asking absolutely the wrong person. I do not spend a lot of my time seeing agents. The people in this room spend a lot of their time seeing agents. I spend a lot of my time imagining agents, if that works for you. That would still be a valuable insight on how you imagine agents competing for belief systems.

23:13I'll put it this way. I think that... So the title of the book that I wanted was Learning What to Learn and How. Because in this new world, the higher level activity of thinking about what it is that you should be trying to learn and how you should be trying to learn it is really sophisticated.

23:43But then I could go one stage further and say, how do I learn about learning how to learn? And who do I ask? And so you have these hierarchies of, like, I don't understand how to operate in this new world with this new set of potentials. My own guess is that the winners are going high level.

24:09That they're asking questions that... I mean, I have found that the more I can ask how should I learn how to learn what I need to learn, the better I do.