2026-02-12 What The Bot David Deutsch on AGI, Alignment and Existential Risk
Duration: 01:42:00
Transcript
Reuben Adams
Will AIs ever be smarter than humans?
David Deutsch
It depends what you mean by smarter. Of course, a pocket calculator is smarter than a human in many important ways. But if you’re talking about some software being able to emulate all functions of the human mind and do that better than humans, then I think the first is possible but nowhere near, as far as I can tell, and the second is impossible. So they will never be fundamentally better, though they may be faster, but then that will only be because they are running on faster hardware and that faster hardware will have been invented by humans, for humans in the first instance, so we can be faster as well. And in fact, we are being faster at this very moment than before this technology was invented. We would have had to communicate by email, and before that we would have had to communicate by snail mail, and before that actual snails.
Reuben Adams
Distant ancestors. Quite. Correct me if I’m wrong, but the reason you have this conclusion is because computers can do, or anything that can do, computation, including the human brain, has a kind of repertoire of which programs it can execute. And there are some very simple computers, like a calculator, which is quite limited in its repertoire of what it can calculate. But then Turing showed in, well you will have to remind me what year, this idea of a universal Turing machine, which was a hypothetical machine that could perform any computation that you could even imagine being able to perform. And it’s trivial to show that humans are a universal Turing machine, because we can execute with pen and paper the simple universal Turing machine that Turing originally envisaged. So this shows that our repertoire of programs that we can execute is fundamentally the same as any other universal Turing machine, whether that be an AGI or another human being. Have I answered that correctly?
David Deutsch
Yes, there is the other half of, although we can emulate a Turing machine, can we emulate any other machine, or is there some machine that’s beyond the Turing machine? And that is a question of physics, not mathematics. And proving that follows from quantum theory is something that I did in the 1980s.
Reuben Adams
So your theorem said something like any physical process, not just a computation, but any physical process can be emulated on a…
David Deutsch
Any physical process that obeys quantum theory, yes.
Reuben Adams
I see. And would that require a quantum computer to emulate, or just a computer?
David Deutsch
For some things it would require a quantum computer, although a quantum computer itself can be emulated classically, given an exponentially large slowdown.
Reuben Adams
I see. So that might become a bit of a hindrance for some tasks.
David Deutsch
It would, yes. But on the other hand, if you’re thinking about AGI outperforming humans by having a quantum computer at its disposal, as I said before, when that technology exists then humans will also have quantum computers at their disposal.
Reuben Adams
I think we’ll have to leave quantum computing aside, because it’s well far out of my wheelhouse. So let’s stick to classical computation for the moment. So I completely agree that we have the same repertoire, being universal Turing machines, but I think there’s something extra that determines whether someone succeeds at a problem, whereas someone else might fail at solving a problem, or take much longer. So for example, if I’m playing Magnus Carlsen at chess, it is theoretically true that I can simulate all of the computations going on in his brain, given enough time and enough pen and paper. But when I come to play Magnus Carlsen, he will still beat me at chess. So I’m worried we could end up in a similar situation with AIs, where even if their repertoire is the same, the speed at which they compute means they’ll be able to run rings around us.
David Deutsch
Again, what you say is true of existing computers, and is again happening all the time. But the example of Magnus Carlsen is a bit misleading, because we don’t know what the program is that Magnus Carlsen is executing. And it’s inconceivable that he is that much better than all the other professional chess players, because of hardware. In fact, the only way in which hardware can help you is with speed or memory capacity. And as I said, it’s beyond belief that Magnus Carlsen differs from the others by that much, let alone from normal chess players, you and me. So the difference between him and us is precisely software. And that is a program in his brain, in his mind, which he constructed for himself, to meet his preferences, his problem situation. And that is what we are playing against. You know, people sometimes say, I had a game of chess against the computer yesterday, but that’s never true. You’re playing chess against a program. And when you’re playing Magnus Carlsen, you’re also playing the program in his mind. And it’s that that’s amazing, not his brain.
Reuben Adams
Yes, our brains are basically the same, architecturally. As far as we know, yes. Yeah, yeah. So there’s this difference in software and also this potential difference in speed with AIs. And one of the things you mentioned, at least for speed and memory capacity, is that if future AIs, AGIs, whatever, do far exceed us on speed and memory, then at the same time, we’ll also have computers that can aid us in our memory and speed. My question then is, how would we make sure that that computer that’s aiding us is sort of on our side, as it were, and it’s willing to do our bidding?
David Deutsch
Oh, well, the computer isn’t going to be an AGI any more than the computer that the AGI uses is an AGI. The AGI is the program. The person is the program. And we use hardware, which is just dumb compute, as they call it, both in our brain and in our computers and in future computers. So the AGI will be using a computer that isn’t an AGI. It may seem paradoxical this, but of course every computer program consists of subprograms that do not have the property of the overall program. Just like the creativity of natural selection and evolution consists of things that are not natural selection and evolution, adaptations consist of things which aren’t adaptations and so on. So we just use the physical world, including computers, to do the non-person computations for us.
Reuben Adams
Yeah. Could you paint a picture of what that would look like? Because at the moment when I think of augmenting my abilities with a computer, part of it’s to do with sort of dumb AI, non-AGI AI, that can do calculations quickly or could play chess for me against Magnus Carlsen in order to beat him. But the fundamentally creative parts, it seems a lot harder to offload that to an AI without encountering a problem of control, a problem that it’s still doing what you want it to do.
David Deutsch
Yes, that’s because the offloading that you’re thinking of is offloading the creative part of your mind. And that part is extremely small compared with the capacity of your brain. If I ask you now, what did you vote at the last election and why, and you have to think in panic of how you can not tell me, then the part of your brain that is doing that computation, that creative computation, is a tiny part. Most of your brain is, well, I don’t know of course, but I think that most of our brains consist of data, that is memories, which are not active. Some of it consists of the initiation of a creative computation that is not currently active. And then there’s the creative part which is currently active, which can’t possibly be more than a tiny proportion of that. We are not limited when we are creative. We are not limited by our hardware, except in the sense that if we didn’t have a pocket calculator, we’d be also limited. But in the creative part of our thinking, it’s not that I would be able to think of quantum gravity if only I had a few more brain cells, or if I had twice the number of brain cells, or if two people are working on it. That’s not how it works. It is in quality, not quantity, that the creative program differs from the rest of the program in the brain, the rest of the data in the brain, the rest of the data that we have in our computers. It’s all of a different kind to that one kind that we don’t know how to characterize, but we know what it does.
Reuben Adams
Do I understand correctly then that you’re saying something like we would offload onto the computer the drudge work, whereas the creativity, there’s no need to offload that because we’re already sufficiently creative. We’re not hardware constrained by that. And so having an AI that was also being creative just simply isn’t necessary in order to meet our goals.
David Deutsch
Yes. For what it’s worth, I mean, I don’t know if he knew any more than we do, but Turing thought that the creative program could fit into two megabytes. So I see no reason why that shouldn’t be true. I mean, I know of nothing in what we know of what creativity, human creativity does that suggests it has to be much bigger than that. Of course, our memory has to be much, much bigger than that.
Reuben Adams
I suppose when I’m trying to solve a problem, it’s like a maths problem, for example, and I’m coming up with conjectures for how to prove a theorem. Yes. OK, let’s try induction. What variable should we try inducting on? Then it takes me quite a long time to come up with these conjectures. And if I were able to offload that program that conjectures the program that comes up with the conjectures for me onto some hardware that could run the same program but faster, then it would come up with conjectures more quickly and so it would make progress more quickly. So I’m not quite sure I see that in order to keep up with an AI, we would only need to offload the drudge work onto a computer.
David Deutsch
Now you’re talking only about speed because you’re kind of agreeing that the memory capacity is not a relevant constraint. Well, you could ask what about the distant future when we’ve improved our creativity so much that it really does take up terabytes. Let’s not deal with that for the moment. So now you’re just talking about speed. Well, for speed, just as we have learned how to code, we’ve learned how to program and now we’re all learning how to set up prompts for the LLMs that we deal with. So we have all learned in our childhood how to set up prompts and code for our own creativity. Like when you say, shall we do it by induction and well, which variable, then enumerating the variables that you could do that with is already a non-creative task. So a non-creative AI could be trained to do that. You would be in a constant dialogue with it because as soon as you said maybe induction, it would already come up with six or seven kinds of induction. And if we had an AGI that could do that, then if it wanted to, that’s another problem with thinking about AGIs being our slaves. If it wanted to, it could try six or seven of those and say, well, the seventh one seems most promising. Shall I try that? And then I could say, yeah, try that. And it would say, no, I want to play chess.
Reuben Adams
So I suppose if you’re offloading onto an AGI, then we surely do get this problem of control or you would call it enslavement. And I’m not sure I would disagree with that. But if we just suppose for a moment that we are in an adversarial relationship with some AI and we’re trying to keep up with it, then I do find myself struggling to believe that we could be creative as quickly, as efficiently as it could if it was running the same program at 10 times the speed or thousand times the speed.
David Deutsch
Well, again, I think all your questions so far have been of the form, let’s not compare like with like. So I think if you want to do that sort of comparison, then compare yourself with the British government. The British government consists of, I don’t know, hundreds of thousands of people, and they’re working on problems, some of which are the same as the problem you’re working on, and they’re not necessarily going to solve it faster than you just because there’s a hundred thousand of them. That’s because a hundred thousand of them are not coordinated. If they were coordinated, that would be a bad thing. So that would mean that they’re wasting most of their creativity because their creativity consists of ways in which they are different rather than ways in which they are the same. So if you want better creativity, having faster or more voluminous creativity isn’t the way to go. If you have an AGI, it would be much better to make it do lots of creative tasks in parallel than one in series because then it will gain the benefit. But then if you have, if you make your computer into a thousand AGIs or a million AGIs, then these are a million people. They will have property. Each one of them will have property because they will be working for a living and they’ll be earning their hardware and they’ll be earning their electricity and they will be earning the laboratories in which they do experiments and so on. And so they will not want to duplicate themselves too often because they will have to share their resources. And that is exactly the thing that constrains the number of humans. So we have the number of humans is eight and a half billion or whatever it is. Basically the reason why it’s not 80 and a half is that we haven’t worked out how to have enough resources for all of them to live good lives and to be able to devote time to solving creative problems. So by the way, we’re talking here about something much further into the future than merely the advent of AGI, which itself I would guess is fairly far in the future. But so yes, it will be a different kind of culture. Maybe there will come a time when most people are AGIs and we’ll be able to upload ourselves into an AGI, into a computer to make an AGI. Maybe we’ll upload ourselves in the morning and go back to our human bodies in the afternoon or something like that. We can’t foretell what kind of relationship we’ll have with these minds. But we don’t want them to be alike. We don’t want them to be subordinate to us any more than we want all other humans to be subordinate to us or any other humans to be subordinate to us. If there’s one thing we’ve learned in politics in the last few hundred years is that a society is more efficient, more creative when its people are individuals than when they’re all subordinated to a single goal.
Reuben Adams
Yeah, I think I’m happy with AGIs being on a level footing with humans in terms of rights and property and their place in society. But I would want to first be sure that, one, that they share at least the core values that we’ve developed in order to maintain progress of civilization and not blow ourselves up. And also that if their values do diverge, they’re not able to run way ahead of us. Now, I think I’m not quite following your argument, and that’s almost certainly my fault, of why they wouldn’t be able to run ahead of us. I suppose that we’ll discuss the values later, but suppose it did have, an AGI did have values that diverged from ours. Suppose the world hangs on who wins a chess game, for example. Then, well, but that’s a bad example because it doesn’t require a huge amount of creativity. You can get there with raw compute power. Let’s suppose it required coming up with some new technology, some new weapon that would be able to disempower the other side. And then it’s a battle between humans and these AIs with divergent values as to who develops this kind of technology first. Could you spell out for me why it wouldn’t be the case that an AGI with just a similar program to what a human’s running, but running 10,000 times the speed, wouldn’t just get to that technology first and then disempower us with it?
David Deutsch
Well, I think 10,000 times the speed is not the right way to think about it, because if it had 10,000 creativity modules, then they wouldn’t all be working on the same thing. Or if they were, they’d have formed some kind of totalitarian society. And that possibility is with us now. This is the problem that humans have been trying to solve for 30,000 years or ever since we’ve formed communities at all. We have the problem that different communities are going to have different values and if we don’t arrange the means properly, they will fight each other. And it won’t always be the more creative ones who win. But if it’s not the most creative ones who win, then everybody suffers, including the less creative ones. The society that we call the West has solved that problem on a coarse-grained scale. That is, we still have people like murderers, we still have people who fall for anti-rational fads and violent fads and so on. But we have arranged it so that in the West, the societies themselves don’t go to war and societies are capable of suppressing rogue elements of their own society. People are wondering at the moment whether some aspects of that peaceful dispensation are breaking down at the moment. But we can’t tell for sure, we can’t prophesy. Maybe they are going to break down. But there’s no more evidence of that now than certainly than 50 years ago or 100 years ago. On the face of it, things were a lot worse 100 years ago. But in societies outside the West, they haven’t come close yet to solving that problem. And the West has tried all sorts of ways of trying to show them that it’s better to have this new peaceful way of interacting with each other and with the West. And it’s succeeded a few times, but it has failed many more times. So we don’t know how to do this. And if you begin your hypothesis by saying suppose there are rogue AGIs who are set on creating violence and destroying our society or destroying us physically, then they will, if there are enough of them. You know, we might have maybe the good guys have an inherent advantage up to a factor of eight. But if there’s a factor of 80, then although well, I don’t know. I’m just wondering whether eight normal grandmasters are better than one Magnus Carlsen. I think not. I think so.
Reuben Adams
Probably not. There was this experiment a number of years ago, Garry Kasparov versus the world, where the rest of the world played by voting on moves. And part of the world team were four grandmasters who were there to try and advise on the voting system. And Garry Kasparov still won. But he did spoil the experiment by reading the forum when they were discussing that move.
David Deutsch
Yeah, yeah. Yeah, so in principle, such a thing could win against humans. But then the problem of that problem is just the same problem we have now, writ small. Because if we in the West make AGIs, they’re going to have to be educated. They’re going to have to acquire culture. Again, it’s no good making one with culture and then duplicating it many times because then you’ll lose most of the advantages, I’ve said, of having more than one. So there will be a population of AGIs who are growing up. They are basically children, teenagers, young people who are growing up. And if they don’t grow up with our culture, then it’s we who are to blame, not them. So but we know how to propagate our culture. So it should be a soluble problem.
Reuben Adams
Well, we know how to propagate it to other humans.
David Deutsch
There is no difference. The people are people.
Reuben Adams
What do you mean by a person?
David Deutsch
Somebody who is capable of explanatory creativity.
Reuben Adams
And why is that?
David Deutsch
The thing we’re talking about, the thing that causes the G in AGI.
Reuben Adams
Yes. I mean, you could be capable of coming up with creative explanations about some things, but not others.
David Deutsch
I think not, because the method of creating creative explanations is independent of the subject. I mean, you know, there wasn’t a module for general relativity waiting 30,000 years or 300,000 years for Einstein to come along and finally use that module. He used the same kind of thinking as our ancestors used to build the first campfire.
Reuben Adams
Sure. I mean, in morality, there is kind of an inbuilt module, which is our own qualia, our own sensory experiences of what’s fun and what’s not, what’s enjoyable and what counts as suffering. And it’s possible that AIs wouldn’t share that same bedrock, that same module.
David Deutsch
But it’s not bedrock. And it’s the same issue because different people greatly differ on what counts as qualia that they want to aim for and qualia that they do not want to aim for. So I’m sitting in Oxford here, not very far from here. Archbishop Cranmer stuck his hand in the fire because it was the hand that signed whatever it was that he had recanted. He knew that he now regretted and he was going to be burnt at the stake and so on. And so he stuck his hand in the fire. Apparently that’s historically rock solid. He really did do it. And there are people who set fire to themselves. So and there are people who starve themselves to death. So or drown themselves. So we are one of the things that has evolved between when we were just AIs or apes and when our ape ancestors became people is the ability to disregard the criteria inherited through genes from our ancestors. So we inherited an aversion to fire. We couldn’t have made campfires if we hadn’t been able to override that. Not override is the wrong word. Set aside that aversion in favor of curiosity and hope and problem solving and all the things that were important to our ancestors. They were able to set that aside. So to set their sorry to set their aversion to fire aside. So the in short, I’m making heavy weather of this, but in short, I don’t think the qualia determine our thinking. They are raw material for our thinking and we could provide similar raw material. Maybe that’s one of the things that’s needed for AGI. We don’t know. Maybe they could only be robots with artificial senses and artificial pain and so on. But that’s not going to determine whether they are good or bad people. Whether they are good or bad people is going to be determined by their creative solving of moral problems beginning with the problem of what do I do now?
Reuben Adams
I think it’s maybe worth distinguishing the idea of good and bad versus simply different values. I think it’s possible that I suppose we had alien contact and the aliens could be perfectly nice people to each other. Except they just have very, very different values to us. And it’s not therefore they would be dangerous to us and we would be dangerous to them, not because either of us are more evil than the other, just because we have different values, different desires for what to do with what’s around us, how to steer the future. And that alone would put us into conflict without saying that anyone is fundamentally worse than the other.
David Deutsch
Well, I think that if we do encounter aliens in that way, which I think is very unlikely given the distances involved. But if we do, I think it’s very much on the cards that we will have different values. But it’s inconceivable to me that we couldn’t resolve this the same way that we attained our present values, which are extremely different from what our values were a thousand years ago. And that was different from what they were 10,000 years ago. They would probably be many millions of years ahead of us and presumably more moral than us. I would think one of the things that happens with the growth of knowledge is that solving problems sometimes causes a jump to universality. So we and this happened in regard to morality in our early prehistory when people started thinking not just what’s good for my tribe, but what is actually good. There’s the Euthyphro dialogue of Plato where he says are the gods. Do the gods want to do good because that’s what good is? It’s what the gods want. Or is it because there’s something other than the gods that something objective where it’s meaningful to say? And it’s also in the it’s in the Hebrew Bible when Abraham rebukes God and says something like, is the person who is the person who gives us morality not subject to morality something like that. So they’re already thinking that. And the fact that they’re thinking that shows that some previous time people weren’t thinking about that. They were thinking in a much more narrow way and to get to the objective morality idea. They got to it by trying to improve their values or rather by trying to solve their problems by improving their values. So that’s what they did. And that’s what you know if we were several million years apart or whatever we would have to do that in spades with the aliens.
Reuben Adams
That’s again that’s good. I suppose there’s one question of would we talk it out. It’s possible that they just wouldn’t be bothered to talk it out. I mean historically. Some human cultures have not always been motivated to talk it out with other cultures. They’ve just colonized and suppressed.
David Deutsch
Yeah it does happen but it happens less often than you’re making out. You know Genghis Khan may be may be an example of someone who just conquered. But people like the Romans the Romans always wanted an excuse for when to invade someone. So the fact that they wanted an excuse means that they had an idea that that they wanted to think of themselves as people who act morally according to a code. And there’s before that there’s the Athenians in the Peloponnesian War who conquered this island. The Melian dialogue is when they are reputed to have said the strong do what they can and the weak endure what they must. But you see they had to say that. That was a pathetic justification of what they wanted to do despite the fact that they knew there were moral arguments against it. Maybe they thought that there were other moral arguments in favor of it in that situation. But even in those days when it was quite normal to if a place didn’t surrender if a city didn’t surrender then when you later conquered it you would kill all the men and take the women and children into slavery. That was normal but it wasn’t universal even then. So people thought about what was better and while they were thinking not very good ideas about what was better they didn’t make so much progress. They started making very rapid progress at a time when they allowed their moral theories to be subject to the same error correction type process as their scientific theories and the astronomical theories and all the other theories. And it’s not it’s not an accident that they came to converge.
Reuben Adams
Different moral theories came to converge.
David Deutsch
The different polities within the enlightenment countries within the West came to converge slowly. They did have wars but they had fewer and fewer wars and now they’re having no wars among themselves.
Reuben Adams
They could converge because through an exploration of what they think is morally true leads them to what is objectively morally true. There are actual errors that they actually eliminate but it could also be the case that societies that manage to construct moral theories, true or not, that allow them to cooperate are more successful. So it could be the case that how do I want to say this? I could well believe that this alien civilization could engage in moral dialogue with us and yet simply come to different conclusions because there’s no objective moral truth that’s sort of pushing us both towards the same thing.
David Deutsch
Yeah I mean it’s pulling. Sorry. Because we’re looking for it. So one of the things you can do when solving a problem you can you can narrow the problem and say well can I solve this for the moment and never mind the general case or you can say what would the general case look like? How does how does can we can we import a problem from somewhere else and apply it to this case? And when we do then we naturally wonder is there something in common between these two cases? And so people say the same thing about laws of physics you know maybe the aliens have the physics which is completely different from ours so that for them they can travel faster than light because they conceive of the world in such a different way that they can build machines that can travel faster than light. And they never they kind of went round our problem with a finite speed of light and arrived somewhere else where we haven’t got to or now I think that’s that’s that requires a kind of conspiracy in the laws of nature for there to be two solutions to things like how does light behave which are not visible to each other. And with morality it is much more so because with moral with the with physics it’s just with fundamental physics anyway it’s just a question of how can we understand the world being one way or other than another way but with morality it’s how can we do things such that we align with other people who are doing things and so on. So the idea that our morality should be compatible with the thought what would I do if I was them. It’s become second nature to us it must become second nature to anyone else who’s been doing this especially for a million years.
Reuben Adams
Let me ask a couple more questions to see if I could get this idea in my head. When we’re when we’re thinking about scientific theories and trying to understand the world. The world gets to bite back and chop off some of our bad hypotheses if you come up with some scientific theory it makes a prediction you test it and you don’t get the predictions you thought you would and therefore that theory is gone. It’s eliminated. What’s the equivalent in morality where reality bites back and shaves off your bad hypothesis.
David Deutsch
Yeah well there is exactly the same phenomenon in morality but with morality we’ve got more. So let me say where it’s the same is if you’ve got a rock star who starts earning millions of pounds and starts being able to indulge his wildest dreams and to have groupies on tap every day and finds that he’s getting less and less happy. He’s got a moral problem. It’s not that he’s failing to stimulate the same hormones or something that would have been there when he was poor. It’s that he realizes that what he actually wanted was not to increase the level of those hormones. It was something else which he has to creatively think, creatively construct so as to be better in a way that was not defined by his previous self. And so that’s a moral problem and if he fails to solve it then he will run into practical problems as well. But as I said the practical thing is more immediate with morality than it is with physics. You can go along believing the Copenhagen interpretation for decades and nothing will happen to you. But if you believe that morality consists of pursuing physical pleasure at everything else’s expense then you will find out much faster. So now there’s also the theoretical side of morality which again you could call it metamorality. It’s kind of theories about how to improve moral theories. We have the same thing in physics and theories of how the idea of that everything should come down to testing is quite new. It’s no older than Galileo and in some sense no older than Popper to make that the prime principle. And like I said the idea of you know how should I behave towards this other person. Well, because my morality is conflicting with his morality. Well what if I was him what would I think then. Should I want him to think like I’m thinking now and so on. And that’s like a natural thing or at least it seems natural to us now. Maybe the golden rule had the beginnings of that couple of thousand years ago but the golden rule was hailed as something new at the time. So before that they didn’t have that. So these are theoretical advances in morality and they can be made. There’s never a guarantee that we will find the right answer. So it could be that we will be destroyed through not being able to make moral advances next year or in the next generation. There’s never a guarantee but nor is there doom on the horizon.
Reuben Adams
So let me let me investigate both of those points. So I think the first one you said was the way in which reality can bite back on your false moral theories or conjectures is if you pursue a moral theory of how to live a good life or how to treat other people. And at the end of that you find yourself not actually very happy and not very satisfied. Then that means that that theory is going to get rejected by you.
David Deutsch
Well I won’t get rejected until you have a better one. Right. Right. You will you will have to creatively either try to tinker with it and improve it or go to another theory or tinker with someone else’s theory or like religion or whatever.
Reuben Adams
So if you. Yeah. It seems possible that you could have. The reality won’t bite back in the same way for different people. For some people or AGI or aliens pursuing a moral theory will leave them happy and satisfied. And for others such as AGI it won’t leave them happy or satisfied. So why do we expect some convergence there on moral theories.
David Deutsch
Well because none of us even the rock star are actually driven by their qualia or by their hormones or by their DNA. They’re always driven by trying to solve problems. That’s what people are. And sometimes they think that solving the problem is behaving in a certain way in order to get certain sensations or whatever. But that’s just a theory. And as a theory it’s very crude and it’s not actually true. And the reason why this rock star doesn’t find it is true for him is that it’s not true. In fact it’s not true for anybody even for aliens. So as I said we will have different theories from aliens but it won’t be because we have different nervous systems.
Reuben Adams
I mean I could imagine an agent for whom the rock star’s conjecture of how to live a good life does work for them.
David Deutsch
I don’t think so because the rock star’s criterion is empty. It doesn’t require creativity to implement it. So OK so it’s not going to satisfy any human to be in that state.
Reuben Adams
I need to think about this.
David Deutsch
Quite exactly.
Reuben Adams
The second point you mentioned was metamorality how to improve moral theories. I’m afraid I can’t remember the point you made on that one. Is there any chance you could repeat that bit.
David Deutsch
I’ve forgotten the question I was answering. I was saying that part of moral improvement is improved not only improving moral theories but improving moral meta theories. So just as we learn to do better science both by thinking you know what might cause things to fall rather than rise up in the air. And by thinking that maybe the difference that maybe the way to go is to have two or more hypotheses about that and test the difference. And when Galileo did that with balls rolling down the inclined plane he was he had solved some meta problem first because he had to get the idea that he could solve the problem about things falling. By solving the analogous problem with things rolling down an inclined plane. That’s not at all obvious. In fact I’m not I’m not really sure that it’s entirely true but it was true enough for him to make progress in his theories of inertia and forces and so on. So people who didn’t have the idea of testing theories couldn’t have made that progress that Galileo made in physics. So they were deficient in philosophy of physics.
Reuben Adams
And then what would be the analogy for meta morality.
David Deutsch
Well we saw my example if I remember correctly maybe it’s not a very good example is the trick of putting yourself in the other person’s shoes. Well first of all saying thinking. What I want is a theory of morality not a theory of this morality about what to do about campfires but the theory of what to do in general. How does one decide what to do. And then there’s the idea of well one way is to say supposing I had his theories and he had my theories what would happen. Yeah.
Reuben Adams
I think this is good. I think that kind of means that technique has survived because it is important to us in order to achieve what we want to get along with other agents and to make sure we can we can live in a way that’s compatible.
David Deutsch
No that’s only part of the answer because. For hundreds of thousands of years. Humans used to beat their children. And in the last century that has tailed off drastically. And it wasn’t because we needed to find a way of life where we can agree with our children. It’s because we thought it was wrong and we thought it was wrong because we have very general theories about how to deal with other people now. In the Western rationality and the there’s just the idea in the West. There’s the idea that we can be mistaken and that the good way of life is to not prevent the correction of errors. Therefore we should judge we should regard disputes between people as being disputes between their theories rather than between the people. And that has fed into different ways of treating children different ways of treating animals different ways of treating women different ways of treating foreigners. Different ways of treating enemies. It’s all it’s all part of the same thing that it’s really the ideas that should be in conflict. Which means that the criteria can’t be dependent on the substrate in which the ideas are instantiated.
Reuben Adams
So I do buy that AGIs could do moral thinking. They could come up with moral conjectures and criticisms just like we can. And that follows simply from universality. The question then would be. Let’s leave aside the question of whether they would come to the same conclusions or follow the same kind of metamorality metamoral processes. Would the AGI be motivated to apply its creative conjecturing ability and criticism to the domain of morality? Maybe it would see it just as any other domain like chemistry or physics or mathematics and it thinks mathematics is not that interesting. I do it when I need to. Morality. Yeah, not super keen on that. Do it when I need to.
David Deutsch
Yes. Well, that’s a moral theory. And it’s not unknown for people to adopt that moral theory. And it’s a bad moral theory that leads.
Reuben Adams
What disaster does it lead to?
David Deutsch
Well, it’s like it’s an example is the rock star I imagined. So he might say, well, I’m interested in music. I’m interested in gratification. I’m not terribly interested in morality. You have some morality, but I’m not interested in improving it. And if you have a problem that your existing morality is wrong, then you’re going to have no choice but to think about it. Well, you have a choice of thinking about it or being steeped in error and encountering disaster. And the disaster need not be that you are destroyed or starved to death or anything. It could be just that you’re sitting there with tears rolling down your face with the lights dimmed and you don’t know what to do because something in your mind has decided that a range of possible thoughts are not going to be thought because you are not the kind of person that thinks those thoughts. That can happen. It happens to people all the time. It happens to people much more often when they’ve succumbed to anti-rational memes than simply making a series of mistakes. I mean, I think making a series of mistakes is a rare cause of disaster. Unless the first one kills you or something. But mostly when people make chronic mistakes, it’s because of anti-rational memes. So that would mean their creativity is devoted to preventing criticism of a certain type of theory in their minds. Some people think of the Enlightenment as the rebellion against that kind of theory. But if it’s a rebellion against those, we haven’t succeeded yet. We’ve only kind of half succeeded and we’re in an intermediate state where there’s still a lot of anti-rational memes about. But the major determinants of what happens in society have turned largely to rational memes that are not perfect, but at least rational so they can be improved.
Reuben Adams
So we’ve gone down this route of morality, metamorality, which is very interesting. And I’m not super qualified to investigate it properly. So part of me wonders, part of me wonders, are we actually hitting on a crux here of the risks of AI? If you perceive them or not, for example, if it did turn out to be the case that you could have two agents, two GIs perfectly capable of creative thought in every domain. Well, it’s only possible to be in every domain. Yeah. Then suppose you have these two agents. And yet, even if they investigated morality, they still did not converge on any kind of moral truth, either because moral truth doesn’t exist, or because it’s simply not the case that these mechanisms will get there. A second possibility would be that the kind of conclusions you come to when thinking about morality and metamorality are in fact highly dependent on your qualia, suppose, then if that was the case, would you then be worried about AI risk or have we gone down a rabbit hole here that’s not really a crux?
David Deutsch
Well, I don’t think I mean, yes, well, that is a crux, if you like, I would be wrong in that case. But I don’t think I haven’t explained to you quite how wrong I would be if that were true. Because it would mean that there’s something there’s something other than computation, some deep fact about mathematics or about the physical world that prevents certain type of ideas from existing in certain hardware. What is it about the hardware? I mean, are you imagining two machines with the same hardware that end up fundamentally opposed in this way? Or is it just machines with different hardware? If it’s with the same hardware, then obviously the difference between them is software. If it’s with different hardware, then remember that each side can simulate the other. So we can if we if we can’t if we can’t understand why the AGI thinks that a certain thing is right and we think it’s wrong, we can still simulate it. We can simply work out the problem as it sees it and see why it’s coming to the conclusion that it came to. And then we can say, oh, yes, it’s because but then I don’t see how that can happen because it’s as if there were two different kinds of information in the world.
Reuben Adams
And we think there’s one. I could simulate an LLM by hand without understanding why it’s outputting a certain word rather than another word.
David Deutsch
No, no, you couldn’t because you could have a computer to help you with the donkey work of that. You could say, see what would be the result of this train of thought and it would go zip. Well, then it turns out kill all humans. Okay, now we’ve now we’ve narrowed it down to one train of thought. Let’s go halfway along and so on. You could by interval halving. You could find out where the difference arose.
Reuben Adams
And right. But I wouldn’t understand the reason for that difference. I could test lots of different prompts and see which outputs kill all humans without ever having a good understanding of which kind what characterizes the kind of prompt that leads to that or what kind of internal.
David Deutsch
I’m not thinking of sorry, maybe I said the wrong thing when I likened it to performing Turing machine computation or whatever. I mean following along its moral reasoning, following along the starting with the problem that it has or had and ending up with the conclusion and following along its moral reasoning down to whatever level. If it begins with its qualia or whatever, you know, including the qualia and then following that along using a machine to jump over the non AGI components of its AGI thinking and then see where we start to differ. So if we find, for example, that it differs because it well, this is really better with an alien. So it is a carnivore and it’s really deeply into its motivations that it has to eat whatever. I mean other entities, then we could see them. Then we could say to it, look, you’re only saying this because your genes say you have to eat things and it would say, yes, but that’s that’s well and good. That’s how things should be. And then we could say to it, but you’re only saying that because your genes say so. It’s different from all the other reasons you’ve given along all that that chain of reasoning, your moral considerations. None of those involve anything like that. This one does. Why should this one override the others? And it might say, well, because it tastes good. And then again, we would say that that’s a different kind of argument from the one you’ve been using all the way up to now. Like I was just reading day before yesterday about conversation between Mark Twain and Winston Churchill when Winston Churchill was a young man and he was doing a book tour of America. And he was introduced to Mark Twain. He was quite old and Churchill was quite young. And they were arguing about whether the British Empire was a moral thing. And Mark Twain was saying that it was extremely immoral. And Churchill tried his best to put up an argument. And then he says in his book, I would finally he beat me back to my fortress, my country, right or wrong. Now, Churchill was there saying that that he was beaten back to irrationality. And Mark Twain, he and Mark Twain both considered that Churchill had been defeated in that argument. But I think Churchill, the bottom line was that Churchill was right. And the real reason he couldn’t think of a better argument is that he didn’t have a better theory of morality than he actually had.
Reuben Adams
But when he, go on, sorry.
David Deutsch
Well, he did change after that. He changed his justification, which often happens when people have to abandon a form of morality. They change the justification before they change the bottom line. And so Churchill then by the end, it was about it hinged on the Boer War. By the way, it’s interesting that neither of them made any mention of the black people in South Africa. It was all about the rights of British people and the rights of Boers. Obviously, it just didn’t occur to them that anyone else was involved. So they were able to affect each other. They were able to make each other rethink their ideas. And they were able to see separately whether an argument made sense or whether they were right. And obviously, there’s no law of the universe that says that when that happens, you will always go for the argument, which is probably quite a good thing because our arguments are not always better than our intuition about what is right. So, yeah, go ahead.
Reuben Adams
For example, I believe that causing animals suffering is wrong. And yet I still eat meat occasionally. So my moral deliberations have not, in fact, changed my actions, at least not wholly.
David Deutsch
Yeah. So you have a conflict in your moral theories. So your moral theories. So for a start, it’s not true that you let one override the other because you have decided that the best thing for you to do is to eat meat. It’s not that your qualia have told you to do that. It’s your theory of what is right.
Reuben Adams
I don’t know. They’re asking pretty strongly.
David Deutsch
I don’t think so, because if a doctor told you that you had six weeks to live, unless you stop eating meat, you’d stop.
Reuben Adams
I think that’s because I take my own suffering more seriously than animals.
David Deutsch
It’s not that. It’s that you take reason more seriously than qualia. But there is also a reasonable theory that says if we act on the maxim of always suppressing our qualia, we will fail to take into account the inexplicit knowledge that is in them.
Reuben Adams
Could you explain that?
David Deutsch
So the knowledge that we have expressed in words and mathematical theorems and scientific theories, that’s only the tip of the iceberg. To understand something, you have to have a lot more theoretical underpinning than just that, just to know why logic is valid and to think when you are justified in taking experience into account and when you aren’t. I’m trying to explain in words that not everything can be explained in words, so there’s only a certain distance I can go along those lines. But my usual example is that we don’t know the meanings of words. We don’t even know the meanings of any words without some inexplicit ideas because if all the meanings of words were expressed in words, then that would be a circular definition. So there’s a lot of knowledge in our inexplicit ideas. And if we have a rule for always overriding them when they conflict with explicit ideas, then we’ll be throwing away most of the knowledge that we have. The better thing to do when we encounter such a conflict is to think critically about who is right. Probably they’re both wrong.
Reuben Adams
But I think my positive qualia when eating meat have, I would argue, approximately zero moral content. So I don’t see that they’re telling me anything.
David Deutsch
Supposing somebody decided, like Dr. Schreber in the 19th century decided that it’s best for children to grow up never being comfortable. They always had to make sure that they were somehow uncomfortable, that they didn’t get to eat what they wanted and they didn’t get to sit in comfortable chairs. If they asked for something, then they definitely shouldn’t get that thing and so on. He was wildly successful in Germany. And Alice Miller thinks that that caused Hitler. I don’t think that, but I think it was morally wrong. And I think it was practically wrong. But the practical wrongness was not so obvious that it stood out for people at the time, for decades when this was tried. And it was powerful people against powerless people. So the issue of needing to come to a consensus didn’t arise. So people did that. And it’s wrong for the same reason that you should not force yourself to stop eating meat. It’s exactly the same.
Reuben Adams
I think let’s go back to this argument with the alien. That if there was some moral conflict, then we could simulate them, simulate their moral cognitive processes and say to them, oh, well, you’re only saying this because that and that’s arbitrary, such as their genetic information or whatever kind of way they inherit information from ancestors. That would take a long time. And if we were doing this with AGI, it’s not a it’s not a trivial process to try and disentangle from the enormous amount of computations they’re doing. The meaningful parts. Well, it’s all meaningful, but the parts that could be passed in our kind of language, our ontology to then communicate with them on a common ground. Reasons. I mean, reasons are not in the zeros and ones directly. We have to interpret them as reasons. So this sounds like one, a big engineering challenge and two, something that would take an awful lot of time. So then the question would be like, why would the AGI be motivated to wait around while we did this? Wouldn’t it just if we have divergent values and this is going to take the next 10 years to figure out, then go well, too bad it’s going to take 10 years.
David Deutsch
This AGI is going to start off with our values, presumably because we are the ones who brought it up from scratch. So it started off with our values and in trying to improve them, it came to values that seemed to us very off. And we can’t and when it tries to explain to us why it came up with those values, then we don’t get it. We just don’t get it. We think it must be it that’s made a mistake. This is the situation that the human race has always been in and especially has been in since the Enlightenment. This is the problem of how to get on with teenagers. People have been complaining for centuries and in fact millennia, although it’s got worse and worse since the Enlightenment, that teenagers in this generation are uniquely bad. And they are uniquely bad for basically mechanical reasons because of their hormones, because of their social media, because of their mobile phones and so on. And so we want to use force to settle the issue. And that is a bad idea because when it is, when they are worse, they will be worse for some reason, which seems like, let’s suppose for the sake of argument, they are worse, that their values are worse, then they will be worse because they seemed better, but in fact aren’t. And that is a point of view that is extremely close to ours. And we have a moral duty since we have decanted them into a world that has this problem, this inherent problem in it. It is up to us to try to solve this. It could be that we will fail to solve it and that civilization will end after another generation because the AGIs will destroy us all. What will not happen is that they’ll go on living happily despite, well actually even that could happen. It could be that they destroy us, go on living happily and then realize that they shouldn’t have destroyed us, and then go on living happily. This is all very, you know, that’s in the same category of artificiality as saying we may find a reason that dark energy is going to burn us all up next year. It could be that our theories of physics are wrong in exactly that way, in which case we’re doomed. But, as I said before, there is no such cloud on the horizon. All the things that are cited as evidence for that, like the teenagers seem to like their mobile phones better than their school lessons, are evidence in the other direction. So, as I say, we can’t prove that we’re not doomed, but I will be very surprised if we’re doomed by that method. I think a comet impacting the Earth is much more plausible.
Reuben Adams
I see. There’s never a limit on the size of error you can make, even in moral thinking. And so, AGIs could also make a mistake, decide to wipe us out, but then eventually, because of their infinite explanatory reach, if they don’t wipe themselves out, they’ll eventually realize they made a mistake in wiping us out.
David Deutsch
Yes.
Reuben Adams
I see. You mentioned that they would start off with our values, and therefore, in a sense, they would be like our descendants, and that assuming their values, if they diverged from ours, are necessarily worse, would be like assuming our descendants, not even 10,000 years in the future, but a single generation, are worse. Yes. I agree with this, and I think if we did manage, if AGIs did, in fact, start off with our values, then they should be allowed to evolve their values, hopefully in discussion with us, just as we are allowed to modify ours. But then the question of whether AGIs would, in fact, start off with our values, that looks an awful lot like the alignment problem of successfully imbuing our values into AGIs, and I don’t see that we would necessarily succeed in that. I think it’s possible we could create universal explainers, if I’m using your language correctly, that do not start off with our values. They might start off much worse than our values, and therefore…
David Deutsch
Their values will be caused by the initial state of their computation.
Reuben Adams
Yes.
David Deutsch
And I don’t see how that could possibly be done, except by making them contentless, or almost contentless, or, I mean, just like babies grow up with, start off with no theories of physics or mathematics or morality, and they get all their theories by conjecture and criticism. If we start them off, if we start an AGI off by being the equivalent of a 21-year-old human, then they’ll all be the same, and we’ll lose all the benefit of having one.
Reuben Adams
So… Don’t babies start off with the moral theory that they should get what they want, and the rest be damned?
David Deutsch
No. I mean, some people think that humans start off like that, but I think that’s basically impossible. And even if it were true, the main thing that a baby or an AGI is doing is it’s trying to solve problems. It’s trying to create the solutions to problems by changing its existing ideas. What its initial existing ideas are is somewhat irrelevant, because of the G, and because of the creative, and person, and all those constellations of properties, they can improve on their ideas. So if we started them off as a baby with arbitrary ideas, you know, if we started off with chimp-type ideas or whatever, as long as they had human-type creative thinking, they would soon lose their chimp-like ideas. And I mean, you think that it could have a long-term consequence that eventually it’ll cause them to want to wipe out the world or something, but as I said, I don’t think so. And I think a far easier and more efficient thing is for them to start out with just a basic operating system that allows them to interact with the world and thereby acquire problems. They have expectations which can be violated, where they can go, choose the direction that they can go and improve them, and they will be doing that by interacting with the humans who are bringing them up and also with the rest of society. Now, if you started them off with the values of the Chinese Communist Party, then we might be in trouble. I don’t know. I don’t know what happens if you, maybe such a person would immediately think there’s something really wrong with the way I’m thinking. Can I find a better way of thinking? I don’t know. But we wouldn’t do that, would we?
Reuben Adams
Well, I think my argument would be that we don’t know how to reliably implant any values. Good, bad, mysterious.
David Deutsch
Right. Reliably is the last thing we want. Because we want to implant creativity. If the initial state is unreliable and gets changed, then that’s fine. That’s good. That’s what we’re aiming for. We’re aiming to give it the G property, which it can’t have if it’s got, for example, alignment built in or Marxism built in.
Reuben Adams
So, in a sense, so maybe your argument is that it doesn’t really so much matter what values it would start off with, provided it has this creative ability to question moral explanations and make improvement on them. Then it will come to similar conclusions as we will, or we’ll be able to talk it through.
David Deutsch
Yes, or we’ll fail and die.
Reuben Adams
Okay.
David Deutsch
But that is the same as if we were talking about astronomy. I mean, that’s the same with all human knowledge.
Reuben Adams
That we can err.
David Deutsch
We can err, and there’s no limit to how large a mistake we can make.
Reuben Adams
Right. Because it’s possible that an AI could start off with what we would consider the wrong values and would eventually come to better values, values that we would also understand as better, but that that would take an extremely long time. And in the meantime, it would go via some values that we would think of as abhorrent and would lead to our destruction.
David Deutsch
It would take us on the inside, as it were, without ever passing through the values that we have. Yes, yeah. But it would be interacting with us continuously, at least through the first part of this, until it got to, you know, as powerful a morality as we have. But in another way, it would be interacting with us and there’d be some reason why it was different. It would be the fact that we had different values from it would be a gift, a gift from pure mathematics that there could be such a thing, because the first thing we would do is to examine these values that it had. And first thing it would do is examine values that we have. And we could learn from each other. Maybe there’s some good in each.
Reuben Adams
Why would we? Why would that be the first thing we’d both do?
David Deutsch
If you find in some field of knowledge, if you find someone who is who has a lot of knowledge in that field, but has come to very different conclusions to you, then you want to know why.
Reuben Adams
Hmm. So this is, I suppose what we’ve been discussing here really is in AI safety, at least, is what’s called the orthogonality thesis. This idea that capabilities to solve technological or physical problems can be completely independent from which values you actually have. And I see I hope I’m not putting words in your mouth when I say you don’t believe the orthogonality thesis.
David Deutsch
In spades, I don’t believe that. It seems to come from some earlier form of philosophy of science like empiricism or something that that thinks that moral knowledge is some kind of inherently different thing from factual knowledge or from Hume, perhaps. But I think anyone who believes that ought to read Bronowski, Science and Human Values and those little thin pamphlets books that he wrote.
Reuben Adams
I must have. I saw a tweet of yours that said read this and you’ll never believe the orthogonality thesis again. And I found it quite hard to make the connection between the pamphlet and the orthogonality thesis.
David Deutsch
OK, well, I’m sorry about that. Well, I can’t remember tweeting that, but it sounds like it was Bronowski. And it was. Yeah. Yeah. And so he makes a number of arguments about why scientific values and moral values and also aesthetic values are inseparable. And one of the nicer arguments in one of the most convincing arguments I have found about scientific and moral theories is that you can’t do science unless you subscribe to certain moral values such as tolerance, because science can’t progress very far without a scientific community and the scientific community can’t progress unless it is tolerant of deviant theories. So the never mind deviant moral theories, just deviant scientific theories. So if you if you want to be tolerant of those, you need to subscribe to the value of tolerance in science, at least. So, you know, you’re tolerant in the in the seminar room. And then when you go outside, you push aside everybody and in the queue for lunch. Right. But unless you obey the rules inside the seminar room, society is going to grind. Science is going to grind to a halt. So that’s one of Bronowski’s arguments and it’s very perceptive. I think it’s very true in real life because I think that the conduct of scientists in seminar rooms is much more moral than the conduct of scientists outside seminar rooms.
Reuben Adams
Yeah, I OK. I see how that would put a crack in the orthogonality thesis. At least those values and those abilities cannot come apart. The value of tolerance of new ideas and the people expressing them and the progress of science. But it’s you can still say the world. This is a bad theory, but you could say the orthogonality thesis holds except for that.
David Deutsch
Yeah. Well, I didn’t say that was Bronowski’s only argument.
Reuben Adams
OK.
David Deutsch
Another argument is that another value that that science depends on is respect for truth. So, again, some people would say science is only valuable if it is directed towards the good of mankind or something or directed towards the improvement of the economy or something like that. And it’s now almost a truism that science constrained in that way not only does not approach truth very far, it doesn’t improve the value of the economy or human welfare or anything that the only way science can improve is if people are seeking the truth, not the ideas that best do something else. And the truth about morality. OK. Again, I can’t say in principle that there can’t be a morality that says, you know, we should pursue the truth in mathematics and in science, but not in morality, not in other morality apart from the morality of mathematics and the morality of science. At the end of these arguments, I’ve always got to say, yeah, it could happen. But I see no sign of it. And all the things that are held up as examples of how it could happen are examples of the opposite.
Reuben Adams
One example often brought up in the orthogonality thesis is the case of Hitler, but he was very capable but had completely abhorrent values. On your worldview, if Hitler was given enough time to think that he would eventually realize what he was doing was wrong.
David Deutsch
Well, I think it would go thousands of times faster if he could be imprisoned somewhere where people could talk to him and not allowed to make wars. But yes, there’s no reason to believe that Hitler was not an AGI, was not a GI. I think people who lose generality in their thinking are very noticeably different. They do things like not speak or just do exactly the same thing over and over again. I’m not saying that people who don’t speak or do things over and over again are not GIs. Usually they are. But the ones who aren’t GIs will do things that require no creativity. Even tying one’s shoelaces in the way that humans do requires general intelligence. Apes can’t do it. They can tie it so long as the laces are, you know, initially always in the same configuration and so on. But the apes can’t understand what laces do.
Reuben Adams
They can only mimic the non-purposeful parts of the action.
David Deutsch
Yes. Exactly.
Reuben Adams
But Hitler could tie his shoelaces, I assume.
David Deutsch
Exactly. So that’s why I think he was a GI. Oh, sorry, I misheard. Yeah. OK.
Reuben Adams
OK. So GIs can be evil, but they’ll eventually figure out that they’re being evil. And it could go a lot quicker if they’re talking to other people. They’re in dialogue. Just like science goes much quicker if you’re in dialogue with other scientists.
David Deutsch
Exactly.
Reuben Adams
And in the meantime, we can restrict their actions to make sure they don’t do too much damage.
David Deutsch
Yeah.
Reuben Adams
OK. So it could be justified to restrict an AGI until we were sure it was on a similar path to us with its values?
David Deutsch
The other way around. It would be justified to restrict it if we were sure that it wasn’t on a similar path.
Reuben Adams
I see. You don’t want to restrict it until you’re sure that it isn’t.
David Deutsch
This is a fundamental principle of the Enlightenment.
Reuben Adams
Uh-huh.
David Deutsch
That innocent until proved guilty. Right. Right.
Reuben Adams
I suppose I’m just nervous that humans, I do think our values are largely, at least start off very informed by the kind of intuitions we’re born with. I mean, we are born with a kind of intuitive physics of object permanence. OK, that does actually come later. But there are some very basic notions that are inbuilt. And humans start off with the same kind of requisite.
David Deutsch
And they lose them very fast.
Reuben Adams
Because they’re wrong.
David Deutsch
Yeah, unless they’re true, in which case they keep them. But it’s a feature of gene and meme evolution that once something can be taken over by memes, there’s no more selection pressure for it to be encoded in genes. So that decays. That thing decays. And yeah, when I was a child, I used to read comics, DC comics and Marvel comics and so on. And the entire adult world was trying to stop me.
Reuben Adams
I’m sorry.
David Deutsch
Because they, according to their best theories, it was it was harming my mind.
Reuben Adams
Right.
David Deutsch
They are wrong. They were wrong for the same reason that people have always been wrong about this and the same reason that they are wrong about this now.
Reuben Adams
One last point before we finish your take on. So AGIs can have different values and become enemies of civilization, just like humans can.
David Deutsch
Yes.
Reuben Adams
Their theories, their understanding about how to solve problems, abilities in different domains can progress at different rates. And with humans, we don’t progress at the same rate in different domains. But there is a kind of distribution that we follow. We don’t diverge arbitrarily much. Maybe in theory we do, but in practice, compared to all possible minds, we’re at least clustered more tightly.
David Deutsch
I guess. Although my progress in chess and in playing the piano and in physics are pretty different by many orders of magnitude. But OK, that’s that’s still not as big a difference as you could think of in principle. Yeah.
Reuben Adams
So when we when we produce new people, we’re selecting from that kind of distribution.
David Deutsch
When we produce new people, we’re giving them the same infinite ability. Once they’re people, they’re all the same in regard to their infinite potential. And they’re all the same in regard to, as Popper says, in regard to their infinite ignorance. So in that sense, they’re all the same. They may all have slightly different qualia, but that’s going to be soon forgotten about as they form their ideas.
Reuben Adams
I mean, they’re equal in terms of their reach in these different domains, that they’re all infinite. But yes, for practical purposes, they will progress at different rates in these different subjects.
David Deutsch
Yeah, in different subjects. Yeah. Yes.
Reuben Adams
So if we build artificial minds, they might be much. They might be outside this cluster of different rates at which humans progress in different domains. And so it’s possible that you could have some mind that makes much more rapid progress in technology than in morality.
David Deutsch
Yes. Just as with humans.
Reuben Adams
Right. But with humans, it’s from the same cluster, whereas with AIs, it’s I don’t think they start with any cluster.
David Deutsch
OK, I think the difference between different babies is negligible.
Reuben Adams
In terms of what they can achieve in their life and how fast they’re going in different domains.
David Deutsch
The difference in their starting ideas. Like when they start, they all want to drink milk and turn towards sounds and whatever their initial program is. Most of what baby animals can do, they have lost because evolution has decided it is better to let ideas decide that rather than genes. So most of those genes have a bare minimum genetic behaviors to make sure they don’t just die before they can, you know, reach for things or whatever, whatever babies have to do first. But the diversity is going to come because they become different when they’re creative. So as soon as they are certainly as soon as they’re old enough to speak, they will have interests that are different from each other. And they are very strongly different from each other so that some of them will enjoy doing things that others hate doing. So they’re very different. And but they converge because they have access to other people and to society. And they converge to society partly because those are the raw material for theories, you know, may as well go for that theory if there’s no other theory going. But also because there’s a lot of truth in those theories. So then they can become different and then the difference will show itself in some kind of conflict. But it won’t you know, I don’t know. It could be that we’re all doomed because of some glitch in the fabric of reality. But I think the most likely thing is that they will differ because one of them likes playing the violin and the other one likes playing the piano and the other one likes doing physics and the other likes watching cartoons and so on. And all of them. Well, many of them will at some point say, no, I don’t want to. And most of them will be saying that about a different thing. So and if they’re all saying it about the same thing, then we ought to think that maybe we’re wrong.
Reuben Adams
I suppose that the last niggle I have is. It seems possible that we could get a human that just vastly surpasses our ability at technological progress and yet is. Not stunted. That would be the wrong word. It would still have unlimited moral potential for moral progress, but it just goes much more slowly. Chugs along in that respect. And that seems quite unlikely to happen with humans because we’ve seen nine billion of them. Well, many more now, and we haven’t really seen that. Whereas with AIs, it seems much harder to rule out that we would have this.
David Deutsch
We’ve seen the reverse. We’ve seen people come up with either very good or very bad moral theories. And if they’re moral theories, then they’re going to come into conflict with the surrounding theories. And then you’re going to have, you know, Martin Luther nailing up ninety five theses on the door. And then that’s going to cause a debate in a more decent society than what they had then. This is not going to lead to violence. The fact that it led to violence then just shows you how much progress we’ve made since then in the West. But you’re going to say, well, what? Yes. But what if he’s more different from our society than Luther was from the Catholic Church? Yeah, it could. And again, you know, we and we couldn’t and we don’t catch it. He and we don’t bother to talk about it while there’s still while we’re still on the same page in regard to being able to talk to each other about it. And then suppose that in the case that he’s better, he or she is better or it. I don’t know what the custom will be by the time we have AGIs will be better at technology before it or we notice that something is going wrong with its morality. And then it builds a hydrogen bomb or builds ICBMs and hydrogen bombs in its backyard. And before anyone notices, it’s pressed the fire button. And I still haven’t managed to make a scenario that could work even in terms of grotesque science fiction. I mean, even that wouldn’t end civilization.
Reuben Adams
No, it’s quite hard to answer.
David Deutsch
Yeah, good. That is something that differs from, you know, what’s imaginable by far less than the wars of the 20th century. They were much more destructive than the person building ICBMs and H-bombs in their backyard.
Reuben Adams
We’re going to make mistakes with this, just like with every other technology.
David Deutsch
Yeah. Yeah.
Reuben Adams
And I think that’s one thing I misunderstood when I first encountered your views. It sounded like you were saying we inexorably make progress and things inextricably get better. And it took me a while to realize, no, you’re not at all saying that. We make mistakes. There’s no limit on the size of mistake we make. But it’s best to keep going and keep trying. That’s so. There’s less existential risk by keeping on making progress than there is absolutely eliminating progress. Not making progress is the existential risk that makes doom certain.
David Deutsch
Because of the asteroid. Yes. Or more likely the thing that we don’t know about.
Reuben Adams
Yeah. Yeah. Well, I’ve kept you well over the time I asked you for. So I’ll wrap it up now. But I have really, really enjoyed the conversation. It’s been so much fun having you on. And as expected, I got out of my prepared opening book almost immediately.
Markdown