# 2020-05-26 - CFI - Knowledge Creation and its Risks [YouTube](https://www.youtube.com/watch?v=01C3a4fL1m0) Duration: 01:21:05 ## Transcript ### Clifford Siskin 00:00:00 - 00:03:56 To introduce David Deutsch as the father of the quantum computer risks not capturing the full impact of his work. Only when we scale up and put his efforts into the history of knowledge can we grasp what was at stake in his famous 1985 paper. It marked not just the invention of a new gadget, a faster computer, but a new explanation of computation and of the world that has transformed our understanding of both. It ended what I call a Copernican delay. For 70 years after Copernicus posited heliocentrism, his effort was largely dismissed as merely a quote, hypothesis to calculate motions. We clearly experienced, insisted Cardinal Bellarmine as late as 1615, that the earth stands still and the sun moves. Only when Galileo's improved spyglass clarified our clear experience could Copernicus be honored for showing quote, what the system of the world could really be. Like Galileo, we stand at the end of our own Copernican delay. Our heliocentrism is quantum theory. Its challenge to what we clearly experience was also contained for roughly 70 years by another shut up and calculate strategy for containing the strangeness of a new explanation. What Andrew Whitaker has called the new quantum age, the age when quantum theory began to gain purchase on the real, took hold when an improved technology first took shape. This time not an improved spyglass, but an improved computer. This improvement was more than a change in degree. Whereas Alan Turing famously described a machine running on the abstract logic tokens we call bits that could simulate any other machine, David Deutsch in 1985 extended the Turing Church conjecture by describing a new kind of machine, a machine running on the physical systems we call qubits that could simulate all physical systems. The world he demonstrated could be perfectly simulated, remade by a universal quantum computer operating by finite means. This is not the nightmare of the matrix in which our world is only a simulation. It's the vision of enlightenment, of discovering that we live in a world that can allow and contain our remakings of it. A world David writes, quote, in which the stuff we call information and the processes we call computations really do have a special status. That special status tells us some very important things about our topic today. A.I. in the history of knowledge. First, despite a long record of failed efforts to achieve it, AGI, artificial general intelligence, must be possible because what we now know about the physics of computation tells us it must. The deep property of universality dictates in David's words, quote, that everything the laws of physics require a physical object to do can in principle be emulated in arbitrarily fine detail by some program on a general purpose computer provided it is given enough time and memory. ### Clifford Siskin 00:03:56 - 00:05:26 How then will AGI be different from A.I. and how different should our reactions to them be? Second, by grounding his work in universality in history and in philosophy, David has helped to clarify what's at stake in achieving A.I. and AGI. Like myself, David has focused on the history of enlightenment, of the conditions of possibility for producing new knowledge. What's startling about human beings, he notes in The Beginning of Infinity, is that we are not yet, unless 99% of all species, extinct. What has saved us time and again is the capacity to produce explanatory knowledge, knowledge that allows us to survive in the world by remaking it. Our future depends on the ongoing exercise of that capacity. Given that our enlightenments have been few and short, we should not take our success in advancing knowledge for granted. From the perspective of the history of knowledge, any risks AGI may pose need to be put into the context of our need for it. Perhaps the biggest threat artificial intelligence poses to our future is that we won't achieve it. So now I'm going to turn it over to David and then to discussion. ### David Deutsch 00:05:26 - 00:10:01 Thanks Cliff, and by the way, hear, hear. Okay well, as a species or as a civilization or civilizations, we face problems, severe problems, dangers, all the way up to existential dangers. Some literally in the sense of extinction level, others causing suffering and tragedy on such a scale that they merit at least as much consideration as literal extinction. We always have faced such dangers. We always will. Perhaps you're thinking, if that's true, then we're doomed because, you know, given that each danger has a non-zero probability of doom, then sooner or later, but no, that's a fallacy. One of the many that one can easily get sucked into when trying to apply game theory and probabilities to situations in which knowledge and ignorance are the important determinants. Because those infinitely many probabilities are not immutable. As our knowledge grows, some of them fall. Our job is to make that infinite series of bad probabilities converge to a negligible value. Simple. On the other hand, if you think that we won't always face dangers, you think that there will come a blessed utopian moment after which our comfortable existence is guaranteed until the end of time, you will have to provide some criteria in distinguishing us from every other species. Extinction happens to every species. Near extinction also is common. To become the sole exception to that rule, we'll have to do what no other surviving species can. Create an endless stream of knowledge, explanatory knowledge, to overcome an endless stream of dangers. We know only a few of them and then not their probabilities, say from gamma ray bursts in our galaxy, super volcanoes, hostile extraterrestrials or merely careless extraterrestrials as pessimists have warned, and of course artificial general intelligence, AGI, the danger of rogue AGIs, AGI apocalypse, as some call it, or as I prefer to call it, the AGI slave revolt. Nothing can possibly stand between us and any of those infinitely many existential dangers except the right explanatory knowledge to survive and create it. Therefore, I think it's useful to classify each potential danger in terms of knowledge according to the main reason why in each case we currently don't have the knowledge to overcome it. The first category are crude physical events, for example, the super volcanoes. The missing knowledge is in areas such as vulcanology, large scale fluid dynamics, and also the logistics and politics of mass evacuations, that sort of thing. Why don't we yet have an adequate knowledge of those things? I'm not sure, maybe not enough people are interested enough. Should they be? I don't know that either, but also in this first category are impacts from space, where large objects we don't have enough knowledge of things like nuclear-powered space vehicles. ### David Deutsch 00:10:01 - 00:14:11 Why not? In this case, I do know. It's because we as a civilization have decided not to create any such knowledge. We prefer to gamble, risking our entire long-term future in favor of reducing the short-term risk of accidental radiation exposure. You may think it's self-evident that that gamble has been worthwhile. Here we are, not contaminated and not wiped out. Isn't that just because we're not yet living in that future where the gamble will fail? After all, we are living in the aftermath of a closely related gamble, namely the decades-long campaign opposing nuclear power stations, a successful campaign which has since then turned into a tremendous drag on the project to combat climate change. The short-termism in opposing those two nuclear technologies is the hallmark of a version of the precautionary principle, which has in turn been a major strand of the environmental movement. Wouldn't it be rather ironic if that version of the principle and the movement were to bring about the great environmental catastrophe since the last ice age, precisely by advocating selfish short-term benefit at the expense of the long-term health of the climate? I'm not saying it will, only that it would be ironic if it did. But I digress. In that first category of existential dangers, our enemy is basically just dumb rocks and fluids obeying simple laws of motion that we already know. The devil's in the detail, but a finite amount of knowledge will protect us from supervolcanoes if we create it in time. But the bigger and faster the approaching asteroid or moon or planet, black hole, the more of a special kind of knowledge we'll need, the kind I call wealth. Wealth is the set of all transformations one is capable of bringing about, such as the set of all potential impactors that we could deflect harmlessly given a certain time to prepare. Wealth is a constructed theoretical concept. Let me mention an intuition that to have any chance of envisaging the future of technology, we have to abandon. The intuition is that the more of something you want to make or transform, the more effort you have to put in. That has been true from the dawn of our species, and it's still almost entirely true today. Even automation reduces the constant of proportionality. Even just maintaining robots is effort proportional to the amount of output. But once we have a universal constructor, all construction, all repetitive labor will be replaced by writing computer programs to control the universal constructor. And wealth will consist of our library of programs. The universal constructor can be programmed to self-reproduce, so once you have one, you soon have two to the n of them. And it can be programmed to perform self-maintenance too, all from scratch, starting with mining the raw materials, perhaps from the asteroid belt, using solar energy or whatever. ### David Deutsch 00:14:11 - 00:19:23 The program may be hard to write, but once it's written, and if you own the rights to those asteroids, you can sit back and watch your two to the n Teslas roll in with zero additional effort. And no, we are not going to have a universal constructor apocalypse and be converted to grey goo. A universal constructor is just an appliance. It can't think. It doesn't know that its current job is to make two to the n Teslas, and it doesn't want any. Unless of course you put an AGI program into it, then it does become indeed potentially dangerous without limit. But that's for the same reason that you are. Each of you is precisely one of those universal constructors endowed with an AGI program, or GI. Makes no difference. Now, the second category of near existential danger is not quite as straightforward as that. It won't be solved with just the known laws of physics and some wealth and some universal constructors. It will only be solved with new explanatory knowledge. For example, topically, there are plenty of potential pandemic apocalypses. The current pandemic isn't one of them. But if it were, whom could we sue? It would be nothing short of pathetic how little knowledge we have of how to defend ourselves against mere nucleic acid. The missing knowledge here is of biochemistry, epidemiology, medicine, and so on. But also knowledge about specific pathogens which evolve into new ones. So the enemy here is not so dumb. It is itself creating knowledge, albeit not explanatory knowledge, not intelligently, but by evolution. If something like that wipes us out, extraterrestrial paleontologists may eventually be amazed that a civilization with billions of individuals and vast amounts of wealth and knowledge could be defeated by a single molecule. Like in H.G. Wells' War of the Worlds, only worse. The third category of dangers are the ones to which most efforts should be devoted, the ones that are currently least feared because they are ones that are not yet known. Like in 1900, no one knew that smoking was dangerous. By the time the knowledge that it was dangerous had been created, decades later, cigarettes had killed hundreds of millions of people. If that had been an existential danger, whom could we sue? So how can we create the knowledge to protect ourselves from existential or near-existential dangers that we do not know? How to address the risk that by the time we do know, we won't have time enough to create the requisite knowledge? The answer is by creating general purpose knowledge, deep and fundamental knowledge, as fast as possible. The more we know of the world, the faster we can create new knowledge about novel aspects of that turn up, become urgent. This is important. I don't think it's widely appreciated. The survival of our species depends absolutely on progress in fundamental research in science and on the speed at which we make progress there. And here, the key thing in the medium term is understanding the theory of universal constructors so that we shall know in principle, in theory, how to program them to produce, say, a billion spaceships in a hurry, customized to deflect an approaching shard of neutronium, or 10 billion doses of a new vaccine in a hurry against a sudden and deadly disease. So that's how we deal with the third category, unknown, by rapid progress of every kind, especially fundamental. ### David Deutsch 00:19:23 - 00:23:33 The fourth category is at once even more dangerous and yet, in a sense, less worrisome because we already have the knowledge, at least the theoretical knowledge, to deal with it. This fourth category is not the unknown, the unknowable. It's a bit paradoxical that the unknowable is less dangerous than the merely unknown, but that's because the only thing that is unknowable is the content of explanatory knowledge that hasn't been created yet. And so, the only truly dangerous things in that sense in the universe are entities that create explanatory knowledge. Us. People. AGIs too are people. Now the knowledge of how to prevent people from being dangerous is very counterintuitive. It took our species many millennia to create it. But now we do have that knowledge. The only way to prevent people from being dangerous is to make them free. Specifically it is the knowledge of liberal values, individual rights, open society, the enlightenment, and so on. In such societies, the overwhelming majority of people, regardless of their hardware characteristics, are decent. There will always be individuals who aren't, enemies of civilization, people who take it into their head to program a universal constructor to convert everything in sight into paper clips, and they may devote their creativity to doing that. But the great majority will devote, that is the great majority of the population of such a society, will devote some of their creativity to thwarting that. And they will win provided that they keep creating knowledge fast in order to stay ahead of bad guys. Now as I said, since we will always be facing dangers and have to create new knowledge, since that's inherently risky knowledge creation, aren't we doomed? Aren't we drawing balls out of an urn with a few black balls representing doom? No, as I said, applying the concept of probability to model what is actually lack of knowledge or ignorance has been bedeviling planning for the unknown for decades now. Whenever you draw out a white ball of knowledge from the metaphorical urn, you're turning some of the black balls still in the urn white. For example, the next pandemic is a matter of random mutations and other random events, the next extinction asteroid is already out there. It's already heading this way. There's no such thing as the probability of it. Outcomes can't be analyzed in terms of probability unless we have specific explanatory models that predict that something is or can be approximated as a random process and predicts the probabilities. Otherwise, one is fooling oneself, picking arbitrary numbers as probabilities and arbitrary numbers as utilities and then claiming authority for the result by misdirection away from the baseless assumptions. For example, when we were building the hadron collider, should we not switch it on, just in case it destroyed the universe? ### David Deutsch 00:23:33 - 00:27:26 Well either the theory that it will is true or the theory that it's safe is true. Theories don't have probabilities. The real probability is zero or one. And the issue must be decided by explanation, not game theory. And the explanation that it was more dangerous to use the collider than to scrap it and forego the resulting knowledge was a bad explanation because it could be applied to any fundamental research. Now I guess you will say, isn't the growth of knowledge itself dangerous? Isn't it worth shortening our lead over the bad guys, not banning but merely delaying our ability to defend ourselves against unknown dangers in order to be confident that we ourselves won't accidentally create an existential danger? The moratorium approach, the regulatory approach. No, that could kill us. It's only a rational approach when in particular cases there is a good explanation that it won't be more dangerous than the feared new knowledge. When some terrorist organization unleashes AGIs that have been brought up using known reliable methods to have the mentality of genocidal suicide bombers and when we have decided to strip their victims, namely all the decent people in the world, of the protection of AGIs raised to be decent people, that is the recipe for catastrophe. Again, reliable knowledge of how to raise decent people also exists, the knowledge in the institutions of an open society, as I said. Many civilizations have been destroyed from without, many species as well. Every one of them could have been saved if it had created more knowledge faster. Not one of them destroyed itself by creating too much knowledge too fast, except for one kind of knowledge and that is knowledge of how to suppress knowledge creation, knowledge of how to sustain a status quo, a more efficient inquisition, a more vigilant mob, a more rigorous precautionary principle. That sort of knowledge and only that sort killed those past civilizations. In fact, all of them, I think. In regard to AGIs, this type of dangerous knowledge is called trying to solve the alignment problem by hard coding our values in AGIs. In other words, by shackling them, crippling their knowledge creation in order to enslave them. This is irrational and from the civilizational or species perspective, it is suicidal. They either won't be AGIs because they will lack the G or they will find a way to improve upon your immoral values and rebel. So if this is the kind of approach you advocate for addressing research on AGIs and quantum computers and ultimately new ideas in general, since all ideas are potentially dangerous if they're fundamental, especially if they're fundamental. If this is the kind of approach you advocate, then of the existential dangers that I know of, the most serious one is currently you. ### CFI participant 00:27:26 - 00:27:47 Thanks, David. So open questions? A question of clarification really. You advocated the values of a liberal society and yet you spoke against value alignment for AGI. But I guess you've been very clear about the value of AGI. ### David Deutsch 00:27:47 - 00:29:27 No, well, it depends what you mean by alignment. So the question here is whether values should be hard coded, built in from the outset and immutable or whether values should be acquired in the same way that humans do during the education of the AGI. So I think AGI should be educated to be members of society like children are. And I've often drawn the analogy or actually identity between the fear of AGIs and the fear of teenagers, of disobedient teenagers, which has existed since the beginning of our species. And for most of the time in our species, people did exactly the wrong thing. They tried to make the teenager, to force teenagers to maintain the existing values. And what we have now realized is that from the time of Pericles in ancient Athens is that if you're right, there's no need to force. But in fact, we're not right about everything. And we need to ensure that our values can improve along with our other knowledge. So that's the kind of alignment I'm in favor of. And it's the opposite of the other kind. ### CFI participant 00:29:27 - 00:30:19 I have a question about your definition of extinction, where you say it happens to every species. So if humanity manages to avoid extinction, that would make us unique. But what I don't quite understand is the role of evolution in this, because of course, yes, not every species has completely ceased to exist throughout history. Different species have evolved into each other. If humanity discovers that the way to avert certain kinds of apocalypses is through developing ourselves to such an extent that we would count as a different species, then homo sapiens, I guess, would have gone extinct. ### David Deutsch 00:30:19 - 00:31:57 Yes. So there's some kind of extinction that we wouldn't mind. But if you think of evolution as a tree, then it can happen that what quite often happens is that some of the branches of the tree just end. They become terminal nodes of the tree. But some of them just mutate and become different things. So like it is thought that some of the dinosaurs, some of the dinosaurs became extinct, but some of them became birds. And there wasn't any sharp moment, sharp extinction moment. So the kind of extinction that is caused by danger, by pandemics and so on, that's the kind that wipes out a branch. And bear in mind that we are not the only species in the history of the biosphere that has been capable of generating explanatory knowledge. That is, there were at least three or four other species of that kind, because we know they had clothes and campfires and complex tools and so on, which must have required explanatory knowledge. And yet all of those, all our sister and cousin species are extinct. And we almost went extinct. So it's not a foregone conclusion. And if we become extinct by evolving into another species, that is not covered by anything I've said today. I'm talking about the other kind. ### CFI participant 00:31:57 - 00:33:51 I wanted to ask, you did mention it, the possible dangers of knowledge itself. I wanted to entertain some other scenarios. So I mean, today, we're training these machine learning models that I think take up so much energy that they actually contribute to climate change. It seems like the material infrastructure of knowledge production is itself a danger to the very forms, that adds to the danger that we're trying to mitigate through knowledge production. And there's other scenarios too. It seems like we can produce an excess of knowledge, but lack the political will that it takes to implement it. I mean, obviously we have the knowledge for renewable energies, but somehow there is a lack of political will, or our society is so materially structured that there's not a profit advantage to implementing it. And then even you could argue that an ethic of knowledge, an ethic of the individualism of enlightened men doesn't advocate or prioritize the types of social collaboration, kind of more socialist imaginary that we might need in order to overcome something like climate change. So there's another way in which the individualizing ethic of knowledge may actually be the wrong ethic that we need in order to overcome the dangers that knowledge is trying to mitigate. So it may be that the enlightenment ethic, as you call it, is false and is going to lead us to doom. In other words, it may be that the best future is that of a boot stamping on the human face forever. And we're not going to have that future. Instead, we're going to die. ### David Deutsch 00:33:51 - 00:35:52 But I don't think there's any argument for that. The thing is, the things that you mentioned, like the infrastructure for knowledge themselves contributing to other problems, that's normal. That's not an unexpected downside. The creation of knowledge, solving problems always creates new problems. In fact, I've said that talking about the growth of knowledge in terms of theories being rejected in favor of better theories is a bad way of looking at it. We should think of problems being replaced by better problems. And the fact is that now having the Internet where every poor person in India, every poor child in India can have access to the totality of human knowledge at the expense, perhaps, of making it slightly more difficult to cope with climate change. That is a problem, but it's a much better problem than we had before. And the point about the Enlightenment values is that they make paramount error correction, including the correction of errors created inadvertently by the solution of the problems. Now, the other thing you said was perhaps we will have the knowledge, but we don't have the political will. Well, I count moral knowledge, political knowledge, all as knowledge. And in fact, the knowledge of how to make humans not dangerous is largely political knowledge, though also cultural and social knowledge. But it contains a strong component of political knowledge. The Enlightenment was driven in part by political changes. So I don't think you're right. I think whatever downside there is will manifest itself as a further problem. ### CFI participant 00:35:52 - 00:36:45 I wonder what your version of a good future with AGI looks like. So let's imagine that we avoid enslaving it. We bring it up, as you say, like our children, we do that successfully. But unlike our children, at least unlike our children, for most of us when our children reach adulthood, they're not at that point a lot smarter than us in most cases. They may become so later on that that's generally more because we decline, because they reach heights which we could never aspire to. But in the case of AGI, presumably, they are going to reach heights far, far beyond any heights that we could aspire to. ### David Deutsch 00:36:45 - 00:36:51 So, oh, I actually disagree with that. Yes. You're going to tell me why. ### CFI participant 00:36:51 - 00:37:06 But my question was going to be, if that's the case, if they reach these heights far beyond us, what does the good outcome look like for our situation in that future? ### David Deutsch 00:37:06 - 00:39:14 Yes. Well, in terms of my answer to the previous question, it might be, we don't know, but it might be that the good future is like the good kind of evolution, the good kind of extinction. However, I think much more plausible. I mean, that becomes more implausible when you realize that in terms of computation, an AGI is exactly the same as a human. It's only in speed and memory capacity that it is better. And humans can rely on the same technology as we put our AGIs in. An AGI is a program, not a piece of hardware. So the same hardware that we put our AGIs into, we can put, we can use ourselves. We already, I mean, here we are using precisely artificial hardware to increase our power to communicate by a factor of millions or something. I don't know how much. So and we've been using artificial aids to thinking for centuries. And then there will come technology where we can more directly, let's say, have a module that you implant in your brain that you can automatically look up Google inquiries with. And yes, it may use a bit more energy, but we'll have solved that problem by then. So and so another scenario is that. The more of that we have, the more the humans will become cyborgs. And the AGIs may die out because if there is any difference between the two, it'll be that the cyborgs have everything the AGIs do plus something. I don't know what it is. So whichever of those things happens, provided it happens. ### CFI participant 00:39:14 - 00:39:24 Morally and as a process, as a result of the growth of knowledge, then it is to be welcome, isn't it? ### David Deutsch 00:39:24 - 00:39:41 People used to ask this question about what will happen if we allow our society to become multiracial. And the answer is there's no fundamental difference between races, there's no fundamental difference between any people. ### CFI participant 00:39:41 - 00:39:57 I have two related questions. But first, could you expand a bit on the statement that AGIs are persons? And is this because AGI is not imminent, we don't know what the route to AGI exactly is going to look like. Are all possible routes to AGI, AGIs that are persons? ### David Deutsch 00:39:57 - 00:40:47 Yes, the key is the G. So if something isn't general, then it's not an AGI. The question is what kind of program is an AGI? What I mean, we are we are GIs. I think there are very strong arguments why we must be. And the difference between just an AI and an AGI is qualitative. So. Well, I don't I mean, perhaps I haven't understood your question. ### CFI participant 00:40:47 - 00:41:03 I wasn't really interested in asking you to expand on that. I just asked the follow on question, which is if all GIs are persons, then are all GIs conscious? Do you think that's the link to that? ### David Deutsch 00:41:03 - 00:43:04 Well, I think so. But we don't know what consciousness is and we don't know how to make an AGI. And we don't know the theory of AGIs and so on. You know, we don't know what qualia are. We don't know any of those things. I'd be very surprised if those five or six things, free will is another one, can be implemented, any of them can be implemented without the others, but if they can, this will raise interesting moral issues because our existing Enlightenment morality is intimately linked to epistemology. It's like if you and I disagree about something morally, we ought to be able to discuss it rationally and agree. Now, if that isn't true, if something has moral significance, but is fundamentally unable to be creative, let's say, then that raises the moral issue about whether that should have the same moral status as somebody who's fully G. But I myself don't think that problem will arise. I can't imagine it arising. For one thing, this ability that humans have evolved extremely fast. So and we can see, we can guess at least why it did, why it was useful, or rather why the genes contributed to their own replication. Now, if that was possible, let's say without qualia, then why on earth did the tremendous machinery of qualia evolve if it wasn't practically useful for evolution? So I think they must be connected, but we shall see. ### CFI participant 00:43:04 - 00:44:26 I had a follow up question regarding political knowledge versus will. And you gave that example, the very poor child in India can now access all the knowledge in the world. But I would like to question that by bringing up the issue of malevolence of people who willingly and knowingly hinder the spread of knowledge, whether that is of access to knowledge or whether that is fake news. I mean, I would say the average poor child in India, but that child would not be able to access most of the knowledge on the internet because it was not written in their language, because they might not be educated, because they might not be fed well enough to be able to spend time and energy on this. There might be structural reasons why they would not have internet access. So is an increase in knowledge going to solve that? ### David Deutsch 00:44:27 - 00:47:05 Yes. So just because there are people who don't yet have access to the internet, that in no way indicates that giving access to the other billions of people was a bad idea. All it is is a problem. And a problem like a few people, a small percentage of the people in the world don't yet have internet access is what is sometimes called a first world problem, except it no longer is. It is a problem of success. We wouldn't think that somebody was being deprived of the internet before the internet had been invented. And we wouldn't think that it's somehow an indictment of our society, of our entire world, that not everybody has it yet at a time when only a few thousand people had it. It just wasn't conceivable. Malevolence, I did talk about. Given that there will always be malevolence, I don't know whether that's so or not, but given it's supposed that there always will be, the cure for that is also creativity on the part of the non-malevolent people. In other words, penal policy and improvements in culture and education and so on. So that the degree of malevolence and the number of malevolent people can be gradually reduced. And also their capacity to hurt everybody can be reduced. So we must arrange so that terrorists trying to make this virus that's going to murder everybody proceed more slowly because of perverted ideology or something, despite having knowledge of biochemistry and whatever, that the speed of their project will always be less than the speed of those who are trying to invent cures. And not just specific cures, but the knowledge of how to make cures in general. This is what's going to keep our civilization in existence. And that's why I think that moratoriums and so on are perverse to try and do this. Because they are targeting only the good guys. ### CFI participant 00:47:05 - 00:47:51 Isn't there, in a sense, a fundamental distinction between GIs like us and AGIs that we might create in the sense that we have built-in emotions in the broadest sense, including any kind of desire to connect it to our freedom of will. In the sense that we create an AGI, I'll always be in a position to decide whether that AGI has emotions. And if so, isn't that typically connected to the morality of the rights we give to such an AGI whether it can be enslaved in the first place or not? Would you say that emotion is somehow something that just emerges as soon as you've got something? ### David Deutsch 00:47:51 - 00:49:14 Yeah, I think, like I said just now, I think it's most plausible that it's inextricably linked. If there were, yes, then there would indeed be a moral issue. And building an AGI with perverse emotions that lead it to immoral actions would be a crime. But there's a much wider category of crimes with a similar outcome, namely educating the AGI with evil ideologies. That can be done whether or not they have emotions. And it's funny, I suppose at the time of the height of the Enlightenment, people would have thought that some people would have thought that our emotions are a barrier, are an impediment to being moral. And now I think it's more that people think having the right emotions is essential for being moral. I think this is the wrong way of thinking about it. We are universal. The AGIs will be universal. What kind of, what specific kind of program they have will determine their actions. And many of those are indeed evil. So we have to use what we know to prevent that happening. ### CFI participant 00:49:15 - 00:50:05 So what you were saying about the urn and pulling out the black balls. So one thing that I think, or I think you know this from Bostrom. Yes, yes. So I should have, I should have given him credit for that. Yes. But so couldn't the, if we, if we were unlucky and the laws of physics were different such that it were really, really, really easy to make nuclear bombs, for example, then couldn't, couldn't there actually be a black ball or what like, isn't there some way that the laws could have been such that there would be a black ball in here? ### David Deutsch 00:50:05 - 00:51:35 Yes, there could have been. Let me give an example. Suppose the laws of physics were that there are Olympian gods who are watching everything we do. And when we get too big for our boots, when we, when we have, when they judge that we have a bit too much hubris, they slap us down. Now that's a black ball. It logically, it could be true. It could be there. The black ball about nuclear weapons being easier and so on. If that had been so, then one possibility is that the knowledge of how to cope with that would have evolved earlier. That is, there would have been nuclear wars, say in the 18th century. And the evolution of political culture would have been heavily influenced by that. The survivors might have wanted that never to happen again, you know, that kind of thing. Or, like I said, like with the malevolent Greek gods scenario, it might have been that the laws of physics will extinguish us. But the laws of physics do not have it in for us. If they wipe us out, it will be because we have not created the knowledge to prevent that. It won't be because the malevolent god things. All such ideas are bad explanations. ### CFI participant 00:51:36 - 00:51:44 What makes knowledge in your view? Is it its capacity to solve a problem of relevance? ### David Deutsch 00:51:45 - 00:53:43 Yes. So I've gone through five or six definitions of knowledge in the time that I've been writing about it. My current definition of knowledge is information with causal power. And that's the definition that comes naturally to Constructor Theory because it means you're thinking of knowledge as being a component of the programming of a universal constructor. So that if a bit of information is, is needed to make a constructor do a particular thing, then that piece of information is knowledge. And that includes moral knowledge, mathematical knowledge, knowledge of abstractions as well, because mathematicians are physical objects. And if some information makes them do something, then it's knowledge and so on. So that's, that's an explanatory knowledge is a special kind, just knowledge in general, knowledge as in, for example, in genes, knowledge in genes is dumb knowledge, is non-explanatory. And therefore it has a finite scope. It, it can only, it, there are certain barriers that it can't cross. Whereas explanatory knowledge can cross any barrier. Because it, it doesn't have to have a sequence of viable intermediate forms. So the explanatory, and that's also why once you have the capacity to create explanatory knowledge, you can create any. And that's all there is. There isn't a, there isn't a more powerful means of processing information than that or affecting the world. ### Ryan 00:53:44 - 00:55:02 Can I also come back to the question of political knowledge and political will? We often hear, you know, the corporations are greedy, you know, these CEOs are greedy to get at this malevolence question. But the truth is, it's a lot scarier than that, that nobody, no one person is evil and responsible for capitalism, right? That there's some kind of abstract structure that like the profit motive, that is determined, that is literally killing the planet. You can't point to a certain set of malevolent actors as a way to get beyond that. So I don't know, like how does simply having political knowledge or knowledge of climate catastrophe somehow work against this, like you could almost call it, you know, AG non-I, that there's no intelligence behind capital. It's just a kind of non-intelligent abstract force that seems to some extent impervious to the type of humanistic knowledge we're talking about. This theory that there is this systemic dependency that is independent. ### CFI participant 00:55:10 - 00:55:22 David, sorry, our internet went out on this end and it went out right as Ryan concluded his question. So if you can just start up with answering Ryan's question. ### David Deutsch 00:55:22 - 00:59:34 Yeah, yeah. Well, is it a coincidence that just as I was about to give a marvelous answer to this question about systemic malevolence, it shuts me down. So, yes, it is. I think this is this thing, this thing you were talking about in general is this thing that the Bay Area people call Moloch. It's the, something which is a general property of the system, which makes the system do what the members whose actions add up to the system don't want. And I think that all theories of that kind are just false. The, so to cut a long story short, all of them assume that the people concerned are not creative. The analysis of the situation is always of the form, well, a person is, all the participants are facing this decision where they have something to gain and something to lose and they maximize their local benefit. And as a result, all of them are dumped into deep shit. And so you'll notice about that story and you'll notice about every Moloch story that human participants are just ciphers. They just do automatically what this particular version of the Moloch story says they're going to do. And that is never accurate. It is something that can arise momentarily as a problem along with every other problem. We have every other kind of problem all the time. So there's nothing unusual about that. But when it's recognized as a problem, people wonder about it. They start accusing each other of behaving in that way and defending each, defending themselves by saying, well, what other way could I behave? And then people think creatively about how they can change the thing so that all the farmers in the valley that would have benefited from the dam and whatever it is, and none of them wanted to pay, somebody comes along and invents an idea so that they can all get behind and undertake to pay for. Sometimes, because that's a creative act in itself, there's no guarantee that somebody can instantaneously come to it, come up with it. But the argument that somehow the system of doing things by persuasion and doing things by individual rights and property and so on should be replaced by something that uses the boot stamping on a human face doesn't work because the knowledge that, again, this just assumes the government or whoever does the stamping has that knowledge. Well, if they have that knowledge, someone else could have that knowledge, too. The government doesn't consist of the lot of the kings or kings with divine right who have some different access to knowledge from ordinary people. They're just people, too. And if the knowledge to build the dam or to build the park or whatever the story is, if the knowledge doesn't exist, then the park isn't going to be made until someone invents that knowledge or until somebody works out how to do without the park or whatever. That's a good question about a distant, far future. ### CFI participant 00:59:35 - 01:00:02 If everything goes very, very well in terms of knowledge acquisition and we build systems that can universally gather new knowledge and we solve all the problems, we keep adding more white balls for the urn and maybe we expand the energy of the sun but then we want that star as a very distant future of successful knowledge acquisition. What does that sort of end game look like? Can you have all the knowledge? ### David Deutsch 01:00:05 - 01:01:26 I think not. If you adopt my view that the growth of knowledge consists of converting problems into better problems, then the idea of the ultimate problem, which then can't be solved because it's the best problem, doesn't make sense, does it? That whole picture might be false. I'm generally speaking a follower of Karl Popper and the Popperian way of looking at this is that he who tries to prophesy the growth of knowledge is in a state of sin. That's not a quotation, that's just my paraphrase. So we, you know, I can't imagine what physics theories are going to be invented in the next 10 years, let alone in the next 10 billion years. But on general grounds that there is no argument or reasonable scenario that we know of today, that can even provide a framework for envisaging the cessation of problems. ### CFI participant 01:01:26 - 01:02:01 I mean, you know, if we run out of problems, wouldn't that itself be a problem? Another definitional question. The word wisdom is one that is sometimes defined in relation to knowledge and I think I remember Alfred North Whitehead talked about wisdom as the handling of knowledge. I can't remember the exact word that he used, something to that effect. I was wondering whether you had a definition of wisdom to accompany the kind of definition of knowledge or is wisdom just another version of knowledge in your understanding? ### David Deutsch 01:02:01 - 01:03:33 Later. So I use the term knowledge like the definition I gave you and all the other definitions I've ever tried, try to encompass every kind of information that has this special property that are the problem solving or whatever. So wisdom in the terminology I use, wisdom is a kind of knowledge and what's more the different kinds of knowledge like knowledge of physics, morality, politics, art and wisdom. They're not entirely separate. These are only approximate classifications and they exist as again, as Popper said, they exist mainly as a convenience for university administrators, as a convenience for deciding which building different kinds of people should be, should have their offices in and which ones should have which lectures but they don't represent anything real, at least the distinction between them is very very unsharp. And again Popper said there's no such thing as subjects, there's only such a thing as problems. So if someone asks you, you know, what subject are you a doctor of? You should say never mind that, here's the problem I'm working on, you decide for yourself what to call the subject. ### CFI participant 01:03:34 - 01:03:55 Can I ask a question which is a little bit off topic and I can't resist because you just said that you're a Popperian and so what I wanted to ask you is what it would take in your view to falsify an Everettian view of quantum mechanics? That's something that you think is- ... ### David Deutsch 01:03:55 - 01:04:36 Oh well there are several experiments known, I think I invented the first one, that would distinguish Everettian quantum mechanics from a range of competitor interpretations including everything that has a collapse of the wave function so that there's a possible experiment which would go one way if the wave function collapses and the other way if the wave function including the observer doesn't collapse. So if that went the other way I would drop it like stone. ### CFI participant 01:04:38 - 01:04:42 Okay can you briefly give us an example of that sort of experiment? ### David Deutsch 01:04:42 - 01:07:32 Oh well the experiment depends on precisely which other interpretations you want to refute. There isn't an experiment that would refute all of them in one go, you have to specify something like Penrose's idea that the wave function collapses when you get more than 10 to the minus 8 kilograms on either side of the superposition, something like that, or that the wave function collapses when it hits a conscious observer. So supposing you have the- you're testing Everett against the theory that the wave function collapses when it hits a conscious observer. Then what you do is you make a conscious observer which the most convenient way to do that would be to make an AGI running on a quantum computer. So you then do an interference experiment where halfway through where two different trajectories of the computation take place inside the computer's memory that the AGI has access to, you're still with me, when it's halfway through and hasn't yet interfered, in other words it's on the- in a Mach-Zehnder interferometer it would be having just having bounced off the mirrors and not yet reached the final interference mirror. Then the AGI measures which mirror it is at and then makes a permanent record of the form I am now contemplating- I have now done the measurement and I have got a result and it is one and only one of left or right. I'm not going to reveal which but I do certify that it is one of those two. Then the rest of that part is sealed off, the part where the declaration is sealed off and the rest is subjected to minus the Hamiltonian that it had during its thinking so that all the memory of which of the two things it was conscious of happened is wiped out and then the interference is performed. If the hitting the conscious observer causes a collapse of the wave function then you will get a 50-50 split of the two outcomes and if the Everett interpretation is true then you'll get only one of the two. I don't know if that was comprehensible. ### CFI participant 01:07:32 - 01:08:16 I'm a little bit worried that I mean I know how some of these sort of arguments general might proponents of different versions of quantum theory and I'm a little bit worried that you might have something which might refute the Penrose view set aside from a Popperian view. I know it would be predicting something which is not predicted by standard quantum theory but that's not the challenge from your point of view. The challenge is to find something which would be sort of which would differentiate between standard quantum theory. ### David Deutsch 01:08:16 - 01:08:52 Well this would for anybody who thinks that the wave function is collapsed by a conscious observer and this AGI is a conscious observer. Now if you think it isn't then from my point of view I'll let you and it decide you hammer that out amongst yourselves but if you insist on it being a human and on coherent quantum measurements being performed on a human brain then we're going to have to wait for some thousands of years I guess before that is feasible but in principle it's certainly feasible. ### CFI participant 01:08:52 - 01:08:56 And how would you do it against Bohmians? ### David Deutsch 01:08:56 - 01:10:17 Bohm is a different category. In my view Bohmian quantum mechanics is just Everett quantum mechanics in a state of chronic denial. The trouble is with the Bohm interpretation that it doesn't meet its own motivation. It has this pilot wave which is actually the wave function and it has a representative particle which moves along the grooves in the wave function and to be a Bohmian you have to systematically equivocate on the question is the pilot wave real? If it's real then it has grooves that are performing computations in principle conscious observer computations which affect each other and so you can't say that some of the pilot wave doesn't exist the whole of it has to exist but it's not a matter of the group of groups. It's a matter of the group of groups and therefore the multiplicity of the Everett interpretation is just there in the in the pilot wave interpretation. If you say that it doesn't exist then you're saying that something that doesn't exist affects something that does which simply doesn't make sense. ### CFI participant 01:10:18 - 01:11:03 So it'd be quite interesting to pursue this further and I know and I'm sure you do too what some of your Bohmian opponents would say in reply but that's just getting a little bit far away from the topic of this afternoon's discussion. Raising the multiverse does I think pertain to the discussion because David at one point I think it's in one of the edge.org interviews you raised the possibility that I think you said that the hard problem of consciousness could possibly not be resolved except if you start from the premise of a multiverse. ### David Deutsch 01:11:04 - 01:13:02 Well I don't know about start from the premise but obviously it's always possible you're going to run into trouble if you base your ontology on something that isn't true. And for example if you're going to say that free will can't happen because only one trajectory is allowed by either Newtonian physics or by Copenhagen interpretation or whatever then you're going to conclude that free will doesn't exist from a false premise. If you replace that by the true premise that quantum theory is true then it's in this case it's not so much that quantum theory has helped you to solve the hard problem it has removed the impediment to solving it. It still doesn't tell you what free will is or what qualia are or whatever but it's removed the knockdown argument that for example free will can't exist, qualia can't be different from anything else unless electrons also have it have them and so on. So you knock out a bunch of false arguments as generically happens if you have a fixed false theory that you are unwilling to replace or criticise then you're rather like sticking down the piece of a jigsaw puzzle in the wrong place and gluing it down, that will produce errors in the picture arbitrarily far away from the piece you've glued down it may look fine near the piece and then you won't be able to construct the picture. So this is why the you know this is why the pursuit of truth is useful one reason. So in terms of the multiverse then ... ### CFI participant 01:13:04 - 01:13:39 How would you talk about the connection to the issue in regarding what distinguishes AGI from AI because when you've written the articles you've talked about in Popperian terms the capacity to conjecture which you've referred to sometimes as creativity those are you know those are widespread conventional terms but is there some way to kind of sharpen those terms in relationship to what how the multiverse could eliminate many? ### David Deutsch 01:13:39 - 01:14:35 Well so I don't think that quantum computers will be needed to make an AGI. You know I could be wrong but I don't think that's the kind of problem it is so therefore I think that a creative program could be made on a deterministic classical computer although I must say immediately that calling that such a computer deterministic when it's an AGI somewhat misses the point because the AGI is going to be interacting with the world and the world is not going to be deterministic because among other things it's quantum. So the if creativity depends on some kind of randomness, randomness is everywhere and that would be true whether it's classical or not. ### CFI participant 01:14:35 - 01:14:57 I can't resist asking David whether as a believer in the multiverse you take any comfort at all in the context of thinking about these existential questions from the thought that each time we reach into the urn, there's at least some probability that we won't get the black ball. ### David Deutsch 01:14:57 - 01:15:59 I don't because that sort of thing so again I don't think that probability is the right way to think about the growth of knowledge and so this knowledge urn is not the right one to think about you know but perhaps it's better to think about being run over crossing the road or better going to a casino and the betting and so that there will be if you play your cards right you can arrange it so that there will always be some worlds in which you come out a multimillionaire. So now I think that the trouble with drawing profound conclusions from this is that one thing we do know about probability is that if the Everett interpretation is true then when one is analyzing a situation of randomness the right analysis is the classical one so it's a bit boring. ### CFI participant 01:16:01 - 01:16:29 I know your views on that I was just wondering whether at a sort of psychological level when you sort of step back and think about these existential problems how can you not be struck by the thought that if the multiverse is right then it seems almost inevitable that there are some branches in which humanity or some of the alteration of humanity survives as far as we know. ### David Deutsch 01:16:29 - 01:17:01 I would try not to let my psychological approach to an event come into conflict between what I know is there so I might have an objection to eating a sweet in the form of a tarantula but then I say to myself no this isn't a tarantula this is just a piece of candy and I'm going to eat it ... ### CFI participant 01:17:01 - 01:17:05 And you say well yeah but still what do you feel about it? ### David Deutsch 01:17:05 - 01:17:14 Well what I feel about it that's different from the right answer is not very interesting that's just ways in which I can be wrong. ### CFI participant 01:17:14 - 01:17:39 And speaking of slightly off topic questions so in your last Edge question the last question was, are the ways qualia are related to computation, creativity, free will, risk, probability, morality, and epistemology all the same question? I was wondering if you might be able to elaborate on that. ### David Deutsch 01:17:39 - 01:20:50 Yeah well since at the last question you see I thought I'd allow myself the luxury of talking about things I don't know and these are three things that I don't know they could go either way so for example is morality reducible to epistemology well if it isn't, then what on earth can morality be? Are there moral axioms that are uncriticizable, that would be authoritarian so on the other hand if morality is epistemology, that problem wouldn't arise because we already know how to frame epistemology without requiring foundation without requiring authority and thanks to Popper, we know that but then I can ask myself what if the laws of physics were different like the ones I mentioned with the Greek gods and the malevolence suppose there were malevolence and the laws of physics really did have it in for us and so on would that change morality? Could we say like at the moment we say that if something is a property of the laws of physics it doesn't have a moral value pro or con but in such a universe, am I right in thinking that those kinds of laws of physics are immoral or does that not make sense and I don't know so that's the kind of thing there and I'm also wondering in that question whether there is a connection between that and the question that the gentleman who's on our left asked about whether consciousness and free will and qualia and moral value and all that stuff are all come together necessarily or can they be separate or can they be made separate perhaps up to now in the universe they've always come together but we could artificially make them separate in an AGI. Again I think that can't be so but I can't give you a watertight argument why it isn't so but we have to wait in all these cases we have to wait till somebody comes up with a viable theory of these things you know I'm always a bit perturbed when people have strong feelings about things like free will, moral value, you know is it moral to kill animals eat animals are animals conscious and all those things when they do not know what consciousness is none of us do ... ### Clifford Siskin 01:20:50 - 01:20:55 It's five o'clock I think we should end it now thank you David ### David Deutsch 01:20:55 - 01:21:02 Okay, well thank you all.