The philosopher of artificial intelligence Eliezer Yudkovsky about singularity, Bayesian brain and goblins in the closet

Eliezer Shlomo Yudkovsky is an American expert in artificial intelligence who studies the problems of technological singularity and advocates the creation of Friendly AI. In non-academic circles, he is best known as the author of the fanfiction of “Harry Potter and Rational Thinking” under the auspices of Less Wrong.



I have always been amazed by smart people who believe in things that seem absurd to me. For example, the geneticist and director of the National Institutes of Health, Francis Collins, believes that Jesus has risen from the dead. AI theorist Eliezer Yudkovsky believes that cars ... But I would rather give the floor to him. In 2008, I interviewed him on Bloggingheads.tv, but nothing good came of it, since I decided that he was a follower of the singularity guru Ray Kurzweil . But Yudkovsky did not follow anyone and never attended college. He is a stubborn and original theorist of intelligence, both human and artificial. His work (for example, an essay that helped me understand, or gave the illusion of understanding, the Bayes theorem) exudes the arrogance of a self-taught, whose sharp edges were not polished by formal education - but this is part of his charm. Even when he annoys you, Yudkowski is funny, fresh, provocative. For details of his biography, see his personal website or the website of the Institute for the Study of Machine Intelligence, in the basis of which he participated. And read this interview with the bonus in the form of comments from his wife, Briena.



Horgan : When at the party they ask you what you do, what do you answer?



Yudkovsky : Depends on the event. “I am a specialist in decision making theory,” or “Co-founder of the Institute for the Study of Machine Intelligence,” or, if it is a different type of party, I talk about my artistic works.



X: What is your favorite AI movie and why?



J: The AI in films is horribly standard. Ex Machina came as close to exclusion from this rule as one would expect.



X: Is the college utility overrated?



Yu: I would be surprised if its usefulness was underestimated, given the social requirements for completing it. As far as I know, there is no reason not to believe economists who believe that the college has become a kind of “prestigious product” and that attempts to increase the volume of student loans simply increased the cost of the college and the burden of student debts.



H: Why do you write artistic stories?



Y: To rephrase Wondermark comics: “At first, I tried not to do this, but it didn't work.”



In addition, serious literature conveys knowledge, and fiction - experience. If you want to understand the proofs of the Bayes formula , I can use diagrams. If you want to feel how it is to use Bayesian logic, I need to write a story in which the character does.



H: Are you religious in any sense?



U: No. When you make a mistake, you need to avoid the temptation to go into defense, try to find some point of view from which you are at least a little right. It is much wiser to say, “Oh,” admit that you were not even a little right, swallow the bitter pill whole and continue to live. That is how humanity should relate to religion.



H: If you were the “King of the World,” what would be on the top of your to-do list?



Yu: I once wrote: “The test for libertarian works like this: Imagine that you got the power; What do you think about first of all - about the laws that you will accept, or about the laws that you will repeal? ”I am not 100% libertarian, because not all of my Wishlist are expressed in the abolition of laws and the loosening of restrictions. But I imagine how I would try to create a world in which some unemployed person could offer you a ride up to work, get $ 5 for a 20 minute drive, and nothing bad would happen to him because of this. He would not need to lose unemployment insurance, register a business permit, lose medical insurance, undergo an audit, ask a lawyer to certify the compliance of his work with the rules of the Office of Occupational Safety, etc. He would have just added $ 5.



I would try to return to the state in which it would be as easy to hire an employee as in 1900. Perhaps now there is a sense in some security measures, but I would try to create a security that does not hold back a person and does not produce securities as a result of a simple return of a person to the economy.



I would try to do everything that smart economists have long been shouting about, and that no state does. Replace investment taxes and profits with consumption taxes and real estate taxes. Replace the minimum wage with negative payroll taxes. Establish a nominal GDP targeting policy for central banks and stop supporting structures “too large to go bankrupt”. Require that the loser in court pay for court costs during patent proceedings [ following the so-called English rule - in contrast to US law, according to which each of the parties by default deals with its own costs - approx. trans. ] and return the duration of copyright to 28 years. Remove obstacles to the construction of houses. Copy Singapore medical insurance. E-government in Estonia. Replace committees and complex decision-making processes with specific individuals who make publicly documented decisions and are responsible for this. Conduct controlled experiments with different options for managing countries and take into account their results. I can continue the list for hours.



All this may not matter in two hundred million years. But nominal assets derived from an economic boom can serve well as I try to figure out what we will do with artificial intelligence. The obvious thing is the Manhattan project somewhere on the island, based on competition between the largest hedge funds, in which people can explore the problem of generalized artificial intelligence without publishing the results of their work to automatically bring the world to the end. And unless we accept that I have magical abilities or a fundamentally non-overthrowed regime, I do not see how any law I would accept would have delayed the AI ​​approaching strongly enough on a planet where computers are common everywhere.



But all this can still be considered an impossible mental experiment and in real life the likelihood of such an experiment is zero.



H: What is so good about Bayes theorem?



Yu: Well, for example, it is very deep. Therefore, it is difficult to answer this question briefly.



I could answer that the Bayes theorem can be called the second law of thermodynamics for cognition. If you have concluded that the probability of some assumption is 99%, whether it is milk in the supermarket or the anthropogenic cause of global warming, then you have a combination of fairly good a priori probabilities and fairly reliable evidence. This is not a requirement of regulations, it is the law. Just as a car cannot drive without dissipating entropy, you cannot get a good image of the world without performing a process where there is a Bayesian structure somewhere inside, even if the probabilities are not used directly in the process.



Personally, I think that the most important thing Bayes can offer us is the existence of rules, iron laws that determine whether the way of thinking works to mark reality. Mormons are told that they learn the truth of the Book of Mormon through a burning sensation in the heart. Conservatively, we will accept the a priori probability of the Book of Mormon as one in a billion. Then we estimate the likelihood that the Book of Mormon is not true, and anyone experienced a burning sensation in the heart after being told that this should be expected. If you understand the Bayesian formula, we will immediately understand that the small probability of proof is incommensurable with the small probability of the hypothesis that we are trying to prove with its help. You don’t even need to come up with specific numbers to understand that they don’t converge - as Philip Tetlock discovered in his study of “ super- predictors,” they often knew Bayes formula, but rarely gave certain numbers. In a sense, it is harder to deceive you if you realize that there is some kind of mathematics with which you can accurately determine the strength of the proof and see if it is enough to overcome the small probability of a hypothesis. You can't just invent something and believe it, because it doesn't work that way.



H: Does the Bayesian brain hypothesis impress you?



Yu: I think that some people arguing on this topic talk about different things. Asking whether the brain is a Bayesian algorithm is like asking if the Honda Accord works on the Carnot heat engine . If one person says: “Every car is a thermodynamic process that requires fuel and dissipates parasitic heat,” and another person hears: “If you build a diagram of a Carnot cycle and show its mechanics, he must agree that it looks like the inside of a Honda Accord "Then violent disputes are inevitable.



Some people will be very happy to open the internal combustion engine, find cylinders in it and say: "I am sure that they convert heat into pressure and help move the car forward!" And they will be right, but other people will say: “You focus on the only component of a much larger set of car parts. The catalytic converter is also very important, and it is not on your Carnot cycle diagrams. And sometimes we have an air conditioner that works in exactly the opposite way to how you say the heat engine works. ”



I don’t think it would be surprising if I say that people who say down upon it: “You are clearly unfamiliar with modern cars; you need a whole set of different methods to build a motor, such as candles and catalytic converters, and not just your thermodynamic processes, ”miss the key level of abstraction.



But if you want to know whether the brain can be considered literally Bayesian, and not a device that performs cognitive work, whose nature we can understand using Bayesian methods, then I can answer your question: "No, of course." In this "engine" it is possible and there are several Bayesian "cylinders", but a lot there will look as strange as seat belts and air conditioner. But these additions will not change the fact that in order to properly define an apple based on sensory evidence, something must be done that can be interpreted as a result of induction, which can understand the concept of an apple and be updated on the basis of evidence that distinguishes apples from non-apples.



H: Is it possible to be too rational?



U: You can get into the so-called. "The Valley of Bad Rationality." If before that you were irrational in a few things that balanced each other, then if you become rational, you may become worse than before. The more you become rational, the worse you can become if you choose the wrong direction to apply your skills.



But I would not recommend too much to take care of such an opportunity. In my opinion, people who talk about how to be smartly irrational, they are assholes. It is difficult to come up with a realistic, lifeless situation in which you can decide to be irrational, and the outcome of which is unknown to you. In real life, it's better to tell yourself the truth and not be smart.



It is possible that the ideal representative of Bayesian thinking is incompatible with an interesting and fun life. But this is clearly not such a big problem as our tendency to self-destruction.



H: How does your point of view on singularity differ from that of Kurzweil?



YU:

• I do not think that Moore's law can be applied to AI. AI is a software problem.

• I do not think that the first superhuman intellect will appear from the merging of machines with people. A hundred years have passed since the advent of cars, and we are only now trying to make an exoskeleton for a horse, and an ordinary car is still faster.

• I do not think that the first strong AI will be based on algorithms from neurobiology, just as the planes were not based on birds.

• I don’t think that fusion of nano, info and biotechnology is possible, inevitable, well defined or necessary.

• I think that from 1930 to 1970 there were more changes than from 1970 to 2010.

• I think that in developed countries, productivity stagnates.

• I think the extrapolation of Moore's law to technological progress, which allegedly predicts everything that will happen after the appearance of AI is smarter than a person, is a very strange thing. AI smarter than man destroys all your graphics.

• Some analysts, for example, Illka ​​Tuomi , believe that Moore's law broke in the early 2000s. Not sure I can argue.

• The only technological threshold that interests me is where the AI ​​gains the ability to self-improve. We do not have a schedule going to this threshold, and it is unclear what it will be (although it should not greatly exceed the level of a person, because a person understands computer science), so that his attack cannot be predicted.

• I don’t think that the result of such progress will be good by default. I think it can be made good, but we will need to work hard on this, and key figures are not interested in this. To tell people that we are on a natural trajectory to great and wonderful times will be a lie.

• I think that “singularity” became a suitcase word with too many incompatible values ​​and details inside, so I stopped using it.



X: Do you have the chance to become a super-intelligent cyborg?



Yu: The law of probability conjunction says that P (A & B) <= P (A). The probability of simultaneous occurrence of events A and B is less than the probability of the occurrence of one event A. In experiments in which people consider that P (A & B)> P (A) for two events A and B, there is a " conjunction error " - for example, in 1982, experts from the International Congress of Predictions appointed a greater likelihood of the event “Russia invades Poland and diplomatic ties with the USSR break up” than the probability of a separate event “the breakup of diplomatic ties with the USSR”, appointed by another group. Similarly, another group has assigned a greater likelihood to the event “California earthquake leads to a flood causing thousands of victims”, than another - the probability of the event “Somewhere in North America there is a flood with thousands of victims”. Although the introduction of additional details in the story unambiguously makes it less likely, it makes it more believable. For me, understanding this fact is like a " bridge of donkeys " for serious futurism - the difference between carefully weighing each individual assumption and finding out whether you can support this refinement independently of everyone else and simply composing a wonderful and vivid history.



This is all I say in the context of the answer to the question: “Why do you drag in such a clarification like a cyborg? I don't want to be a cyborg. ” You need to be very careful about adding additional details to the statements.



H: Do you have a chance for immortality?



Y: The literal? Literal immortality is difficult to achieve. To live much longer than a few trillion years, you need to reconsider the expected fate of the expanding universe. To live longer than the googolplex , it is necessary that we make a mistake about the fundamentals of physical laws, and not just in details.



Even if some of the unusual reasoning turns out to be true and our Universe can generate daughter universes, this will not give us immortality. In order to live a lot more years and not repeat a googolplex, you will need computers with more elements than googol, and such a machine will not fit in the Hubble sphere .



And googolplex is not infinity. Paraphrasing Martin Gardner, the Graham number is still quite small, since most of the finite numbers are much larger than him. If you want to demolish the roof, read about the fast-growing hierarchy , and infinity will still be longer. Only very strange and frightening anthropic theories will allow you to live long enough to watch the longest-working Turing machine stop with hundreds of conditions.



However, I do not think that from an emotional point of view, I want to live long enough to see the hundredth number in the game " hunting for a beaver-hard worker ." I can somehow empathize with myself, who has lived a hundred years from now. That future I will be able to empathize with the future myself in another hundred years. And maybe somewhere in this sequence there will be someone who will face the prospect of ending their existence, and he may become very sad about this. But I'm not sure that I can imagine this man. “I want to live another day. Tomorrow I will also want to live another day. Therefore, I want to live forever, proven by the induction of positive integers. ” Even my desire for a long life in a physically possible universe is an abstraction generated by induction. I can not imagine myself in a trillion years.



: , , , . ?



: , . .



, , , , . , , . . – , , ? ?



, () , *, > 1, , . , « ». (), , , . , ? , - , , .



. – , . , , .



– , « » , , , , . – . , , , , , .



, : « , , , , , , ? ?»



, , , , . , , , – . , , , .. etc.



, , , , . ; . , , .



, . , .



, , , 300 , . , , ? , , . , .



, , « », , , . , , , , , , , , .



:



. , , -, . , , – . . , .



, , , . , – , , , .



, , , , . , . – , , , . – .



, , – . , – . , , , , , . , – , , , . , , , 30 , 30 .



: ?



: - , , , . . , , . , , , , , . . «» .



: , . .



: , , ?



: , , , . , , , , . – , , .



: ? - ?



: , . , . , - , - . , , , , , .. , , , . , .



, « » – . – , , . – . , , , , , , , .



– , , , , , , , , , « » « », , , , , , , .



, , , , .



, . .



, -.



: , , , – . ?



: , . , , «» . , , , - . -, , , «».



[ , , , , – . trans. ] , , . , , , . , , , « ?». , .



– , , , - . . , , , , «» «». , -, , .



: ?



: , , .



: ?



: , .



: ?



: " ", , - .