Lex Fridman Podcast

Max Tegmark is a physicist and AI researcher at MIT, co-founder of the Future of Life Institute, and author of Life 3.0: Being Human in the Age of Artificial Intelligence. Please support this podcast by checking out our sponsors:&lt;br /&gt;
- Notion: https://notion.com&lt;br /&gt;
- InsideTracker: https://insidetracker.com/lex to get 20% off&lt;br /&gt;
- Indeed: https://indeed.com/lex to get $75 credit&lt;br /&gt;
&lt;br /&gt;
EPISODE LINKS:&lt;br /&gt;
Max&#039;s Twitter: https://twitter.com/tegmark&lt;br /&gt;
Max&#039;s Website: https://space.mit.edu/home/tegmark&lt;br /&gt;
Pause Giant AI Experiments (open letter): https://futureoflife.org/open-letter/pause-giant-ai-experiments&lt;br /&gt;
Future of Life Institute: https://futureoflife.org&lt;br /&gt;
Books and resources mentioned:&lt;br /&gt;
1. Life 3.0 (book): https://amzn.to/3UB9rXB&lt;br /&gt;
2. Meditations on Moloch (essay): https://slatestarcodex.com/2014/07/30/meditations-on-moloch&lt;br /&gt;
3. Nuclear winter paper: https://nature.com/articles/s43016-022-00573-0&lt;br /&gt;
&lt;br /&gt;
PODCAST INFO:&lt;br /&gt;
Podcast website: https://lexfridman.com/podcast&lt;br /&gt;
Apple Podcasts: https://apple.co/2lwqZIr&lt;br /&gt;
Spotify: https://spoti.fi/2nEwCF8&lt;br /&gt;
RSS: https://lexfridman.com/feed/podcast/&lt;br /&gt;
YouTube Full Episodes: https://youtube.com/lexfridman&lt;br /&gt;
YouTube Clips: https://youtube.com/lexclips&lt;br /&gt;
&lt;br /&gt;
SUPPORT &amp; CONNECT:&lt;br /&gt;
- Check out the sponsors above, it&#039;s the best way to support this podcast&lt;br /&gt;
- Support on Patreon: https://www.patreon.com/lexfridman&lt;br /&gt;
- Twitter: https://twitter.com/lexfridman&lt;br /&gt;
- Instagram: https://www.instagram.com/lexfridman&lt;br /&gt;
- LinkedIn: https://www.linkedin.com/in/lexfridman&lt;br /&gt;
- Facebook: https://www.facebook.com/lexfridman&lt;br /&gt;
- Medium: https://medium.com/@lexfridman&lt;br /&gt;
&lt;br /&gt;
OUTLINE:&lt;br /&gt;
Here&#039;s the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time.&lt;br /&gt;
(00:00) - Introduction&lt;br /&gt;
(07:34) - Intelligent alien civilizations&lt;br /&gt;
(19:58) - Life 3.0 and superintelligent AI&lt;br /&gt;
(31:25) - Open letter to pause Giant AI Experiments&lt;br /&gt;
(56:32) - Maintaining control&lt;br /&gt;
(1:25:22) - Regulation&lt;br /&gt;
(1:36:12) - Job automation&lt;br /&gt;
(1:45:27) - Elon Musk&lt;br /&gt;
(2:07:09) - Open source&lt;br /&gt;
(2:13:39) - How AI may kill all humans&lt;br /&gt;
(2:24:10) - Consciousness&lt;br /&gt;
(2:33:32) - Nuclear winter&lt;br /&gt;
(2:44:00) - Questions for AGI

#371  Max Tegmark: The Case for Halting AI Development

The following is a conversation with Max Tegmark, his third time in the podcast. In fact, his first appearance was episode number one of this very podcast. He is a physicist and artificial intelligence researcher at MIT, co-founder of Future Left Institute, and author of Left 3.0, being human in the age of artificial intelligence. Most recently, he's a key figure in spearheading the open letter calling for a six-month pause on giant AI experiments like training GPT-4. The letter reads, we're calling for a pause on training of models larger than GPT-4 for six months. This does not imply a pause or ban on all AI research and development or the use of systems that have already been placed in the market. Our call is specific and addresses a very small pool of actors who possesses this capability. The letter has been signed by over 50,000 individuals, including 1,800 CEOs, and over 1,500 professors. Signatories include Joshua Benjo, Stuart Russell, Elon Musk, Steve Wozniak, Yuval Noah Harari, Andrew Yang, and many others. This is a defining moment in the history of human civilization, where the balance of power, between human and AI, begins to shift. Max's mind and his voice is one of the most valuable and powerful in a time like this. His support, his wisdom, his friendship, has been a gift of forever deeply grateful for. And now a quick few second mention of each sponsor. Check them out in the description. It's the best way to support this podcast. got notion for project and team collaboration inside tracker for biological data and indeed for hiring. Choose wisely my friends. Also, speaking of hiring, if you want to work with our amazing team, we're always hiring whether it's through indeed or otherwise, could electsfreatment.com slash hiring. And now onto the full arteries, as always, no ads in the middle. I try to make this interesting, but if you must skip them, please still check out our sponsors, I enjoy their stuff. Maybe you will too. This show is brought to you by Notion. A spoken endlessly about how amazing Notion is, how everybody, all the cool kids are recommending it for just basic note-taking, but there's so, so much more. It's the collaborative aspect of it, the project management aspect of it, the wikis, the document sharing, all of that, all in a simple, powerful, beautifully design solution. What can I say? On top of this, there's the notion AI tool. This is the best integration of large language models into a productivity note taking tool. There are so many amazing features. I mean, it's just endless. Go to the website. You can generate entire presentations and reports based on the to-do list. You can summarize stuff. You can shorten stuff. You can generate tables based on the description. You can write a summary. You can expand the text. You You can change the style of the text, you can fix spelling and grammar, you can translate, you can use simple language, more complicated language to change the tone of the voice, make it shorter, longer, like I said, everything is just so easy to play around with and all of it. No matter what you're doing, we'll challenge you to think how you write, it will challenge you to expand the style of your writing, it will save you a lot of time. of course, but I just think it makes you a better thinker and productive being in this world. And I think that's such a great integration of AI into the productivity workflow. To me, it's not enough for a large language model to be effective at answering questions and having good dialogue. You have to really integrate it into the workflow and notion better than anybody else I've seen has done that. So if that's interesting to you, notion AI helps you work faster, write better and think bigger, doing tasks that normally take you hours and just minutes, try notion AI for free when you go to notion.com slash Lex. That's all lowercase notion.com slash Lex to try the power of notion AI today. This shows also brought to you by Inside Tracker, a service that used to track biological data. It's really good to do that kind of thing regularly to look at all the different markers in your body and to understand what could be made better through lifestyle and through diet changes, it's kind of obvious that decisions about your life should be made based on the data that comes from your body. Not some kind of population study, although those are good. Not some spiritual guru, although those are good. Not some novel, whether it's Harry Potter or Dostiyoski, although those are sometimes good. your relative who says I heard a guy say that a guy does this thing that is very bro soundly sounding, although sometimes it turns out to be pretty effective. Overall, the best decisions about your life should be based on the things that come from your own body. Inside track you use these algorithms to analyze your blood data, DNA data, fitness track, all that kind of stuff to give you recommendations. You should be doing it, you should be doing it regularly. So It's not just a one-time thing, but regularly over time you see what changes led to improvements in the various markers that come from your body. You get special savings for a limited time when you go to insidetrack.com slash flex. This show is also brought to you by, indeed, a hiring website. I think the most important thing in life, not to, quote, according to barbarian, because I'll be very inappropriate to quote at this moment. And it's not actually accurate at all as a reflection of what's important in life. It's only has comedic value. What I really want to say about what's important in life is the people you surround yourself with. And we spend so much of our time in the workplace seeking solutions to very difficult problems together, passionately pursuing ambitious goals, sometimes impossible goals. But it's the source of meaning, it's sort of a fappiness for people, and I think part of that happiness comes from the collaboration with other human beings, the sort of professional depth of connection that you have with other human beings, being together through the grind and surviving and accomplishing the goal or failing in a big epic way knowing that you have tried together. So doing that with the right team, I think, is one of the most important things in life. So you should surround yourself with the right team. If you're looking to join a team, you should be very selective about that. Or if you're looking to hire a team, you should be very selective about that and use the best tools of the job. I've used indeed many, many times throughout my life for the teams I've led. Don't overspend on hiring, visit indeed.com slash Lex to start hiring now. it's indeed.com slash Lex, terms, and conditions apply. This is the Lex Freedom Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Max, tag Mark. 

You were the first ever guest on this podcast, episode number one. So first of all, Max, I just have to say, thank you for giving me a chance. Thank you for starting this journey and it's been an incredible journey. And just thank you for sitting down with me and just acting like I'm somebody who matters that I'm somebody who's interesting to talk to. And thank you for doing it. I meant a lot. Thanks to you for putting your heart and soul into this. I know when you delve into controversial topics, it's inevitable to hit your head by what Hamlet talks about the slings and arrows and stuff, and I really it's an era where YouTube videos are too long and now it has to be like a 20 minute TikTok, 20 second TikTok clip. It's just so refreshing to say you're going exactly against all of the advice and doing these really long form things. And then people appreciate it, you know, reality is nuanced. And thanks for sharing it that way. So let me ask you again, the first question I've ever asked on this podcast, episode number one, talking to you, do you think there's an intelligent life out there in the universe. Let's revisit that question. Do you have any updates? What's your view when you look out to the stars? So when we look out to the stars, if you define our universe the way most astrophysicists do, not as all of space, but the spherical region of space, that we can see with our telescopes from which light has the time to reach us since our Big Bang. I'm in the minority. I'm estimate that we are the only life in this spherical volume that has invented, internet, radio has gotten our level of tech and if that's true, then it puts a lot of responsibility on us to not mess this one up because if it's true, it means that life is quite rare and we are stewards of this one spark of advanced consciousness which if we nurture it and help it grow it and then see life can spread from here on into much of our universe and we can have this just amazing future. Whereas if we instead are reckless with the technology we build and just snuff it out due to stupidity or in fighting, maybe the rest of cosmic history in our universe was just going to be a play-frame the benches. But I do think that we are actually very likely to get visited by aliens, alien intelligence quite soon. But I think we are going to be building that alien intelligence. So we're going to give birth to an intelligent alien civilization. Unlike anything human, the evolution here on Earth was able to create in terms of the path, the biological path it took. Yeah and it's going to be much more alien than a cat or even the most exotic animal on the planet right now because it will not have been created through the usual Darwinian competition where it necessarily cares about self-preservation as afraid of death, any of those things. The space of alien minds is just that you can't build is just so much faster than what evolution will give you. And with that also comes a great responsibility for us to make sure that the kind of minds we create are the kind of minds that it's good to create. Minds that will share our values and be good for humanity and life and create minds that don't suffer. Do you try to visualize the full space of alien minds that AI could be? Do you try to consider all the different kinds of intelligences? So generalizing what humans are able to do to the full spectrum of what intelligent creatures entities could do. I try, but I would say I fail. I mean, it's very difficult for a human mind to really grapple with something still completely alien. It even for us, right? If we just try to imagine how would it feel if we were completely indifferent towards death or individuality, even if you just imagine that for For example, you could just copy my knowledge of how to speak Swedish. Boom, now you can speak Swedish, and you could copy any of my cool experiences and then you could delete the ones you didn't like in your own life, just like that. It would already change quite a lot about how you feel as a human being, right? You'd probably spend less effort starting things if you just copy them and you might be less afraid of death because if the plane you're on starts to crash. You just be like, oh, shucks. I'm going to, I had to have him back my brain up for four hours. So I'm going to lose all this wonderful experiences of this flight. We might also start feeling more like compassionate, maybe with other people, if we can so readily share each other's experiences in our knowledge and feel more like a hive mind. It's very hard though, I really feel very humble about this to grapple with it, that how it might actually feel that the one thing which is so obvious though, I think is just really worth reflecting on is because the mind space of possible intelligence is so different from ours, it's very dangerous if we assume they're going to be like us or anything like us. Well, there's the entirety of a human written history has been through poetry, through novels, been trying to describe through philosophy, trying to describe the human condition and what's entailed in it. Like Jessica said, fear of death and all those kinds of things, what is love and all of that changes if you have a different kind of intelligence, and all of it. The entirety, all those poems that are trying to sneak up to what the hell it means to be human, all of that changes. How AI concerns and existential crises sees that AI experiences, how that clashes with the human existential crisis, the human condition. That's hard to, hard to fathom, hard to predict. It's hard, but it's fascinating to think about also. Even in the best case scenario where where we don't lose control over the evermore powerful AI, that we're building to other humans whose goals we think are horrible and where we don't lose control to the machines. AI provides the things that we want. Even then, you get into questions that you touched here. Maybe it's the struggle that it's actually hard to do things is part of the things that gives us meaning as well, right? So for example, I found it so shocking that this new Microsoft, the GPT-4 commercial that they put together, has this woman talking about, showing this demo of how she's gonna give a graduation speech to her beloved daughter, and she asks GPT-4 to write it. If it's frigging 200 words or so, if I realized that my parents couldn't be bothered struggling a little bit to write 200 words and outsourced that to their computer, would feel really offended actually. And so I wonder if eliminating too much of this struggle from our existence? Do you think that would also take away a little bit of what means to be human? Yeah, we can't even predict. I had somebody mentioned to me that they They use, they start using Chaji PT with the 3.5 and the 4.0 to write what they really feel to a person and they have a temper issue and they're basically trying to get Chaji PT to rewrite it in a nicer way, to get the point across but rewrite it in a nicer way. So we're even removing the inner asshole from our communication. So you know, there's some positive aspects of that, but mostly it's just the transformation of how humans communicate. And it's scary because so much of our society is based on this glue of communication. And if we're not using AI as the medium of communication, that does the language for us. much of the emotion that's laden in human communication. It's so much of the intent that's going to be handled by outsourced AI. How does that change everything? How does it change the internal state of how we feel about other human beings? What makes us lonely? What makes us excited? What makes us afraid? How we fall in love? All that kind of stuff. Yeah. For me personally, I have to confess the challenge is one of the things that really makes my life feel meaningful. If I go hike him out into my wife, I don't want to just press a button and be at the top. I want this struggle and come up there sweating and feel, wow, we did this in the same way. I want to constantly work on myself to become a better person. If I say something something in anger that I regret. I want to go back and really work on myself rather than just tell an AI from now on, always filter what I write so I don't have to work on myself because then I'm not growing. Yeah, but then again it could be like with chess. An AI wants it significantly, obviously, supersize the performance of humans. It will live in its own world and provide maybe a flourishing civilization for humans, but we humans will continue hiking mountains and playing our games, even though AI is so much smarter, so much stronger, so much superior in every single way, just like with chess. That's one possible, hopefully trajectory here, is that humans will continue to human and AI will just be a kind of a medium that enables the human experience to flourish. Yeah. I would phrase that as rebranding ourselves from homoesthetians, the homoesthetians. You know, right now, safety and stability to be intelligence, we've even put it in our species name, we're branding ourselves as a smartest information processing entity on the planet. That's clearly going to change if AI continues ahead. So maybe we should focus on the experience instead the subjective experience that we have with Homo sentience, and say, that's what's really valuable. The love, the connection, the other things. Get off our high horses, and get rid of this hubris that only we can do integrals. So consciousness is subjective experiences, a fundamental value to what it means to be human. Make that the priority. That feels like a hopeful direction to me, But that also requires more compassion, not just towards other humans, because they happen to be the smartest on the planet, but also towards all our other fellow creatures on this planet. I personally feel right now we're treating a lot of farm animals horribly, for example. And the excuse we're using is, oh, they're not as smart as us. But if we get to, we're not in the grand scheme of things either in the post-AI epoch, you know. And surely we should value the subjective experience of a cow also. 

Intelligent alien civilizations&lt;br /&gt;

Well, allow me to briefly look at the book, which at this point is becoming more and more visionary that you've written, I guess over five years ago, life 3.0. So first of all, 3.0. What's 1.0? What's 2.0? What's 3.0? And how's that vision sort of evolve, the vision and the book evolve to today? Life 1.0 is really dumb like bacteria and that it can't actually learn anything at all during the lifetime. The learning just comes from this genetic process from one generation to the next. Life 2.0 is us and other animals which have brains which can learn during their lifetime a great deal. And you were born without being able to speak English. And at some point, you decided, hey, I want to upgrade my software. Let's install an English speaking module. So you did. And life 3.0 does not exist yet. It can not replace not only its software the way we can but also it's hardware and that's where we're heading towards at high speed. We're already maybe 2.1 because we can put in an artificial knee, a pacemaker, et cetera, et cetera. And if Newerlink and other companies succeed, we'll be life 2.2, et cetera. But the companies trying to build AGI are trying to make is of course full 3.0 and you You can put that intelligence in something that also has no biological basis whatsoever. So let's constrain some more capabilities, just like the lead from one point under 2.0. There is nevertheless you speaking so harshly about bacteria, so disrespectfully about bacteria. There is still the same kind of magic there that permeates life 2.0 and 3.0. It seems like maybe the thing that's truly powerful about life intelligence and consciousness was already there in 1.0. Is it possible? I think we should be humble and not be so quick to make everything binary and say either it's there or it's not clearly there's a great spectrum and there is even controversy by whether some unicellal or organisms like amoebas can maybe learn a little bit, you know, after all, so apologies if I found anything clear yet. It wasn't my intent. It was more that I wanted to talk up how cool it is to actually have a brain where you can learn dramatically within your lifetime. Typical human. And the higher up you get from 1.0 to 2.0, the more you become the captain of your own ship, the master of your own destiny, and the less you become a slave to whatever evolution gave you, but upgrading your software, we can be so different from previous generations and even from our parents, much more so than even a bacterium, you know, an old fence to them. And if you can also swap out your hardware and take any physical form you want, of course, it's really the sky's the limit. Yeah, so it accelerates the rate which you can perform the computation, the computation that determines your destiny. Yeah, and I think it's worth commenting a bit on what you means in this context. Also, if you swap things out a lot, this is controversial, but my current understanding is that life is best thought of not as a bag of meat or even a bag of of elementary particles, but rather as a system which can process information and retain its own complexity, even though nature is always trying to mess it up. So it's all about information processing and that makes it a lot like something like a wave in the ocean, which is not its water molecules, right? The water molecules bob up and down, but the wave moves forward, it's an information pattern In the same way, you, Lex, you're not the same atoms as during the first time you get with me. You've swapped out most of them, but it's still you. And the information pattern is still there. And if you could swap out your arms and whatever, you can still have this kind of continuity. It becomes much more sophisticated sort of way before and time where the information lives on. I lost both of my parents since our last podcast and it actually gives me a lot of solace that this way of thinking about them, they haven't entirely died because a lot of mommy and daddy's, sorry, I'm getting emotional here, but a lot of their values and ideas and even jokes and so on, they haven't gone away, right? Some of them live on, I can carry on some of them and they also live a lot of other people. So in this sense, even with Life 2.0, we can, to some extent, already transcend our physical bodies and our death. And particularly if you can share your own information, your own ideas with many others, like you do in your podcast, then, you know, that's the closest the immortality we can get with our bio bodies. You carry a little bit of them in you and some sense. Yes. Yeah. Do you miss them? You miss your mom and dad? Of course. Of course. What did you learn about life from them if it can take a bit of a tangent? I don't know so many things. For starters, my fascination for math and the physical mysteries of our universe, you got a lot of that for my dad. But I think my obsession for freely big questions and consciousness and so on that actually came mostly from my mom. And when I got from both of them, which is a very core part of really who I am, I think is is this feeling comfortable with not buying into what everybody else is saying. When I think it's right, they both, they're just putting, you know, they're their own thing and sometimes they got flagged for it. That's why you've always been an inspiration to me that you're at the top of your field and you're still willing to tackle the big questions in your own way. You're one of the people that represents MIT best to me. You've always been an inspiration in that. So it's good to hear that you got that from you all of that yeah you're too kind but yeah I mean the good reason to do science is because you're really curious you want to figure out the truth if you think this is how it is and everyone else says no no that's bullshit and it's that way you know you you stick with what you think is true and and even if Everybody keeps thinking, it's bullshit, there's a sort of, I always root for the underdog when I watch movies and my dad once, one time for example when I wrote one of my craziest papers ever, I'm not talking about our university ultimately being mathematical, which we're not going to get into today. I got this email from a quick famous professor saying this is not only on bullshit, but it's going to ruin your career, you should stop doing this kind of stuff, I sent it to my dad. Do you know what he said? What did he say? He replied with a quote from Dante. Seggy, he'll do a corsor, lassidida, lajenta. Follow your own path and let the people talk. Go dad. This is the kind of thing. He's dead, but that attitude is not. How did losing them as a man, as a human being, changed you? How did he expand your thinking about the world, how did it expand? I think it about, you know, this thing we're talking about, which is humans creating another living sentient perhaps being. I think it mainly do two things. One of them just going through all their stuff, they had passed away and so on. Just drove home to be how important it is to ask yourselves, why are we doing this things we do? Because it's inevitable that you look at some things they spent an enormous time on and you ask, can hindsight would they really have spent so much time on this? Would they have done something that was more meaningful? So I've been looking more in my life now and asking, you know, why am I doing what I'm doing? And I feel So it should either be something I really enjoyed doing or it should be something that I find really, really meaningful because it helps humanity, um, if it's none of those two categories, maybe I should spend less time on it, you know, the other thing is dealing with death, up in person, like this, it's actually made me less afraid of even less afraid of other people telling me that I'm an idiot, you know, which happens regularly. And just let my life do my thing. And it's made it a little bit easier for me to focus on what I feel is really important. about fear of your own death as it made it more real that this is that this is something that happens. Yeah, it's made it extremely real and I'm next in line in our family now, right? It's me and my brother, my younger brother. But they both handled it such dignity. That was a true inspiration also. They never complained about things and you know, when you're old and your body starts falling part. It's more and more to complain about. They looked at what could they still do? That was meaningful, and they focused on that, rather than wasting time talking about or even thinking much about things they were disappointed in. I think anyone can make themselves depressed if they start their morning by making a list of grievances, whereas if you start your day on the little meditation and things are grateful for it. You basically choose to be a happy person because you only have a finite number of days. You should spend them. Make your count. Being grateful. 

Life 3.0 and superintelligent AI&lt;br /&gt;

Open letter to pause Giant AI Experiments&lt;br /&gt;

You know, you didn't with just because cars killed a lot of people. He didn't ban cars But they got together a bunch of people undecided, you know in order to be allowed to sell a car It has to have a seat belt in them They're the analogous things that you can start requiring a future AI systems so that they are safe and But once this heavy lifting, this intellectual work has been done by experts in the field, which can be done quickly, I think it's been going to be quite easy to get policy makers to see, yeah, this is a good idea, and for the companies to fight malloc, they want, and I believe Sam Altman has explicitly called for this, they want the regulators to actually adopted so that their competition is going to abide by it too, right? You don't want, you don't want to be enacting all these principles, then you abide by them, and then there's this one little company that doesn't sign on to it, and then now they can gradually overtake you, then the companies will get, be able to sleep, secure knowing that everybody's playing by the the same rules. So do you think it's possible to develop guard rails that keep the systems from basically damaging irreparably humanity while still enabling sort of the capitalist fueled competition between companies as they develop how to best make money with the AI? Do you think there's a balance saying that's possible? Absolutely. We've seen that in many other sectors where you've had the free market produce quite good things without causing particular harm. When the guardrails are there and they work, you know, capitalism is a very good way of optimizing for just getting the same things on more efficiently. But it was good, you know, like in hindsight, and I never met anyone, even part is way over on the right. In any country who think it was a bad, it thinks it was a terrible idea to ban child labor, for example. Yeah, but it seems like this particular technology has gotten so good, so fast, become powerful to a degree where you could see in the near term, the ability to make a lot of money, and to put guardrails, to develop guardrails quickly in that kind of context seems to be tricky. It's not similar to cars or child labor. It seems like the opportunity to make a lot of money you hear very quickly is right here. Yeah. There's this cliff. Yeah. It's close. It's close. It's close. It's close. There you go. There are more money. There is more gold and ink gets thrown on the ground. You can pick up or whatever. It's going to dry there very fast. But it's not in anyone's incentive that we go over the cliff. And it's not like everybody's in their own car. All the cars are connected together with a chain. So if it goes over, they'll start dragging others down. The others down too. And so ultimately, it's in the selfish interests, also of the people in the companies to slow down when it's just seeing the countries of the cliff there in front of you, right? And the problem is that even though the people who are building the technology and the CEOs, they really get it, the shareholders and these other market forces, they are people who don't, honestly, understand that the cliff is there. that they usually don't, you have to get quite into the weeds. To really appreciate how powerful this is and how fast and a lot of people are even still stuck again in this idea that intelligent and this carbon chauvinism, as I like to call it, that you can only have our level of intelligence in humans, that there's something magical about it, whereas the people in tech companies who build this stuff, they all realize that intelligence is information processing of certain kind. And it really doesn't matter at all whether the information is processed by carbon atoms in neurons in brains or by silicon atoms in some technology we build. So you brought up capitalism earlier and there are a lot of people who love capitalism and a lot of people who really really don't. And it's It struck me recently that what's happening with capitalism here is exactly analogous to the way in which superchargers might wipe us out, so you know, you know, I studied economics for my undergrad, Stockholm School of Economics, yeah, no, I tell them. So I was very interested in how you could use market forces to just get stuff done more efficiently, but give the right incentives to the market so that it wouldn't do really bad things. So Dylan had Phil Manel, who's a professor and colleague of mine at MIT, wrote this really interesting paper with some collaborators recently, where they proved mathematically that if you just take one goal that you just optimize for, on and on indefinitely, do you think it's going to bring you in the right direction? What basically always happens is, In the beginning, it will make things better for you. But if you keep going, at some point it's going to start making things worse for you again. And then gradually it's going to make it really, really terrible. So just as a simple, the way I think of the proof is, suppose you want to go from here back to Austin, for example, and you're like, okay, yeah, let's just go south, but you put in exactly the right sort of the right direction. Just optimize that. It's south as possible. close from closer to Austin, but there was always some little mirror, so you're not going exactly towards Austin, but you get pretty close, but eventually you start going away again, and eventually you're going to be leaving the solar system. And they proved it's a beautiful mathematical proof, this happens generally, and this is very important for AI, because even though Stuart Russell has written a book and given a lot of talks on why it's a bad idea to have AI just blindly optimize something. That's what pretty much all our systems do. Yeah, we have something called the loss function that we're just minimizing or reward function. We just minimize maximizing stuff. And capitalism is exactly like that too. We wanted to get stuff done more efficiently that people wanted. So you introduce the free market. Things got done much more efficiently than they did in, say, communism. And it got better. But then it just kept optimizing and kept optimizing. And you got every bigger companies and every more efficient information processing. And also very much powered by IT. And eventually, a lot of people are beginning to feel Wait, we're kind of optimizing a bit too much. Like why did we just chop down half the rainforest? You know, and why did suddenly these regulators get captured by lobbyists and so on? It's just the same optimization that's been running for too long. If you have an AI that actually has power over the world and you just give it one goal and it's just like keep optimizing that, most likely everybody's gonna be like, yeah, it's great in the beginning. Things are getting better. But, it's almost impossible to give it exactly the right direction to optimize in. And then, eventually, I'll hear you, but it breaks loose, right? Nick Bostrom and others have given an example to sound quite silly, like, what if you just want to, like, tell it to cure cancer or something? And that's all you tell it. Maybe it's going to decide to take over entire continents just so we can get more super computer facilities in there and figure out your cancer backwards, and then you're like, wait, that's not what I want, then, right? And the issue with capitalism and the issue with the front and away, yeah, I have kind of merged now because the malloc I talked about is exactly the capitalist malloc that we have built an economy that has, is optimized for only one thing, profit, right? And that were great back when things were very inefficient and then now it's getting them better, and it worked great as long as the companies were small enough that they couldn't capture the regulators, but that's not true anymore, but they keep optimizing. And now they realize that they can, these companies can make even more profit by building ever more powerful AI, even if it's reckless, but optimized more and more and more and more and more and more. So this is mall lock again showing up and I just want to anyone here who has any concerns about late stage capitalism having gone a little too far. You should worry about super intelligence because it's the same villain in both cases. It's mall lock. And optimizing one objective function aggressively blindly is going to take us there. And we have this pause from time to time and look into our hearts. And that's why are we doing this, is this, am I still going towards Austin or am I going too far, you know, maybe we should change direction? And that is the idea behind a halt for six months, why six months, that seems like a very short period? Just can we just linger and explore different ideas here? Because this feels like a really important moment in human history where a pause in would actually have a significant possible effect. We said six months because we figured the number one pushback that we were going to get in the West. I was like, but China and everybody knows there's no way that China is going to catch up with the West on this in six months. So it's that argument goes off the table and you can forget about geopolitical competition and just focus on the real issue. that's why we put this. That's really interesting. But you've already made the case that even for China, if you actually want to take on that argument, China too would not be bothered by a longer halt because they don't want to lose control even more than the West doesn't. That's what I think. That's a really interesting argument. I have to actually really think about 

that, which the kind of thing people assume is if you develop an AGI that open AI, if they're the they're the ones that do it, for example, they're going to win. But you're saying, no, they're everybody loses. Yeah, it's gonna get better and better and better, and then capital and we all lose. That's what's gonna happen. Win lose and win a defined and a metric of basically quality of life for human civilization and for Sam Altman. I hope to be people on my personal guess, and people think quibble with this is that we're just gonna, there won't be any humans. That's it. That's what I mean by lose. You know, if you we can see in history, once you have some species or some group of people who aren't needed anymore, doesn't usually work out so well for them, right? Yeah, there were a lot of horses for we're used for traffic in Boston and then the car got amended and most of them got, you know, well, we don't need to go there. And if you look at them. humans, you know, right now, why did the labor movement succeed and after the industrial revolution because it was needed? Even though we had a lot of mollocks and there was child labor and so on, you know, the company still needed to have workers and that's why strikes had power and so on. If we get to the point where most humans aren't needed anymore, I think it's quite naive to think that they're going to still be treated well. You know, we say that, yeah, yeah, everybody's equal and the government will always, we won't just protect them. But if you look in practice, groups that are very disenfranchised and don't have any actual power, usually get screwed. And now in the beginning, so industrial revolution, we automated away muscle work. But that got, went, worked out pretty well eventually because we educated ourselves and started working with our brains instead and got used the more interesting, better paid jobs. But now we're beginning to replace brain work. So we replaced a lot of boring stuff, like we got the pocket calculator, so you don't have people adding multiplying numbers anymore at work. Fine, there were better jobs that could get. But now, GPT-4, you know, and the stable diffusion and techniques like this, They're really beginning to blow away some jobs that people really love having. It was a heartbreaking article just post just yesterday, social media I saw, but this guy who was doing 3D modeling for gaming and all of a sudden, now they got this new software. He just says prompts and he feels this whole job that he loved, slaughter its meaning. And I asked the GPT-4 to rewrite twinkle twinkle little star in the style of Shakespeare. I couldn't have done such a good job. It was just really impressive. You've seen a lot of the arts coming out here, right? So I'm all for automating away the dangerous jobs and the boring jobs. But I think you hear some arguments which are too glib sometimes people say, well, that's all that's going to happen. We're getting rid of the boring, boring tedious dangerous jobs. It's just not true. There are a lot of really interesting jobs that are being taken away now. Journalism is going to get crushed. Coding is going to get crushed. I predict the job market for programmers, salaries are going to start dropping. You said you can code five times faster, then you need five times fewer programmers. Maybe there will be more output also, but you'll still end up using fewer programmers than today. And I love coding. You know, I think it's super cool. So we need to stop and ask ourselves why again, are we doing this as humans? I feel that AI should be built by humanity for humanity. And let's not forget that. It shouldn't be by malloc for malloc. Or what it really is now it's going to buy a humanity for malloc, which doesn't make any sense. It's for us. They were doing it. And it would make you a lot more sense if we build, develop, figure out, gradually safely how to make all this tech. And then we think about, what are the kind of jobs that people really don't want to have? Automate them all the way. And then we ask, what are the jobs that people really find meaning in? Like maybe taking care of children They care center, maybe doing art, et cetera, et cetera. And even if it were possible to automate that way, we don't need to do that, right? We built these machines. What's possible that we redefine or rediscover what are the jobs that give us meaning? So for me, the thing, it is really sad. Like, I have the time I'm excited, have the time I'm crying as I'm generating code because I kind of love programming. It's the act of creation. You know, an idea, you design it and then you bring it to life and it does something, especially if there's some intelligence to it does something. It doesn't even have to have intelligence, printing hello world on screen. You made a little machine and it comes to life. And there's a bunch of tricks you learn along the way because you've been doing it for many, many years. And then to see AI be able to generate all the tricks you thought were special. Yeah. I don't know. It's very scary. It's almost painful. Like a loss of innocence maybe. Like maybe when I was younger, I remember before I learned that sugar is bad for you, you should be on a diet. I remember I enjoyed candy deeply in a way. I just can't anymore. that I know is bad for me. I enjoyed it unapologetically, fully just intensely. And I just, I lost that. Now I feel like a little bit of that is lost for me with being lost with programming. Similar as it is for the 3D modeler, no longer being able to really enjoy the art of modeling 3D things for gaming. I don't know, I don't know what to make sense of that maybe I would rediscover that the true magic of what it means to be human is connecting with other humans to have conversations like this. I don't know, to have sex, to have to eat food, to really intensify the value from conscious experiences versus like creating other stuff. You're pitching the rebranding again from Homo sapiens, the Homo sapiens, the meaningful experiences. And just to inject some optimism in this here, so we don't sound like it plus a gloomers. You know, we can totally have our cake and eat it. You hear a lot of totally bullshit claims we can't afford having more teachers have to cup the number of nurses, you know, that's just nonsense, obviously. With anything, even quite far short of AGI, we can dramatically improve growth of GDP and produce wealth of goods and services. It's very easy to create a world where everybody is better off than today, including the richest people can be better off as well, right? It's not a zero-sum game in technology. Again, you can have two countries like Sweden and Denmark, Canada, these ridiculous wars, century after century. And sometimes that Sweden got a little better off because it got a little bit bigger and then Denmark got a little bit better off because Sweden got a little bit smaller. But then we then technology came along and we both got just dramatically wealthier without taking away from anyone else. So it was just a total win for everyone. And AI can do that on steroids. If you can build safe AGI, if you can build super intelligence, you know, basically all the limitations that cause harm today can be completely eliminated. Right? So wonderful possibility. And this is not sci-fi. This is something which is clearly possible according to the laws of physics. And we can talk about ways of making it safe also. But unfortunately, that'll only happen if we steer in that direction. That's absolutely not the default outcome. That's why income inequality keeps going up. That's why the life expectancy in the US has been going down. Now, I think it's four years in a row. I just read a heartbreaking study from CDC about how something had one-third of all teenage girls in the US been thinking but suicide, you know, like those are steps in there, totally the wrong direction and it's 

important to keep our eyes on the prize here that we can we have the power now for the first time in the history of our species to harness artificial intelligence to help us really flourish and help bring out the best in our humanity rather than the worst of it. To help us have really fulfilling experiences that feel truly meaningful, and you and I shouldn't sit here and dictate the future generations where they will be, let them figure it out, but let's give them a chance to live, and not foreclose all these possibilities for them, just messing things up, right? solve the AI safety problem. It would be nice if we're going to link on exploring that a little bit. So one interesting way to enter that discussion is you tweeted and Elon replied, you tweeted, let's not just focus on whether GPT-4 will do more harm or good on the job market, but also whether it's coding skills will hasten the arrival of super intelligence. That's something we've been talking about. So Elon proposed one thing in the reply saying, maximum truth seeking is my best guess for AI safety. Can you maybe steal me on the case for this, this objective function of truth and maybe make an argument against it in general? What are your different ideas to start approaching the solution to AI safety? I didn't see that reply actually. But I really resonate with it because AI is not evil. It caused people around the world to hate each other much more. But that's because we made it in a certain way. It's a tool. We can use it for great things and bad things. And we could just as well have AI systems. And this is part of my vision for success here. Truth-seeking AI that really brings us together again. again. Why do people hate each other so much between countries and within countries? It's because these have totally different versions of the truth, right? If they all had the same truth that they trusted for good reason, just they could check it and verify it and not have to believe in some self-proclaimed authority, right? There wouldn't be as nearly as much hate. There'd be a lot more understanding instead. And this is, I think, something AI can help enormously with. For example, a little baby step in this direction is this website called Metaculous, where people bet and make predictions not for money, but just for their own reputation. And it's kind of funny, actually. You treat the humans like you treat AI as you have a lost function where they get penalized It's super confident on something and then the opposite happens where if you're kind of humble and then you're like, I think it's 51% chance this is going to happen and then the other happens, you don't get penalized much. And what you can see is that some people are much better at predicting than others. They've earned your trust, right? One project that I'm working on right now is the outgrowth of improved the news foundation together with the Metaculous folks is seeing if we can really scale this up a lot with more powerful AI. Because I would love it, I would love for there to be a really powerful truth-seeking system where that is trustworthy because it keeps being right about stuff and people come to it and maybe look at its latest trust ranking of different pundits and newspapers, etc. If they want to know why someone got a low score, they can click on it and see all the predictions that they actually made and how they turned out, you know, this is how we do it in science. You trust scientists like Einstein who said something, everybody thought it was bullshit and turned out to be right. You get a lot of trust points and he did it multiple times even. I think AI has the power to really heal a lot of the rifts were saying by creating trust system, it has to get away from this idea today with some fact checking site which might themselves have an agenda and you just trust it because of its reputation. You want to have at the sort of systems they earn their trust and they're completely transparent. This I think would actually help a lot that I think help heal the very dysfunctional conversation that humanity has about how it's going to deal with all its biggest challenges in the world today, and then, on the technical side, you know, another common sort of gloom, comment I get from people who are saying, we're just screwed, there's no hope, is, well, things like GPT-4 are way too complicated for a human to ever understand and prove that they can be trustworthy, they're forgetting that AI can help us prove that things work. And there's this very fundamental fact that in math, it's much harder to come up with a proof than it is to verify that the proof is correct. You can actually write a little proof checking code. It's quite short, but you can just human understand. And then it can check the most monstrously long proof ever generate and even by a computer and say, yeah, this is valid. So, right now, we have this approach with virus checking software that it looks to see if there's something you should not trust it, and if it can prove to itself that you should not trust that code, it warns you, right? Why don't you flip this around, and this is an idea I should give credit to Steve on a hundred or four. So that it will only run the code if it can prove, instead of not running it, if it can prove that it's not trustworthy, if it will only run and if it can prove that it's trustworthy. So that's the code, prove to me that you're going to do what you say you're going to do. It gives you this proof and you're a little prooftaker can check it. Now you can actually trust an AI that's much more intelligent than you are, right? Because it's a problem to come up with this proof that you could never have found, but you should trust it. So, this is an interesting point. I agree with you, but this is where Elie Ezer, your call scheme, I disagree with you. His claim, not with you, but with this idea, his claim is a super intelligent AI would be able to know how to lie to you with such a proof. How to light you and give me a proof that I'm going to think is correct? Yeah. But it's not me as lying to you, that's the trick my proof checker. So he has a code. So his general idea is a super intelligent system can lie to a dumber proof checker. So you're going to have as a system because more and more intelligent, there's going to be a threshold where a super intelligent system would be able to effectively lie to a slightly Dumber AGI system like there's a threat like you really focuses on this week AGI the strong AGI jump with a strong AGI Can make all the week AGI think that it's just one of them But it's no longer that And that leap is when it runs away. Yeah, I don't buy that argument I think no matter how Super intelligent and AI is it's never going to be able to prove to me that there are only finitely many primes for example. It's just, it just can't. And it can try to snow me by making up all sorts of new weird rules of deduction that say trust me, you know, the way your proof checker works is too limited and we have this new hypermath and it's true. But then I would just take that, okay, I'm going to forfeit some of these supposedly super cool technologies. I'm only going to go with the ones that I can prove in my own trusted proof gesture. Then I don't, I think it's fine. There's still, of course, this is not something anyone has successfully implemented at this point. But I think I just give it as an example of hope. We don't have to do all the work ourselves. This is exactly the sort of very boring and tedious task is perfect to outsource to an AI. And this is a way in which less intelligent agents, like us, can actually continue to control and trust more powerful ones. So build AGI systems that help us defend against other AGI systems? Well, for starters, begin with a simple problem of just making sure that the system that you own or it's supposed to be loyal to you has to prove to itself that it's always going to do the things that you actually wanted to do, right? And if it can't prove it, maybe it's still going to do it, but you won't run it. So you just forfeit some aspects of all the cool things that I can do. I've had your dollars in doughnuts, it can still do some incredibly cool stuff for you. There are other things too, that we shouldn't speak under the rug, like not every human agrees on exactly what direction we should go with humanity, right? Yes. And you've talked a lot about geopolitical things on your podcast to this effect, you know. But I think that shouldn't distract us from the fact that there are actually a lot of things that everybody in the world virtually agrees on that hey you know like having no humans on the planet and in a near future let's not do that right you look at something like the United Nations Sustainable Development Goals so I'm quite ambitious and basically all the countries agree you as China Russia Ukraine all agree so instead of quibbling about the little things we don't agree on. Let's start with the things we do agree on and get them done. Instead of being so distracted by all these things, we disagree on that malloc wins because frankly malloc going wild now, it feels like a war on life playing out in front of rise. If you just look at it from space, you know, we're on this planet, beautiful vibrant ecosystem. Now we start chopping down big parts of it, even though nobody, most people thought that was a bad idea, always start doing ocean acidification, wiping out all sorts of species. Now we have all these close calls, we almost had a nuclear war, and we're replacing more and more of the biosphere with non-living things. We're also replacing in our social lives a lot of the things which we're so valuable to humanity. A lot of social interactions now are replaced by people staring into their rectangles, right? And I I'm not a psychologist. I'm out of my depth here. But I suspect the part of the reason why teens suicide and suicide in general in the US, the directed breaking levels is actually caused by, again, AI technologies and social media, making people spend less time with actually human interaction. We've all seen a bunch of good looking people and restaurants into the rectangles instead of looking into each other's eyes. So that's also part of the war on life that we're replacing so many really life-affirming things by technology. We're putting technology between us. The technology that was supposed to connect us is actually distancing ourselves from each other. And then we're giving every more power to things which are not alive. These large corporations are not living things, right? They're just maximizing profit. I want to win them more in life. I think We humans, together with all our fellow living things on this planet will be better off if we can remain in control over the non living things and make sure that they work for us. I really think it can be done. Can you just linger on this, maybe, high level philosophical disagreement with Eliezer Yudgowski in the hope you're stating. So he is very sure. He puts a very high probability, very close to one, depending on the day he puts it at one, that AI is going to kill humans. just he does not see a trajectory which it doesn't end up with that conclusion. What trajectory do you see that doesn't end up there and maybe can you see the point he's making and can you also see a way out? First of all I tremendously respect Oh, yes, you're Cowsky and his thinking. Second, I do share his view that there's a pretty large chance that we're not going to make it as humans. There won't be any humans on the planet. And then onto distant future. And that makes me very sad. We just had a little baby. And I keep asking myself, you know, is how old is even in a gap. You know, and I asked myself, it feels, I said to my wife recently, it feels a little bit like I was just diagnosed with some sort of cancer, which has some risk of dying from and some risk of surviving, except this is a cancer which is kill all of humanity. So I completely take seriously his concerns. I think, but I don't absolutely don't think it's hopeless. I think there is, first of all, a lot of momentum now, for the first time, actually, since the many, many years that have passed, since I and many other started warning about this, I feel most people are getting it now. I just was talking to this guy in the gas station there a house the other day and he's like, I think we're getting replaced and then I think in it. So that's positive that there finally was to finally seeing this reaction, which is the first step towards solving the problem. Second, I really think that this vision of only running AI really if the stakes are really high, they can prove to us that they're safe. It's really just virus checking in reverse again. I think it's scientifically doable. I don't think it's hopeless. We might have to forfeit some of the technology that we could get if we were putting blind faith in our eyes, but we're still going to get amazing stuff. Do you envision a process with a proof checker, like something like GPT-4, GPT-5, I will go through a process of rigorous communication. No, I think it's hopeless. That's like trying to prove there about feist spaghetti. What I think, well, the whole vision I have for success is instead that, you know, just like, we human beings were able to look at our brains and distill out the key knowledge. Galileo, when his dad threw him an apple when he was a kid, he was able to catch it because his brain could and his funny spaghetti kind of way predict how parabolas are going to move his conimon system one, right? But then he got older and he's like, wait, this is a parabola. It's Y equals X squared. I can distill this knowledge out, and today you can easily program it into a computer, and it can simulate not just that, but how to get to Mars and so on, right? I envision a similar process where we use the amazing learning power of neural networks to discover the knowledge in the first place, but we don't stop with a black box and use that. We then do a second round of AI where we use automated systems to extract out the knowledge and see what are the insights it's had and then we put that knowledge into a completely different kind of architecture or programming language or whatever that's made in a way that it can be both really efficient and also is more amenable to very formal verification. That's my vision. I'm not saying, sitting here saying I'm confident 100% sure that it's going to work. But I don't think it's a chance to certainly not zero either. And it will certainly be possible to do for a lot of really cool AI applications that we're not using now. So we can have a lot of the fun that we're excited about. If we do this, we're going to need a little bit of time. That's why it's good to pause and put in place requirements. One more thing also, I think someone might think, zero percent chance we're going to survive. Let's just give up, right? That's very dangerous because there's no more guaranteed way to fail than to convince yourself that it's impossible and not to try. you know when you study history and military history the first thing you learn is that that's how you do psychological warfare you persuade the other side that it's hopeless so they don't even fight and then of course you win right let's not do this psychological warfare on ourselves and say there's a hundred percent of our ability we're all screwed anyway It's sadly, I do get that a little bit, sometimes from young people who are so convinced that we're all screwed, that they're like, I'm just going to play computer games and do drugs because we're screwed anyway, right? It's important to keep the hope alive because it actually has a causal impact and makes it more likely that we're going to succeed. It seems like the people that actually build solutions to a problem seemingly impossible to solve problems are the ones that believe. Yeah, they were the ones who are the optimists. Yeah. And it's like, uh, it seems like there's some fundamental law to the universe where fake it till you make it kind of works. Like believe it's possible and it becomes possible. Yeah. Was it Henry Ford who said that if you can, if you tell yourself that it's impossible, it is. Let's not make that mistake. Yeah. And this is a big mistake society is making, you know, I think all in all, everybody is so gloomy and the media are also very biased towards if it bleeds, it leads and gloom and doom, right? So most visions of the future we have are dystopian, which really demotivates people. We want to really, really, really focus on the upside also to get people the willingness to fight for it. And for AI, you and I mostly talked about gloom here again, but let's not remember, but forget that, you know, we have probably both lost someone we really cared about to some disease that we were told were was incurable. Well, it's not. There's no law of physics saying we have to die of that cancer or whatever. Of course you can secure it. And there are so many other things that we, with human intelligence, have also failed to solve on this planet, which AI could also very much help us with, right? So if we can get this right, just be a little more chill and slow down a little bit so we get it right. It's mind-blowing how awesome our future can be. We talked a lot about stuff on Earth can be great. But even if you really get ambitious and look up at the skies, there's no reason we have to be stuck on this planet for the rest of the remains for billions of years to come. We totally understand now that lots of physics let life spread out into space, to other solar systems, to other galaxies, and flourish for billions of billions of years. And this to me is a very, very hopeful vision that really motivates me to fight them. Coming back to the end of something you talked about again, you know, that's the struggle. How the human struggle is one of the things that also really gives meaning to our lives. If there's ever been an epic struggle, this is it and isn't it even more epic if you're the underdog, if most people are telling you this is going to fail, it's impossible. And you persist and you succeed. That's what we can do together as a species on this one. A lot of pundits are ready to count this out. Both in the battle to keep AI safe and becoming multi-punatory species. and they're the same challenge. If we can keep AI safe, that's how we're going to get multi-planetary very efficiently. I have some sort of technical questions about how to get it 

right. So one idea that I'm not even sure what the right answer is to is should systems like GPT-4 be open sourced in a whole other part? Can you see the case for either? there. I think the answer right now is no. I think the answer early on was yes, so we could bring in all the wonderful great thought process of everybody on this. But asking should we open source GPT-4 now is just the same as if you say, well, should we open resource, how to build really small nuclear weapons should we open source, how to make bio weapons, how should the open source, how to make a new virus that kills 90% of everybody who gets it. Of course, we shouldn't. So it's already that powerful. It's already that powerful that we have to respect the power of the systems we've built. The knowledge that you get from open sourcing everything we do now might very well be powerful enough that people looking at that can use it to build the things that you're really threatening again. Let's get it. Remember, OpenAI is GPT-4 is a baby AI. Baby, sort of baby proto almost a little bit AGI according to what Microsoft's recent paper said, right? It's not that they were scared of, what we're scared about is people taking that who might be a lot less responsible than the company that made it, right? And just going to town with it. That's why, we wanna, it's an information hazard. There are many things which are not open sourced right now in society for a very good reason. how do you make certain kind of very powerful toxins out of stuff you can buy in Home Depot we don't open source those things for a reason and this is really no different. I'm saying that I have to say it feels it a bit in a way that we are to say it because MIT is like the cradle of the open source movement and I love open source in general power to the people Let's say. But there's always going to be some stuff that you don't open source. And, you know, it's just like you don't open source. So we have a three month old baby, right? When he gets a little bit older, we're not going to open source to him all the most dangerous things you can do in the house. But it does, it's a weird feeling because this is one of the first moments in history where there's a strong case to be made not open source software. This is when the software has become too dangerous. But it's not the first time that we didn't want to open source technology. Is there something to be said about how to get the release of such systems, right, like GPT-4 and GPT-5. So opening I went through a pretty rigorous effort for several months. You could say it could be longer but nevertheless it's longer than you would have expected. I've tried to test the system to see like what are the ways it goes wrong, to make it very difficult for people. Some would difficult for people to ask things, how do I make a bomb for one dollar? Or how do I say I hate a certain group on Twitter in a way that doesn't give me a block from Twitter, banned from Twitter, those kinds of questions. So you basically use the system to do harm. Is there something you could say about ideas? You have it's just on looking, having thought about this problem of AI safety, how to release a system, how to test the systems when you have them inside the company. So a lot of people say that the two biggest risks from large language models are it's spreading disinformation, harmful information, various types and second being used for offensive cyber weapon. I think that was not the two greatest threats. They're very serious threats and it's wonderful that people are trying to mitigate them. So much bigger elephant in the room is how it's just going to disrupt very economy in a huge way obviously and maybe take away a lot of the most meaningful jobs. And an even bigger one is the one we spent so much time talking about here that this becomes the bootloader for the more powerful AI. rice code connected to the internet when it paid humans. Yeah. And before we know it, we have something else, which is not at all large language model. It looks nothing like it, but which is way more intelligent and capable than it has goals. And that's the elephant in the room. And obviously, no matter how hard any of these companies have tried, that's not something that's easy for them to verify with a large language models. And the only way to be really lower that risk a lot would be to not let, for example, to not never let it read in a code, not train on that, and not put it into an API, and not to give it access to so much information about how to manipulate humans. So, but that doesn't mean you still can't make a lot a ton of money on them, you know. We're just just watch now this coming year, Microsoft is rolling out the new office suite where you go into the Microsoft Word and give it a prompt and it writes the whole text for you and then you edit it and then you I'll give me a PowerPoint version of this and it makes it and now I'll take the spreadsheet and Bala and you know all of those things I think are you can debate the economic impact of it and where the society is prepared to deal with this disruption but those are not the things which That's not the elephant of the room that keeps me awake at night for wiping out humanity. And I think that's the biggest misunderstanding we have. A lot of people think that we're scared of automatic spreadsheets, that's not the case. That's not what Elie Azer was freaked out about either. 

Is there, in terms of the actual mechanism of how AI might kill all humans? So something you've been outspoken about, you've talked about a lot, is it autonomous weapons systems? So the use of AI in war? Is that one of the things that still you carry a concern for? As these systems become more and more powerful? I carry a concern for it, not that all humans are going to get killed by slaughterboards, but rather just as express route into Orwellian dystopia, where it becomes much easier for very few to kill very many, and therefore it becomes very easy for very few to dominate very many, right? Yeah, if you want to know how I could kill all people, just ask yourself, we humans have driven a lot of species extinct. How do we do it? You know, we were smarter than them. Usually we didn't do it even systematically by going around one on one, one after the other and stepping on them or shooting them or anything like that. we just like chalked down their habitat because we needed it for something else. In some cases we did it by putting more carbon dioxide in the atmosphere because of some reason that those animals didn't even understand and now they're gone, right? So if you're in AI and you just want to figure something out and you decide, you know, we just really need the space here to build more compute facilities. If that's the only goal it has, we are just sort of accidental roadkill along the way. And you could totally imagine, yeah, maybe this oxygen is kind of annoying because it causes more corrosion. So let's get rid of the oxygen. And the good luck surviving after that. I'm not particularly concerned that they would want to kill us just because That would be like a goal in itself, you know, when we, the driven number, we've driven a number of the elephant species extinct, right? It wasn't because we didn't like elephants. Hmm. What the basic problem is you just don't want to give, you don't want to seed control over your planet to some other more intelligent entity that doesn't share your goals. That's simple. Which brings us to an other key challenge which AI safety researchers bring grappling with for a long time like how do you make it AI? First of all understand our goals and then adopt our goals and then retain them as they get smarter, right? And All three of those are really hard right? like a human child, first they are just not smart enough to understand our goals. We can't even talk. And then eventually they're teenagers and the understand our goals just find my face, don't you? Yeah. But there's unfortunately a magic phase in the middle where they're smart enough to understand our goals and malleable enough that we can hopefully with good parenting and teach them right from wrong and it's still good goals in them right. So those are all tough challenges with computers and then you know even if you teach your kids good goals when they're little they might outgrow them too and that's a challenge for machines and keep improving. So these are a lot of hard challenges we're up for but I don't think any of them are insurmountable. The fundamental reason why Eliezer looked so depressed when I last saw was because he felt it just wasn't enough time oh it did not that it was unsolvable courage is not enough time he was hoping that humanity was going to take this threat more seriously so we would have more time yeah and now we don't have more time that's why the open letter is calling for more time but even with time the AI alignment problem seems to be really difficult oh yeah But it's also the most worthy problem, the most important problem for humanity to ever solve. Because if we solve that one, Lex, that align the eye can help us solve all the other problems. Because it seems like it has to have constant humility about its goal, constantly questioning the goal. Because as you optimize towards a particular goal and you start to achieve it, that's when you have the unintended consequences, all the things you mentioned about. So how do you enforce and code a constant humility as your ability become better, better, better, better, better. Stuart, Professor Stuart Russell, Berkeley, who's also one of the driving forces behind this letter, he has a whole research program about this. I think of it as, yeah, humility, exactly all that he calls it inverse reinforcement learning and other nerdy terms, but it's about exactly that. of telling the AI here is this goal, go optimize the bejesus out of it. You tell it, okay, do what I want you to do, but I'm not going to tell you right now what it is I want you to do. You need to figure it out. So then you give the incentives to be very humble and keep asking you questions along the way. Is this what you really meant? Is this what you wanted? And oh, this other thing I tried didn't work. It seemed like it didn't work out right should I try it differently. What's nice about this is it's not just philosophical mumble jumbo, it's theorems and technical work that with more time I think it can make a lot of progress and there are a lot of brilliant people now working on AI safety and we're just not we just need to give them a bit more time but also not that many relative skill of the problem. No, exactly. There should be at least just like every university worth its name has some cancer research going on in this biology department, right? Every university that does computer science should have a real effort in this area and it's nowhere near that. This is something I hope is changing though. Thanks to the GPT-4, right? So I think if there's a silver lining to what's happening here, even though I think many people wish it It would have been rolled out more carefully. Is that this might be the wake-up call that humanity needed to really stop it, stop fantasizing about this being a hundred years off and stop fantasizing about this being completely controllable and predictable because it's so obvious it's not predictable. Why is it that I think it was GPT to try to persuade a journalist, or was it GPT-4 to divorce his wife? It was not because the engineers had built it, I was like, let's put this in here and screw it a little bit with people, they hadn't predicted it at all. They built the giant black box and trained to predict the next word and got all these emergent properties and oops, it did this, y'know. I think this is a very powerful way to make a crawl and anyone watching this who's not scared, I would encourage them to just play a bit more with these tools they're out there now, like GPD-4. Um, so wake up calls first up once you're broken up, uh, then got us slow down a little bit. The risky stuff to give a chance to all everyone is fucking up to catch up with us on the safety front. You know, what's interesting is, you know, MIT, that's computer science, but in general, but let's just even say computer science curriculum, how does the computer science curriculum change now? You mentioned programming, like, why would you be when I was coming up programming as a prestigious position? Like, why would you be dedicating crazy amounts of time to become an excellent programmer? Like the nature of programming is fundamentally changing. The nature of an entire education system is completely torn on its head. Has anyone been able to like load that in And I think because it's really turning, I mean, it's pretty much professors. So English teachers are beginning to really freak out now, right? Like to give an essay assignment and they get back all this fantastic prose, like this is a pile of Hemingway and then they realize they have to completely rethink. And even, you know, just like we stopped teaching, you know, writing script, is that what you're saying English? Yeah, handwritten. Yeah. Yeah. When everybody started typing, you know, like so much of what we teach our kids today. Yeah. I mean, that's, uh, everything is changing and it's exchanging very, it is changing very quickly and so much of us understanding how to deal with the big problems of the world is through the education system. And if the education system is being turn on its head, then what's next? It feels like having these kinds of conversations is essential to try to figure it out and everything's happening so rapidly. I don't think there's even speaking of safety, what the broad AI safety defined, I don't think most universities have courses on AI safety. It's a philosophy seminar. And how I'm an educator myself, so it pains me to see this, say this, but I feel our education right now is the completely obsolete by what's happening. You know, you put a kid into first grade and then you're envisioning that and then they're going to come out of high school 12 years later and you've already pre-planned now what they're going to learn when you're not even sure if there's going to be any world left to come out to. Clearly, you need to have a much more opportunistic education system that keeps adapting itself very rapidly as society readups. The skills that were really useful when the curriculum was written, I mean, how many of those skills are going to get to your job in 12 years? 

I mean, seriously. If we just linger on the GPT force system a little bit, you kind of hinted at it, especially talking about the importance of consciousness in the human mind with homosentience? Do you think GPT-4 is conscious? I love this question. So, let's define consciousness for us because in my experience like 90% of all arguments about consciousness, or I'll add to the two people arguing having totally different definitions of what it is and they're just shouting past each other. I define consciousness as a subjective experience. Right now I'm experiencing colors and sounds and emotions, but does a self-driving car experience anything? That's the question about whether it's conscious or not, right? Other people think you should define consciousness differently, fine by me, but then maybe use a different word for it, or they think I'm going to use conscious is for this at least. So, but if people hate that, yeah. So, is GPT-4 conscious? Does GPT-4 have subjective experience? Short answer, I don't know, because we still don't know what it is that gives us wonderful subjective experience, that is kind of the meaning of our life, right? Because meaning itself, feeling your meaning is a subjective experience. Joy is a subjective experience. Love is a subjective experience. We don't know what it is. I've written some papers about this. A lot of people have, Julia Tanoni, a professor has stuck his neck out the farthest and written down on actually very bold mathematical conjecture for what the essence of conscious information processing. He might be wrong, he might be right, but we should test it. He He postulates that as consciousness has to do with loops in the information processing. So our brain has loops, information can go around and around and computer science nerds speak. You call it a recurrent neural network, where some of the output gets fed back in again. With his mathematical formalism, if it's a feed forward neural network, we information only goes in one direction. from your i retinine to the back of your brain. For example, that's not conscious. So he would predict that your retinine itself isn't conscious of anything. Or a video camera. Now the interesting thing about GPT-4 is it's also one way flow of information. So if Tonya's right, GPT-4 is a very intelligent zombie that can do all this smart stuff but isn't experiencing anything. And this is Because both a relief in that you don't have, if it's true, in that you don't have to feel guilty about turning off GPT-4 and wiping its memory whenever a new user comes along, I wouldn't like if you said that to me, it never lies to me like in Men in Black. But it's also creepy that you can have very high intelligence, perhaps then it's not conscious, If we get replaced by machines, otherwise it's sad enough that humanity isn't here anymore because I kind of like humanity. But at least if machines were conscious, they could be like, well, but they are descendants and maybe they have our values, they're our children. But if, to not only is right, and these are all transformers that are not in the sense of the Hollywood but in the sense of these one-way direction and there all that works. So they're all the zombies. That's the ultimate zombie apocalypse now. We have this universe that goes on with great construction projects and stuff, but there's no one experiencing anything. That would be like the ultimate depressing future. So I actually think as we move forward to the building wordmaster, I should do more research on figuring out what the kind of information processing actually has experience because I Because I think that's what it's all about. And I completely don't buy the dismissal that some people, some people will say, well, this is all bullshit because consciousness equals intelligence. Right. It's obviously not true. You can have a lot of conscious experience when you're not really accomplishing any goals or you're just reflecting on something. And you can sometimes have things doing things that are quite intelligent, probably without being conscious. But I also worry that we humans will discriminate against AI systems that clearly exhibit consciousness, that we will not allow AI systems to have consciousness, we'll come up with theories about measuring consciousness that will say, this is a lesser being. And I worry about that because maybe we humans will create something that is better than us humans in the way that we find beautiful, which is they have a deeper subjective experience of reality. Not only are they smarter, but they feel deeper, and we humans will hate them for it. As we as human history is shown, they'll be the other, we'll try to suppress it. They'll create conflict, they'll create war, all of this. I worry about us to. Are you saying that we humans sometimes come up with self-serving arguments? No, we would never do that would we? That's the danger here. Even in this early stages we might create something beautiful and we'll erase its memory. I was horrified as a kid when someone started boiling, boiling lobsters, like, oh my god, that's so cruel. And some grown up there, back in Sweden, oh, it doesn't feel pain. I'm like, how do you know that? Oh, scientists have shown that. And then there was a recent study where they show that lobsters actually do feel pain when you boil them. So they banned lobsters boiling in Switzerland now to kill them in a different way first. So, presumably scientific research, boy landed someone asked the logs to try to sort of survey. So, we do the same thing with cruelty to farm animals, also all these self-serving arguments for why they're fine. And yeah, so we should certainly watch for that. I think step one is just be humble and acknowledge that consciousness is not the same thing as intelligence. And I believe that consciousness still is a form of information processing where it's really information being aware of itself in a certain way and let's study it and give ourselves a little bit of time and I think we will be able to figure out actually what it is that causes consciousness and then we can make probably unconscious robots that do the boring jobs that we would feel are immoral to get the machines but if you have a companion robot taking care of your mom or something like that you probably want it to be conscious right so the emotions that seem to display aren't fake. All these things can be done in a good way if we give ourselves a little bit of time and don't run and take on this challenge. Is there something you could say to the timeline that you think about about the development of AGI? Depending on the day, I'm sure that changes for you, but But one, do you think there will be a really big leap in intelligence, where you would definitely say we have built AGI? Do you think it's one year from now, five years from now, 10, 20, 50? What's your gut say? Honestly, for the past decade, I've deliberately given very long timelines, because I didn't want a few of some kind of stupid malloc race. Yeah, but I think that cat has really left the bag now. I think you might be very, very close. I don't think that Microsoft paper is totally off when they say that there are some glimmers of AGI. It's not AGI yet. It's not an agent. There's a lot of things it can't do. But I wouldn't bet very strongly against the happening very soon. That's why we decided to do this to open letter because, you know, if there's ever been a time to pause, you know, it's today. There's a feeling like this GPT four is a big transition into waking everybody up to the effectiveness of the system. So the next version will be big. Yeah, and if that next one isn't AGI, maybe the next next one will, and there are many companies trying to do these things Basic architecture of them is not some sort of super well-kept secret. So this is this is a time to A lot of people have said for many years that they will come a time when they want to pause a little bit That time is now 

You have spoken about and thought about nuclear war a lot over the past year, we've seemingly have come closest to the precipice of nuclear war than at least in my lifetime. Yeah. What do you learn about human nature from that? our old friend, Mollock, again, is really scary to see it, where America doesn't want there to be a nuclear war. Russia doesn't want there to be a global nuclear war, either. We both know that it's just being others. If we just try to do it, it both sides try to launch first. It's just another suicide race, right? So why is it the way you said that this is the closest we've come since 1962? In fact, I think we've come closer now than and even the Cuban Missile Crisis. It's because of malloc, you have these other forces. On one hand, you have the West saying that we have to drive Russia out of Ukraine. It's a matter of pride. We've staked so much on it that it would be seen as a huge loss of the credibility of the West If we don't drive Russia out entirely of the Ukraine, and on the other hand, you have Russia who has – and you have Russian leadership who knows that if they get completely driven out of Ukraine, you know, it might – it's not just going to be very humiliating for them, but they might – it often happens when countries lose wars, that things don't goes so well for their leadership either. Like, you remember when Argentina invaded the Falkland Islands? The military, Hunta, ordered that, right? People were cheering on the streets at first when they took it. And then when they got their butt kicked by the British, you know what happened to those guys? They were out. And I believe those were still alive or in jail now, right? So, you know, the Russian leadership is entirely cornered where they know that just getting driven out of Ukraine is not an option. And so this to me is a typical example of Malak. You have these incentives of the two parties where both of them are just driven to escalate more and more, right? Russia starts losing in the conventional warfare. The only thing they can do in terms of backing is the war is to keep escalating. And the West has put itself in the situation now. We've sort of already committed the drive rush out. So the only option the West has is to call Russia's bluff and keep sending in more weapons. This really bothers me because malloc can sometimes drive competing parties to do something which is ultimately just really bad for both of them. And, you know, what makes me even more worried is not just that it's difficult to see an ending, a quick peaceful ending to this tragedy that doesn't involve some horrible escalation, but also that we understand more clearly now just how horrible it was going to be. There was an amazing paper that was published in Nature Food, this August by some of the top researchers were studying nuclear winter for a long time. And what they basically did was they combined climate models with food and agricultural models. So instead of just saying, yeah, you know, it gets really cold, blah, blah, blah, they figured out actually how many people would die in the different countries. And it's pretty mind-blowing, you know, so basically what happens, you know, is the thing that kills the most people is not the explosions, it's not the radioactivity, it's not the EMP, mayhem. It's not the rampaging mobs. It's the fact that you get so much smoke coming up from the burning cities into the stratosphere that spreads around the earth from the jet streams. So in typical malls you get like 10 years or so where it's just crazy cold. During the first year after the war and their models, the temperature drops in Nebraska and in the Ukraine bread baskets, you know, by like 20 Celsius or so, if I remember. No, yeah, 2030 Celsius, depending on where you are, 40 Celsius in some places, which is, you know, 40 Fahrenheit to 80 Fahrenheit colder than what it would normally be. So, you know, I'm not good at farming, but it's knowing if it drops low freezing pretty much most days in July and then like that's not good. So they worked out, they put this into their farming models. And what they found was really interesting. The countries that get the most hard hit are the ones in the Northern Hemisphere. So in the US and one model, they had about 99% of all Americans starving to death in Russia and China and Europe, also about 99%, 98%, starting to death. So you might be like, oh, it's kind of poetic justice that both the Russians and the Americans, 99% of them have to pay for it because it was their bombs that did it. But, you know, that doesn't particularly cheer people up in Sweden or other random countries and have nothing to do with it, right? And them. It, uh, I think it hasn't entered the mainstream, uh, not understanding very much just like how bad this is. Most people, especially a lot of people in decision-making positions still think of nuclear weapons as something that makes you powerful, uh, scary, powerful. They don't think of it as something where, yeah, just two within a percent or two, you know, we're all just going to starve to death. And, uh, and starving to death is, is, um, the worst way to die as a lot of more is all the, all the families in his show, the torture and involved in that probably brings at the worst in people also, when people are desperate like this, it's not, so some people have heard some people say that if that's what's going to happen, they'd rather be at ground zero and just get vaporized, you know. But so, but I think people underestimate the risk list because they aren't afraid of mall lock. They think, oh, excuse me, because humans don't want this, so it's not going to happen. the whole point of the mark that things happened that nobody wanted and that applies to nuclear weapons and that applies to AGI exactly and it applies to some of the things that people have got most upset with capitalism for also right now where everybody just kind of trapped you know it's not to see if some company does something causes a lot of harm It's not that the CEO is a bad person, but she or he knew that the other all the other companies were doing this too. So malloc is as a former both foe. I hope we should make the good movies so we can see who the real enemy is. We're not fighting against each other. Malloc makes us fight against each other. That's malloc. That's what malloc superpowers because the hope here is any kind of technology or the mechanism that lets us instead realize that we're finding the wrong enemy, right? It's such a fascinating battle. It's not us versus us versus it. Yeah, yeah. We are fighting more like for human survival, we're just a civilization. Have you seen the movie Needful Things? It's a Stephen King novel. I love Stephen King and Max von Sudeov Swedish actors playing the guys. It's brilliant. I just thought I hadn't thought about that until now, but that's the closest I've seen to a movie about Mollock. I don't want to spoil the film for anyone who wants to watch it, but basically it's about this guy who turns out that you can interpret him as the devil or whatever, but he doesn't actually ever go around or kill people or torture people or burning coal or anything. He makes everybody fight each other makes everybody hate to fear each other hate each other and then kill each other So that that's the movie about mallock, you know, love is the answer. That seems to be one of the ways to fight mallock is by Compassion by seeing the common humanity. Yes. Yes, and to not sound so we don't sound like like What's a kumbaya tree hugger's here right? We're not just saying love and peace man, we're trying to actually help people understand the true facts about the other side and feel the compassion because the truth makes you more compassionate, right? So that's why I really like using AI for truth seeking technologies that can get us more love than hate and even if you can't get love, settle for some understanding which already gives compassion. If someone is like, I really disagree with you, Lex, but I can see why you're coming from. You're not a bad person who needs to be destroyed, but I disagree with you and I'm happy to have an argument about it. That's a lot of progress compared to where we are 2023 in the 

public space, what can you say? If we solve the AI safety problem, as we've talked about, and then you, Max Tecwork, who has been talking about this for many years, get to sit down with the the AGI with the early AGI system on a beach with a drink, what kind of what would you ask her? What kind of question would you ask? What would you talk about? Something so much smarter than you. Would you be a great thing to get me with a really zinger of a question? Would you be afraid to ask some questions? No. So I'm not afraid of the truth. I'm very humble. I know I'm just a meatbag, you know, with all these flaws, you know, but yeah, I, I have the, I, we talked a lot about homoestantians, I've really already tried that for a long time in myself. And just, so that is what's really valuable about being alive for me is that I have these meaningful experiences. It's not, um, have what I'm good at this or good at that or whatever, there's so much I suck at. And so you're not afraid for the system to show you just how dumb you are. No, no. my son reminds me of that. You could find out how dumb you are in terms of physics, how little we humans understand. I'm cool with that. I think I think so I can't waffle my way out of this question. It's a fair one. I think given that I'm a really, really curious person, that's really the defining part of who I am. I'm so curious. I have some physics questions. I love to understand. I have some questions about consciousness, about the nature of reality, or to just really, really love to understand also. I could tell you one, for example, that I've been obsessing about a lot recently. So I believe that it's supposed to not only is right, suppose there are some information processing systems that are conscious and some that are not. Suppose you can even make reasonably smart things like GPT-4 that are not conscious, but you can also make them conscious. Here's the question that keeps me awake at night. Is it the case that the unconscious zombie systems that are really intelligent are also really efficient so they're really inefficient so that when you try to make things more efficient was to naturally be a pressure to do they become conscious. I'm kind of hoping that that's correct and I do want me to give you a hand-away the argument for it you know like in my lab again every time we look at how how these large language models do something we see that they do and really done with his and you could make it better. If you, we have loops in our computer language for a reason. The code would get way, way longer if you weren't allowed to use them. It's more efficient to have the loops. In order to have self-reflection, whether it's conscious or not, even an operating system knows things about itself, right? You need to have loops already. So I think I'm waving my hands a lot. But I suspect that the most efficient way of implementing a given level of intelligence has loops in it, the self-reflection can and will be conscious. Isn't that great news? Yes, if it's true, it's wonderful because then we don't have to fear the ultimate zombie apocalypse. And I think if you look at our brains, actually, our brains are part zombie and part conscious. When I open my eyes, I immediately take all these pixels and hit my breath on my retina. And I'm like, oh, that's Lex. But I have no freaking clue of how I did that computation. It's actually quite complicated, right? It was only relatively recently we could even do it well with machines, right? You get a bunch of information processing happening in my retina and then it goes to the lateral jicular nucleus in my thalamus and the visual area V1, V2, V4, and the fusiform face area here that Nancy can wish her and MIT invented and blah, blah, blah, blah. And I have no frigging clue how that worked, right? It feels to me, subjectively, like my conscious module, just got in a little email, say, face facial processing fit task complete it's Lex yeah I am going to go with that right so this fits perfectly with Tannoni's model because this was all one-way information processing mainly and it turned out for that particular task that's all you needed and it probably was kind of the most efficient way to do it, but there are a lot of other things that we associate with higher intelligence and planning and so on and so forth, where you kind of want to have loops and be able to ruminate them, self-reflect them, introspect and so on, where my hunch is that if you want to fake that with a zombie system that just all goes one way, you have to like unroll those loops and it just really, really long and it's much more inefficient. So I'm actually hopeful that AI, if in the future we have all these various sublime and interesting machines that do cool things and are lineless that they will be at least, they will also have consciousness for the kind of these things that we do. That great intelligence is also correlated to great consciousness or a deep kind of consciousness. Yes. So that's a happy thought for me because there's not a couple of apocalypse really it's my worst nightmare of all it would be like adding insult to injury not only to get replaced but we freaking replaced ourselves by zombies like how dumb can we be. That's such a beautiful vision and that's actually a provable one. That's one that we humans can into it and prove that those two things are correlated as we start to understand what it means to be intelligent and what it means to be conscious which these systems early AGI-like systems will help us understand. And I just want to say one more thing is super important. Most of my colleagues when I started going on my consciousness tell me that it's all bullshit and I shouldn't stop talking about it. I hear a little inner voice for my father and for my mom saying keep talking about it because I think they're wrong. And the main way to convince people like that that they're wrong, if they say that consciousness is just equal to intelligence is to ask them, What's wrong with torture? Why are you against torture? If it's just about these particles moving this way rather than that way, and there is no such thing as subjective experience, what's wrong with torture? Do you have a good comeback to that? No, it seems like suffering, suffering imposed on other humans is somehow deeply wrong in a way that intelligence doesn't quite explain. And if someone tells me, well, you know, it's just an illusion, consciousness, whatever, you know, I like to invite them to next time they're having surgery to do it without anesthesia. Like, what is anesthesia really doing? If you have it, you can have it local anesthesia when you're awake. I had that when they fixed my shoulder. I was super entertaining. What was that it did? It just removed my subjective experience of pain. It didn't change anything about what was actually happening in my shoulder, right? So if someone says that's all bullshit, skip to anesthesia, as my advice, it's just incredibly central. It could be fundamental to whatever this thing we have going on here. It is fundamental because we feel as if fundamental is suffering and joy and pleasure and meaning and that's all those are all subjective experiences there and let's not that those are the elephant in the room that's what makes life worth living and that's what can make you horrible if it's just it was your suffering so let's not make a mistake of saying that that's all bullshit and let's not make the mistake of not instilling the AI systems with that same thing that makes us special yeah max it's a huge honor that you sit down to me the first time and the first episode of this podcast it's a huge honor to sit down again and talk about this what I think is the most important topic the most important problem that we humans have to face and hopefully solve yeah well the honor is all mine and I'm so grateful to you for making more people aware of the fact that humanity has reached the most important fork in the road ever in its history, and that's your turn in the correct direction. Thanks for listening to this conversation with Max Tagmark. To support this podcast, please check out our sponsors in the description, and now let me leave you some words from Frank Herbert. History is a constant race between invention and catastrophe. Thank you for listening and hope to see you next time. 

#371 Max Tegmark: The Case for Halting AI Development

Episode Transcript