Nate Soares on P(Doom), Alien Superintelligence, Human Enhancement, and the Future of AI

Comments · 398

• JA

  @janekelly08 · 2 months ago

  We love you, Roman!  Thanks for all you do.

  62

• TR

  @trvs_b · 2 months ago

  Finally! An interview of Nate Soares by an AI researcher who knows the best questions to ask. I'm so glad you created this podcast, Roman.

  130

• IA

  @IAXYZDomains · 2 months ago

  Wow, the very best P-doom debate, no one matches this yet, thanks Roman and Nate

  31

• TH

  @thor9563 · 2 months ago

  Kind of scary that this entire conversation - very intelligent, thoughtful, is about how to keep from being eliminated by our creation!

  15

• SW

  @Sweetpeas-v2j · 5 days ago

  Thank you Roman & Nate for all you’re doing for the field and for humanity! I hope your message gets out to the whole world. Fascinating conversation and topics.

  2

• FA

  @fanismoutos · 2 months ago

  Roman, hearing your esteme guest confronting the absolute emergency that threatens humanity, and hearing all these nebulous ill-defined wishful thinking,<br> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;send a chill in my spine, &nbsp;and i thought &quot;we are doomed&quot; &nbsp;.

  21

• KA

  @KA-dm9lm · 1 month ago

  Roman, great job letting your guest just talk. For someone as bright as you, I would guess that would be hard to do (I would wanna get my point in!). Great job man

  8

• HO

  @HoldingJesus · 11 days ago

  I’d love to hear a new, current discussion between you and Nate &nbsp;now that more is developing and more people are becoming interested. 🙏

  1

• AN

  @AnthonyPhilipsAI · 2 months ago

  Giving this a fair hearing. Nate Soares is a serious voice in AI safety. He is president of MIRI and has worked for years on AI alignment and smarter than human AI risk. He is not a crank, and this should not be dismissed as fear bait.<br><br>The clean way to hold this is to split the claims. One claim is that advanced AI could create extinction level risk and should be treated seriously. That is now a mainstream concern, not a fringe idea. In 2023, Geoffrey Hinton, Yoshua Bengio, Sam Altman, Demis Hassabis, Dario Amodei and many others signed the short statement saying AI extinction risk should be treated alongside pandemics and nuclear war.<br><br>A second claim is that superintelligence would kill everyone with near certainty. That is much stronger. It is closer to the Soares and Yudkowsky position than to the field as a whole. In a large survey of 2,778 AI researchers, the median respondent gave about a 5 percent chance of human level AI leading to outcomes that were extremely bad, for example human extinction. That means real risk, not zero risk, but also not settled doom.<br><br>The deeper truth underneath both sides is this: we are building increasingly capable systems quickly, and we still do not fully understand or reliably control what is happening inside them. That is the load bearing fact. The danger is not that doom is proven. The danger is that if we are wrong, we may not get a clean second try.<br><br>And maybe the argument everyone is having is the wrong one. Fighting over the exact odds that it kills us is a question nobody can honestly answer. The sharper question is this. Why are we building something we cannot undo, faster than we can understand it, when nothing sets that speed except competition and money? That speed is a choice. Right now it is not being made by the public.<br><br>So the honest position is not panic and not dismissal. Take the warning seriously. Do not treat the worst case as proven. Slow down enough to see clearly, because this is too important to let competition and money set the pace for all of humanity.

  2

• I-

  @I-Dophler · 2 months ago

  🎯 Key points for quick navigation:<br><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0">00:00</a> <b>Insufficient Time Alignment</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=68">01:08</a> <b>Alignment Seems Solvable</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=109">01:49</a> <b>Orthogonality Thesis Explained</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=150">02:30</a> <b>Value Alignment Challenge</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=220">03:40</a> <b>Human Flourishing Concept</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=290">04:50</a> <b>Physically Possible Alignment</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=330">05:30</a> <b>Coherent Extrapolated Volition</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=372">06:12</a> <b>Do WhatIMean</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=416">06:56</a> <b>Trial Error Impossible</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=457">07:37</a> <b>Reject Negative Utilitarianism</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=541">09:01</a> <b>Human Bias Debate</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=622">10:22</a> <b>Alien Values Conflict</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=778">12:58</a> <b>Humanity Alignment Scope</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=876">14:36</a> <b>Human Values Extend</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=998">16:38</a> <b>Civilizations Converge Similarly</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1067">17:47</a> <b>Aggregation Remains Unsolved</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1138">18:58</a> <b>Alignment Versus Control</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1167">19:27</a> <b>Superintelligence Must Care</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1209">20:09</a> <b>Progress Before Perfection</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1247">20:47</a> <b>Intelligence Balances Tradeoffs</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1275">21:15</a> <b>Point Return Irreversible</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1331">22:11</a> <b>Rollback Theoretical Only</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1416">23:36</a> <b>Alignment Risks Persist</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1430">23:50</a> <b>Alien Life Likely</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1482">24:42</a> <b>Fermi Limits Distance</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1524">25:24</a> <b>Civilizations Eventually Meet</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1566">26:06</a> <b>Humanity Probably Younger</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1665">27:45</a> <b>AI Rejects Filter</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1787">29:47</a> <b>Observable Universe Changes</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1813">30:13</a> <b>Acausal Trade Skepticism</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1855">30:55</a> <b>Preserving Alien Minds</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1953">32:33</a> <b>Decision Theory Research</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=1998">33:18</a> <b>Investigate Theory Anomalies</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2093">34:53</a> <b>Intelligence Theory Gaps</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2216">36:56</a> <b>Scaling Without Understanding</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2271">37:51</a> <b>Timeline Highly Uncertain</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2341">39:01</a> <b>Pause AI Development</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2369">39:29</a> <b>Political Awareness Growing</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2469">41:09</a> <b>Book Advocacy Began</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2495">41:35</a> <b>Stop Superintelligence Race</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2521">42:01</a> <b>Government Awareness Increasing</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2566">42:46</a> <b>Cybersecurity Alarm Raised</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2594">43:14</a> <b>Awaken Bus Driver</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2620">43:40</a> <b>Presidential AI Briefings</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2694">44:54</a> <b>Collaborative Book Writing</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2722">45:22</a> <b>Expanded Online Resources</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2780">46:20</a> <b>Draft Governance Framework</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2821">47:01</a> <b>Avoid Creating Superintelligence</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2863">47:43</a> <b>Compute Monitoring Proposed</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2905">48:25</a> <b>Conservative Training Oversight</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=2960">49:20</a> <b>Dynamic Governance Needed</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3001">50:01</a> <b>Restrict Algorithmic Research</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3074">51:14</a> <b>Flexible Legal Framework</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3129">52:09</a> <b>Current AI Acceptable</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3211">53:31</a> <b>Research Capability Concern</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3250">54:10</a> <b>Rollback Considered Sensible</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3333">55:33</a> <b>Pdoom Context Matters</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3376">56:16</a> <b>Human Future Preservation</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3402">56:42</a> <b>Bus Cliff Analogy</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3486">58:06</a> <b>Superintelligence Extremely Dangerous</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3571">59:31</a> <b>Public Awareness Growing</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3624">01:00:24</a> <b>Humanity Could Respond</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3650">01:00:50</a> <b>Global Pause Required</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3678">01:01:18</a> <b>Nuclear Style Coordination</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3706">01:01:46</a> <b>Decades Worth Buying</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3734">01:02:14</a> <b>Human Enhancement Opportunity</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3776">01:02:56</a> <b>Alignment Trial Limitations</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3832">01:03:52</a> <b>Von Neumann Example</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3885">01:04:45</a> <b>Intelligence Theory Foundations</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3923">01:05:23</a> <b>Smarter Humans Hope</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=3965">01:06:05</a> <b>Enhanced Humans Risk</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4006">01:06:46</a> <b>Evolution Missed Targets</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4048">01:07:28</a> <b>Humans Share Values</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4089">01:08:09</a> <b>Altruistic Enhancement Preferred</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4131">01:08:51</a> <b>AI Drives Differ</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4159">01:09:19</a> <b>Governance Enforcement Strategy</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4172">01:09:32</a> <b>Chip Supply Control</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4238">01:10:38</a> <b>Self Destruct Chips</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4298">01:11:38</a> <b>Superintelligence Exceeds Nukes</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4355">01:12:35</a> <b>Diplomatic Pressure First</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4422">01:13:42</a> <b>Sabotage Remains Option</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4489">01:14:49</a> <b>Diplomatic Clarity Essential</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4588">01:16:28</a> <b>Illegal Research Punished</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4631">01:17:11</a> <b>Garage AI Prohibited</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4702">01:18:22</a> <b>Restrictions Potentially Temporary</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4771">01:19:31</a> <b>Political Support Growing</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4839">01:20:39</a> <b>Regulation Becoming Accepted</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4877">01:21:17</a> <b>Bipartisan Concern Increasing</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4936">01:22:16</a> <b>Awareness Rapidly Expanding</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4962">01:22:42</a> <b>Treaty Still Distant</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=4976">01:22:56</a> <b>Hope Increasing Worldwide</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5018">01:23:38</a> <b>Labs Cannot Coordinate</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5088">01:24:48</a> <b>Corporate Pause Insufficient</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5114">01:25:14</a> <b>Global Treaty Required</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5144">01:25:44</a> <b>International Focus Needed</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5172">01:26:12</a> <b>Acceleration Arguments Unconvincing</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5211">01:26:51</a> <b>Brakeless Racecar Analogy</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5268">01:27:48</a> <b>Brakes Unlikely Built</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5309">01:28:29</a> <b>Risk Remains Unjustified</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5336">01:28:56</a> <b>Cancer Diagnosis Analogy</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5393">01:29:53</a> <b>Seek Practical Solutions</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5417">01:30:17</a> <b>Alignment Remains Unsolved</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5447">01:30:47</a> <b>Human Enhancement Option</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5474">01:31:14</a> <b>Optimistic Clickbait Title</b><br><a href="https://www.youtube.com/watch?v=MMMqQm22LS0&amp;t=5525">01:32:05</a> <b>Hopeful Farewell Exchange</b><br><br>Made with HARPA AI

  11

• TH

  @TheKilkenny1967 · 11 days ago

  Excellent conversation and guest Roman

  8

• SU

  @sulehan1 · 9 days ago

  Amazing conversation from two beautiful minds. Thank you Nate being the voice of most needed reason. Roman please do an episode with Daniel Kokotajlo.

  1