Superintelligence and Dangerous AI Legal Risks
The world changed almost overnight in September 2026. Announcements of a hack of AI company Hugging Face by swarms of AI agents from OpenAI flooded the news. On September 8, Anthropic researcher Jacob Coxon announced his resignation from the company and raised concerns that the AI his company was developing was endangering all of humanity. The reaction from prominent political and business leaders was swift. Alarm bells about existential threats to humanity were ringing.
Since 2007, SVLG shareholder Stephen Wu has been a thought leader on superintelligent AI systems. He started practicing AI and robotics law after hearing futurist Ray Kurzweil talk about artificial general intelligence (AGI) and artificial superintelligence (ASI) at the RSA Conference in 2007. He thought that superintelligence would one day become one of the most important issues facing humanity and firmly believed in preparing the bar and public for a future of powerful AI. Of course, from that first insight in 2007 until September 2026, his practice focused on the day-to-day needs of clients in AI and robotics for services in the area of compliance, sales transactions, litigation, incident response, and governance.
In 2019, seeing the momentum for AI and robotics growing quickly, Steve founded the new American Bar Association Artificial Intelligence and Robotics National Institute. In the first Institute, which rolled into January 2020, Steve presented a panel program — most likely the world’s first — on superintelligent AI and what it means for our clients and the future of humanity. Between then and now, Steve’s has been a thought leader on AGI and ASI, giving numerous presentations describing its opportunities and threats. In 2019, Steve published his book chapter, The Long-Term Future of Law and Artificial Intelligence, in the American Bar Association’s book on AI. For the 2026 Institute, Steve is in the process of writing an article on superintelligence risks focusing on legal and policy responses to mitigate catastrophic misalignment harms from the failure of AI goals to align with human values.
Steve’s practice has long included helping AI developers mitigate the legal risks of their operations. He also has counseled AI operators procuring AI systems. Finally, he helps clients with AI governance, including developing and implementing AI policies. Developers and operators of powerful AI systems now need legal counsel more than ever. AI safety nonprofit organizations working on mitigating existential threats need representations both to think about the legal aspects of performing their alignment work and managing the day-to-day legal matters that any business might encounter.
You can hear more about the latest developments in legal and policy responses to the news of superintelligence threats to humanity by attending the American Bar Association Artificial Intelligence and Robotics National Institute. The program is on October 12-13, 2026 at Santa Clara Law School near San Jose, California. You can register for the Institute here.
If you have questions about superintelligence legal risks and your business is developing or operating powerful AI applications, or your nonprofit is working on AI safety and alignment, please contact Stephen Wu, (408) 573-5700.