OpenAI’s Superalignment team, responsible for ensuring the safe development and control of “superintelligent” AI systems, has faced significant internal challenges leading to several high-profile resignations. Despite being promised substantial resources by CEO Sam Altman, the team struggled to secure the necessary compute power to carry out their mission. This has raised concerns about the company’s commitment to AI safety and alignment, with key figures like co-leads Jan Leike and Ilya Sutskever stepping down.
One former employee, Daniel Kokotajlo, who joined OpenAI in 2022 with hopes of steering it toward the safe deployment of AI, refused to sign the offboarding agreement so he could freely criticize the company. He worked on the governance team until he quit last month. This article delves into the reasons behind these resignations, the internal conflicts within OpenAI, and the implications for the future of AI safety at the company.
OpenAI’s Superalignment team, formed in July, aimed to solve technical challenges in controlling superintelligent AI within four years. Led by Jan Leike and OpenAI co-founder Ilya Sutskever, the team was promised 20% of the company’s compute resources. However, according to a team member, requests for even a fraction of that compute were often denied, hindering their work. This lack of resources led to significant frustration among the team members and was a primary reason for several resignations, including Leike’s.
Jan Leike, a former DeepMind researcher involved with the development of ChatGPT and GPT-4, publicly shared his reasons for resigning. He stated that he had been in disagreement with OpenAI leadership over the company’s core priorities for some time. Leike emphasized the importance of preparing for future AI models, focusing on security, monitoring, safety, alignment, and societal impact. He expressed concern that OpenAI was not on the right path to address these issues adequately.
The internal discord at OpenAI escalated with the attempted firing of CEO Sam Altman last year by Ilya Sutskever and the board. Although Altman was reinstated, the incident strained relationships within the company. Altman’s efforts to raise funds for a new AI chip-making company and partnerships with regimes like Saudi Arabia further raised concerns among safety-minded employees.
Daniel Kokotajlo, who joined OpenAI in 2022 with hopes of steering it toward the safe deployment of AI, worked on the governance team until he quit last month. Kokotajlo refused to sign the offboarding agreement so he could freely criticize the company. He cited a loss of trust in OpenAI’s leadership and their ability to handle AGI responsibly as his reasons for leaving. Kokotajlo expressed disappointment in OpenAI’s shift in priorities and the diminishing focus on safety.
Following the departures of Leike and Sutskever, OpenAI co-founder John Schulman will head the work previously done by the Superalignment team. However, the team has been disbanded and is now a loosely associated group of researchers embedded in various divisions. An OpenAI spokesperson described this change as integrating the team more deeply.
Concerns have arisen about the future focus on AI safety at OpenAI. While current products like ChatGPT may not pose an immediate threat, the potential risks of future AI models remain a critical concern. Leike stressed the need for significant investment in preparing for these challenges, expressing doubts about OpenAI’s trajectory in addressing them effectively.
OpenAI did not immeditaly respond to a request for comment about the resource allocation for the Superalignment team or the resignations of key team members. However, CEO Sam Altman did respond to the resignation of Jan Leike in a post on social media
i'm super appreciative of @janleike's contributions to openai's alignment research and safety culture, and very sad to see him leave. he's right we have a lot more to do; we are committed to doing it. i'll have a longer post in the next couple of days.
— Sam Altman (@sama) May 17, 2024