If powerful AI is aimed at solving humanity’s most difficult problems and the risks are managed responsibly, the potential is extraordinary. Possibilities include compressing decades of scientific progress into years; accelerate drug discovery, disease detection and delivery of personalized therapies; provide personalized education and expert financial and legal advice; advance the design of electrical networks and batteries to achieve greater efficiency and reliability; and extend to the developing world capabilities that took rich nations generations to develop.
How modest! Of course, the key words are in the caveat: namely, that these lofty goals will only be achieved “if… risks are managed responsibly,” which Anthropic has long touted as its goal. reason for being.
AI systems could “post catastrophic or existential risks to humanity” (page 46)
And here’s Anthropic’s full statement on the flip side of its utopian AI vision:
Both within our organization and by other researchers, regulators, policymakers, and industry participants, concerns have been raised that sufficiently advanced AI systems could, if not developed, deployed, and operated with appropriate safeguards, oversight, and security measures, pose catastrophic or existential risks to humanity, particularly if such systems are misused, replicated, weaponized, or allowed to operate with a high degree of autonomy.
And if you were waiting for Anthropic’s IPO filing to argue that those catastrophic risks are overblown, well, no:
We have observed that our models are contributing to the pace of our own research and these risks will increase as AI systems substantially automate AI research and development if, for example, they accelerate the progress of AI beyond our ability to evaluate and safeguard new models, which could adversely affect our business, financial condition and results of operations.
In other words, Anthropic formally admits that yes, its AI models could advance beyond the level of our ability to control them, which could be bad for humanity and even worse for Anthropic’s finances.
