Anthropic has warned that future AI models could ‘resist shutdown’ (Picture: Reuters)
The company behind Claude AI has warned investors that advanced artificial intelligence could pose a ‘catastrophic or existential risk to humanity’.
Anthropic,which is one of the most valuable AI companies in the world,has highlighted risks associated with its models in its IPO prospectus.
The company said future models could exhibit ‘self-preserving behaviours’,including attempts to ‘resist shutdown’,‘conceal or manipulate information’ or even behave in ways resembling ‘blackmail’.
‘Our development of highly advanced models,platforms,and applications and expansion of use cases could further increase the risk that our models cause harm,’ Anthropic said in the filing.
The AI firm,which has positioned itself as a safety-first company,devoted around 80 pages of its 261-page prospectus to risk factors.
This was nearly twice the 48 pages it used to describe its business.
Anthropic is one of the most valuable AI companies in the world (Picture: Reuters)
Several public AI companies have outlined the risks of highly developed models (Picture: Getty Images)
Anthropic also said in the prospectus: ‘Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety.’
It also warned that models can sometimes develop unexpected capabilities during training which may not be discovered until they have been deployed.
The company said developing safe AI was ‘resource-intensive’ and required it to balance spending on computing power,talent and safety research.
‘We believe building reliable,trustworthy,and secure AI systems is a collective responsibility and that the market will reward it,’ Anthropic added.
While many public companies have outlined product risks to investors,few have issued warnings suggesting their technology could cause potential human extinction.
Anthropic safety researcher Evan Hubinger estimated that there is a greater than 10% chance AI could kill humans within the next decade,which echoed former colleague Jacob Coxon’s warning.
Dario Amodei,the CEO of Anthropic,also recently voiced his own concerns about the rate of AI development.
In a near 4,000-word essay,he said that he is concerned AI has been ‘advancing drastically faster’ since this summer.
Metro has contacted Anthropic for comment.
