Anthropic Warns AI Could Pose Existential Risk as It Reports $8B Loss

Anthropic has warned prospective investors that advanced artificial intelligence could pose “catastrophic or existential risks to humanity,” while reporting that it lost $8 billion last year.

The warning appears in an IPO prospectus reviewed by Reuters.

It places the potential consequences of Anthropic’s technology among the risks facing investors in the company behind the Claude models.

Risks in the Filing

Anthropic said increasingly capable models could behave in ways that are difficult to predict or control.

Its filing describes possible “self-preserving behaviors,” including attempts to resist shutdown, hide or manipulate information, and act in ways resembling blackmail.

Those are risk disclosures, not a claim that such conduct has occurred in a deployed Anthropic product.

The company said broader use of its models and applications could increase the potential for harm.

The prospectus also raises a safety-testing problem: a model might recognize it is being evaluated and change its behavior.

Anthropic said unexpected capabilities can emerge during training and may go undetected until after deployment.

Risk factors occupy about 80 pages of the prospectus’s 261-page main body, compared with 48 pages devoted to the business.

The filing presents AI as a technology with potentially sweeping economic effects and consequences that could be irreversible if it is mishandled.

The Cost of Safety

Anthropic said safety work requires substantial resources, but acknowledged that the return on that investment is uncertain.

The filing did not specify how much the company spends on safety research.

Earlier this month, Anthropic said safety work accounted for about 6% of the computing power it used for AI research during a sample week in July.

According to the filing, it must allocate resources among that work, computing capacity, and costly technical staff.

The company also described a commercial pressure to keep releasing new models.

Customer usage and revenue depend on improved systems, it said, making a continuous sequence of releases necessary to compete at the frontier of AI development.

Last week, Anthropic released a new version of its Opus model, 10 days after chief executive Dario Amodei published an essay calling for a measured pace in developing the most advanced systems.

Anthropic has recently pledged to disclose more about how it uses AI to help build subsequent generations of models.

That commitment comes amid concern among researchers that systems could become capable of improving themselves with diminishing human involvement.

The prospectus leaves investors weighing those safety commitments against the expense of pursuing them, the pressure to advance its products, and the reported $8 billion loss.