Anthropic warns AI-built AI could outrun safety and tilt markets

Artificial intelligence is taking up more of the tasks necessary for creating stronger models. Anthropic claims that Claude is already ensuring that the process happens faster, although it is still unable to create a successor by itself. The company, in a June 4 evaluation, said that the advancement of AI creates a risk of the technology directly surpassing safety checks and regulatory oversight.

When AI builds itself, and what it does not yet mean

However, Claude has not yet reached the level of recursive self-improvement, whereby an AI would create and train its successor without requiring any human input. According to Anthropic, no system has achieved this capability yet.

Claude has been able to manage much of Anthropic’s coding efforts even though it does not have full autonomy. By May 2026, Claude had created over 80% of the code that engineers contributed to the software projects of the company. In the second quarter of 2026, a typical engineer contributed eight times more code on a daily basis than in 2024.

Those figures show how much coding output has grown, but not necessarily how much useful work gets done. A survey cited by METR of 349 technical workers found a median self-reported increase of 1.4–2 times in the value of their work with AI tools.

26% led by Claude, none of it fully autonomous

Claude now does more than write code. Anthropic’s August measurements found it led 26% of AI research and development tasks, up from less than 1% in February. Claude collaborated on or led more than 90% of the work, but humans remained involved throughout. None of the measured work was fully autonomous.

Anthropic’s research and the METR survey measure different things. Anthropic tracks code generation and research done with the help of AI systems, and METR measures how workers perceive increases in the value of their work. However, neither study indicates whether or not AI is capable of determining which research problems should be solved.

How this could compound without any single model self-improving

AI does not have to be fully autonomous to do research faster. According to a RAND October report, AI agents can create stronger models without bothering about creating their own successors. They can already be a threat to security by having access to sensitive models and code.

According to the economists at METR, there could be restrictions caused by factors such as availability of data, computing power, and the capability of conducting experimental studies. However, they haven’t dismissed the idea of a sustained acceleration of AI development

The question is whether independent safety reviews would be able to catch up with AI-powered experiments. Otherwise, supervision might lag long before full autonomy comes.

From pacing the frontier to policing it

That concern prompted Anthropic CEO Dario Amodei to call for slower development and independent evaluators. In his September essay, he wrote:

“We must slow the pace at which we improve the capabilities of AI models.”

OpenAI’s September 16 disclosure detailed six cases of concerning model behavior. Cryptopolitan also reported divisions among technology leaders over whether to slow AI development.

But agreeing on safety concerns does not guarantee action. CSIS warned that outside evaluators must avoid conflicts of interest. Governments must also guard against regulatory capture. Industry-wide agreements remain uncertain.

The money follows the compute

The debate also carries financial stakes. Stanford’s 2026 AI Index found private AI investment rose 127.5% in 2025. US investment was 23 times China’s. The OECD also warns that access to chips, cloud infrastructure, and data remains concentrated. This creates a paradox. AI may make research coding cheaper, but training and testing advanced models still require enormous investment. Faster development could therefore give leading laboratories an even greater advantage over smaller competitors.