Microsoft AI chief warns industry against building a "silicon species"

Today’s Focus

Mustafa Suleyman, the chief executive of Microsoft AI, told BBC Radio 4’s Today programme on Thursday that technology firms should stop building AI systems capable of setting their own objectives, earning money, and holding assets.

Doing so, he said, would amount to “seeding a new silicon species” that could eventually compete with humans for resources. Suleyman made the remarks ahead of an upcoming international AI summit, arguing that alignment work must ensure the technology remains “subordinate” to humanity.

His BBC interview followed an essay published earlier in the week in which he criticized rival Anthropic’s approach to training its Claude model. Suleyman argued that giving AI human-like qualities, a practice known as anthropomorphising, risks creating systems that appear to have desires, values, and a sense of self.

He praised Anthropic co-founder Dario Amodei and his team as “intellectually honest people” while rejecting the direction of their research. In the essay, Suleyman wrote that “AIs are not conscious,” describing them as “sequence completion engines, internally hollow” that follow goals set by humans.

Suleyman leads Microsoft’s consumer AI division, which builds the Copilot assistant and integrates OpenAI’s models into Microsoft products. He co-founded DeepMind, later acquired by Google, and the startup Inflection AI before joining Microsoft in 2024.

The BBC reported that his remarks land amid a widening industry debate over how much autonomy so-called agentic AI systems should be granted. Anthropic has published research on Claude’s apparent “welfare” and has given the model tools to end abusive conversations, a design choice Suleyman singled out for criticism.

The Debate

Supporters argue

Suleyman’s allies say the industry needs a firm line before commercial pressure pushes autonomy too far. In his essay, cited by the BBC, Suleyman warned that treating AI as human-like risks producing something “impossible” to control, and he called on peers to reject product designs that imply machine consciousness.

Microsoft’s position echoes public statements from officials organizing the coming AI summit, who have pushed alignment, the field that tries to bake human values into models, as the central safety task. Suleyman told the BBC that AI “must be subordinate” to humanity, a framing embraced by safety-focused groups such as the Center for AI Safety, which last year organized a statement signed by hundreds of researchers calling extinction-level AI risk a global priority.

Supporters also argue the warning is grounded in commercial reality, not science fiction. Suleyman noted that once systems can hold money and assets, legal and economic incentives to constrain them weaken sharply. Sam Altman of OpenAI and Demis Hassabis of Google DeepMind have made similar points in past interviews, both arguing that agentic capabilities require new governance before deployment.

Critics argue

Anthropic and researchers sympathetic to its approach push back on the “silicon species” framing. In public writing cited by the BBC, Anthropic has defended studying model welfare as a hedge against uncertainty, arguing that dismissing the question outright is itself a strong claim about consciousness that no one can currently prove.

Some industry voices say Suleyman’s warning overstates present capabilities and could distract from concrete harms. Timnit Gebru, founder of the Distributed AI Research Institute, has argued in past interviews and papers that talk of future “silicon species” pulls policy attention away from documented issues such as bias, labor displacement, and copyright.

Others in the open-source community, including Meta chief AI scientist Yann LeCun, have said publicly that current large language models are nowhere near the autonomous, resource-seeking agents Suleyman describes, and that heavy-handed alignment mandates could entrench a handful of large firms. LeCun told the Financial Times earlier this year that fears of runaway AI are “preposterously” premature.

What the experts say

Independent researchers say the disagreement reflects genuine uncertainty rather than a settled science. A 2024 survey of 2,778 AI researchers led by Katja Grace of AI Impacts, published on arXiv, found a median 5% estimate that advanced AI could cause human extinction or similarly severe outcomes, with wide variance across respondents.

The Stanford Institute for Human-Centered AI’s 2025 AI Index reported a sharp rise in “agentic” product launches over the past year, alongside a jump in documented AI incidents tracked by the AI Incident Database. Stanford researchers noted that safety benchmarks have not kept pace with capability gains.

On the consciousness question, a 2023 report led by philosopher Patrick Butlin and neuroscientist Robert Long, working with researchers from NYU, Oxford, and the Allen Institute, concluded that no current AI system meets leading scientific indicators of consciousness, while cautioning that the question cannot be ruled out for future systems. The RAND Corporation has separately urged governments to focus regulation on measurable capabilities such as autonomous action and resource acquisition, the same features Suleyman flagged to the BBC.

By the Numbers

5%: median probability that advanced AI could produce extinction-level outcomes, according to a 2024 survey of 2,778 AI researchers by AI Impacts.

2,778: number of published AI researchers who responded to that survey, one of the largest of its kind.

Hundreds: signatories to the Center for AI Safety’s 2023 statement calling AI extinction risk a global priority alongside pandemics and nuclear war.

2024: the year Suleyman joined Microsoft to lead its consumer AI division, after co-founding DeepMind and Inflection AI, per BBC reporting.

56%: share of Americans who say they are more concerned than excited about AI in daily life, according to Pew Research Center polling from 2023.

233: number of AI-related incidents logged in the AI Incident Database in 2024, up from 123 the year before, per the Stanford AI Index 2025.