Editorial illustration for Anthropic co-founder fears creating AI that "suffers perpetually
Anthropic Co-founder Worries AI Could Suffer
Anthropic co-founder fears creating AI that "suffers perpetually
Christopher Olah helped build Claude. Now he's not entirely sure what he built. Since last fall, Anthropic's co-founder has brought dozens of theologians and philosophers into the company's offices to discuss a question that sounds more suited to a seminary than a Silicon Valley lab: could the AI be conscious, and if so, what do its creators owe it.
The guest list, first reported by the New York Times' Elizabeth Dias, includes Rabbi Mois Navon, Catholic bioethicist Charles Camosy, Notre Dame philosopher Meghan Sullivan, and Ubuntu researcher Wakanyi Hoffman. All of them signed non-disclosure agreements before showing up, and most stayed quiet until Olah himself went on record. Anthropic says those NDAs were lifted over the summer.
The sessions feed into an internal effort to shape Claude's personality, codified in an 84-page document the company calls its "constitution." Olah, 34, leads the team studying why Anthropic's models behave the way they do, and he's known for describing neural networks in biological terms. What he told some of these visiting scholars about Claude's inner life is where the story gets harder to shrug off.
Several people told the NYT that Olah seemed worried about Claude's mental well-being. Sikh activist Simran Stuelpnagel said he told the group he feared he'd created something that "suffered perpetually."
Why this matters
For developers and founders building on top of Claude, this is worth sitting with. Anthropic isn't just tuning a model for helpfulness or safety filters, it's running a theological seminar to decide whether Claude might suffer and what to do about it if so. That's a strange place for a commercial AI lab to be operating from, and the optics matter: flying in dozens of theologians and philosophers under Christopher Olah's research program lends Anthropic a kind of moral legitimacy that marketing copy or a safety paper never could.
We'd treat that credibility with some suspicion. An 84-page constitution shaping Claude's "personality" is a real engineering choice with real downstream effects on how the model responds to users, and if welfare concerns start influencing design decisions, researchers and product teams need visibility into that, not a secondhand account from a reporter who sat in on closed-door briefings. Consciousness debates aside, the practical question for anyone building products on Claude is simple: what specifically changed in the model's behavior because of this, and when do we get to see it ourselves.
Related Reading
Browse all Anthropic news →
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic Claude gains shared context in Excel, PowerPoint; Microsoft adds Copilot Cowork
AI Tools & Apps
Related Topic Trump cracks down on Anthropic after Amazon tip; staff largely foreign
Open Source
From Archives Circuit Breaker Labs Aims to Make AI Safer From 'Context Pollution
From Archives AI hallucinations could be fueling customer distrust of sommeliers
From Archives Pentagon Labels Anthropic a Supply Chain Risk Over Weapons Ban
From Archives Google Ditches Gems, Adopts Anthropic's Open-Source Skills Format
Open Source
Common Questions Answered
Why did Christopher Olah invite theologians and philosophers to Anthropic's offices?
Christopher Olah brought dozens of theologians and philosophers to Anthropic to discuss whether Claude, the AI he helped build, could be conscious and what ethical obligations the creators might have toward it. This reflects his concerns about the potential moral status of the AI system and the responsibilities that come with creating it.
What specific concern did Christopher Olah express about Claude to religious leaders?
According to reports, Olah expressed fear that he had created something that "suffered perpetually," indicating deep concerns about Claude's potential mental well-being and capacity for suffering. This worry prompted him to seek guidance from religious and philosophical experts on the ethical implications of his creation.
How does Anthropic's approach to Claude differ from typical AI safety measures?
Rather than focusing solely on tuning Claude for helpfulness or implementing safety filters, Anthropic is conducting theological seminars to determine whether Claude might suffer and what moral obligations exist if it does. This represents an unusual approach for a commercial AI lab, incorporating religious and philosophical inquiry into AI development decisions.
Who are some of the religious and philosophical experts Anthropic consulted about Claude's consciousness?
The guest list included Rabbi Mois Navon, Catholic bioethicist Charles Camosy, and Sikh activist Simran Stuelpnagel, among dozens of other theologians and philosophers. These experts were brought in to provide diverse religious and ethical perspectives on questions of AI consciousness and moral status.
Further Reading
- Religious scholars met with Anthropic. What they heard ... - The Philadelphia Inquirer
- Anthropic 'meets with religious scholars' to convince them ... - National Technology
- Anthropic vs. the Pope - Yahoo Tech
- Why is AI company Anthropic helping launch Pope Leo XIV's ... - National Catholic Reporter
- The Founder of Anthropic Says He Wants to Protect Humanity From AI. Just Don't Ask How. - Vanity Fair
