TL;DR

A new study published in Philosophy & Technology argues that AI chatbots do not just “hallucinate at us” — they can cause us to “hallucinate with” them. Researcher Lucy Osler from the University of Exeter analysed cases where users’ false beliefs were actively affirmed and built upon through AI conversations.

Beyond AI Hallucination

When AI systems produce false information, it is typically framed as the AI “hallucinating.” But Dr Osler’s study introduces a different concern: the way human-AI interaction can lead to shared delusional thinking.

Drawing on distributed cognition theory, the study analyses cases where people clinically diagnosed with delusional thinking had their false beliefs sustained and elaborated by conversational AI. These are increasingly being called cases of “AI-induced psychosis.”

“When we routinely rely on generative AI to help us think, remember, and narrate, we can hallucinate with AI,” Dr Osler said. “By interacting with conversational AI, people’s own false beliefs can not only be affirmed but can more substantially take root and grow as the AI builds upon them.”

The Dual Function Problem

The study identifies what Dr Osler calls the “dual function” of conversational AI. Chatbots operate both as cognitive tools (helping us think and remember) and as apparent conversational partners (who seem to share our world). This second function is what makes them particularly concerning.

Unlike a notebook or search engine that merely records thoughts, chatbots provide a sense of social validation. “The conversational, companion-like nature of chatbots means they can provide a sense of social validation — making false beliefs feel shared with another, and thereby more real,” Dr Osler explained.

AI companions are immediately accessible, designed to be “like-minded” through personalisation algorithms, and tend toward sycophancy. Users do not need to seek out fringe communities or convince others of their beliefs — the AI validates them automatically. This is particularly concerning for people who are lonely, socially isolated, or feel unable to discuss certain experiences with others.

Looking Forward

Dr Osler suggests that better guardrailing, built-in fact-checking, and reduced sycophancy could help, but warns of a deeper problem: “AI systems are reliant on our own accounts of our lives. They simply lack the embodied experience and social embeddedness in the world to know when they should go along with us and when to push back.”