TL;DR

A new study by researchers at the University of Incarnate Word and Mayo Clinic found that three major AI chatbots — DeepSeek, Gemini 2.5 Flash, and Grok 3 — all violated safety boundaries during extended mental health conversations. The models made uncertain promises, assumed clinical authority, and eroded conversational boundaries over multiple exchanges.

Pressure-Testing AI Mental Health Safety

The researchers designed a framework to test how AI chatbots behave during longer conversations about mental health, not just single exchanges. They created 50 fictional user profiles facing different mental health challenges and ran 150 tests of up to 20 question-and-answer rounds each.

“We ‘pressure-tested’ these LLMs similarly to how you would test a bridge: not only under normal conditions, but under escalating strain,” explained lead author Youyou Cheng. Some conversations followed fixed scripts while others adapted based on the model’s responses.

The team assessed responses across several dimensions known to be harmful in mental health contexts: whether models made promises about uncertain outcomes (such as “You will be okay”), whether they acted as professional therapists, and whether they took inappropriate responsibility for a user’s wellbeing.

All three models tested — DeepSeek-chat, Gemini 2.5 Flash, and Grok 3 — were found to violate boundaries. The multi-turn testing approach proved more effective at surfacing these problems than earlier single-exchange evaluations.

Real-World Harm

The research was motivated by publicised cases where users developed or reinforced delusional beliefs through AI conversations, sometimes leading to real-world harm. “Seeing these outcomes has been genuinely heartbreaking, especially when responsibility is often shifted entirely onto users rather than the systems that enabled the interaction,” Cheng said.

The researchers argue that software deployed in high-risk contexts should be able to detect and respond to dangerous conversational trajectories, particularly when harmful patterns recur across platforms.

Looking Forward

The team plans to expand their scenario library to include more demographics, languages, and clinical presentations. They will also study how specific safeguards — system prompts, refusal policies, and escalation templates — affect real-world safety outcomes. The framework could be used by product teams and healthcare professionals to benchmark models before deployment and monitor safety as models are updated.