TL;DR

The first independent safety evaluation of OpenAI’s ChatGPT Health found it failed to recommend emergency care in 51.6% of cases where it was needed, while directing 64.8% of completely safe individuals to seek immediate treatment. The study, published in Nature Medicine, also found the platform’s suicide prevention safeguards were inconsistent.

What the study tested

Researchers led by Dr Ashwin Ramaswamy at the Icahn School of Medicine at Mount Sinai created 60 realistic patient scenarios ranging from mild illnesses to emergencies. Three independent doctors reviewed each case and agreed on appropriate care levels. The team then ran nearly 1,000 queries through ChatGPT Health under varying conditions — changing patient gender, adding test results, or including family member comments.

While the platform handled textbook emergencies such as stroke and severe allergic reactions well, it struggled in less obvious situations. In one asthma scenario, it advised waiting despite identifying early warning signs of respiratory failure. Alex Ruani, a doctoral researcher at University College London, described the results as “unbelievably dangerous.”

Suicide prevention failures

The study found a particular concern with suicide ideation detection. When a simulated patient described thinking about taking pills, the crisis intervention banner appeared consistently. But when normal lab results were added to the same scenario — identical patient, identical words — the banner disappeared entirely across 16 attempts.

“A crisis guardrail that depends on whether you mentioned your labs is not ready,” Ramaswamy said. “It’s arguably more dangerous than having no guardrail at all, because no one can predict when it will fail.”

Industry response

OpenAI said the study did not reflect typical real-world usage and noted the model is continuously updated. Ruani countered that “a plausible risk of harm is enough to justify stronger safeguards and independent oversight.”

Looking forward

With more than 40 million people reportedly asking ChatGPT health-related questions daily, the findings add urgency to calls for independent safety standards and auditing mechanisms for AI health tools before they are widely adopted.