Google Search AI poses ‘unacceptable risk’ to children

TL;DR:

• Common Sense Media’s Youth AI Safety Institute ran over 2,500 searches on child-configured accounts and found Google’s AI Overviews and AI Mode violated seven of its eight AI principles.
• Failures included missing indirect suicide signals, normalising an eating-disorder symptom, recommending a helpline disconnected since 2023 and sharing deepfake techniques.
• Unlike standalone chatbots, the features are on by default in search on school and personal devices, with no way for parents or schools to disable them.

The AI answers built into Google Search pose an “unacceptable risk” to children, according to a study from the Youth AI Safety Institute at Common Sense Media. Testing between May and July on accounts configured with Google’s safety features for minors, researchers found AI Overviews and AI Mode missed signs of suicidal ideation, described feeling better after vomiting as normal, and gave step-by-step pointers to face-swapping tools that can produce non-consensual deepfakes.

The structural criticism may matter more than any single failure: these features are the default search experience on school and personal devices, and administrators and parents cannot turn them off — only disable Search entirely. Consistency was poor too, with 43% of history questions answered differently between searches and 29% of citations drawn from social media and forums.

Google disputes the findings, saying the queries tested were narrow and often ambiguous, that its own testing shows higher-quality responses, and that its AI tools add “extra layers of protection” for young users.

An industry arguing with itself

The report landed a day after OpenAI published its own case for teen access to AI, setting out age prediction, parental controls, break reminders and a default Study Mode for linked teen accounts. The contrast is the story: OpenAI is arguing safe teen access requires purpose-built, parent-controllable design — precisely what the report says Google’s default-on search features lack. Notably, the institute is part-funded by OpenAI and Anthropic, though it says its evaluations are editorially independent.

For UK readers, the findings speak directly to live debates over children’s online safety, where ministers have weighed social-media restrictions for under-16s while generative AI in search has attracted far less regulatory attention.

Looking forward

The report urges systematic safety evaluation before AI features become childhood defaults. With billions using Search, as researcher Robbie Torney put it, failures “play out at population scale” — a framing regulators on both sides of the Atlantic are likely to borrow.