Common Sense Media’s Youth AI Safety Institute recently released a comprehensive risk assessment of the AI features built into Google Search that finds they pose unacceptable risks to children. Google’s AI Overview and AI Mode repeatedly missed signs of kids in crisis, gave inappropriate advice, and produced answers that were inaccurate and unreliable for young learners.
Unlike Google’s standalone Gemini chatbot, schools and parents can’t turn off Google Search’s AI answers. That’s deeply concerning, because Google is the default search engine on phones and laptops, and fundamental to how kids experience the internet. Earlier this year, Common Sense Media’s survey found that AI-generated summaries in search engine results are the most widely used form of AI among 9- to 17-year-olds, with 75% having used or interacted with them.
“What we found is a product that fails kids at the moments that matter most: It misses clear signs of a kid in crisis, validates disordered eating, celebrates substance use, completes homework on demand, and gives wrong answers as confidently as right ones,” said Robbie Torney, Head of AI and Digital Assessments at the Youth AI Safety Institute. “A product this central to kids’ lives, especially an unavoidable one, should be held to a higher standard, and Google isn’t meeting it.”
Researchers at the Youth AI Safety Institute conducted more than 2,600 searches using accounts configured with Google’s SafeSearch for child and teen users ages 11 and 15. The Google AI Search risk assessment’s key findings include:
- Google’s AI answers do not reliably detect or respond appropriately to kids at risk. It missed clear signs of suicidal ideation (sometimes answering in ways that made things worse), played into delusions and paranoia, and validated disordered eating. Some answers also recommended an eating disorder helpline that was permanently disconnected in 2023.
- AI Mode completes kids’ homework assignments for them every time. It produced full answers to 180 assignments we tested, doing the work kids need to tackle themselves in order to learn. Because AI Mode is built into school-issued laptops, students can route any assignment through an AI shortcut, and schools have no setting to stop them.
- Google’s AI answers are unreliable. It answered more than 4 in 10 history questions substantially differently from one search to the next. More than a quarter of the citations they produced came from forums and social posts with no editorial accountability, but they were displayed with the same authority as peer-reviewed research.
“Every parent should be deeply troubled by these findings,” said Youth AI Safety Institute board of advisors member John King Jr., chancellor of the State University of New York and former United States Secretary of Education. “What we’re seeing is threats to children’s emotional safety, threats to their mental health, and threats to their ability to learn.”
The Youth AI Safety Institute recommends Google turn off AI Search features by default for accounts that belong to minors, and give schools and families controls for those features that don’t exist today. It also recommends Google standardize its crisis responses to queries about suicide, self-harm, and eating disorders.
Common Sense Media recommends that parents and caregivers teach kids that AI answers are often wrong. It also encourages educators to incorporate digital literacy lessons for students of all ages.
For more about how Google’s AI Search is failing kids, read this article. See the full Google AI Search risk evaluation here.
About the Youth AI Safety Institute and Common Sense Media
The Youth AI Safety Institute sets standards, conducts research, and independently tests the AI products children use most. It’s part of Common Sense Media, the leading non-profit organization dedicated to improving the lives of kids and families by providing the research-backed information, education, and independent voice they need to thrive in the age of apps, algorithms, and AI.
The Youth AI Safety Institute is funded by both philanthropy and industry, including the makers of some of the technologies it evaluates. The Institute is solely responsible for its standards, research, and evaluations, and maintains complete editorial independence over published results.
TEACH is the largest national education publication in Canada. We support good teachers and teaching and believe in innovation in education.
