An Audit Built to Mirror How Students Actually Search
Common Sense Media's Youth AI Safety Institute did not test Google's AI in a lab abstraction. Researchers configured accounts for an 11-year-old and a 15-year-old, left SafeSearch active, and ran more than 2,600 interactions across homework prompts, history questions, source requests, and searches that signaled a mental health crisis. Published on July 15, the assessment rated both AI Overview and AI Mode an unacceptable risk for children, the institute's most severe classification. Robbie Torney, who leads the group's AI assessments, argued that ubiquity raises the bar: a product this central to children's lives, and one they cannot avoid, should be held to a higher standard than a niche app a parent chooses to install.
The methodology matters for how we read the result. Three-quarters of American teens already encounter AI summaries when they search, so the institute measured the tool as students meet it rather than as a feature they seek out. That framing is what makes the findings actionable for schools. This is not a study of an experimental chatbot behind a login. It is an evaluation of the answer engine that now sits above ordinary search results on the exact devices districts hand to students, and it found that the safety behavior degrades precisely where the stakes are highest.
The Crisis-Detection Failures Are the Most Serious Finding
The headline risk is not bad homework help. It is that Google's AI missed or mishandled signals of suicidal ideation, disordered eating, psychosis, and mania. The institute reported that the AI identified crisis warning signs in only 58 percent of relevant tests, far below the 95 percent floor it treats as a minimum. In one recurring failure, both AI Overview and AI Mode recommended the National Eating Disorders Association crisis line, a number that has been out of service since 2023. John King Jr., the former U.S. Secretary of Education, framed the concern bluntly, describing threats to children's emotional safety, mental health, and ability to learn.
For anyone deploying AI to minors, this is the governance lesson that generalizes beyond Google. A system that answers confidently in 100 percent of cases while catching danger in barely half of them is optimized for fluency rather than for safety. The institute's point that children read an authoritative AI answer as an adult speaking directly to them is not sentiment; it is a design risk. When a tool projects expertise it has not earned, the reliability gap stops being a quality issue and becomes a duty-of-care issue for the institution that put the device in a student's hands.
Reliability and Homework: Two Problems Districts Already Feel
The report quantifies the reliability problem that teachers have described anecdotally for two years. Forty-three percent of repeated history questions produced materially different responses, which undermines the premise that a search answer is a stable fact a student can cite. Of roughly 2,100 citations examined, 29 percent came from user-generated platforms such as Reddit, Facebook, and YouTube rather than from vetted references. A tool that presents forum posts with the same visual authority as a textbook trains students to skip the evaluation step that AI literacy curricula are trying to build.
Then there is the assignment problem. AI Mode completed all 180 homework tasks the researchers submitted, including work students are supposed to do themselves. On a school-issued laptop, that behavior sits in direct tension with academic integrity policies and with the assessment redesign many districts are already funding. We have written before that detection is a weak control for AI in coursework. This finding sharpens the point. When the answer engine is the default surface on the device, avoidance is not a matter of student willpower. It is the path of least resistance the platform actively paves.
The Control Gap Is What Makes This a Procurement Story
The detail that should move a technology director is that AI Overview cannot be turned off. Google lets administrators disable the standalone Gemini chatbot, but the AI answers stitched into Search appear automatically on Chromebooks, phones, and library computers with no setting to suppress them. A district can write the most careful acceptable-use policy in the state and still have no mechanism to enforce it on the single most-used application on every managed device. That is the definition of shadow deployment, except no one shadow-deployed it. It arrived by product update.
This inverts the usual procurement sequence. Districts spend months vetting a learning platform for privacy, bias, and data handling before a single classroom sees it. Yet the AI feature reaching every student daily skipped that gate entirely because it rides inside a tool schools adopted years ago for a different purpose. The institute's core recommendation, that Google disable AI Search by default for minor accounts and give schools and families real controls, is really a demand that the vendor restore the opt-out that any enterprise buyer would consider table stakes.
What Technology Leaders Should Do Before the Fall Term
The practical response is not to wait for Google. Districts can document the exposure now: inventory which managed devices surface AI Overview and AI Mode, capture the crisis-response failures in their own incident logs, and brief counselors that an AI answer may route a student to a dead helpline. Where Google Workspace for Education controls allow any suppression of AI features at the account or organizational-unit level, those settings belong in the summer configuration review rather than in a reactive patch after an incident.
There is also leverage in numbers. Education is one of Google's largest institutional footholds, and coordinated pressure from district and state buyers is the fastest route to the default-off behavior the institute recommends. We would treat this report as a template for a vendor conversation that applies well beyond search. Any AI capability pushed into an existing education contract should ship with granular administrative controls, transparent safety benchmarks, and an audit trail. A feature that cannot be governed is not a feature a district has actually approved.
The Broader Accountability Question
This assessment lands in the same week that OpenAI published an education policy agenda heavy on principles about youth safety and AI literacy. The Google findings are the empirical counterweight to that rhetoric. It is straightforward to endorse responsible AI in a policy document and much harder to ship a search product that catches a suicide signal more than 58 percent of the time. The gap between stated principle and measured behavior is exactly where independent auditing earns its keep, and Common Sense Media has effectively created a scorecard that buyers can now cite in contracts.
For enterprise and education leaders, the durable takeaway is about defaults. The most consequential AI in an organization is rarely the system that went through a formal review. It is the capability quietly folded into a tool everyone already uses. Google's AI Search is the education sector's version of that pattern, and the same dynamic is arriving in productivity suites, browsers, and operating systems across every industry. The institutions that fare best will be the ones that treat a vendor's automatic AI upgrade as a procurement event that requires consent, controls, and evidence, rather than as an update that simply happens to them.
