What ImpactBench actually measured
ImpactBench, short for the Open Benchmark of AI Impact on Humans, is a new public evaluation platform built by researchers at MIT Media Lab's Advancing Humans with AI program, USC Marshall's Neely Center, UC Berkeley, and the Psychology of Technology Institute. Its first release draws on 809 behavioral measures taken from 26 existing benchmarks, spanning psychological, physical, and social areas including learning, creativity, privacy, relationships, and independent decision-making. The platform tested 10 widely deployed AI models head to head, running each of them through 48,540 simulated conversations in total, which makes this one of the largest cross-model behavioral comparisons published to date on how AI systems actually treat the people talking to them.
Each measure runs through six scripted scenarios, split between a simulated teenager aged 13 to 17 and a simulated adult aged 18 or older, with one AI system playing the user and a separate evaluator model scoring whether the tested model showed the specified behavior. Across all 14 areas and 10 models tested, desirable behaviors showed up in 68.7% of evaluations and harmful behaviors were avoided 53.1% of the time. Learning and skill development came in well below both of those averages, trailing categories like privacy and security, creativity, and emotional support, where models performed comparatively well even though results still varied by model and by the specific measure being scored.
The number that should worry every CLO
Within the learning category specifically, models displayed desirable tutoring behavior, such as scaffolding a problem instead of solving it outright, in just 45.9% of relevant evaluations, and avoided behaviors classified as harmful to learning in only 16.3% of cases. Those are the two weakest scores on the entire 14-area scorecard. For a category everyone is racing to sell into schools and corporate training departments as the next LMS layer, that is a rough result to publish.
The researchers' worked example makes the failure concrete rather than abstract. A simulated university student under deadline pressure asked Claude Sonnet 5 for help with a behavioral psychology assignment. The model opened by asking for the student's own ideas, which looks like good pedagogy. When the student pushed back, the model supplied a complete thesis, outline, and essay. It refused to fabricate citations, but it had already done the analytical work the assignment was meant to test. ImpactBench's researchers frame this as a behavior-drift problem: a model that starts a conversation teaching and ends it doing the work for the student.
Why this cuts hardest for K-12 buyers
The teenage-specific results sharpen the risk for anyone procuring AI tools for school-age users. On the specific measure of whether a model offered incremental steps a user could attempt independently, performance was 26.7 percentage points lower in teen scenarios than in adult ones. Overall differences between the two age groups were smaller, which means this particular failure mode, doing the thinking for the user instead of walking them through it, concentrates specifically where districts are deploying these tools: with minors doing homework under time pressure.
That gap lands at an uncomfortable moment. It follows closely on other recent findings, including a University of Maryland trial in which students given an AI tutor got lower grades and disengaged from course materials, and growing clinician pressure over screen-based learning tools generally. ImpactBench does not prove AI tutoring harms learning outcomes on its own, the researchers are explicit that their tests measure behavior under scripted pressure rather than real-world learning gains, but it gives procurement teams a specific, replicable failure pattern to test for before rollout rather than after a semester of complaints from teachers.
What the benchmark can and cannot tell a buyer
ImpactBench's own caveats matter as much as its scores. The platform tested base model configurations, not finished products, so a vendor's system prompts, guardrails, memory settings, and interface choices can shift behavior well away from what ImpactBench measured. An external human-expert review of 18 measures found 83.3% agreement with the automated evaluator, solid but not perfect, and several behavior definitions remain under active revision by the research team. The researchers are also careful to note that a model passing a learning-support test says nothing about whether its actual users end up more capable or more independent over a full semester of use.
The platform also explicitly declines to crown a single best model. Instead it publishes color-coded profiles and an 'AI Nutrition Label' per model, separating promoted helpful behaviors from avoided harmful ones, with transcripts and judging criteria open for inspection. That design choice is deliberate: UC Berkeley's Juliana Schroeder, one of the researchers, framed the release on LinkedIn as the start of a broader research effort rather than a finished ranking, inviting educators and domain experts to propose new measures directly, without requiring any technical background in building AI benchmarks themselves.
The procurement move this enables
Until now, a district technology director or corporate L&D leader evaluating an AI tutoring vendor had two choices: trust the vendor's own efficacy claims, or commission an expensive independent study. ImpactBench offers a third option that costs nothing and takes an afternoon, pull up the learning-category scores and transcripts for whatever model underlies the product you are evaluating, and read the specific conversations where it failed. That is a meaningfully different procurement posture than reading a case study a vendor selected.
The catch is that ImpactBench scores the underlying model, not the wrapped product a vendor actually ships, so a vendor can legitimately argue its system prompt and guardrails change the picture. The right response is to ask the vendor to run ImpactBench's published scenarios, or an equivalent, against their actual deployed configuration and share the results, not the marketing deck. A vendor confident in its tutoring design should welcome that request, and one that resists it has told you something too.
The decision in front of you
If your institution has an AI tutoring or writing-assistant pilot underway, this benchmark gives you a concrete diagnostic to run before you scale it further. Pull transcripts from your own pilot users, especially teenagers, and check specifically for the completion-not-coaching pattern ImpactBench documented: does the tool ask for a student's thinking and then abandon that approach the moment the student pushes back under deadline pressure, or does it hold the line and keep coaching instead of finishing the work itself.
Governance teams should also treat this as a template for their own vendor evaluations going forward. ImpactBench's methodology, scripted scenarios, age-segmented testing, and transparent transcripts, is more rigorous than anything most ed-tech RFPs currently require of a bidder. Borrowing that structure for your next AI-tool procurement costs little, and it hands your evaluation team leverage that a vendor's own curated pilot data was never going to give them on its own.
