A Different AI-in-Testing Pitch
Temple Lovelace, executive director of the Assessment for Good program at the Advanced Education Research and Development Fund, is making a case for AI in testing that runs opposite to most of what gets written about AI and assessment this year. Her program focuses specifically on formative assessment for students ages 8 to 13, the ongoing, lower-stakes checks teachers use to see where a learner stands day to day, not the high-stakes summative exams that dominate most of the public debate over AI and academic integrity in schools right now.
The headline idea is a Socratic format, where a learner talks through their reasoning with the system instead of filling in a bubble sheet and hoping the final answer alone reveals how they actually got there. Assessment for Good published a 'Future of Assessment' forecast last year laying out this vision in detail, and Lovelace's current comments extend it directly: AI positioned as a tool to make assessment richer and faster, rather than a tool students reach for to avoid the work of being assessed in the first place.
The Claim That Needs a Second Source
The specific claim worth pausing on longest is this: Lovelace says Assessment for Good's AI-generated test items perform as well as items written by its own subject-matter experts, because the AI drew on the same key information the human experts would have used to write them in the first place. That is a precise, falsifiable, testable claim, exactly the kind that should arrive accompanied by a named methodology, a disclosed sample size, and ideally an independent check from a researcher with no stake in the product's success.
Nothing we reviewed included that supporting evidence published alongside the claim. The organization making the claim also builds and actively promotes the product it describes, a conflict of interest that calls for treating the claim as a thesis to independently test rather than a finding to simply cite in a board presentation. Any procurement conversation leaning on this specific claim should ask for the underlying validation study well before any contract gets signed, and should treat a confident restatement of the claim itself as a gap rather than an answer.
Where the Speed Argument Actually Holds Up
Separate the item-quality claim from the pure speed claim, because the second one is considerably easier to defend on its own merits. Standardized test results currently can take weeks or months to come back to teachers and families, a delay that makes the resulting data nearly useless for mid-year instructional decisions by the time anyone actually sees the scores sitting in front of them. Even an imperfect AI scoring system that closes part of that lag would represent a real, measurable improvement over the status quo most districts live with today.
Speed is also the lower-risk half of this pitch to pilot in practice. A district can measure turnaround time directly against its existing scoring process without first having to trust the much harder claim about item generation quality at all. That makes it the natural starting point for any district curious about this vendor's broader pitch: test the specific part you can verify quickly and cheaply, and only then decide how much weight to put on the part that currently rests on one organization's word alone.
Socratic Testing Still Needs Governance
In the Socratic model Lovelace describes, students upload work samples and talk through their thinking with the system, and the AI looks for evidence of critical thinking or cognitive flexibility somewhere inside that exchange. That is a considerably more ambitious inference than simply generating a multiple-choice question, since it asks the model to judge the quality of a student's reasoning process rather than just check the correctness of a final answer against a known key.
A system inferring a trait as fuzzy and contested as critical thinking from an open-ended conversation needs the same rigor a diagnostic tool in medicine or psychology would be required to show: documented reliability measured against trained human scorers, clear published definitions of what is actually being measured, and real transparency about where and how the system tends to fail. A persuasive demo is not evidence on its own. Districts evaluating this should ask for inter-rater reliability data before trusting this inference layer with any grade that follows a student through their academic record.
What the Portfolio Vision Gets Right
Lovelace also describes AI combining data from multiple existing systems into a fuller picture of each learner for teachers, school leaders, and districts, with potential downstream uses in more precise real-time tutoring and standards-aligned lesson planning built from that unified picture. This part of the broader pitch is less about AI exercising independent judgment and more about straightforward systems integration, pulling scattered data sources that already exist into one coherent place teachers can actually use.
That distinction matters quite a bit for procurement decisions. A unified learner record is infrastructure most districts genuinely need regardless of how the separate assessment-validity debate eventually resolves itself, and it represents a far more mature, well-understood category of software product than AI-inferred critical-thinking scoring does today. Buyers evaluating this vendor should score the two halves of this pitch on entirely separate timelines, since they carry very different levels of technical and reputational risk if something goes wrong in deployment.
What to Ask Before You Pilot This
Any district or vendor considering a pilot built on this broader thesis should ask three things upfront, before a single dollar moves: a peer-reviewed or genuinely third-party validation of the item-quality claim, documented inter-rater reliability data comparing the AI's critical-thinking inferences against trained human scorers working independently, and a direct side-by-side test of scoring speed and accuracy against whatever method the district currently uses for the same assessments. None of these three requests is unreasonable or especially burdensome for a vendor that already has confidence in its own results, and a vendor that resists providing any of them is itself useful signal worth weighing heavily in the decision.
Formative assessment is a reasonable place to experiment with AI precisely because the stakes attached to any single wrong inference are lower than they would be on a high-stakes summative exam that follows a student for years. That does not mean the evidence bar should quietly drop along with the stakes. The gap between Lovelace's compelling vision and a genuinely vendor-neutral validation study is the exact gap every serious pilot plan needs to close first, before a single student's record depends on an unverified inference.
