"Research Shows" Is Not an Answer

Originally written in Traditional Chinese: 研究證明 AI 讓孩子學得更好？那批研究被一篇一篇查了

"Research shows AI helps kids learn better." How many times have you heard that this year?

On November 2, Education Week reported on an audit that should make every parent, teacher and school leader pause. Patrick O'Neill, an associate professor at Ivy Tech Community College in Bloomington, Indiana, went through the meta-analyses claiming AI improves learning, the "studies of studies" most often cited as proof.

According to The Hechinger Report, there are 39 of them. All 39 reported substantial benefits. O'Neill examined 14 in depth. Those 14 have been cited more than 3,000 times on Google Scholar, and gain about 200 citations a month.

What he found: some included studies reported "impossible" statistics. Some references were likely hallucinated. One included study even cited ChatGPT studies dated before ChatGPT was publicly released. One of these widely cited meta-analyses, published in a Nature Portfolio journal in May 2025, was retracted in April 2026. As MIT's Justin Reich put it: "Peer review's not working. Meta-analysis is not working."

Stanford's AI Hub for Education (SCALE Initiative) tells a similar story. Of more than 800 papers (as of October 2025) on AI in K-12 education, only 20 were high-quality causal studies, and none were high-quality causal studies of student AI use in U.S. K-12 classrooms.

We often say the most dangerous thing about AI is that it can be confidently wrong. This time, what was confidently wrong was not the AI. It was the research we use to persuade each other.

"Research shows" is not an answer. It's a claim you should question.

To be clear, this does not mean AI is useless for learning. Stanford notes that tools with instructional guardrails show more promising outcomes than chatbots that simply hand out answers. O'Neill's own preliminary analysis of 44 well-designed experiments averaged roughly zero, with results ranging widely from very helpful to very harmful, and that is preliminary too. The honest summary: the evidence is not yet strong enough to support how loudly "it works" is being said.

Next time someone tells you "research shows", ask three questions:

  1. Who was studied? College students or young children? Where? How many?
  2. What was measured? Finishing homework faster, or still knowing it a month later?
  3. Can you find it? Search the authors and year on Google Scholar. Does it exist? Has it been retracted?

If none of the three has an answer, set the claim aside for now.

Get the Human-First practice card

Four steps, ten minutes, with copy-ready AI prompts. The card and my weekly letter are currently written in Traditional Chinese.

👉 Subscribe free to 碩學苑 (Traditional Chinese)

Follow in English

I also post in English on LinkedIn.

👉 Follow Haishuo Lee on LinkedIn

References