Do AI Tutors Actually Work? What the Studies Show

Theo Marchetti

GenM Offers Free Marketing for Businesses With Digital Apprenticeships

Online learning spent a decade as a cheaper substitute for a classroom: a recorded lecture, a quiz, a certificate nobody checked. That version is largely dead, and what replaced it is more interesting and more unsettling. The question worth asking is no longer whether a child can learn from a screen. It is whether a machine that adapts to one child in real time can outperform a good teacher addressing thirty.

Two studies published recently suggest it can, at least on a narrow measure, and both come with caveats that the headlines dropped.

What actually changed

The shift is from broadcast to conversation. Older online courses delivered the same material to everyone and tested at the end. An AI tutor works the other way round: it asks a question, reads the answer, and decides what to do next. If a student misunderstands one step of a problem, it can stay on that step without the social cost of holding up a room. That difference in pacing, rather than any improvement in content, is what the research keeps pointing to.

The Harvard physics study

Researchers at Harvard, led by physics lecturer Gregory Kestin with senior lecturer Kelly Miller, ran a trial with 194 students in a physics course for life-sciences majors in autumn 2023. The design was a crossover: each group took one instructor-guided active-learning lesson in class and one AI-supported lesson at home, then swapped the following week, with pre-tests and post-tests measuring mastery.

Learning gains for the AI-tutored lessons came in at roughly double those of the in-class lessons, with mean post-test scores of 4.4 against 3.6. Students also reported markedly higher engagement and motivation.

That is a striking result, and it needs three caveats attached. The findings were preliminary, posted ahead of final publication. The comparison was two lessons, not a term. And Kestin himself warned that AI could just as easily undermine learning, that these tools should not think for students, and that the sensible role is to supplement in-person teaching rather than replace it. Harvard Gazette has the detail.

The Nigeria programme

The larger claim comes from Edo State, Nigeria, where a World Bank-supported six-week after-school programme paired AI tutoring with teacher guidance for secondary students. Participants gained 0.31 standard deviations, a figure the programme leads equate to roughly one and a half to two years of ordinary schooling.

If that holds up it is remarkable. Read the write-up carefully, though, and the interesting finding is not the technology. The account comes from World Bank staff who co-led the work rather than an independent evaluator. It reports nothing on retention past six weeks, nor on cost or scalability. The conversion from standard deviations into “two years of schooling” is not explained. And the authors attribute the result not to the tutor but to everything built around it: a defined problem, teacher co-design, AI literacy training, local-language support, and delivery over mobile phones because devices were scarce. Their own summary is blunt — simply incorporating AI does not guarantee usage or impact.

What both studies have in common

A teacher. In Harvard’s trial the AI lesson sat inside a taught course with a lecturer who had built the tutor himself. In Edo, teachers guided the sessions and helped design them. Neither study tested a machine teaching children on its own, and in both cases the researchers went out of their way to say that is not what they were recommending.

So the honest reading of the current evidence: an adaptive tutor, supervised by a competent adult, appears to beat a lecture. That is a real and useful finding. It is not the same claim as the one in the callout below.

Our call

Robots will be the primary teacher for most children, and sooner than parents expect. Not an assistant, not a homework helper — the thing delivering the instruction, with the adult in the room supervising rather than teaching. The economics point one way: a tutor that adapts to one child costs almost nothing to run and a teacher who does the same costs a salary, and the pacing advantage in the studies above is the whole ballgame in a class of thirty.

What the evidence currently says against us: every study worth citing kept a teacher in the loop and the researchers running them explicitly warn against replacement. We are predicting past the data, deliberately, and saying so.

What would prove us wrong: if measured gains depend on the supervising adult rather than the software, the model collapses — you cannot remove the expensive part. Also if retention turns out poor past six weeks, or if what children lose in a machine-taught classroom (motivation, the social work of being confused together, learning to be taught by a person) shows up as a cost nobody priced.

What this means if you have a child now

Practically, the useful move is not waiting for the future. It is using an adaptive tutor for the thing it demonstrably does well — drilling a specific stuck point at the child’s own pace — while treating the claim that it can run an education as untested. Sit with them for the first few sessions. Watch whether the tool explains or simply supplies answers, because a model that completes the work is worse than no tool at all.

The problems nobody has solved

Access. The Edo programme ran on mobile phones because devices and electricity were scarce. Any model assuming a laptop and reliable power is not a model for most of the world.
Retention. Six-week gains are not the same as knowledge a year later, and nobody has published the long-run number.
Assessment. If the tutor can write the essay, the essay has stopped measuring anything.
The adult in the room. Both studies had one. Neither tested what happens without.

Online learning has genuinely advanced, and the advance is real rather than marketing. What has not happened yet is the part everyone is arguing about.

More of our calls: our predictions for 2030, and how the 2014 ones held up