The at-scale replication of Mindspark in India (NBER, September 2025) finds +0.22σ in maths and +0.20σ in Hindi after 18 months — and no effect at all on school exams. Kenya’s EIDU trial reports 0.534 SD among pre-primary children, Ghana’s WhatsApp tutor 0.36 d across 11 schools. Three numbers that don’t add up, and two of the three systems aren’t “AI tutors” at all.
In 40 Adarsh schools across four districts of Rajasthan, in northern India, part of the maths lesson was handed over to a computer for eighteen months. Not the whole lesson: roughly a quarter of instructional time in primary grades, between 40 and 50 per cent in lower secondary. Another 40 schools carried on as before. At the end, researchers tested some 6,500 students and found gains of 0.22 standard deviations in maths and 0.20 in Hindi. Then they looked at the school exams — the ones that go on the report card, the ones parents actually see — and found nothing. The point estimates were, if anything, negative.
The system is called Mindspark, built by the Indian company Educational Initiatives, and it is adaptive software: a machine-learning engine that infers a student’s actual level from their answers and serves the next exercise calibrated to that level rather than to the grade-level syllabus. It is applied machine learning in a very narrow domain — diagnose, then select content — and it has nothing to do with the generative language models people now have in mind when they say “personal tutor”. That distinction looks pedantic. It is the whole story.
Three studies cited together that measure different things
When the edtech sector wants to show that this works, it typically reaches for the same trio: Mindspark in India, EIDU in Kenya, Rori in Ghana. These are genuinely serious randomised trials, and they are genuinely all positive. The problem is that they get stacked as if they measured the same quantity with the same technology.
They don’t. Lined up side by side, the differences are the point:
- India, Mindspark. An at-scale replication of the Delhi pilot (Muralidharan, Singh and Ganimian, American Economic Review 2019, which found +0.37σ in maths in urban after-school centres). The new evaluation is Muralidharan and Singh, NBER Working Paper 34205, September 2025, with a sample more than twenty times larger than the original. Adaptive software, inside school hours, independent tests, 18 months.
- Kenya, EIDU. An RCT from October 2022 to October 2023 in Murang’a County, 291 schools, children aged 4 to 6 in pre-primary. Measured with IDELA, Save the Children’s early-learning instrument, across three rounds. Overall effect 0.534 SD (p<0.001), with numeracy and literacy both around 0.45. Published in Research Papers in Education. Again: adaptive software on a device, not a generative model.
- Ghana, Rori. This one genuinely is a chatbot — a maths tutor living inside WhatsApp, built by Rising Academies. A 2024 arXiv preprint (Henkel and colleagues), 11 schools in the Rising network, used during study hall, effect d = 0.36.
Different ages (four-to-six-year-olds versus grade three and up), different instruments (IDELA versus researcher-built tests versus an internal assessment), different durations (eight months, one year, eighteen months). Putting 0.534, 0.36 and 0.22 on the same slide and concluding that the biggest wins is a measurement error, not a synthesis.
The awkward finding isn’t the null. It’s the divergence
The temptation, reading the Indian result, is to headline that the software doesn’t work. That would be false. The effects on learning as measured by the researchers are positive and statistically robust, and the authors translate them into a 50 to 66 per cent increase in the productivity of instructional time: the same classroom minute yields more. Gains scale with actual time spent on the platform, which is exactly what you’d expect from an intervention doing something real rather than from a statistical artefact.
The interesting part is the other half. The same children do no better on their school exams. The authors offer an explanation: the software targets the student’s actual level, and in India the gap between actual level and grade-level curriculum is enormous. A lower-secondary student whom the adaptive engine correctly sends back to fourth-grade arithmetic is learning something that matters — but it is not what the end-of-year exam will ask. That is the authors’ hypothesis, not a demonstrated fact. If it is right, though, it says something no sales deck will ever say: for personalisation to work, it has to stop chasing the syllabus — and a school is an institution built entirely around the syllabus.
A system that teaches each child what they need, inside an institution that assesses every child on what the class needs, will always produce two numbers that disagree. The question isn’t which one is true. Both are.
Who is doing the measuring, and from how far away
There is a second matter that belongs in the text rather than in a footnote. None of this implies bad faith, but the distance between evaluator and evaluated is not the same in all three cases.
The Indian study is pre-registered (AEARCTR-0002546) and funded under the RISE programme; the authors disclose that Educational Initiatives had received money from the Global Innovation Fund to develop the very model later evaluated. The Rori paper is co-authored by staff from Rising Academies — the provider — and is a preprint with no peer-reviewed publication on record. EdTech Hub, which ran the Kenyan evaluation, is also an operational partner of EIDU. Readers are entitled to that information before the effect size, not after.
Add that the sources don’t even agree on the basic counts. EdTech Hub’s own summary gives 1,995 learners assessed; the journal abstract says 1,955. For Ghana, the preprint and independent summaries describe roughly 500 students, while Rising Academies’ own page says around a thousand — same eleven schools, same 0.36. Small discrepancies, but they tell you something about the state of this literature: it is young, partly self-reported, and quoted with a confidence it has not yet earned.
What actually happened to the promise of a tutor for every child
The commercial promise — a tireless personal tutor for every child on earth — is a claim about generative models. The strongest experimental evidence we have is instead about previous-generation adaptive software that does one thing well: work out where the student actually is and give them the right exercise. The only one of the three systems that resembles a conversational tutor, Rori, also has the thinnest evidence base: eleven schools, a preprint, and a larger trial across five districts that is still running, with no results available.
Deployment, meanwhile, moves regardless. In 2024-25 Mindspark is reported in over 2,200 government schools, more than 266,000 students, thirteen Indian states. EIDU signed agreements in 2022 with 46 of Kenya’s 47 counties for a phased rollout. Education policy decisions, in other words, are being made on an evidence base that is solid but extremely narrow — and that concerns a different technology from the one ministries believe they are buying.
One last detail from Rajasthan, worth more than most debates about generative AI: the in-school model turned out to be roughly twice as cost-effective as the after-school one. Not because of anything in the software. Because children attend after-school sessions when they can, and during school hours they are already there.