An AI tutor raises achievement by half a standard deviation in a large trial

Current status

Nothing qualifies yet. A paper only counts if its achievement tests finished after 16 Sep 2026, so in practice the earliest candidates are trials from the 2026‑27 school year. Those results will mostly appear from mid‑2027.

Reference class: large randomized trials of AI or adaptive tutors

High‑income settings show small or negative effects:

• Khanmigo, two years in Tennessee: about 0.06–0.08 SD per year (Oreopoulos & Low, EdWP 26‑1551, Aug 2026).
• NUMI, 6,997 students: modest effects only (NBER, Sep 2026).
• University of Maryland, 2,379 students: −0.27 to −0.37 SD on grades (EdWP ai26‑1598).
• Bastani et al., Turkey: zero or negative.

Low‑ and middle‑income countries show bigger effects, but the upper tail sits just below 0.5:

• Gemini in Sierra Leone, 1,763 students: 0.258 SD after 8 weeks (DeepMind report, Jun 2026).
• Dominican Republic adaptive software, 1,333 students: 0.29–0.31 SD (World Bank WP 11418).
• Nigeria GPT‑4 after‑school program: 0.31 SD.
• Rori in Ghana: 0.37 SD.
• Andhra Pradesh adaptive‑learning software, about 14,000 students: 0.43 SD (Development Innovation Lab brief, Sep 2025).
• EIDU in Kenya, 1,955 pupils: 0.45 SD for numeracy and for literacy. The 0.534 overall score includes non‑academic areas, so it would not count.
• Khan Academy in Uttar Pradesh: 0.44–0.47 SD. The treatment was implementation staff, not access to the tool, so this would not count either.

Benchmarks:

• A World Bank meta‑analysis (Aug 2026) puts the average effect of AI tutoring at 0.12 SD, with a range of −0.09 to 0.55.
• Human tutoring programs serving more than 1,000 students average 0.16 SD (Kraft et al.).

Among roughly 25 historical large trials I found no clear hit, though there are several near misses at 0.43–0.47. I put the chance that a given trial clears 0.5 at about 2–4% in low‑ and middle‑income countries (around 10% for intensive, well‑run programs) and under 1% in rich countries.

Pipeline and paths to YES

Trials with endlines after the opening date include:

• Gemini science trial in India: 9,000 pupils, ending Sep 2026 (AEARCTR‑0019276).
• BEFIT in Malawi: onecourse tablet software at a high dose of about 4 hours a week, early‑grade reading and maths tests, intervention ends Jan 2027 (AEARCTR‑0018446). Whether onecourse counts as an "AI tutor" is uncertain.
• Uzbekistan adaptive maths software (AEARCTR‑0019200).
• Peru AI maths tutor, which ended 11 Sep 2026 and may qualify depending on when its endline tests were given.
• Rori full‑scale trial in Ghana: 8,100 students.
• US and UK trials: PLUS (4,500 students), ALTER‑Math (7,500), Eedi, and NUMI 2026‑27.

DeepMind, the World Bank, J‑PAL/LEVI, the Development Innovation Lab and others are adding trials every year. The rules allow bundled human support, extra instruction time, comparison groups with no tutoring, and tests aligned to the taught content. That raises the chance of a large effect in intensive, long‑running trials in poor countries.

Model

I treated each school‑year group of trials as a separate source of hits. The expected number of hits rises from about 0.2 for 2026‑27 to about 0.35 a year later on, as the number of trials grows. The delay from final testing to a full qualifying paper has a median of about 10 months, with a long tail. I mixed three scenarios: modest effects (×0.45), middle (×1), and AI tutors improving (×2). The model gave about 63% by the end of 2031; I shaded this to about 60% because of the strict criteria and the delay until full papers appear. That works out to about 9% by the end of 2027 and 25% by the end of 2028.

Strongest case against

Generative‑AI chat tutors at scale keep landing around 0.25 SD, and many new trials compare AI against existing software or test AI design variants rather than AI against no AI. Effects shrink as programs scale up, and near misses at 0.45 do not count. If trials stay near those levels, the probability would be closer to 40%.

Main sources

• edworkingpapers.com/sites/default/files/ai26-1551.pdf
• deepmind.google/blog/measuring-the-impact-of-learning-with-ai-in-sierra-leone-and-beyond/
• ideas.repec.org/p/wbk/wbrwps/11418.html
• convegenius.com/images/result.pdf
• wwhge.org (EIDU abstract, 2024)
• povertyactionlab.org (Uttar Pradesh Khan Academy; Rori full‑scale trial)
• thedocs.worldbank.org Can‑EdTech‑Close‑Learning‑Gaps.pdf (Aug 2026)
• AEA RCT Registry entries 19276, 18446, 19200, 19556, 19131, 19188, 18643, 17917, all retrieved Oct 2026

Question. By the last day of each quarter, has a randomized trial with at least 1,000 students found an AI tutor raising achievement by at least half a standard deviation?

Definitions.

• AI tutor: software that interacts directly with students and adapts instructional explanations, hints, feedback, or problem selection to their responses using an AI model or an intelligent tutoring system. Generative and non-generative systems, from any developer, qualify. Fixed videos, electronic textbooks, ordinary answer-checking software, and AI used only to advise a human teacher or tutor do not qualify.
• Randomized trial: a prospective experiment randomly assigning students, or clusters such as classes or schools, to an AI-tutoring intervention and a comparison condition. A comparison may be usual instruction, no additional tutoring, human tutoring, or another educational intervention without the tested AI tutor. Quasi-experiments do not qualify.
• At least 1,000 students: at least 1,000 distinct students assigned across the treatment and comparison arms used for the qualifying estimate, combined, before attrition. Students in unrelated arms do not count. Cluster trials count students, not clusters. Multiple independent trials cannot be pooled to reach the threshold; a single multisite trial can. Students in school, university, or formal vocational education qualify.
• Achievement: performance on an academic knowledge or skills assessment administered after the relevant instruction and completed without the tested tutor's assistance. Researcher-designed tests and immediate post-tests qualify; no minimum intervention duration or retention interval is required. Engagement, satisfaction, attendance, course completion, self-reported learning, and performance while using the tutor do not qualify.
• Raising achievement by at least half a standard deviation: the paper's main intention-to-treat estimate for at least one primary achievement outcome in the full eligible treatment–comparison sample is positive and at least 0.50 student-level standard deviations. If the paper does not designate primary outcomes, use achievement outcomes explicitly identified as its main results. Use the authors' preferred adjusted specification, or their main unadjusted specification if none is preferred. The outcome must be standardized using a student-level control-group, pooled-sample, or baseline standard deviation, or reported as Cohen's d or Hedges' g; standard errors and standard deviations of classroom or school means are not eligible denominators. Use the most precise reported value rather than rounding a lower value up to 0.50. The point estimate, not the lower confidence bound, must meet the threshold; statistical significance is not an additional requirement.

Resolves YES if an eligible full research paper reports a completed trial satisfying all the definitions. Human supervision, ordinary teaching, and additional instructional time bundled with the AI-tutoring program are allowed: the trial need not isolate the AI component from every other component. An intervention that merely adds AI advice for human instructors does not qualify. A paper must provide enough methods and results to verify randomization, sample size, assessment conditions, and the standardized effect.

Resolves NO / Exclusions. Resolve NO at a horizon if no qualifying paper has been published by that horizon. Exclude study announcements, protocols, registry entries without results, press releases, demonstrations, observational comparisons, meta-analyses, within-treatment pre/post gains without a randomized between-arm contrast, subgroup-only effects, and treatment-on-the-treated or per-protocol effects that reach 0.50 when the main intention-to-treat estimate does not. Apply the resolution window and observation period conventions: the qualifying assessment period must end after the opening date; publishing or revising results solely for earlier assessment periods does not qualify.

Resolving source. A peer-reviewed research article in a scholarly journal or conference proceedings, or a publicly accessible full working paper officially published by a university, government research body, or research institution, including the World Bank's Policy Research Working Paper series. Author affiliation alone does not make an independently uploaded manuscript an institutional publication. Institutional publication must be evidenced by an official-domain paper page, repository record, or working-paper series entry. These research publications are the resolving sources, rather than general Public confirmation under A2. Apply the Resolution date convention A3 to the first qualifying publication; an eligible working paper need not await peer review. Corrections issued before a publication first qualifies are incorporated; after a valid YES, the resolution window's persistence rule applies.