Expert
If you've been skeptical about whether AI tutors actually work, a 2024 study from Harvard University should change your mind.
Researchers found that students using an AI tutor learned more than twice as much as students in an active learning classroom, and they did it in less time.
The control group was an actively taught physics course using peer instruction, small-group activities, and research-based teaching methods. The kind of class most educators would consider best-in-class.
The AI tutor still beat it by a wide margin.
What the Study Found
The research team, led by Gregory Kestin and Kelly Miller from Harvard's Physics Department, conducted a randomized controlled experiment with 194 undergraduate students in an introductory physics course.
Each student experienced both conditions: learning from an AI tutor at home and learning in an active classroom. This crossover design eliminated individual differences as a factor.
The results:
- Students in the AI tutor group scored significantly higher on post-tests (median 4.5 vs 3.5 out of possible points)
- Learning gains were over double compared to the active lecture group
- The effect size ranged from 0.73 to 1.3 standard deviations—a large effect by any measure
- Results were highly statistically significant (p < 0.00000001)
Perhaps most striking: students using the AI tutor spent less time learning. The median time on the AI platform was 49 minutes, compared to 60 minutes for the in-class session. Seventy percent of AI tutor students finished in under an hour.
Students Preferred the AI Experience
Beyond test scores, the researchers measured how students felt about their learning experience.
Students using the AI tutor reported feeling:
- More engaged (4.1 vs 3.6 on a 5-point scale)
- More motivated when working on difficult questions (3.4 vs 3.1)
Enjoyment and growth mindset were comparable between groups, meaning the AI didn't make learning feel like a chore. Students found it just as enjoyable as the classroom, while actually learning more.
When asked about explanation quality, 83% of students said the AI tutor's explanations were as good as or better than those from human instructors.
Why This AI Tutor Worked When Others Haven't
Previous studies on AI in education have shown mixed results. Some found that students using ChatGPT without guidance learned less and engaged in less critical thinking.
So what made this different?
The Harvard team didn't just point students at a chatbot. They built a carefully designed system that followed pedagogical best practices:
1. Active learning, not answer delivery
The AI was instructed not to give away solutions immediately. Instead, it encouraged students to work through problems, only confirming answers or providing hints when needed. The system prompt explicitly stated: "DO NOT give away the full solution... if the student wants the answer in the first message, encourage them to give it a try first."
2. Structured scaffolding
Rather than free-form conversation, students moved sequentially through each part of each problem. This mirrored how an experienced instructor would guide a student through complex material.
3. Step-by-step solutions to prevent hallucinations
The researchers didn't rely on GPT-4 to generate solutions on the fly. They provided detailed, pre-written step-by-step answers in the system prompts. This dramatically reduced inaccuracies, a known problem with large language models in technical subjects.
4. Managed cognitive load
The AI was instructed to keep responses brief and only provide one step at a time. This prevented information overload and let students process concepts before moving on.
5. Growth mindset reinforcement
The system was designed to be supportive and encouraging, promoting the belief that effort leads to understanding rather than treating ability as fixed.
6. Self-pacing
Unlike a lecture that moves at one speed for everyone, students could take as much time as they needed or move quickly through material they already understood.
AI Tutors Complement Teachers, They Don't Replace Them
Despite these impressive results, the Harvard researchers are clear: AI tutors should not replace in-person teaching.
As the authors write: "An AI tutor should not replace in-person teaching—rather, it should be used to bring all students up to a level where they can achieve the maximum benefit from their time in class."
Their recommended approach mirrors the "flipped classroom" model. Use AI tutors for the initial introduction of material, getting everyone to a baseline level of understanding. Then use precious classroom time for what humans do best: advanced problem solving, project-based learning, group work, and critical thinking.
This also solves a common problem with AI in education. When students use AI for homework or assignments without structure, they often take shortcuts that bypass learning. But when AI is used for initial instruction before class, instructors can then assess higher-order skills in person, where AI can't be used as a crutch.
The study notes: "We advise against the notion that AI, solely due to its efficacy in enhancing teaching and learning, should entirely supplant traditional instructional methods. Our demonstration illustrates how AI can bolster student learning beyond the confines of the classroom. We advocate harnessing this capability to enable instructors to use in-class sessions for activities and projects that foster advanced cognitive skills such as critical thinking and content synthesis."