'Intelligent tutoring systems' use AI to boost student IQ

Do you know when you're learning?

During his presentation at Harvard University, Koedinger asked the 50 or so attendees at the Institute for Applied Computational Science's seminar, "Do you know when you're learning?"

The response was a 50-50 mix, which Koedinger described as a typical response to the question. However, research suggests that the answer is no -- students don't know when they're learning and that much of what they believe to be indicative of learning is an illusion.

Take the relationship between liking a subject or a college course and mastery. Students might think that the more they like a class, the more they learn. But according to Koedinger, the correlation between liking and learning is actually quite low. Indeed, studies using artificial intelligence software to detect student engagement show that students who appear engaged could be focused on the wrong content and students who appear confused could be in the midst of a productive struggle, according to Koedinger.

Part of the difficulty in answering Koedinger's question is that students have not acquired the expertise they need "to compare what they know to whether they really know it or whether they just think they know it," Koedinger said.

On the flip side, teachers have acquired so much knowledge that they may not be cognizant of how much their students don't know. Koedinger conducted research on what makes a high school algebra problem hard. He provided students with the same problem in three forms -- as a story problem, as a word problem and as an equation. Teachers thought the equation would be the easiest for students to solve. But students found it to be the hardest because they struggled with basic mathematical lexicon such as the order of operations.

Koedinger's point is that experts underestimate their knowledge and tend to overestimate what their students know. "There's literature that suggests experts are unaware of about 70% of their knowledge," he said. "This literature comes from various cognitive tasks analyses."

'Starts and ends with data'

Intelligent tutoring systems -- and, more specifically, the data generated by intelligent tutoring systems -- might be able to help fill the gap. Koedinger talked specifically about Cognitive Tutor, an intelligent tutoring system developed at Carnegie Mellon University and a project Koedinger has been involved in advancing.

Cognitive Tutor, which has been around for decades, is based on machine-generated cognitive models and performance models. The cognitive models are built on the skills needed to solve a problem, how the necessary skills are acquired, the rules that govern the problem and the errors students can make.

The cognitive models use a couple of algorithms to measure performance. A model-tracing algorithm follows students through their individual approach to a problem to dynamically provide help based not only on where a student gets stuck -- but on how a student gets stuck. A Bayesian knowledge-tracking algorithm models what rules a student has mastered, according to Koedinger.

All the intelligent tutoring system models generate data, which is vital to education research and to improving the learning system. Koedinger said the improvements are made in a continuous loop that starts with data and ends with data. In between, researchers create a kind of A/B test called an "in vivo test." These are "randomized, controlled experiments inside of the course where we test version one against version two and use the data to evaluate the improvement," he said.

Data also can reveal how students learn, he said. Collectively, the data can be used to produce learning curves, which map opportunities to learn against student error rate. Rather than telling students that if they want to do better at geometry, they should do more geometry, the data generated by an intelligent tutoring system can help educators break down broad topics into nuanced components.

By looking at the learning curves of each component, educators and researchers can better identify where students struggle or where they fail to transfer what they learned from one component to another. "Imagine you want to get better at something like playing tennis," Koedinger said. Traditionally, if tennis players have problems with serving the ball, they practice serving. But what if the problem with the serve was caused by something specific, such as the way the player tossed up the ball.

"So the idea is that we use data to find a particular problematic thing and now what we should do is find a way to isolate practice on that particular problem," he said.