Cognitive tutor could teach employees new skills

Transcript - Cognitive tutor could teach employees new skills

Hey, I'm Nicole Laskowski, and this is Schooled in AI.

Ken Koedinger began a seminar he was giving at Harvard University like this:

I'd be interested to know: Do you know when you're learning?

Quick aside: The audio is courtesy of the Harvard Institute for Applied Computational Science and the Harvard John A. Paulson School of Engineering and Applied Sciences. OK, so …

Ken Koedinger: Do you know when you're learning?

It looks like a 50/50 split among attendees -- based on a show of hands, and Koedinger said that's a pretty typical response. But the question he poses does a couple of things: It sets the stage for the rest of his talk, and it raises a fundamental quandary about learning -- whether it's humans doing the learning or it's machines.

Koedinger: Much of what we feel like are indicators of learning are illusions.

Koedinger is a professor of human-computer interaction and psychology at Carnegie Mellon University and is known for developing what's called Cognitive Tutor software, which relies on machine learning to guide and personalize student instruction.

Koedinger: It's a subjective experience that feels like you do know yourself. You see what's in front of you, right?

After I saw his lunch seminar at Harvard, I invited him to be a guest on this podcast.

Koedinger: But you don't see what's in front of you. Your brain is actually making up a good share of what's in front of you.

Because comments like this one made me wonder: If we don't know how we learn, can we really build machines that learn like we do? That's one of the things Koedinger is trying to do: Build models that learn how to master a task or a subject like we do. And here's the thing, when the models are successful -- and it takes experimentation to get there -- they give researchers insight into how humans learn and where that learning breaks down.

Koedinger: It's challenging for sure. I think as far as questions on the scientific frontier, we probably know more about the universe than we know about our brains -- particularly when it comes to questions of, like, how the heck are humans able to learn complex things like algebra, calculus, chemistry, to be an engineer, to write a poem?

Building a simulated student

One estimate, and this was made famous by journalist Malcolm Gladwell, suggests that it takes 10 years to master a skill. A part of Koedinger's research could help uncover how a human can become an expert at something.

Koedinger: One way to make it easier, and this is sort of science in general, is to slow things down and look closely. So, we try to capture data and then look at it closely.

He uses AI models to generate that data. Koedinger calls the models 'synthetic students' or 'simulated students.' And before we get into what that is, let me explain why I think CIOs should know about his work: IT departments -- and companies in general -- are in the midst of a skills crisis that's not going away anytime soon. Rather than companies having to hire their way out of a talent shortage, software like this could provide the training employees need to step into a new role and to modernize the workforce. OK, that's my pitch. Now, let's get back to Koedinger.

Koedinger: A synthetic student is a simulation of human student capabilities.

Let's take algebra, for example. And I know, this is an education example and not an enterprise example, but I think you'll see the bigger picture. An algebra instructor might start by showing students how to solve simple problems and then ask the students to solve problems on their own. And they're given feedback along the way. The better a student does, the harder the problems become. Koedinger simulates this process in computers.

Koedinger: We give it problems. It says, 'I don't know what to do.' We show it what to do. It starts doing stuff. We say yes or no. And from that emerges in the system a competence in whatever domain we're tutoring the computer -- the simulated student or synthetic student.

Synthetic students can provide a window into big theoretical questions: How do humans learn? But they also can help shed light on a more practical question:

Koedinger: Where does learning break down? Why do some students get stuck along the way when they're trying to learn to program or even become a good counselor, let's say? And there are various techniques -- empirical techniques -- to try to figure out where those sticking points are.

But what we've discovered is that in addition to those techniques, if we try to make these learning algorithms, and they get stuck in certain points, we can look inside.

Koedinger doesn't need a brain imaging device to look inside a person's head and see what's going on. Instead, he can analyze the model and focus on where it had trouble.

Koedinger: And in many cases we can later confirm with real students that it's having trouble because that's just a hard part of learning in the domain. We didn't know that. We weren't teaching toward that. But now we can. And now the experience for real students is going to be better.

Koedinger refers to the sim students as cognitive crash test dummies that can show teachers whether the instruction is working or not and …

Koedinger: … where is it going smoothly and where is the analog of a crash right where the instruction just isn't working.

He builds synthetic students using a mix of machine learning techniques. These include more old-school machine learning techniques, which are based on more symbolic or logical kinds of structures, and more modern-day machine learning techniques such as deep neural networks, which are more statistically oriented.

Koedinger: Essentially, what we've found is that if we want to model human learning, we've got to bring these things together. We have to have a more logical learning mechanisms as well as more statistical learning mechanisms.

How, where, when, what learning

The sim students have four different learning sub-problems.

Koedinger: There's how learning, there is where learning, there is when learning and there is what learning. The what learning is closer to, like, deep learning neural net probabilistic grammar learning as one of the technical approaches we use for learning the representation of your domain.

And for each sub-problem, Koedinger and his team are using a different machine learning algorithm or multiple machine learning algorithms. While Koedinger and his team are not focused on algorithm development, here's what they are focused on.

Koedinger: We've been focusing more on developing an architecture that brings algorithms together. I think a lot of machine learning is solving what are often called classification tasks. You know, like, a label this picture. I've been using algebra as one example -- that's not a labeling thing. You really have to construct the next step. We've applied it in English grammar where there's some constructive capabilities. That's the how part of the learning mechanism chemistry domain. We've done this and so we really have to go beyond the sort of typical statistical classification approach to machine learning, but we're often doing that by combining multiple methods and innovating in that sense. But at the same time, we're building on the shoulders of other machine learning work out there.

One method is called transfer learning, where knowledge accumulated in one area, such as algebra, can be applied to another area that isn't algebra.

Koedinger: If you want to build an agent that's going to, say, sit on your phone and help you do various tasks -- like, you can teach it how to order a cappuccino from the Starbucks app -- you don't want to have to then teach it again. And better yet, if you go to another food app and want to teach it to order a pizza, you'd hope that it would transfer some of what you taught it about ordering a latte to ordering a pizza and this other application domain.

Four types of learning and complicated programming like transfer learning isn't for the typical developer. It was a truth that Koedinger and his team had to come to terms with.

Koedinger: In the early years, we spun off a company called Carnegie Learning. We were building those AI systems you know by programming them -- by writing the rules in the 'expert system.' But what we recognized is to scale that kind of technology, it would be great if more non-AI programmers could be engaged in building an intelligent tutor.

So, they developed Cognitive Tutor Authoring Tools. The tools are designed to enable developers who have limited AI programming experience to build an intelligent tutoring system -- a machine that can essentially teach to a specific task.

Koedinger: The human expert can teach the computer, this simulated student, how to do it. And now that computer is the expert that can teach others how to do it -- many others -- and do it in a highly interactive way.

Koedinger called this the 'grand vision' -- one that seems to be taking hold.

Koedinger: We have instances of that now that are beginning to work in real time in real-world applications -- at the K-12, higher education and in training settings.