Researchers Are Teaching AI to be Curious Using Super Mario Bros.
Artificial intelligence is really good at doing what we tell it to do. It can solve problems at lightning speed and find patterns in huge swathes of data, all at our command. But what if we didn't have to command it? What if AI was motivated to solve problems all on its own, just to sate its own curiosity? That's what many researchers are trying to make happen, and some are using something plenty familiar: 8-bit video games.
It Was In You All Along
Think back to the first time you ever played a video game. You probably were timid and clumsy at first as you figured out which buttons did what and determined how exactly to proceed in the virtual world before you, but eventually you honed your skills and managed to beat a level or two. Importantly, you worked at improving those skills not for money or a grade, but because it felt good. You had what's called intrinsic motivation — a drive that comes from within. That's what drives curiosity.
Money and grades — and debt and academic probation — are examples of extrinsic motivations, or drives that come from your environment. That's what computer scientists use to train an artificial intelligence algorithm to learn: when it does the correct task, it gets "points"; when it does something wrong, there's a penalty. This approach, called reinforcement learning, has led AI to do some remarkable things, from beating humans at the world's most complex board game to merging an autonomous vehicle on the freeway.
But it has its limits. To reward the correct tasks, you have to know which tasks are correct. That's easy when you're training an AI on a well-defined goal like winning a board game, but not so easy when it comes to things we'll soon need them to do, like rescuing people after a disaster. Computer scientists need a way to help an AI figure out which tasks are "correct" on its own — something akin to curiosity.
Ready Player One
To teach AI to be curious, researchers at the Berkeley Artificial Intelligence Research Lab took inspiration from the masters of curiosity: babies. Research shows that when babies and toddlers make sense of the world, they're drawn most to the things that surprise them. That's what makes them reach out and touch a new object to see how it feels or throw a toy to the ground to see what happens.
The Berkeley researchers programmed this surprise-driven exploration into their AI by making it predict an outcome, then generate an intrinsic reward depending on how wrong that prediction turns out to be. The bigger the error, the bigger the surprise, and the bigger the reward. Because it gets the biggest rewards from the predictions it knows the least about, it's drawn toward the things it doesn't know. There you have it: the AI version of curiosity.
When they had this curious AI play Super Mario Bros., it learned in much the same way you or I did when we first played. It tried out all of the buttons, and quickly learned that while "down" doesn't do anything, pressing "right" takes Mario to unpredictable places. And those unpredictable places yielded big rewards.
This curiosity algorithm isn't perfect, as demonstrated by the fact that the AI can't yet beat the first level. There's still fine-tuning to do. But one day, this kind of intrinsic motivation could drive AI to rescue disaster victims, explore extraterrestrial worlds, and answer the open-ended questions we don't even know to ask yet.