Ying Xu designs interactive dialogues for AI-assisted cartoon

Lyla, a curious and dynamic 7-year-old girl, is always ready to solve everyday issues in her community. She is never alone in these challenges, surrounded by helpful family, friends and neighbors—and viewers.

Lyla will be able to interact with young viewers because of programming devised by Ying Xu, assistant professor of learning sciences and technology at the University of Michigan’s Marsal Family School of Education.

Focused on improving learning outcomes for children, the new PBS KIDS animated series, “Lyla in the Loop,” will feature interactive digital episodes that incorporate AI-assisted conversations between the audience and the show’s titular character, Lyla Loops, who lives in a city inspired by Philadelphia with her family and “fantastical” sidekick, Stu.

Xu and colleagues have partnered with PBS KIDS and worked on the interactive components for this new show using real-time conversational artificial intelligence to support STEM learning among young children. They will provide a group of 200 children access to the AI episodes during fall 2024. After a new period of data collection, the researchers will release the interactive episodes more broadly to the public.

In these interactive videos, which will be available after a PBS KIDS testing period on mobile devices—like smartphones or tablets—the show’s characters engage in dialogues with children as curious peers. As the episode unfolds, Lyla asks questions to encourage children to think, organize, interpret and use data to make decisions.

Using automatic speech recognition and natural language processing, the character listens to children’s responses, interprets their level of understanding and provides responsive feedback to the children. Researchers crafted the dialogues to intentionally align them with the episodes’ learning goals.

The interaction will be similar to how kids already engage with AI-powered tools in their homes, like voice assistants on Google Home devices or Amazon Alexa and internet-enabled toys. Smart TVs may be able to handle the technology in the future, too.

Xu has been studying the impact of interacting with an AI-powered television character on kid’s learning since 2018 based on another PBS KIDS show, "Elinor Wonders Why." One of her studies, published in the ACM Digital Library, found that offering children opportunities to converse with an AI-powered interactive media character in a science video leads to better learning outcomes than for children watching videos with no interactions.

“The central theme of my research is to use AI to support children’s learning through engaging them in conversations,” Xu said. “Before I initiated a partnership with PBS KIDS, I was already studying children’s use of voice assistants, like Siri or Alexa, and contemplating how these technologies could be better designed to offer rich learning opportunities.”

According to Xu, most people were not aware of the potential of interactive dialogues a few years back. To maximize its impact, Xu and her collaborators at the University of California, Irvine decided to find a partner who had already produced high-quality media and cared about how new technologies like AI could enhance children’s learning.

“We pitched our idea to PBS KIDS and were excited to see that we had the same vision and would be good research partners,” said Xu, who has been working on the interactive videos for “Lyla in the Loop” for about a year-and-a-half.

The NSF Advancing Informal STEM Learning program funds the project.

“The integration of this technology into our content is a natural progression, and helps further our mission to develop safe, meaningful, and high-quality media experiences for children and families,” said Silvia Lovato, PBS KIDS senior director and head of learning and research.

The team has studied and developed conversational videos incorporating novel forms of support for children, including extended back-and-forth conversation that builds upon a child’s responses, visual scaffolding that facilitates verbal communication, and bilingual language processing.

“We worked closely on looking at how we could adapt the show to incorporate AI to facilitate children’s interactions,” Xu said. “There are a lot of design considerations to create the best questions to facilitate children’s engagement, especially if they are a little bit more hesitant to share their thoughts. And if we identify the children who have misconceptions on a question, how do we scaffold those kids?"

Xu and her team conducted several phases of testing with a small group of kids, ages 4-8, showing them the interactive videos and observing the kids’ interaction with AI. They then asked them questions like, “How do you like this process? Do you feel that it was confusing and have nothing to say? Do you have any ideas to help us make it even better?”

“This was when we entered the second phase: the randomized control trial, working with a larger number of kids, sometimes from 200 to 300,” Xu said. “We randomly divided those kids into two to three different groups and had them watch the same content in slightly different formats.”

Some children watched the original episodes aired on PBS KIDS with no AI questioning, while others watched the interactive video and had conversations with AI. The researchers then compared the groups to see how much they learned. They found that higher engagement via AI meant greater learning.

According to PBS KIDS, one of Lyla’s stronger characteristics is that she doesn’t give up, even when it takes several tries to get it right.

“We already know that children form a relationship with their favorite TV characters,” Xu said. “When the character could listen to and talk back to them, it meant much more. If these interactions lead children to form stronger bonds with their favorite characters, would they view those characters as role models for science learning?

“And Lyla is an African-American girl, so we’re now curious to see if kids’ interaction with her would increase girls from minoritized backgrounds’ desire to become scientists. This will be the next step of our research.”

This article originally appeared in Michigan News.