Let’s Chat! Socialbot Called Sounding Board Wins for Being Most Engaging

University of Washington engineering students win Amazon’s inaugural Alexa Prize.

(Image courtesy of Sounding Board.)

(Image courtesy of Sounding Board.)

A team of University of Washington engineering students and faculty has won Amazon’s inaugural Alexa Prize, a university competition designed to produce an artificial intelligence agent capable of coherent and sustained conversation with humans.

The UW team developed Sounding Board, a conversational agent designed to provide engaging and informative conversation and to transform how people interact with everyday devices in their homes. The team from the UW Department of Electrical Engineering and the Paul G. Allen School of Computer Science & Engineering took home the $500,000 first prize, which will be shared among the students.

Their challenge was to produce a socialbot — an AI agent capable of coherent conversation — that could converse about popular topics and current events for a goal of 20 minutes. Teams built their socialbots using the Alexa Skills Kit and received continuous, real-world feedback from millions of Amazon customers who interacted with teams anonymously through Alexa.

Amazon selected the winning team from three worldwide finalists on Tuesday at the AWS re:Invent 2017 conference in Las Vegas.

“Our philosophy in developing Sounding Board was to bring a variety of relevant content into a natural conversation,” said team leader and electrical engineering doctoral student Hao Fang.  “Ultimately, we hope Sounding Board can become a conversational gateway to online information that users enjoy talking with.”

The Sounding Board socialbot earned an average score of 3.17 on a 5-point scale from a panel of independent judges and achieved an average conversation duration of 10:22.

The runner up team Alquist from Czech Technical University in Prague, which attained an average score of 2.72 and had an average conversation duration of 3:55, received a $100,000 prize. The third-place team from Heriot-Watt University in Edinburgh, Scotland, received a $50,000 prize for an average score of 2.36 and an average conversation duration of 4:01.

The UW Sounding Board team (left to right: Hao Fang, Hao Cheng, Ari Holtzman, Mari Ostendorf, Maarten Sap, Elizabeth Clark, Yejin Choi) wins Amazon's inaugural Alexa Prize. (Image courtesy of University of Washington.)

The UW Sounding Board team (left to right: Hao Fang, Hao Cheng, Ari Holtzman, Mari Ostendorf, Maarten Sap, Elizabeth Clark, Yejin Choi) wins Amazon’s inaugural Alexa Prize. (Image courtesy of University of Washington.)

The UW Sounding Board team combines expertise in natural language processing, speech technology and human-AI collaboration from additional team members EE doctoral student Hao Cheng and Allen School doctoral students Elizabeth Clark, Ari Holtzman, and Maarten Sap. EE professor Mari Ostendorf is the lead faculty advisor for the team, working in collaboration with professors Yejin Choi and Noah Smith of the Allen School’s Natural Language Processing research group.

“The students started from scratch, with no experience building a dialog system or working with Alexa skills, but together they brought a breadth of perspectives on language processing and a passion for understanding both the technical and human factor challenges of conversational AI,” Ostendorf said.

The Sounding Board design is both user- and content-driven. The system aims to understand user comments in multiple dimensions, from directives to sentiment and personality, in order to best serve user interests. At the same time, the system relies on having interesting and timely things to talk about. It actively harvests online content and leverages a knowledge graph to provide connections between related topics that can be used to steer the conversation.  

“Sounding Board is unique in its ability to understand what type of person the user is, and is able to adjust parts of the conversation based on who it thinks the user is,” said Maarten Sap.

As to how Sounding Board works, the team explained on their GitHub page, “Sounding Board is built on top of the Alexa Skills Kit powered by Echo devices, and most of the computation is done in the cloud. At each talk turn, the Echo device sends the recognized text to our cloud system. There, our algorithm decides how to take the conversation further through a two-tiered dialogue manager. Based on what the user said, we may query our databases for recent news or interesting facts about a certain topic. Once ready, our system generates a response combining natural dialogue markers with queried news or facts. The response is then sent back to the Echo device where it is turned into audio signal.”

“To try it out, simply tell your Amazon Echo: ‘Let’s chat!’”

The UW team relied on the collaborative environment at the university, both for getting feedback on technical ideas and for user testing. Faculty and students from across the UW-NLP community — in computer science, electrical engineering and linguistics — provided input on the many different versions of Sounding Board as it evolved.  In addition, a key resource in system development was access to real Alexa users nationwide. “It is impossible to anticipate all the types of reactions and questions people will have, even the different ways that a simple yes-or-no question can be answered. Learning from actual user data is critical,” Ostendorf said.

More than 100 teams from universities in 22 countries applied to be part of the inaugural competition. The finalists were selected from among 12 semi-finalists whose socialbots were evaluated based on customer ratings of their interactions during hundreds of thousands of conversations last summer.

The three finalists were announced in August, and continued to improve their socialbots by leveraging customer interactions through Nov. 7. Amazon selected the winner based on assessments of a panel of judges listening to conversations with three interactors.

Amazon will publish technical papers from all participating teams in the Alexa Prize Proceedings as a way of sharing their work with the broader research community.

“We envision that conversational AI will be integral at the interface between humans and machines, and the Alexa Prize makes an important step toward that vision,” said Choi. “It has been an exciting journey to build Sounding Board, and we look forward to working on crucial research challenges that we have identified along the way.”

For students who want to get in on the next Alexa Prize competition for 2018, applications will be open from December 4th, 2017 to Jan 8th, 2018.

Source: University of Washington Newsroom