DataScience.US
A Data Professionals Community

ParlAI: A new software platform for dialog research

27

One of the long-term goals in AI is to develop intelligent chat bots that can converse with people in a natural way. Existing chat bots can sometimes complete specific independent tasks but have trouble understanding more than a single sentence or chaining subtasks together to complete a bigger task. More complex dialog, such as booking a restaurant or chatting about sports or news, requires the ability to understand multiple sentences and then reason about those sentences to supply the next part of the conversation. Since human dialog is so varied, chat bots must be skilled at many related tasks that all require different expertise but use the same input and output format. To achieve these goals, it is necessary to build software that unifies these tasks, as well as the agents that can learn from them.

Recognizing this need, the Facebook AI Research (FAIR) team has built a new, open source platform for training and testing dialog models across multiple tasks at once. ParlAI (pronounced “par-lay”) is a one-stop shop for dialog research, where researchers can submit new tasks and training algorithms to a single, shared repository. Integration with Mechanical Turk for data collection, training, and evaluation also allows bots in ParlAI to talk to humans. This works toward the goal of unifying existing dialog data sets with learning systems that contain real dialog between people and bots.

ParlAI complements existing FAIR text research efforts like FastText, our quick and efficient text classification tool, and CommAI, our framework for developing artificial general intelligence through increasingly complex tasks.

Tasks in ParlAI

More than 20 public datasets are included in this release, shown in the left panel below.

The tasks are separated into five categories:

  • Question answering: This is one of the simplest forms of dialog, with only one turn per speaker. Question answering is particularly useful because evaluation is simpler than in other forms of dialog: If the answer to the question is known (i.e., the data set is labeled), then we can quickly check whether the answer is correct.
  • Sentence completion (cloze test): In this test, the agent has to fill in a missing word in the next utterance in a dialog. While this is another specialized dialog task, the datasets are cheap to make and the evaluation is simple.
  • Goal-oriented dialog: A much more realistic class of dialog involves having to achieve a goal; for example, a customer and a travel agent discussing a flight, a speaker recommending a movie to another speaker, two speakers agreeing when and where to eat together, and so on.
  • Chit-chat dialog: Some tasks don’t necessarily have an explicit goal but are more of a discussion; for example, two speakers discussing sports, movies, or a mutual interest.
  • Visual dialog: These are tasks that include images as well as text. In the real world, dialog often is grounded in physical objects. In the future, we plan to add other sensory information such as audio.

Choosing a task in ParlAI is as easy as specifying it on the command line. If the dataset has not yet been used, ParlAI will download it automatically. As all datasets are treated the same way in ParlAI (with a single dialog API), a dialog agent can switch between training and testing among any of them. You can also specify many tasks at once (multitasking) by providing a comma-separated list; e.g., the command line -t babi,squad to use those two data sets; all the QA data sets at once -t #qa; or every task in ParlAI at once -t #all. The aim is to make it easy to build and evaluate rich dialog models.

Worlds, agents, and teachers

The main concepts (classes) in ParlAI are worlds, agents, and teachers. A world is the environment the speakers live in, and can vary from a simple two-way conversation to more complex scenarios such as a reactive game environment. Agents are things that can act and speak in the world. An agent can be a learner, such as a machine learned system, or a hard-coded bot designed to interact with learners, such as non-player characters in a game. Finally, teachers are a type of agent that talk to the learner, for example, one that implements a task listed previously.

Continue Reading…

Comments

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More

X