CMU Details 6 Years of Contributions to the National Science Foundation- Funded DialPort Project…

Summary

Carnegie Mellon University researchers have detailed six years of contributions to the NSF-funded DialPort project, which provides free tools for dialog research, including data collection and analysis, to be showcased at SIGDIAL 2022.

Abstract

The DialPort project, funded by the US National Science Foundation (NSF), has been instrumental in supporting the dialog research community by offering free tools for high-quality data collection and system comparison. Over the past six years, Carnegie Mellon University researchers have played a significant role in developing these tools, including the DialPort Portal for data collection and interactive assessments, the DialPort Dashboard for data analysis and organization, and DialCrowd for crowdsourcing detailed annotations. These tools have facilitated significant advancements in dialog system research, as evidenced by their use in the Ninth Dialog System Technology Challenge (DSTC9). The researchers will demonstrate these tools at the upcoming SIGDIAL 2022 conference, highlighting their utility in connecting systems, interacting with the tools, and the benefits they offer to the research community.

Opinions

The DialPort project is recognized as a valuable resource for the dialog research community, providing necessary tools for advancing the field.
The DialPort Portal is highlighted for its role in collecting natural dialog data and enabling real user interactions, particularly in the context of the DSTC9.
The DialPort Dashboard is considered an essential tool for researchers to analyze, filter, and compare dialog systems effectively.
DialCrowd is praised for simplifying the crowdsourcing process and improving the quality of human intelligence tasks (HIT) annotations.
The researchers are optimistic about the impact of their tools and are eager to showcase their capabilities at SIGDIAL 2022, anticipating that attendees will appreciate the practicality and advantages of the DialPort suite.

CMU Details 6 Years of Contributions to the National Science Foundation- Funded DialPort Project for Dialog Research

The DialPort project is funded by the US National Science Foundation (NSF) and aims to develop free tools to fulfill the needs of the dialog research community. DialPort helps researchers gather high-quality data and enables access to and comparisons of dialogue systems.

In the new paper The DialPort tools, researchers from Carnegie Mellon University provide background information and details on the DialPort project contributions over the last six years. These tools — such as the DialPort Portal and DialCrowd — will be demoed at the SIGDIAL 2022 conference next month in Edinburgh.

The DialPort Portal was developed to facilitate the collection of flexible and evolving natural dialog data and interactive assessments by real users. One of its best-known use cases was connecting systems and collecting data for the Interactive Evaluation of Dialog track at the Ninth Dialog System Technology Challenge (DSTC9) in 2020.

Once data is collected from the DialPort Portal interface, the DialPort Dashboard can be used to analyze, filter and organize dialogs and enable users to compare their system with others.

DialCrowd, meanwhile, is a simple and accessible tool that assists with crowdsourcing. DialCrowd helps requesters create well-structured human intelligence tasks (HIT), so crowdsourced workers can provide more detailed and better annotations on the data collected from DialPort.

The researchers will showcase demos and provide examples for the DialPort Portal, Dashboard, and DialCrowd at SIGDIAL, showing how to connect a system, what interactions with each tool looks like, and the advantages these tools provide. SIGDIAL 2022 will be held September 7–9 at the Heriot-Watt University in Edinburgh, UK.

The demos are available on DialPort.org. The paper The DialPort tools is on arXiv.

Author: Hecate He | Editor: Michael Sarazen

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.