Details

Autor(en) / Beteiligte

• Mielke, Sabrina J.
• Szlam, Arthur
• Dinan, Emily
• Boureau, Y-Lan

Titel

Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration

Ist Teil von

• Transactions of the Association for Computational Linguistics, 2022-08, Vol.10, p.857-872

Ort / Verlag

One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA: MIT Press

Erscheinungsjahr

2022

Link zum Volltext

Quelle

Elektronische Zeitschriftenbibliothek - Frei zugängliche E-Journals

Beschreibungen/Notizen

• While improving neural dialogue agents’ factual accuracy is the object of much research, another important aspect of communication, less studied in the setting of neural dialogue, is transparency about ignorance. In this work, we analyze to what extent state-of-the-art chit-chat models are in the sense that their verbalized expression of doubt (or confidence) matches the likelihood that the model’s responses are factually incorrect (or correct). We find that these models are poorly calibrated, yet we show that likelihood of correctness can accurately be predicted. By incorporating such metacognitive features into the training of a controllable generation model, we obtain a dialogue agent with greatly improved linguistic calibration.