Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningpeer review

6 Citationer (Scopus)
14 Downloads (Pure)

Abstract

Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today's NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for language-independent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directly-targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.

OriginalsprogEngelsk
TitelNAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, Proceedings of the Conference
ForlagAssociation for Computational Linguistics (ACL)
Publikationsdato2022
Sider4875-4893
ISBN (Elektronisk)9781955917711
DOI
StatusUdgivet - 2022
Begivenhed2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 - Seattle, USA
Varighed: 10 jul. 202215 jul. 2022

Konference

Konference2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022
Land/OmrådeUSA
BySeattle
Periode10/07/202215/07/2022
SponsorAmazon, Bloomberg, et al., Google Research, LIVE PERSON, Meta

Bibliografisk note

Funding Information:
We would like to thank Miryam de Lhoneux, Con-stanza Fierro, Desmond Elliott and the anonymous reviewers for their valuable feedback.

Publisher Copyright:
© 2022 Association for Computational Linguistics.

Citationsformater