Minimax and Neyman–Pearson Meta-Learning for Outlier Languages

Edoardo Maria Ponti, Rahul Aralikatte, Disha Shrivastava, Siva Reddy, Anders Søgaard

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningpeer review

14 Citationer (Scopus)
9 Downloads (Pure)

Abstract

Model-agnostic meta-learning (MAML) hasbeen recently put forth as a strategy to learnresource-poor languages in a sample-efficientfashion. Nevertheless, the properties of theselanguages are often not well represented bythose available during training. Hence, weargue that the i.i.d. assumption ingrained inMAML makes it ill-suited for cross-lingualNLP. In fact, under a decision-theoretic framework, MAML can be interpreted as minimising the expected risk across training languages(with a uniform prior), which is known asBayes criterion. To increase its robustness tooutlier languages, we create two variants ofMAML based on alternative criteria: MinimaxMAML reduces the maximum risk across languages, while Neyman–Pearson MAML constrains the risk in each language to a maximum threshold. Both criteria constitute fullydifferentiable two-player games. In light ofthis, we propose a new adaptive optimiser solving for a local approximation to their Nashequilibrium. We evaluate both model variants on two popular NLP tasks, part-of-speechtagging and question answering. We reportgains for their average and minimum performance across low-resource languages in zeroand few-shot settings, compared to joint multisource transfer and vanilla MAML. The codefor our experiments is available at https://github.com/rahular/robust-maml.
OriginalsprogEngelsk
TitelFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021
ForlagAssociation for Computational Linguistics
Publikationsdato2021
Sider1245-1260
DOI
StatusUdgivet - 2021
BegivenhedFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021 - Virtual, Online
Varighed: 1 aug. 20216 aug. 2021

Konference

KonferenceFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021
ByVirtual, Online
Periode01/08/202106/08/2021

Citationsformater