Group Fairness in Multilingual Speech Recognition Models

Anna Katrine van Zee; Marc van Zee; Anders Søgaard

Group Fairness in Multilingual Speech Recognition Models

Anna Katrine van Zee, Marc van Zee, Anders Søgaard

Natural Language Processing

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

4 Downloads (Pure)

Abstract

We evaluate the performance disparity of the Whisper and MMS families of ASR models across the VoxPopuli and Common Voice multilingual datasets, with an eye toward intersectionality. Our two most important findings are that model size, surprisingly, correlates logarithmically with worst-case performance disparities, meaning that larger (and better) models are less fair. We also observe the importance of intersectionality. In particular, models often exhibit significant performance disparity across binary gender for adolescents.

Originalsprog	Engelsk
Titel	Findings of the Association for Computational Linguistics: NAACL 2024
Forlag	Association for Computational Linguistics
Publikationsdato	2024
Sider	2213–2226
Status	Udgivet - 2024
Begivenhed	2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024 - Hybrid, Mexico City, Mexico Varighed: 16 jun. 2024 → 21 jun. 2024

Konference

Konference	2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024
Land/Område	Mexico
By	Hybrid, Mexico City
Periode	16/06/2024 → 21/06/2024
Sponsor	Baidu, Capital One, et al., Grammarly, Megagon Labs, Otter.ai

Adgang til dokumentet

FulltextForlagets udgivne version, 490 KBLicens: CC BY

https://aclanthology.org/2024.findings-naacl.143/Licens: CC BY

Citationsformater

van Zee, AK, van Zee, M & Søgaard, A 2024, Group Fairness in Multilingual Speech Recognition Models. i Findings of the Association for Computational Linguistics: NAACL 2024. Association for Computational Linguistics, s. 2213–2226, 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024, Hybrid, Mexico City, Mexico, 16/06/2024. <https://aclanthology.org/2024.findings-naacl.143/>

@inproceedings{0f11785ca7f14cc7bc7339f20f4f695c,

title = "Group Fairness in Multilingual Speech Recognition Models",

abstract = "We evaluate the performance disparity of the Whisper and MMS families of ASR models across the VoxPopuli and Common Voice multilingual datasets, with an eye toward intersectionality. Our two most important findings are that model size, surprisingly, correlates logarithmically with worst-case performance disparities, meaning that larger (and better) models are less fair. We also observe the importance of intersectionality. In particular, models often exhibit significant performance disparity across binary gender for adolescents.",

author = "{van Zee}, {Anna Katrine} and {van Zee}, Marc and Anders S{\o}gaard",

year = "2024",

language = "English",

pages = "2213–2226",

booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024",

publisher = "Association for Computational Linguistics",

note = "2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024 ; Conference date: 16-06-2024 Through 21-06-2024",

}

TY - GEN

T1 - Group Fairness in Multilingual Speech Recognition Models

AU - van Zee, Anna Katrine

AU - van Zee, Marc

AU - Søgaard, Anders

PY - 2024

Y1 - 2024

N2 - We evaluate the performance disparity of the Whisper and MMS families of ASR models across the VoxPopuli and Common Voice multilingual datasets, with an eye toward intersectionality. Our two most important findings are that model size, surprisingly, correlates logarithmically with worst-case performance disparities, meaning that larger (and better) models are less fair. We also observe the importance of intersectionality. In particular, models often exhibit significant performance disparity across binary gender for adolescents.

AB - We evaluate the performance disparity of the Whisper and MMS families of ASR models across the VoxPopuli and Common Voice multilingual datasets, with an eye toward intersectionality. Our two most important findings are that model size, surprisingly, correlates logarithmically with worst-case performance disparities, meaning that larger (and better) models are less fair. We also observe the importance of intersectionality. In particular, models often exhibit significant performance disparity across binary gender for adolescents.

M3 - Article in proceedings

SP - 2213

EP - 2226

BT - Findings of the Association for Computational Linguistics: NAACL 2024

PB - Association for Computational Linguistics

T2 - 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2024

Y2 - 16 June 2024 through 21 June 2024

ER -