Details
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI 2021 |
| Editors | Zhi-Hua Zhou |
| Pages | 552-559 |
| Number of pages | 8 |
| Publication status | Published - 2021 |
| Externally published | Yes |
| Event | 30th International Joint Conference on Artificial Intelligence, IJCAI 2021 - Online, Canada Duration: 19 Aug 2021 → 26 Aug 2021 |
Publication series
| Name | IJCAI International Joint Conference on Artificial Intelligence |
|---|---|
| ISSN (Print) | 1045-0823 |
Abstract
Word embedding models reflect bias towards genders, ethnicities, and other social groups present in the underlying training data. Metrics such as ECT, RNSB, and WEAT quantify bias in these models based on predefined word lists representing social groups and bias-conveying concepts. How suitable these lists actually are to reveal bias-let alone the bias metrics in general-remains unclear, though. In this paper, we study how to assess the quality of bias metrics for word embedding models. In particular, we present a generic method, Bias Silhouette Analysis (BSA), that quantifies the accuracy and robustness of such a metric and of the word lists used. Given a biased and an unbiased reference embedding model, BSA applies the metric systematically for several subsets of the lists to the models. The variance and rate of convergence of the bias values of each model then entail the robustness of the word lists, whereas the distance between the models' values gives indications of the general accuracy of the metric with the word lists. We demonstrate the behavior of BSA on two standard embedding models for the three mentioned metrics with several word lists from existing research.
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI 2021. ed. / Zhi-Hua Zhou. 2021. p. 552-559 (IJCAI International Joint Conference on Artificial Intelligence).
Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review
}
TY - GEN
T1 - Bias Silhouette Analysis
T2 - 30th International Joint Conference on Artificial Intelligence, IJCAI 2021
AU - Spliethöver, Maximilian
AU - Wachsmuth, Henning
PY - 2021
Y1 - 2021
N2 - Word embedding models reflect bias towards genders, ethnicities, and other social groups present in the underlying training data. Metrics such as ECT, RNSB, and WEAT quantify bias in these models based on predefined word lists representing social groups and bias-conveying concepts. How suitable these lists actually are to reveal bias-let alone the bias metrics in general-remains unclear, though. In this paper, we study how to assess the quality of bias metrics for word embedding models. In particular, we present a generic method, Bias Silhouette Analysis (BSA), that quantifies the accuracy and robustness of such a metric and of the word lists used. Given a biased and an unbiased reference embedding model, BSA applies the metric systematically for several subsets of the lists to the models. The variance and rate of convergence of the bias values of each model then entail the robustness of the word lists, whereas the distance between the models' values gives indications of the general accuracy of the metric with the word lists. We demonstrate the behavior of BSA on two standard embedding models for the three mentioned metrics with several word lists from existing research.
AB - Word embedding models reflect bias towards genders, ethnicities, and other social groups present in the underlying training data. Metrics such as ECT, RNSB, and WEAT quantify bias in these models based on predefined word lists representing social groups and bias-conveying concepts. How suitable these lists actually are to reveal bias-let alone the bias metrics in general-remains unclear, though. In this paper, we study how to assess the quality of bias metrics for word embedding models. In particular, we present a generic method, Bias Silhouette Analysis (BSA), that quantifies the accuracy and robustness of such a metric and of the word lists used. Given a biased and an unbiased reference embedding model, BSA applies the metric systematically for several subsets of the lists to the models. The variance and rate of convergence of the bias values of each model then entail the robustness of the word lists, whereas the distance between the models' values gives indications of the general accuracy of the metric with the word lists. We demonstrate the behavior of BSA on two standard embedding models for the three mentioned metrics with several word lists from existing research.
UR - http://www.scopus.com/inward/record.url?scp=85125489618&partnerID=8YFLogxK
U2 - 10.24963/ijcai.2021/77
DO - 10.24963/ijcai.2021/77
M3 - Conference contribution
AN - SCOPUS:85125489618
SN - 9780999241196
T3 - IJCAI International Joint Conference on Artificial Intelligence
SP - 552
EP - 559
BT - Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI 2021
A2 - Zhou, Zhi-Hua
Y2 - 19 August 2021 through 26 August 2021
ER -