SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings

Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review

Authors

  • Jan Engler
  • Sandipan Sikdar
  • Marlene Lutz
  • Markus Strohmaier

Research Organisations

External Research Organisations

  • RWTH Aachen University
  • University of Mannheim
  • Complexity Science Hub Vienna (CSH)
View graph of relations

Details

Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics: EMNLP 2022
Pages4636-4648
Number of pages13
Publication statusPublished - Dec 2022
Event2022 Findings of the Association for Computational Linguistics: EMNLP 2022 - Abu Dhabi, United Arab Emirates
Duration: 7 Dec 2022 → 11 Dec 2022

Abstract

Adding interpretability to word embeddings represents an area of active research in text representation. Recent work has explored the potential of embedding words via so-called polar dimensions (e.g. good vs. bad, correct vs. wrong). Examples of such recent approaches include SemAxis, POLAR, FrameAxis, and BiImp. Although these approaches provide interpretable dimensions for words, they have not been designed to deal with polysemy, i.e. they can not easily distinguish between different senses of words. To address this limitation, we present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings. The resulting interpretable word embeddings achieve a level of performance that is comparable to original contextual word embeddings across a variety of natural language processing tasks including the GLUE and SQuAD benchmarks. Our work removes a fundamental limitation of existing approaches by offering users sense aware interpretations for contextual word embeddings.

ASJC Scopus subject areas

Cite this

SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings. / Engler, Jan; Sikdar, Sandipan; Lutz, Marlene et al.
Findings of the Association for Computational Linguistics: EMNLP 2022. 2022. p. 4636-4648.

Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review

Engler, J, Sikdar, S, Lutz, M & Strohmaier, M 2022, SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings. in Findings of the Association for Computational Linguistics: EMNLP 2022. pp. 4636-4648, 2022 Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, 7 Dec 2022. https://doi.org/10.48550/arXiv.2301.04704
Engler, J., Sikdar, S., Lutz, M., & Strohmaier, M. (2022). SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 4636-4648) https://doi.org/10.48550/arXiv.2301.04704
Engler J, Sikdar S, Lutz M, Strohmaier M. SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2022. 2022. p. 4636-4648 doi: 10.48550/arXiv.2301.04704
Engler, Jan ; Sikdar, Sandipan ; Lutz, Marlene et al. / SensePOLAR : Word sense aware interpretability for pre-trained contextual word embeddings. Findings of the Association for Computational Linguistics: EMNLP 2022. 2022. pp. 4636-4648
Download
@inproceedings{89e1f7bb8bac42dfa174c805f467e1f5,
title = "SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings",
abstract = "Adding interpretability to word embeddings represents an area of active research in text representation. Recent work has explored the potential of embedding words via so-called polar dimensions (e.g. good vs. bad, correct vs. wrong). Examples of such recent approaches include SemAxis, POLAR, FrameAxis, and BiImp. Although these approaches provide interpretable dimensions for words, they have not been designed to deal with polysemy, i.e. they can not easily distinguish between different senses of words. To address this limitation, we present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings. The resulting interpretable word embeddings achieve a level of performance that is comparable to original contextual word embeddings across a variety of natural language processing tasks including the GLUE and SQuAD benchmarks. Our work removes a fundamental limitation of existing approaches by offering users sense aware interpretations for contextual word embeddings.",
author = "Jan Engler and Sandipan Sikdar and Marlene Lutz and Markus Strohmaier",
note = "Funding Information: Sandipan Sikdar was supported in part by RWTH Aachen Startup Grant No. StUpPD384-20.; 2022 Findings of the Association for Computational Linguistics: EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",
year = "2022",
month = dec,
doi = "10.48550/arXiv.2301.04704",
language = "English",
pages = "4636--4648",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",

}

Download

TY - GEN

T1 - SensePOLAR

T2 - 2022 Findings of the Association for Computational Linguistics: EMNLP 2022

AU - Engler, Jan

AU - Sikdar, Sandipan

AU - Lutz, Marlene

AU - Strohmaier, Markus

N1 - Funding Information: Sandipan Sikdar was supported in part by RWTH Aachen Startup Grant No. StUpPD384-20.

PY - 2022/12

Y1 - 2022/12

N2 - Adding interpretability to word embeddings represents an area of active research in text representation. Recent work has explored the potential of embedding words via so-called polar dimensions (e.g. good vs. bad, correct vs. wrong). Examples of such recent approaches include SemAxis, POLAR, FrameAxis, and BiImp. Although these approaches provide interpretable dimensions for words, they have not been designed to deal with polysemy, i.e. they can not easily distinguish between different senses of words. To address this limitation, we present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings. The resulting interpretable word embeddings achieve a level of performance that is comparable to original contextual word embeddings across a variety of natural language processing tasks including the GLUE and SQuAD benchmarks. Our work removes a fundamental limitation of existing approaches by offering users sense aware interpretations for contextual word embeddings.

AB - Adding interpretability to word embeddings represents an area of active research in text representation. Recent work has explored the potential of embedding words via so-called polar dimensions (e.g. good vs. bad, correct vs. wrong). Examples of such recent approaches include SemAxis, POLAR, FrameAxis, and BiImp. Although these approaches provide interpretable dimensions for words, they have not been designed to deal with polysemy, i.e. they can not easily distinguish between different senses of words. To address this limitation, we present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings. The resulting interpretable word embeddings achieve a level of performance that is comparable to original contextual word embeddings across a variety of natural language processing tasks including the GLUE and SQuAD benchmarks. Our work removes a fundamental limitation of existing approaches by offering users sense aware interpretations for contextual word embeddings.

UR - http://www.scopus.com/inward/record.url?scp=85149902132&partnerID=8YFLogxK

U2 - 10.48550/arXiv.2301.04704

DO - 10.48550/arXiv.2301.04704

M3 - Conference contribution

AN - SCOPUS:85149902132

SP - 4636

EP - 4648

BT - Findings of the Association for Computational Linguistics: EMNLP 2022

Y2 - 7 December 2022 through 11 December 2022

ER -