Details
| Original language | English |
|---|---|
| Title of host publication | Natural Language Processing and Information Systems |
| Subtitle of host publication | 29th International Conference on Applications of Natural Language to Information Systems, NLDB 2024, Proceedings |
| Editors | Amon Rapp, Luigi Di Caro, Farid Meziane, Vijayan Sugumaran |
| Publisher | Springer Science and Business Media Deutschland GmbH |
| Pages | 150-160 |
| Number of pages | 11 |
| ISBN (electronic) | 978-3-031-70242-6 |
| ISBN (print) | 9783031702419 |
| Publication status | Published - 20 Sept 2024 |
| Event | 29th International Conference on Natural Language and Information Systems, NLDB 2024 - Turin, Italy Duration: 25 Jun 2024 → 27 Jun 2024 |
Publication series
| Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| Volume | 14763 LNCS |
| ISSN (Print) | 0302-9743 |
| ISSN (electronic) | 1611-3349 |
Abstract
This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.
ASJC Scopus subject areas
- Mathematics(all)
- Theoretical Computer Science
- Computer Science(all)
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
Natural Language Processing and Information Systems : 29th International Conference on Applications of Natural Language to Information Systems, NLDB 2024, Proceedings. ed. / Amon Rapp; Luigi Di Caro; Farid Meziane; Vijayan Sugumaran. Springer Science and Business Media Deutschland GmbH, 2024. p. 150-160 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14763 LNCS).
Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review
}
TY - GEN
T1 - Effective Context Selection in LLM-Based Leaderboard Generation
T2 - 29th International Conference on Natural Language and Information Systems, NLDB 2024
AU - Kabongo, Salomon
AU - D’Souza, Jennifer
AU - Auer, Sören
N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
PY - 2024/9/20
Y1 - 2024/9/20
N2 - This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.
AB - This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.
UR - http://www.scopus.com/inward/record.url?scp=85205459491&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-70242-6_15
DO - 10.1007/978-3-031-70242-6_15
M3 - Conference contribution
AN - SCOPUS:85205459491
SN - 9783031702419
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 150
EP - 160
BT - Natural Language Processing and Information Systems
A2 - Rapp, Amon
A2 - Di Caro, Luigi
A2 - Meziane, Farid
A2 - Sugumaran, Vijayan
PB - Springer Science and Business Media Deutschland GmbH
Y2 - 25 June 2024 through 27 June 2024
ER -