Details
| Original language | English |
|---|---|
| Title of host publication | Natural Language Processing and Information Systems |
| Subtitle of host publication | 29th International Conference on Applications of Natural Language to Information Systems, NLDB 2024, Proceedings |
| Editors | Amon Rapp, Luigi Di Caro, Farid Meziane, Vijayan Sugumaran |
| Publisher | Springer Science and Business Media Deutschland GmbH |
| Pages | 183-194 |
| Number of pages | 12 |
| ISBN (electronic) | 978-3-031-70242-6 |
| ISBN (print) | 9783031702419 |
| Publication status | Published - 20 Sept 2024 |
| Event | 29th International Conference on Natural Language and Information Systems, NLDB 2024 - Turin, Italy Duration: 25 Jun 2024 → 27 Jun 2024 |
Publication series
| Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| Volume | 14763 LNCS |
| ISSN (Print) | 0302-9743 |
| ISSN (electronic) | 1611-3349 |
Abstract
Questions are an integral part of test formats in education. Also online learning platforms like Coursera or Udemy use questions to check learners’ understanding. However, the manual creation of questions can be very time-intensive. This problem can be mitigated through automatic question generation. In this paper, we present a comparison of fine-tuned text-generating transformers for question generation. Our methods include (i) a comparison of multiple fine-tuned transformers to identify differences in the generated output, (ii) a comparison of multiple token search strategies evaluated on each model to find differences in generated questions across different strategies and (iii) a newly developed manual evaluation metric that evaluates generated questions regarding aspects of naturalness and suitability. Our experiments show a difference in question length, structure and quality depending on the used transformer architecture, which indicates a correlation between transformer architecture and question structure. Furthermore, different search strategies for the same model architecture do not greatly impact structure or quality.
Keywords
- Automatic Question Generation, Pre-trained Transformer, Transformer Architecture
ASJC Scopus subject areas
- Mathematics(all)
- Theoretical Computer Science
- Computer Science(all)
- General Computer Science
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
Natural Language Processing and Information Systems : 29th International Conference on Applications of Natural Language to Information Systems, NLDB 2024, Proceedings. ed. / Amon Rapp; Luigi Di Caro; Farid Meziane; Vijayan Sugumaran. Springer Science and Business Media Deutschland GmbH, 2024. p. 183-194 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14763 LNCS).
Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review
}
TY - GEN
T1 - Question Generation Capabilities of “Small" Large Language Models
AU - Berger, Joshua
AU - Koß, Jonathan
AU - Stamatakis, Markos
AU - Hoppe, Anett
AU - Ewerth, Ralph
AU - Wartena, Christian
N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
PY - 2024/9/20
Y1 - 2024/9/20
N2 - Questions are an integral part of test formats in education. Also online learning platforms like Coursera or Udemy use questions to check learners’ understanding. However, the manual creation of questions can be very time-intensive. This problem can be mitigated through automatic question generation. In this paper, we present a comparison of fine-tuned text-generating transformers for question generation. Our methods include (i) a comparison of multiple fine-tuned transformers to identify differences in the generated output, (ii) a comparison of multiple token search strategies evaluated on each model to find differences in generated questions across different strategies and (iii) a newly developed manual evaluation metric that evaluates generated questions regarding aspects of naturalness and suitability. Our experiments show a difference in question length, structure and quality depending on the used transformer architecture, which indicates a correlation between transformer architecture and question structure. Furthermore, different search strategies for the same model architecture do not greatly impact structure or quality.
AB - Questions are an integral part of test formats in education. Also online learning platforms like Coursera or Udemy use questions to check learners’ understanding. However, the manual creation of questions can be very time-intensive. This problem can be mitigated through automatic question generation. In this paper, we present a comparison of fine-tuned text-generating transformers for question generation. Our methods include (i) a comparison of multiple fine-tuned transformers to identify differences in the generated output, (ii) a comparison of multiple token search strategies evaluated on each model to find differences in generated questions across different strategies and (iii) a newly developed manual evaluation metric that evaluates generated questions regarding aspects of naturalness and suitability. Our experiments show a difference in question length, structure and quality depending on the used transformer architecture, which indicates a correlation between transformer architecture and question structure. Furthermore, different search strategies for the same model architecture do not greatly impact structure or quality.
KW - Automatic Question Generation
KW - Pre-trained Transformer
KW - Transformer Architecture
UR - http://www.scopus.com/inward/record.url?scp=85205514065&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-70242-6_18
DO - 10.1007/978-3-031-70242-6_18
M3 - Conference contribution
AN - SCOPUS:85205514065
SN - 9783031702419
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 183
EP - 194
BT - Natural Language Processing and Information Systems
A2 - Rapp, Amon
A2 - Di Caro, Luigi
A2 - Meziane, Farid
A2 - Sugumaran, Vijayan
PB - Springer Science and Business Media Deutschland GmbH
T2 - 29th International Conference on Natural Language and Information Systems, NLDB 2024
Y2 - 25 June 2024 through 27 June 2024
ER -