Details
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 12th International Workshop on Semantic Evaluation |
| Editors | Marianna Apidianaki, Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, Marine Carpuat |
| Pages | 763-772 |
| Number of pages | 10 |
| Publication status | Published - Jun 2018 |
| Externally published | Yes |
| Event | 12th International Workshop on Semantic Evaluation, SemEval 2018, co-located with the 16th Annual Conference of the North American Chapter of the - New Orleans, United States Duration: 5 Jun 2018 → 6 Jun 2018 |
Abstract
A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal is to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of ≈ 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.
ASJC Scopus subject areas
- Computer Science(all)
- Computational Theory and Mathematics
- Arts and Humanities(all)
- Language and Linguistics
- Social Sciences(all)
- Linguistics and Language
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
Proceedings of the 12th International Workshop on Semantic Evaluation. ed. / Marianna Apidianaki; Marianna Apidianaki; Saif M. Mohammad; Jonathan May; Ekaterina Shutova; Steven Bethard; Marine Carpuat. 2018. p. 763-772.
Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review
}
TY - GEN
T1 - SemEval-2018 Task 12
T2 - 12th International Workshop on Semantic Evaluation, SemEval 2018, co-located with the 16th Annual Conference of the North American Chapter of the
AU - Habernal, Ivan
AU - Wachsmuth, Henning
AU - Gurevych, Iryna
AU - Stein, Benno
N1 - Funding Information: This work was supported by the German Research Foundation (DFG) within the ArguAna Project GU 798/20-1, and by the DFG-funded research training group “Adaptive Preparation of Information form Heterogeneous Sources” (AIPHES, GRK 1994/1).
PY - 2018/6
Y1 - 2018/6
N2 - A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal is to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of ≈ 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.
AB - A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal is to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of ≈ 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.
UR - http://www.scopus.com/inward/record.url?scp=85061624457&partnerID=8YFLogxK
U2 - 10.18653/v1/S18-1121
DO - 10.18653/v1/S18-1121
M3 - Conference contribution
AN - SCOPUS:85061624457
SN - 9781948087209
SP - 763
EP - 772
BT - Proceedings of the 12th International Workshop on Semantic Evaluation
A2 - Apidianaki, Marianna
A2 - Apidianaki, Marianna
A2 - Mohammad, Saif M.
A2 - May, Jonathan
A2 - Shutova, Ekaterina
A2 - Bethard, Steven
A2 - Carpuat, Marine
Y2 - 5 June 2018 through 6 June 2018
ER -