Details
| Originalsprache | Englisch |
|---|---|
| Titel des Sammelwerks | Proceedings of the 12th International Workshop on Semantic Evaluation |
| Herausgeber/-innen | Marianna Apidianaki, Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, Marine Carpuat |
| Seiten | 763-772 |
| Seitenumfang | 10 |
| Publikationsstatus | Veröffentlicht - Juni 2018 |
| Extern publiziert | Ja |
| Veranstaltung | 12th International Workshop on Semantic Evaluation, SemEval 2018, co-located with the 16th Annual Conference of the North American Chapter of the - New Orleans, USA / Vereinigte Staaten Dauer: 5 Juni 2018 → 6 Juni 2018 |
Abstract
A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal is to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of ≈ 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.
ASJC Scopus Sachgebiete
- Informatik (insg.)
- Theoretische Informatik und Mathematik
- Geisteswissenschaftliche Fächer (insg.)
- Sprache und Linguistik
- Sozialwissenschaften (insg.)
- Linguistik und Sprache
Zitieren
- Standard
- Harvard
- Apa
- Vancouver
- BibTex
- RIS
Proceedings of the 12th International Workshop on Semantic Evaluation. Hrsg. / Marianna Apidianaki; Marianna Apidianaki; Saif M. Mohammad; Jonathan May; Ekaterina Shutova; Steven Bethard; Marine Carpuat. 2018. S. 763-772.
Publikation: Beitrag in Buch/Bericht/Sammelwerk/Konferenzband › Aufsatz in Konferenzband › Forschung › Peer-Review
}
TY - GEN
T1 - SemEval-2018 Task 12
T2 - 12th International Workshop on Semantic Evaluation, SemEval 2018, co-located with the 16th Annual Conference of the North American Chapter of the
AU - Habernal, Ivan
AU - Wachsmuth, Henning
AU - Gurevych, Iryna
AU - Stein, Benno
N1 - Funding Information: This work was supported by the German Research Foundation (DFG) within the ArguAna Project GU 798/20-1, and by the DFG-funded research training group “Adaptive Preparation of Information form Heterogeneous Sources” (AIPHES, GRK 1994/1).
PY - 2018/6
Y1 - 2018/6
N2 - A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal is to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of ≈ 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.
AB - A natural language argument is composed of a claim as well as reasons given as premises for the claim. The warrant explaining the reasoning is usually left implicit, as it is clear from the context and common sense. This makes a comprehension of arguments easy for humans but hard for machines. This paper summarizes the first shared task on argument reasoning comprehension. Given a premise and a claim along with some topic information, the goal is to automatically identify the correct warrant among two candidates that are plausible and lexically close, but in fact imply opposite claims. We describe the dataset with 1970 instances that we built for the task, and we outline the 21 computational approaches that participated, most of which used neural networks. The results reveal the complexity of the task, with many approaches hardly improving over the random accuracy of ≈ 0.5. Still, the best observed accuracy (0.712) underlines the principle feasibility of identifying warrants. Our analysis indicates that an inclusion of external knowledge is key to reasoning comprehension.
UR - http://www.scopus.com/inward/record.url?scp=85061624457&partnerID=8YFLogxK
U2 - 10.18653/v1/S18-1121
DO - 10.18653/v1/S18-1121
M3 - Conference contribution
AN - SCOPUS:85061624457
SN - 9781948087209
SP - 763
EP - 772
BT - Proceedings of the 12th International Workshop on Semantic Evaluation
A2 - Apidianaki, Marianna
A2 - Apidianaki, Marianna
A2 - Mohammad, Saif M.
A2 - May, Jonathan
A2 - Shutova, Ekaterina
A2 - Bethard, Steven
A2 - Carpuat, Marine
Y2 - 5 June 2018 through 6 June 2018
ER -