TY - GEN

T1 - Justevents

T2 - 39th European Conference on Information Retrieval, ECIR 2017

AU - Ceroni, Andrea

AU - Gadiraju, Ujwal

AU - Fisichella, Marco

N1 - Publisher Copyright: © Springer International Publishing AG 2017.

PY - 2017

Y1 - 2017

N2 - Inspecting text to affirm the occurrence of an event is a nontrivial task. Since events are tied to temporal attributes, this task is more complex than merely identifying evidence of entities acting together and thus defining the event in a document. Manual inspection is a typical solution, although it is an onerous task and becomes infeasible with an increasing scale of documents. Therefore, the task of automatically determining whether an event occurs in a document or corpus, named as event validation, has been recently investigated. In this paper, we present a dataset for benchmarking event validation methods. Events and documents are coupled in pairs, whose validity has been judged by human evaluators based on whether the document in the pair contains evidence of the given event. In contrast to the notion of relevance considered in available datasets for event detection, validity judgments in this work strictly consider whether a document reports an event within its timespan as well as the number of event participants reported in the document. These requirements make the generation of manual validity judgments an onerous procedure. The ground truth, made of multiple judgments for each pair, has been acquired through crowdsourcing.

AB - Inspecting text to affirm the occurrence of an event is a nontrivial task. Since events are tied to temporal attributes, this task is more complex than merely identifying evidence of entities acting together and thus defining the event in a document. Manual inspection is a typical solution, although it is an onerous task and becomes infeasible with an increasing scale of documents. Therefore, the task of automatically determining whether an event occurs in a document or corpus, named as event validation, has been recently investigated. In this paper, we present a dataset for benchmarking event validation methods. Events and documents are coupled in pairs, whose validity has been judged by human evaluators based on whether the document in the pair contains evidence of the given event. In contrast to the notion of relevance considered in available datasets for event detection, validity judgments in this work strictly consider whether a document reports an event within its timespan as well as the number of event participants reported in the document. These requirements make the generation of manual validity judgments an onerous procedure. The ground truth, made of multiple judgments for each pair, has been acquired through crowdsourcing.

KW - Crowdsourcing

KW - Evaluation

KW - Event detection

KW - Event validation

KW - Human computation

UR - http://www.scopus.com/inward/record.url?scp=85018671536&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-56608-5_38

DO - 10.1007/978-3-319-56608-5_38

M3 - Conference contribution

AN - SCOPUS:85018671536

SN - 9783319566078

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 484

EP - 492

BT - Advances in Information Retrieval - 39th European Conference on IR Research, ECIR 2017, Proceedings

A2 - Hauff, Claudia

A2 - Jose, Joemon M.

A2 - Albakour, Dyaa

A2 - Altingovde, Ismail Sengor

A2 - Tait, John

A2 - Song, Dawei

A2 - Watt, Stuart

PB - Springer Verlag

Y2 - 8 April 2017 through 13 April 2017

ER -