TY - JOUR

T1 - No one is perfect

T2 - Analysing the performance of question answering components over the DBpedia knowledge graph

AU - Singh, Kuldeep

AU - Lytra, Ioanna

AU - Radhakrishna, Arun Sethupat

AU - Shekarpour, Saeedeh

AU - Vidal, Maria Esther

AU - Lehmann, Jens

N1 - Funding Information: This work has received funding from the EU H2020 R&I programme for the Marie Sk?odowska-Curie action WDAqua (GA No 642795).

PY - 2020/12

Y1 - 2020/12

N2 - Question answering (QA) over knowledge graphs has gained significant momentum over the past five years due to the increasing availability of large knowledge graphs and the rising importance of Question Answering for user interaction. Existing QA systems have been extensively evaluated as black boxes and their performance has been characterised in terms of average results over all the questions of benchmarking datasets (i.e. macro evaluation). Albeit informative, macro evaluation studies do not provide evidence about QA components’ strengths and concrete weaknesses. Therefore, the objective of this article is to analyse and micro evaluate available QA components in order to comprehend which question characteristics impact on their performance. For this, we measure at question level and with respect to different question features the accuracy of 29 components reused in QA frameworks for the DBpedia knowledge graph using state-of-the-art benchmarks. As a result, we provide a perspective on collective failure cases, study the similarities and synergies among QA components for different component types and suggest their characteristics preventing them from effectively solving the corresponding QA tasks. Finally, based on these extensive results, we present conclusive insights for future challenges and research directions in the field of Question Answering over knowledge graphs.

AB - Question answering (QA) over knowledge graphs has gained significant momentum over the past five years due to the increasing availability of large knowledge graphs and the rising importance of Question Answering for user interaction. Existing QA systems have been extensively evaluated as black boxes and their performance has been characterised in terms of average results over all the questions of benchmarking datasets (i.e. macro evaluation). Albeit informative, macro evaluation studies do not provide evidence about QA components’ strengths and concrete weaknesses. Therefore, the objective of this article is to analyse and micro evaluate available QA components in order to comprehend which question characteristics impact on their performance. For this, we measure at question level and with respect to different question features the accuracy of 29 components reused in QA frameworks for the DBpedia knowledge graph using state-of-the-art benchmarks. As a result, we provide a perspective on collective failure cases, study the similarities and synergies among QA components for different component types and suggest their characteristics preventing them from effectively solving the corresponding QA tasks. Finally, based on these extensive results, we present conclusive insights for future challenges and research directions in the field of Question Answering over knowledge graphs.

KW - Entity linking

KW - Experiment and analysis

KW - Knowledge graph

KW - Question answering

KW - Relation extraction

KW - Relation linking

UR - http://www.scopus.com/inward/record.url?scp=85089276334&partnerID=8YFLogxK

U2 - 10.48550/arXiv.1809.10044

DO - 10.48550/arXiv.1809.10044

M3 - Article

AN - SCOPUS:85089276334

VL - 65

JO - Journal of Web Semantics

JF - Journal of Web Semantics

SN - 1570-8268

M1 - 100594

ER -