“Are machines better than humans in image tagging?” - A user study adds to the puzzle

Research output: Chapter in book/report/conference proceedingConference contributionResearchpeer review

Authors

  • Ralph Ewerth
  • Matthias Springstein
  • Lo An Phan-Vogtmann
  • Juliane Schütze

Research Organisations

External Research Organisations

  • German National Library of Science and Technology (TIB)
  • Friedrich Schiller University Jena
View graph of relations

Details

Original languageEnglish
Title of host publicationAdvances in Information Retrieval
Subtitle of host publication39th European Conference on IR Research, ECIR 2017, Proceedings
EditorsClaudia Hauff, Joemon M. Jose, Dyaa Albakour, Ismail Sengor Altingovde, John Tait, Dawei Song, Stuart Watt
PublisherSpringer Verlag
Pages186-198
Number of pages13
ISBN (print)9783319566078
Publication statusPublished - 2017
Event39th European Conference on Information Retrieval, ECIR 2017 - Aberdeen, United Kingdom (UK)
Duration: 8 Apr 201713 Apr 2017

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10193 LNCS
ISSN (Print)0302-9743
ISSN (electronic)1611-3349

Abstract

“Do machines perform better than humans in visual recognition tasks?” Not so long ago, this question would have been considered even somewhat provoking and the answer would have been clear: “No”. In this paper, we present a comparison of human and machine performance with respect to annotation for multimedia retrieval tasks. Going beyond recent crowdsourcing studies in this respect, we also report results of two extensive user studies. In total, 23 participants were asked to annotate more than 1000 images of a benchmark dataset, which is the most comprehensive study in the field so far. Krippendorff’s α is used to measure inter-coder agreement among several coders and the results are compared with the best machine results. The study is preceded by a summary of studies which compared human and machine performance in different visual and auditory recognition tasks. We discuss the results and derive a methodology in order to compare machine performance in multimedia annotation tasks at human level. This allows us to formally answer the question whether a recognition problem can be considered as solved. Finally, we are going to answer the initial question.

ASJC Scopus subject areas

Cite this

“Are machines better than humans in image tagging?” - A user study adds to the puzzle. / Ewerth, Ralph; Springstein, Matthias; Phan-Vogtmann, Lo An et al.
Advances in Information Retrieval: 39th European Conference on IR Research, ECIR 2017, Proceedings. ed. / Claudia Hauff; Joemon M. Jose; Dyaa Albakour; Ismail Sengor Altingovde; John Tait; Dawei Song; Stuart Watt. Springer Verlag, 2017. p. 186-198 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10193 LNCS).

Research output: Chapter in book/report/conference proceedingConference contributionResearchpeer review

Ewerth, R, Springstein, M, Phan-Vogtmann, LA & Schütze, J 2017, “Are machines better than humans in image tagging?” - A user study adds to the puzzle. in C Hauff, JM Jose, D Albakour, IS Altingovde, J Tait, D Song & S Watt (eds), Advances in Information Retrieval: 39th European Conference on IR Research, ECIR 2017, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10193 LNCS, Springer Verlag, pp. 186-198, 39th European Conference on Information Retrieval, ECIR 2017, Aberdeen, United Kingdom (UK), 8 Apr 2017. https://doi.org/10.1007/978-3-319-56608-5_15
Ewerth, R., Springstein, M., Phan-Vogtmann, L. A., & Schütze, J. (2017). “Are machines better than humans in image tagging?” - A user study adds to the puzzle. In C. Hauff, J. M. Jose, D. Albakour, I. S. Altingovde, J. Tait, D. Song, & S. Watt (Eds.), Advances in Information Retrieval: 39th European Conference on IR Research, ECIR 2017, Proceedings (pp. 186-198). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10193 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-56608-5_15
Ewerth R, Springstein M, Phan-Vogtmann LA, Schütze J. “Are machines better than humans in image tagging?” - A user study adds to the puzzle. In Hauff C, Jose JM, Albakour D, Altingovde IS, Tait J, Song D, Watt S, editors, Advances in Information Retrieval: 39th European Conference on IR Research, ECIR 2017, Proceedings. Springer Verlag. 2017. p. 186-198. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-56608-5_15
Ewerth, Ralph ; Springstein, Matthias ; Phan-Vogtmann, Lo An et al. / “Are machines better than humans in image tagging?” - A user study adds to the puzzle. Advances in Information Retrieval: 39th European Conference on IR Research, ECIR 2017, Proceedings. editor / Claudia Hauff ; Joemon M. Jose ; Dyaa Albakour ; Ismail Sengor Altingovde ; John Tait ; Dawei Song ; Stuart Watt. Springer Verlag, 2017. pp. 186-198 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
Download
@inproceedings{67c2d366560e4ba4b97cf43f4fe8f9f7,
title = "“Are machines better than humans in image tagging?” - A user study adds to the puzzle",
abstract = "“Do machines perform better than humans in visual recognition tasks?” Not so long ago, this question would have been considered even somewhat provoking and the answer would have been clear: “No”. In this paper, we present a comparison of human and machine performance with respect to annotation for multimedia retrieval tasks. Going beyond recent crowdsourcing studies in this respect, we also report results of two extensive user studies. In total, 23 participants were asked to annotate more than 1000 images of a benchmark dataset, which is the most comprehensive study in the field so far. Krippendorff{\textquoteright}s α is used to measure inter-coder agreement among several coders and the results are compared with the best machine results. The study is preceded by a summary of studies which compared human and machine performance in different visual and auditory recognition tasks. We discuss the results and derive a methodology in order to compare machine performance in multimedia annotation tasks at human level. This allows us to formally answer the question whether a recognition problem can be considered as solved. Finally, we are going to answer the initial question.",
author = "Ralph Ewerth and Matthias Springstein and Phan-Vogtmann, {Lo An} and Juliane Sch{\"u}tze",
note = "Publisher Copyright: {\textcopyright} The Author(s) 2017. Copyright: Copyright 2017 Elsevier B.V., All rights reserved.; 39th European Conference on Information Retrieval, ECIR 2017 ; Conference date: 08-04-2017 Through 13-04-2017",
year = "2017",
doi = "10.1007/978-3-319-56608-5_15",
language = "English",
isbn = "9783319566078",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "186--198",
editor = "Claudia Hauff and Jose, {Joemon M.} and Dyaa Albakour and Altingovde, {Ismail Sengor} and John Tait and Dawei Song and Stuart Watt",
booktitle = "Advances in Information Retrieval",
address = "Germany",

}

Download

TY - GEN

T1 - “Are machines better than humans in image tagging?” - A user study adds to the puzzle

AU - Ewerth, Ralph

AU - Springstein, Matthias

AU - Phan-Vogtmann, Lo An

AU - Schütze, Juliane

N1 - Publisher Copyright: © The Author(s) 2017. Copyright: Copyright 2017 Elsevier B.V., All rights reserved.

PY - 2017

Y1 - 2017

N2 - “Do machines perform better than humans in visual recognition tasks?” Not so long ago, this question would have been considered even somewhat provoking and the answer would have been clear: “No”. In this paper, we present a comparison of human and machine performance with respect to annotation for multimedia retrieval tasks. Going beyond recent crowdsourcing studies in this respect, we also report results of two extensive user studies. In total, 23 participants were asked to annotate more than 1000 images of a benchmark dataset, which is the most comprehensive study in the field so far. Krippendorff’s α is used to measure inter-coder agreement among several coders and the results are compared with the best machine results. The study is preceded by a summary of studies which compared human and machine performance in different visual and auditory recognition tasks. We discuss the results and derive a methodology in order to compare machine performance in multimedia annotation tasks at human level. This allows us to formally answer the question whether a recognition problem can be considered as solved. Finally, we are going to answer the initial question.

AB - “Do machines perform better than humans in visual recognition tasks?” Not so long ago, this question would have been considered even somewhat provoking and the answer would have been clear: “No”. In this paper, we present a comparison of human and machine performance with respect to annotation for multimedia retrieval tasks. Going beyond recent crowdsourcing studies in this respect, we also report results of two extensive user studies. In total, 23 participants were asked to annotate more than 1000 images of a benchmark dataset, which is the most comprehensive study in the field so far. Krippendorff’s α is used to measure inter-coder agreement among several coders and the results are compared with the best machine results. The study is preceded by a summary of studies which compared human and machine performance in different visual and auditory recognition tasks. We discuss the results and derive a methodology in order to compare machine performance in multimedia annotation tasks at human level. This allows us to formally answer the question whether a recognition problem can be considered as solved. Finally, we are going to answer the initial question.

UR - http://www.scopus.com/inward/record.url?scp=85018704962&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-56608-5_15

DO - 10.1007/978-3-319-56608-5_15

M3 - Conference contribution

AN - SCOPUS:85018704962

SN - 9783319566078

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 186

EP - 198

BT - Advances in Information Retrieval

A2 - Hauff, Claudia

A2 - Jose, Joemon M.

A2 - Albakour, Dyaa

A2 - Altingovde, Ismail Sengor

A2 - Tait, John

A2 - Song, Dawei

A2 - Watt, Stuart

PB - Springer Verlag

T2 - 39th European Conference on Information Retrieval, ECIR 2017

Y2 - 8 April 2017 through 13 April 2017

ER -