Details
| Original language | English |
|---|---|
| Article number | 56 |
| Journal | Proceedings of the ACM on Human-Computer Interaction |
| Volume | 3 |
| Issue number | CSCW |
| Publication status | Published - 7 Nov 2019 |
Abstract
Complex machine learning models are deployed in several critical domains including healthcare and autonomous vehicles nowadays, albeit as functional blackboxes. Consequently, there has been a recent surge in interpreting decisions of such complex models in order to explain their actions to humans. Models which correspond to human interpretation of a task are more desirable in certain contexts and can help attribute liability, build trust, expose biases and in turn build better models. It is therefore crucial to understand how and which models conform to human understanding of tasks. In this paper we present a large-scale crowdsourcing study that reveals and quantifies the dissonance between human and machine understanding, through the lens of an image classification task. In particular, we seek to answer the following questions: Which (well performing) complex ML models are closer to humans in their use of features to make accurate predictions? How does task difficulty affect the feature selection capability of machines in comparison to humans? Are humans consistently better at selecting features that make image recognition more accurate? Our findings have important implications on human-machine collaboration, considering that a long term goal in the field of artificial intelligence is to make machines capable of learning and reasoning like humans.
Keywords
- Crowdsourcing, Dissonance, Human Intelligence, Humans, Image Understanding, Interpretability, Machine Learning Models, Machines, Neural Networks, Object Recognition
ASJC Scopus subject areas
- Social Sciences(all)
- Social Sciences (miscellaneous)
- Computer Science(all)
- Human-Computer Interaction
- Computer Science(all)
- Computer Networks and Communications
Sustainable Development Goals
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: Proceedings of the ACM on Human-Computer Interaction, Vol. 3, No. CSCW, 56, 07.11.2019.
Research output: Contribution to journal › Article › Research › peer review
}
TY - JOUR
T1 - Dissonance Between Human and Machine Understanding
AU - Zhang, Zijian
AU - Singh, Jaspreet
AU - Gadiraju, Ujwal
AU - Anand, Avishek
N1 - Funding information: We thank all the anonymous crowd workers who participated in our experiments. This research has been supported in part by the Amazon Research Awards, and the Erasmus+ project DISKOW (grant no. 60171990).
PY - 2019/11/7
Y1 - 2019/11/7
N2 - Complex machine learning models are deployed in several critical domains including healthcare and autonomous vehicles nowadays, albeit as functional blackboxes. Consequently, there has been a recent surge in interpreting decisions of such complex models in order to explain their actions to humans. Models which correspond to human interpretation of a task are more desirable in certain contexts and can help attribute liability, build trust, expose biases and in turn build better models. It is therefore crucial to understand how and which models conform to human understanding of tasks. In this paper we present a large-scale crowdsourcing study that reveals and quantifies the dissonance between human and machine understanding, through the lens of an image classification task. In particular, we seek to answer the following questions: Which (well performing) complex ML models are closer to humans in their use of features to make accurate predictions? How does task difficulty affect the feature selection capability of machines in comparison to humans? Are humans consistently better at selecting features that make image recognition more accurate? Our findings have important implications on human-machine collaboration, considering that a long term goal in the field of artificial intelligence is to make machines capable of learning and reasoning like humans.
AB - Complex machine learning models are deployed in several critical domains including healthcare and autonomous vehicles nowadays, albeit as functional blackboxes. Consequently, there has been a recent surge in interpreting decisions of such complex models in order to explain their actions to humans. Models which correspond to human interpretation of a task are more desirable in certain contexts and can help attribute liability, build trust, expose biases and in turn build better models. It is therefore crucial to understand how and which models conform to human understanding of tasks. In this paper we present a large-scale crowdsourcing study that reveals and quantifies the dissonance between human and machine understanding, through the lens of an image classification task. In particular, we seek to answer the following questions: Which (well performing) complex ML models are closer to humans in their use of features to make accurate predictions? How does task difficulty affect the feature selection capability of machines in comparison to humans? Are humans consistently better at selecting features that make image recognition more accurate? Our findings have important implications on human-machine collaboration, considering that a long term goal in the field of artificial intelligence is to make machines capable of learning and reasoning like humans.
KW - Crowdsourcing
KW - Dissonance
KW - Human Intelligence
KW - Humans
KW - Image Understanding
KW - Interpretability
KW - Machine Learning Models
KW - Machines
KW - Neural Networks
KW - Object Recognition
UR - http://www.scopus.com/inward/record.url?scp=85075088294&partnerID=8YFLogxK
U2 - 10.1145/3359158
DO - 10.1145/3359158
M3 - Article
AN - SCOPUS:85075088294
VL - 3
JO - Proceedings of the ACM on Human-Computer Interaction
JF - Proceedings of the ACM on Human-Computer Interaction
IS - CSCW
M1 - 56
ER -