Details
| Original language | English |
|---|---|
| Title of host publication | HT 2021 - Proceedings of the 32nd ACM Conference on Hypertext and Social Media |
| Pages | 165-175 |
| Number of pages | 11 |
| Publication status | Published - 29 Aug 2021 |
| Externally published | Yes |
| Event | 32nd ACM Conference on Hypertext and Social Media, HT 2021 - Virtual, Online, Unknown Duration: 30 Aug 2021 → 2 Sept 2021 |
Abstract
Designing tasks clearly to facilitate accurate task completion is a challenging endeavor for requesters on crowdsourcing platforms. Prior research shows that inexperienced requesters fail to write clear and complete task descriptions which directly leads to low quality submissions from workers. By complementing existing works that have aimed to address this challenge, in this paper we study whether clarity flaws in task descriptions can be identified automatically using natural language processing methods. We identify and synthesize seven clarity flaws in task descriptions that are grounded in relevant literature. We build both BERT-based and feature-based binary classifiers, in order to study the extent to which clarity flaws in task descriptions can be computationally assessed, and understand textual properties of descriptions that affect task clarity. Through a crowdsourced study, we collect annotations of clarity flaws in 1332 real task descriptions. Using this dataset, we evaluate several configurations of the classifiers. Our results indicate that nearly all the clarity flaws in task descriptions can be assessed reasonably by the classifiers. We found that the content, style, and readability of tasks descriptions are particularly important in shaping their clarity. This work has important implications on the design of tools to help requesters in improving task clarity on crowdsourcing platforms. Flaw-specific properties can provide for valuable guidance in improving task descriptions.
Keywords
- BERT-based binary classification, crowdsourcing, feature-based binary classification, task clarity assessment, task design, unclear task descriptions
ASJC Scopus subject areas
- Computer Science(all)
- Artificial Intelligence
- Computer Science(all)
- Computer Graphics and Computer-Aided Design
- Computer Science(all)
- Human-Computer Interaction
- Computer Science(all)
- Software
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
HT 2021 - Proceedings of the 32nd ACM Conference on Hypertext and Social Media. 2021. p. 165-175.
Research output: Chapter in book/report/conference proceeding › Conference contribution › Research › peer review
}
TY - GEN
T1 - What Is Unclear?
T2 - 32nd ACM Conference on Hypertext and Social Media, HT 2021
AU - Nouri, Zahra
AU - Gadiraju, Ujwal
AU - Engels, Gregor
AU - Wachsmuth, Henning
PY - 2021/8/29
Y1 - 2021/8/29
N2 - Designing tasks clearly to facilitate accurate task completion is a challenging endeavor for requesters on crowdsourcing platforms. Prior research shows that inexperienced requesters fail to write clear and complete task descriptions which directly leads to low quality submissions from workers. By complementing existing works that have aimed to address this challenge, in this paper we study whether clarity flaws in task descriptions can be identified automatically using natural language processing methods. We identify and synthesize seven clarity flaws in task descriptions that are grounded in relevant literature. We build both BERT-based and feature-based binary classifiers, in order to study the extent to which clarity flaws in task descriptions can be computationally assessed, and understand textual properties of descriptions that affect task clarity. Through a crowdsourced study, we collect annotations of clarity flaws in 1332 real task descriptions. Using this dataset, we evaluate several configurations of the classifiers. Our results indicate that nearly all the clarity flaws in task descriptions can be assessed reasonably by the classifiers. We found that the content, style, and readability of tasks descriptions are particularly important in shaping their clarity. This work has important implications on the design of tools to help requesters in improving task clarity on crowdsourcing platforms. Flaw-specific properties can provide for valuable guidance in improving task descriptions.
AB - Designing tasks clearly to facilitate accurate task completion is a challenging endeavor for requesters on crowdsourcing platforms. Prior research shows that inexperienced requesters fail to write clear and complete task descriptions which directly leads to low quality submissions from workers. By complementing existing works that have aimed to address this challenge, in this paper we study whether clarity flaws in task descriptions can be identified automatically using natural language processing methods. We identify and synthesize seven clarity flaws in task descriptions that are grounded in relevant literature. We build both BERT-based and feature-based binary classifiers, in order to study the extent to which clarity flaws in task descriptions can be computationally assessed, and understand textual properties of descriptions that affect task clarity. Through a crowdsourced study, we collect annotations of clarity flaws in 1332 real task descriptions. Using this dataset, we evaluate several configurations of the classifiers. Our results indicate that nearly all the clarity flaws in task descriptions can be assessed reasonably by the classifiers. We found that the content, style, and readability of tasks descriptions are particularly important in shaping their clarity. This work has important implications on the design of tools to help requesters in improving task clarity on crowdsourcing platforms. Flaw-specific properties can provide for valuable guidance in improving task descriptions.
KW - BERT-based binary classification
KW - crowdsourcing
KW - feature-based binary classification
KW - task clarity assessment
KW - task design
KW - unclear task descriptions
UR - http://www.scopus.com/inward/record.url?scp=85114834391&partnerID=8YFLogxK
U2 - 10.1145/3465336.3475109
DO - 10.1145/3465336.3475109
M3 - Conference contribution
AN - SCOPUS:85114834391
SN - 9781450385510
SP - 165
EP - 175
BT - HT 2021 - Proceedings of the 32nd ACM Conference on Hypertext and Social Media
Y2 - 30 August 2021 through 2 September 2021
ER -