TY - JOUR

T1 - Crowd Anatomy Beyond the Good and Bad

T2 - Behavioral Traces for Crowd Worker Modeling and Pre-selection

AU - Gadiraju, Ujwal Kumar

AU - Demartini, Gianluca

AU - Kawase, Ricardo

AU - Dietze, Stefan

PY - 2019/9/1

Y1 - 2019/9/1

N2 - The suitability of crowdsourcing to solve a variety of problems has been investigated widely. Yet, there is still a lack of understanding about the distinct behavior and performance of workers within microtasks. In this paper, we first introduce a fine-grained data-driven worker typology based on different dimensions and derived from behavioral traces of workers. Next, we propose and evaluate novel models of crowd worker behavior and show the benefits of behavior-based worker pre-selection using machine learning models. We also study the effect of task complexity on worker behavior. Finally, we evaluate our novel typology-based worker pre-selection method in image transcription and information finding tasks involving crowd workers completing 1,800 HITs. Our proposed method for worker pre-selection leads to a higher quality of results when compared to the standard practice of using qualification or pre-screening tests. For image transcription tasks our method resulted in an accuracy increase of nearly 7% over the baseline and of almost 10% in information finding tasks, without a significant difference in task completion time. Our findings have important implications for crowdsourcing systems where a worker’s behavioral type is unknown prior to participation in a task. We highlight the potential of leveraging worker types to identify and aid those workers who require further training to improve their performance. Having proposed a powerful automated mechanism to detect worker types, we reflect on promoting fairness, trust and transparency in microtask crowdsourcing platforms.

AB - The suitability of crowdsourcing to solve a variety of problems has been investigated widely. Yet, there is still a lack of understanding about the distinct behavior and performance of workers within microtasks. In this paper, we first introduce a fine-grained data-driven worker typology based on different dimensions and derived from behavioral traces of workers. Next, we propose and evaluate novel models of crowd worker behavior and show the benefits of behavior-based worker pre-selection using machine learning models. We also study the effect of task complexity on worker behavior. Finally, we evaluate our novel typology-based worker pre-selection method in image transcription and information finding tasks involving crowd workers completing 1,800 HITs. Our proposed method for worker pre-selection leads to a higher quality of results when compared to the standard practice of using qualification or pre-screening tests. For image transcription tasks our method resulted in an accuracy increase of nearly 7% over the baseline and of almost 10% in information finding tasks, without a significant difference in task completion time. Our findings have important implications for crowdsourcing systems where a worker’s behavioral type is unknown prior to participation in a task. We highlight the potential of leveraging worker types to identify and aid those workers who require further training to improve their performance. Having proposed a powerful automated mechanism to detect worker types, we reflect on promoting fairness, trust and transparency in microtask crowdsourcing platforms.

KW - Behavioral traces

KW - Crowdsourcing

KW - Microtasks

KW - Pre-screening

KW - Pre-selection

KW - Worker typology

KW - Workers

UR - http://www.scopus.com/inward/record.url?scp=85049077761&partnerID=8YFLogxK

U2 - 10.1007/s10606-018-9336-y

DO - 10.1007/s10606-018-9336-y

M3 - Article

AN - SCOPUS:85049077761

VL - 28

SP - 815

EP - 841

JO - Computer Supported Cooperative Work: CSCW: An International Journal

JF - Computer Supported Cooperative Work: CSCW: An International Journal

SN - 0925-9724

IS - 5

ER -