TY - JOUR

T1 - Investigating the impact of violations of the “left and top means first” heuristic on response behavior and data quality

AU - Höhne, Jan Karem

AU - Yan, Ting

N1 - Funding Information: We acknowledge financial support by the German Science Foundation through the Collaborative Research Center 884 ‘Political Economy of Reforms’ at the University of Mannheim (Germany).

PY - 2020/5/3

Y1 - 2020/5/3

N2 - Web surveys are an established data collection mode that use written language to provide information. The written language is accompanied by visual elements, such as presentation formats and shapes. However, research has shown that visual elements influence response behavior because respondents sometimes use interpretive heuristics to make sense of the visual elements. One such heuristic is the ‘left and top means first’ (LTMF) heuristic, which suggests that respondents tend to believe that a response scale consistently runs from left to right or from top to bottom. We conducted a web survey experiment to investigate how violations of the LTMF heuristic affect response behavior and data quality. For this purpose, a random half of respondents received response options that followed a consistent order and the other half received response options that followed an inconsistent order. The results reveal significantly different response distributions between the two groups. We also found that inconsistently ordered response options significantly increase response times and decrease data quality in terms of criterion validity. We, therefore, recommend using options that follow the design strategies of the LTMF heuristic.

AB - Web surveys are an established data collection mode that use written language to provide information. The written language is accompanied by visual elements, such as presentation formats and shapes. However, research has shown that visual elements influence response behavior because respondents sometimes use interpretive heuristics to make sense of the visual elements. One such heuristic is the ‘left and top means first’ (LTMF) heuristic, which suggests that respondents tend to believe that a response scale consistently runs from left to right or from top to bottom. We conducted a web survey experiment to investigate how violations of the LTMF heuristic affect response behavior and data quality. For this purpose, a random half of respondents received response options that followed a consistent order and the other half received response options that followed an inconsistent order. The results reveal significantly different response distributions between the two groups. We also found that inconsistently ordered response options significantly increase response times and decrease data quality in terms of criterion validity. We, therefore, recommend using options that follow the design strategies of the LTMF heuristic.

KW - Criterion validity

KW - field experiment

KW - interpretive heuristics

KW - response behavior

KW - response times

KW - web survey

UR - http://www.scopus.com/inward/record.url?scp=85076491034&partnerID=8YFLogxK

U2 - 10.1080/13645579.2019.1696087

DO - 10.1080/13645579.2019.1696087

M3 - Article

AN - SCOPUS:85076491034

VL - 23

SP - 347

EP - 353

JO - International Journal of Social Research Methodology

JF - International Journal of Social Research Methodology

SN - 1364-5579

IS - 3

ER -