Microworkers who are paid less tend to exaggerate the importance of their participation; express greater enjoyment and experience lower tension; expend lower effort and are more likely to drop out

Microworkers as Research Participants: Does Underpaying Turkers lead to Cognitive Dissonance? Bingjie Liu, S.Shyam Sundar. Computers in Human Behavior, https://doi.org/10.1016/j.chb.2018.06.017

Highlights
•    Turkers who are paid less tend to exaggerate the importance of their participation.
•    Turkers who are paid less express greater enjoyment and experience lower tension.
•    Turkers paid less expend lower effort and are more likely to drop out.
•    Underpaying microworkers undermines ethical practice of scientific research.

Abstract: Social science researchers increasingly rely on microworkers to serve as study participants, paying them very little compared to participants recruited from other venues. This has raised ethical concerns and questioned the validity of research based on microworkers. Informed by cognitive dissonance theory, we conducted two between-subjects experiments to examine the effects of underpaying Amazon Mechanical Turk workers (Turkers) on their perceptions and their actual performance on criteria crucial to online social science research. Data show that underpaid Turkers experienced ‘cognitive dissonance’ such that those paid as low as $0.25 said that their participation was more important (than subjects who were paid higher), which was positively associated with other positive perceptions and demand characteristics. Nevertheless, underpaying Turkers increased dropout rate, reduced their level of effort in answering open-ended questions and undermined perceived agency. We discuss the ethical and practical implications of underpaying microworkers.

Keywords: Amazon’s Mechanical Turk; underpayment; cognitive dissonance; enjoyment; effort