Brief overview
It is generally thought that pay rate does not affect data
quality on Mechanical Turk. For example (Buhrmester, Kwang, & Gosling, 2011) showed that whether Workers
are paid 5 cents or one dollar for a survey study, the internal reliability of
the surveys does not change. They did show however that fewer Workers will take
the surveys that pay less. We recently replicated these findings for both US
and India-based Workers (Litman et al, 2014). Here we show that low pay rates
have two effects on Workers: 1) Workers are more likely to return a HIT before
completing it and 2) Workers spend less time answering questions. We examined
30 MTurk studies that were run over the last 6 months. The findings show that
36% of the dropout rate variance is explained by the length and pay rate of a
survey. These results show that low pay rates do more than just slow down the
rate at which Workers take HITs. Low pay rates may also negatively impact the representativeness of
data due to high participant dropout, and they may also decrease how much
attention participants pay to each question. Based on these findings we
recommend against low paying HIT We also recommend against overly long surveys, unless
Workers are appropriately compensated. To minimize dropout and to maximize time
on task, compensation for HITs should not be below $4 per hour and should be
closer to $6 per hour or more.
Introduction
Dropout on Mechanical Turk occurs when Workers do not
complete a started HIT. Although sometimes a Worker cannot complete a HIT due
to technical difficulties, the most common reason for dropout is that a Worker Returns
the HIT. An important question that has not been previously addressed in the
MTurk literature is whether there are specific factors that may increase the likelihood
with which HITs are returned.
In a previous post we discussed how to monitor dropout rate
in real time on TurkPrime. Here we address a more complex issue – what factors
contribute to dropout rate and what are the ways to minimize dropout. To
examine the factors that may be associated with dropout rate we examined 30
studies that we ran on TurkPrime. Payment, completion time, the number of
questions in the survey, the number of seconds it takes to answer a survey
question, and pay rate per hour, was examined as possible predictors of dropout
rate.
Method
Thirty studies that were run over a period of 6 months on
TurkPrime in 2014 were examined. This study only examined Hits that utilized
survey instruments, and did not examine non-survey tasks, or surveys that
included open-ended questions. The number of unique participants in each study
ranged between 20 and 3991 (mean = 530, SD = 927). Payments ranged between 20
cents and $1.25; corresponding to a pay rate per hour that ranged between $1.88
and $8.57 (mean = $4.50, SD $1.7). The total number of questions per survey
ranged between 28 and 176 (mean = 65, SD
= 42).
Results
Completion rate ranged between 69% and 100%. Average
completion rate was 89.7% (SD = 8.7). Completion rate did not correlate with
payment, r (28) = -.025, p = .896; but completion rate did positively correlate
with pay rate per hour, r (28) = .39, p = .039. Completion rate also negatively
correlated with the number of survey questions in a HIT, r (28) = -.414, p =
.026. Completion rate also positively correlated with the average time that it
took participants to answer survey questions, r (28) = .5, p = .020.
In the next analysis all three variables that were shown to
correlate with completion rate were entered as simultaneous predictors of
completion rate in OLS regression. The results showed that 36% of completion
rate can be predicted from pay rate per hour (Beta = .25), number of questions
in a HIT (Beta = -.3), and average time to answer a question (Beta = .25),
F(3,27) = 3.9, p = .022. However, partial correlations were not significant for
any of the three predictors, indicating that there is a single underlying
process that accounts for completion rate.
Discussion
Three factors were shown to be predictive of completion
rate. Payrate per hour positively predicted completion rates, and so did the length
of the HIT. This indicates that when workers see that a HIT is taking too long
they are more likely to return it. Requesters’ estimates of how long it takes
to complete a HIT varies in accuracy. If a hit that pays 50 cents is advertised
at taking 5 minutes, but in reality takes ten minutes, Workers will Return that
HIT when they become aware of the discrepancy between the advertised and the
actual HIT length. The number of questions in a survey is also a significant
predictor of dropout, indicating that Workers are less likely to complete very
long tasks.
Perhaps the most surprising results was the positive
correlation between the average time it takes to answer a question and
completion rate. This correlation indicates that HITs with a low dropout rate
are also the ones on which Workers take more time per question. This finding
suggests that Hits that increase the likelihood of dropout also increase the
likelihood that Workers will spend less time on the HITs questions, possibly
because they are becoming more impatient and less interested in the task. These
findings suggest that long and low paying HITs have a dual effect on Workers:
some workers dropout and other workers start speeding through the study.
Previous studies have reported that low paying tasks do not
affect data quality but only slow down the speed with which Workers take a HIT.
The data presented here paint a different picture, and suggest that low paying
tasks can decrease data quality in two ways. First, low paying tasks increase
drop out, which may have a systematic effect on the representativeness of the data.
Second, low paying tasks make Workers more likely to speed through the HIT.
This is likely to affect how much cognitive resources a worker allocates to a
task, and may influence the likelihood that a worker will devote their energy
to the task.
Conclusion
Overall, increasing the pay rate of a HIT is likely to minimize
dropout and to increase how much attention your Workers allocate to your study.
To minimize dropout, compensation for HITs should not be below $4 per hour and
should be closer to $6 per hour or more.
No comments:
Post a Comment