A Simple Formula for Predicting the Time to Complete a Study on Mechanical Turk.

Overview

The simple formula

We describe a general formula for predicting the time it takes Workers to complete survey studies on MTurk. The average Worker takes 10.3 seconds to answer a single question. This means that a study with 60 questions should take approximately 10 minutes. At $6 per hour the appropriate pay rate for a 60 question survey would be $1.

The slightly more nuanced approach

We also show that increasing the pay rate and decreasing the length of a survey can increase the average time that Workers spend on each question by 36%. Pay rate and the number of questions in a HIT both influence how long Workers spend answering questions. Workers spend less time on each question for longer surveys and for surveys that pay less. Survey length is also a moderator of the association between pay rate and the time that Workers spend answering questions.

A more detailed approach to predicting the length of a survey is depicted in Figure 1, which takes both survey length and pay rate into consideration when predicting the time it takes Workers to answer a single question. For longer surveys with 108 questions or above, time per question is closer to 8.3 seconds per question, and is independent of pay rate. For medium length surveys with 65 questions, time per survey can range between 9.2 seconds for a pay rate of $1.8 per hour, 10 seconds per question at $3.50 per hour, and 10.6 seconds for a pay rate of over $5 per hour. Shorter surveys with 28 questions are answered at a rate of 10 seconds per question at $1.80 per hour, 11.5 seconds per question at $3.50 per hour and over 13.2 seconds for pay rates of over $5 per hour. Overall, higher  pay rates are most effective at increasing completion time for shorter surveys.

This approach probably also generalizes to non-MTurk online surveys and paper and pencil surveys, but more research should be done to compare completion time across different platforms.

Why determining completion time is important

Completion time refers to the time it takes a Worker to complete a HIT. It is important for a Requester to correctly estimate completion time so as to set the appropriate pay rate.  This is often difficult to do accurately however, and Requesters often end up over or underestimating completion time and, consequently, pay rate. In a previous post we showed that lower pay rates are associated with higher dropout. Thus, misestimating completion time, and not setting an appropriate pay rate as a result, can have negative effects on data quality.

Method

Determining completion time for survey studies

To determine a general formula for predicting study length we adopted the following strategy. We reviewed 30 MTurk survey studies, counting the number of survey questions per survey and the average time it took to complete each survey, across all Workers. We then calculated the average time it takes for MTurk Workers to answer a single question, across all studies. Future studies can use this average time estimate by simply multiplying the number of questions in their study by the number of seconds it takes to complete a question. We additionally examined various factors that may influence how long Workers take to complete a question such as pay rate and the number of questions in the survey.

Thirty 30 MTurk studies were reviewed to examine how long Workers take to answer survey questions on average. The studies had a total of 15, 392 Workers who answered a total of 570, 214 survey questions. The surveys consisted of Likert items presented in a matrix (52.2%), Likert items presented individually (33.1%), Yes/No questions (5.2%) and single check box questions (9.5%).

Results

How long do Workers take to answer a single survey question?

Workers take 10.3 seconds to answer a single survey question on average, with a standard deviation of 3.8 seconds. The study with the shortest average time to answer questions was 5.53 seconds, and for the longest study Workers took 21.2 seconds to answer the average question.

Factors that predict how long Workers spend on survey questions

Various factors were associated with how long Workers take to answer questions (question time). As described in a previous post, question time correlated with dropout rate r (28) = .5, p = .01. Question time also correlated with total number of questions in a survey (survey length), r (28) = -.5, p = .01, and pay rate per hour, r (28) = .4, p = .04.

Pay rate per hour and survey length were used as simultaneous predictors of question time in an OLS regression equation. The results showed that 36% of the question time variance was accounted for by survey length and pay rate, R² = .363, F (2, 28) = 6.3, p. = .007.

Semi partial correlations for both pay rate per hour (Beta = .71) and survey length (Beta = -.04) were both significant (p’s < .05). This indicates that both variables significantly contribute to question time, with the other variable controlled for (co-varied out of the regression equation).

In the next analysis we examined whether the survey length moderates the association between pay rate and question time. Moderation was computed utilizing PROCESS, a mediation and moderation software package (Hayes, 2012). The moderation analysis revealed that the association between pay rate and question time was not significantly moderated by survey length (R² change due to the interaction  = .0255, F (1, 26) = .88, p = .36.

The relationship between pay rate and question time was then explored at different levels of the continuous survey length moderator using the Johnson-Neyman bootstrapping technique (Hayes, 2012). The Johnson-Neyman bootstrapping technique revealed that while pay rate does not significantly influence question time for surveys that are longer than 108 questions, for surveys that are shorter pay rate is associated with significant increases in question time.  Figure 1, shows the slopes (representing the association between pay rate and question time) at three different levels of survey length. The three levels of survey length in Figure 1 correspond to the average number of questions in a survey (65) and one standard deviation above (108) and below (28) the average.

It is possible that the overall moderation by survey length was not significant because the moderating effect of survey length may not be linear. In the absence of an overall moderation effect, the Johnson-Neyman analysis, which probes the interaction across the range of the continuous moderator, suggests that higher pay rate is associated with an increased time that Workers spend per question, particularly for surveys that are shorter than 42 questions.

Interpretation

The results of this study show that Amazon Turk Workers take an average of 10.3 seconds to answer a question for an average survey of 65 questions at a pay rate of $3.50 per hour. However,  question time can vary depending on two factors: 1) how much a study pays, and 2) how many questions a study has. These factors independently contribute to the time that Workers take to answer questions, and they also interact with each other.

The results suggest that higher pay rates can be effective at increasing question time, but that this effect does not generalize to overly long surveys. We did not find evidence that question time increases per  pay rate. In other words, Workers spend less time per question for long surveys independent of pay rate.   However, it may be that pay rates may influence question time even for very long surveys, if the pay rate is higher than those that were utilized in the studies reviewed here. Future studies should explore these issues in more detail.