The New New Demographics on Mechanical Turk: Is there Still a Gender Gap?

Overview

Seventy five Mechanical Turk studies conducted with US-based Workers in  2013 and 2014 were reviewed. From a total of 32, 595 Workers, 15,324 (47%) were female.

Background

It’s been a while since the last update on the demographics of Mechanical Turk Workers, so we thought it’s time for a new look. The current consensus seems to be that MTurk Workers are primarily female. For example Panos Ipeirotis' blog reports that US-based Workers are 65% female. MTurk is always changing, and this report presents data from 75 studies conducted over the last two years.

Method/Results

We examined 75 studies that were conducted since the beginning of 2013. The studies were available only for US-based Workers. The percentage of female workers was computed for each study and is presented in Figure 1. On average, 47% of Workers were female. However, there was considerable variation across the studies (SD = 7.3). The smallest percentage of females in a study was 29.7% and the largest was 62%. A single sample t-test showed that the percentage of females across all studies was significantly smaller than 50%, t (74) = -4.1, p. < .0001. This means that over the last two years MTurk studies were more likely to have more male Workers. 

Interpretation

Overall, these results show that the gender gap on MTurk has shifted over the years.  The majority of studies from 3 to 4 years ago have reported that MTurk Workers are predominantly female. Our results show that since 2013 most HITs have more male Workers.  These differences are not particularly large however, even though they are statistically significant. What is most important for any one study is not the overall ratio of male to female Workers on MTurk but the gender distribution within a specific HIT. Our data show that there is considerable variation of gender across HITs, with some HIT’s having less than 30% female Workers, and some over 60%.

It is not clear whether the male/female ratio systematically depends on other factors. An informal look at some of the data shows that males are particularly more likely to take HITs during working hours from 9 am to 5 pm.  These data may suggest that during working hours Workers are likely to be unemployed males. We are currently conducting a systematic study in which we will examine how Worker demographics vary during different times of day, and days of the week. If it turns out that the time of day is a systematic predictor of the gender composition of a study, it may be worthwhile to sample Workers across different times of the day by pausing a HIT and then restarting at a later time, to reduce common method bias.

What about other demographics?

We also reviewed other demographics, like ethnicity, education, religious background, family composition, income, and many others. For the most part these other demographics have remained similar to what has been reported previously. These data have to be examined more thoroughly and will be presented in a future post.