New Feature: Exclude Highly Active Workers

Some workers on MTurk are extremely active, and take the majority of posted HITs. This can lead to many issues, some of which are outlined in our previous post. Although MTurk has over 100,000 workers who take surveys each year, and around 25,000 who take surveys each month, you are much more likely to recruit highly active workers who take a majority of HITs. About 1,000 workers (1% of workers) take 21% of the HITs. About 10,000 workers (10% of workers) take 74% of all HITs.

TurkPrime now has a feature to allow researchers to exclude the most active workers so that you can collect data from less experienced workers who are less likely to have previously taken part in research similar to your own. Below is a screenshot of the “Naivete (Exclude most active Workers)” feature. You can select what percentage of workers you would like to exclude from the dropdown menu seen below. 

Strengths and Limitations of Mechanical Turk

Hundreds of academic papers are published each year using data collected through Mechanical Turk. Researchers have gravitated to Mechanical Turk primarily because it provides high quality data quickly and affordably. However, Mechanical Turk has strengths and weaknesses as a platform for data collection. While Mechanical Turk has revolutionized data collection, it is by no means a perfect platform. Some of the major strengths and limitations of MTurk are summarized below.

Strengths

A source of quick and affordable data

Thousands of participants are looking for tasks on Mechanical Turk throughout the day, and can take your task with the click of a button. You can run a 10 minute survey with 100 participants for $1 each, and have all your data within the hour.

Data is reliable

Researchers have examined data quality on MTurk and have found that by and large, data are reliable, with participants performing on tasks in ways similar to more traditional samples. There is a useful reputation mechanism on MTurk, in which researchers can approve or reject the performance of workers on a given study. The reputation of each worker is based on the number of times their work was approved or rejected. Many researchers use a standard practice that relies on only using data from workers who have a 95% approval rating, thereby further ensuring high-quality data collection.

Participant pool is more representative compared to traditional subject pools

Traditional subject pools used in social science research are often samples that are convenient for researchers to obtain, such as undergraduates at a local university. Mechanical Turk has been shown to be more diverse, with participants who are closer to the U.S. population in terms of gender, age, race, education, and employment.

Limitations

There are two kinds of potential limitations on MTurk, technical limitations, and more fundamental limitations with the platform. Many of the technical limitations of MTurk have been resolved through scripts written by researchers or platforms such as TurkPrime, which help researchers do things they were not previously able to do on MTurk including

• Exclude participants from a study based on participation in a previous study
• Conduct longitudinal research
• Make sure larger studies do not stall out after the first 500 to 1000 Workers
• Communicate with many Workers at a time.

There are however several more fundamental limitations to data collection on MTurk:

Small population

There are about 100,000 Mechanical Turk workers who participate in academic studies each year. In any one month about 25,000 unique Mechanical Turk workers participate in online studies. These 25,000 workers participate in close to 600,000 monthly assignments. The more active workers complete hundreds of studies each month. The natural consequence of a small worker  population is that participants are continuously recycled across research labs. This creates a problem of ‘non-naivete’. Most participants on Mechanical Turk have been exposed to common experimental manipulations and this can affect their performance. Although the effects of this exposure have not been fully examined, recent research indicates that this may be impacting effect sizes of experimental manipulations, comprising data quality and the effectiveness of experimental manipulations.

Diversity

Although Mechanical Turk workers are significantly more diverse than the undergraduate subject pool, the Mechanical Turk population is significantly less diverse than the general US population. The population of MTurk workers is  significantly less politically diverse, more highly educated, younger, and less religious compared to the US population. This can complicate the way that data can be interpreted to be reliable on a population level.

Limited selective recruitment

Mechanical Turk has basic mechanisms to selectively recruit workers who have already been profiled. To accomplish this goal Mechanical Turk conducts  profiling HITs that are continuously available for workers.  However, Mechanical Turk is structured in such a way that it is much more difficult to recruit people based on characteristics that have not been profiled. For this reason while rudimentary selective recruitment mechanisms exist there are significant limitations on the ability to recruit specific segments of workers.

Solutions

TurkPrime offers researchers more specific selective recruitment opportunities, and has some features in development to help researchers target participants who are less active and therefore more naive to common experimental manipulations and survey measures. TurkPrime also offers access to PrimePanels, which has access to over 10 million participants, who can be selectively recruited, and are more diverse.

References:

Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior research methods, 46(1), 112-130.

Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213-224.

Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior research methods, 49(2), 433-442.

Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk. Behavior research methods, 47(2), 519-528.

Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184-188.

Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior research methods, 46(4), 1023-1031.

Verified US Region Targeting

Verified US State and Region Targeting

Problem: Many researchers wish to target participants from specific states or US regions of the United States like from the Northeast or the West. The issue that they often encounter is that using the MTurk Geographic Qualification specifying the US state is often not adequate to ensure that the participants reside in the specified state. 

The MTurk state may be incorrect since workers move since setting up their MTurk account and it has been reported that MTurk uses the worker bank location as the worker's state which may have never been the state of the worker's residency. TurkPrime internal quality tests have shown that up to 25% of the worker reported states using the  MTurk Geographic Qualification are inconsistent with the worker's state as reported by their IP address.

Solution: The TurkPrime Pro feature to verify location by Ip address and the US Region selector will only qualify workers whose IP address has been verified as being located in the study's required state or US region. All other workers will be disqualified from taking the study.

In addition, to run regionally targeted studies, TurkPrime now includes a region selector which automatically includes IP state verification to endure quality results.

How to Create a Universal Exclude Worker List

Problem: 

Requesters may observe that some workers, even those with high Approval ratings, may not perform to their expectations on a study. Sometimes this may result in rejecting their work which affects the Worker approval rating. But, often the work is not acceptable for research but is not worthy of rejection, or, it may simply be the policy of the research lab to approve all assignments for IRB or some ethical standard they may follow. 

At this point the researcher may wish to exclude these workers from all future studies. MTurk has an option to Block a Worker (available through the API) but our experience has been that this solution is somewhat draconian and extreme: the effect of a Worker Block can trigger the suspension of the Worker's MTurk account. 

(Source: When I was a newbie Requester in 2012 I blocked some workers who gave me inconsistent and poor responses . I learned the hard way by having my Turkopticon rating suffer and the MTurk Worker discussion groups spread the bad word. I responded to the Worker complaints and Unblocked them to undo the damage to their reputation -- and mine!)

Solution:

Create a Universal Exclude List using the TurkPrime Worker Group feature. This exclude list simply excludes all specified workers in this group from taking a study with this Group Requirement. When you design your studies, just add this exclude group to your Worker Requirements and none of the workers in this exclude group will be qualified to take your study. 

This will achieve your goal of blocking undesired Workers without tarnishing their reputation.

Reusable Include and Exclude Groups

Problem: You are running a longitudinal study and have identified 1000 workers who you want to allow to take your second phase studies. How do you easily group those workers for easy access. 

Or you want to exclude certain workers from taking a number of your studies and wish to group them for easy exclusion in future studies. How can you do that?

Solution: Use the TurkPrime Worker Groups feature available in the Manage Workers menu. This allows you to create reusable worker groups for either inclusion or inclusion in studies. You simply select whether this group is an include or exclude group, give it a clear name and specify the workers in the group. 

Workers will not know why they were targeted for inclusion or exclusion since the MTurk Qualification name used is a random sequence of characters. The TurkPrime Group Name will not be visible to Workers. This is important so that you can run blind studies (i.e. the workers who are specified for inclusion will know they were targeted because they are exhibiting a certain personality, for example)

When you design your study, and specify Worker requirements, add this group to the worker requirements and all workers in that group will be wither included or excluded, as needed.

Survey Groups Ensure Unique Workers Across Multiple Concurrent Studies

Problem: Ever want to run multiple simultaneous studies and needed to ensure that each worker only took a single study? There is no simple way to block workers who completed one study from accepting another study run at the same time....until now. 

(The TurkPrime exclude feature will exclude workers who completed one study from taking another subsequent study but not if both studies are run at the same time.)

Solution: The TurkPrime Survey Group (arriving August 25, 2016)  feature allows a user to assign multiple studies to a survey group. Then, any worker who accepts any study in that group will be disqualified from accepting any other study in that group. Your studies will be guaranteed to have a mutually exclusive group of respondents.

The Survey Group option is also a broader exclusion than the standard Exclude Worker option: Survey Group excludes workers who even only accepted one study from taking the other studies in the group, while the standard Exclude Worker option only excludes workers who completed the study but will not exclude workers who accepted but did not complete the first study from taking the second study.

Anonymize Mechanical Turk Worker IDs

We recently launched a ground-breaking feature that helps protect Mechanical Turk worker identities. It has been reported in the literature that Mechanical Turk Worker IDs can be used to identify the worker. This is because Amazon uses the same value for both the Worker ID on Mechanical Turk and elsewhere on Amazon properties like Amazon.com product reviews.

The Anonymize Worker IDs feature anonymizes Mechanical Turk Worker IDs, as discussed below.

When you enable the Anonymize Worker IDs feature, all Worker IDs that appear in your study's downloadable CSV file will appear encrypted. For example, if the Worker ID is "ABCDEFGHJKL", it will instead appear as "TP_1UPKSI2WHSJ4". This encrypted TurkPrime Worker ID begins with letters TP and can be used in all operations on TurkPrime where an Amazon Worker ID can be used:

• Exclude and Include Feature supports both Amazon Worker IDs and the encrypted TurkPrime Worker IDs.
• Reusable Worker Groups can specify both Amazon Worker IDs and the encrypted TurkPrime Worker IDs.
• Bonus Workers can specify both Amazon Worker IDs and the encrypted TurkPrime Worker IDs.

TurkPrime Video Tutorials to Get You Started with Mechanical Turk

TurkPrime now has tutorials to get you started

Setting up your account

Exclude Workers With One Click

Problem:

Suppose you're running a Mechanical Turk survey and need to exclude workers who took a previous survey. How can you quickly set this up. 

Some of the currently used solutions require following multiple steps to set things up and are not turnkey solutions and others require Workers to enter their Worker ID, which may self-filter workers and limit the number of workers taking your survey. 

Solution:

Exclude Workers Feature 

Create your surveys using TurkPrime.com's "Exclude Workers" feature. When your HIT launches it will have a Qualification Requirement that will limit your HIT to only the Workers not in your exclude list. All excluded workers will be unqualified from taking your HIT.