When researchers collect data
online, it’s natural to be concerned about data quality. Participants aren’t in
the lab, so researchers can’t see who
is taking their survey, what those
participants are doing while answering questions, or whether participants are who they say they are. Not knowing is
unsettling.
Recently, the research community has been consumed with
concern that workers on Amazon’s Mechanical Turk (MTurk) are cheating
requesters by faking their location or using “bots” to submit surveys. These
concerns originated and have been driven by reports from researchers that there
are more nonsensical and low-quality responses in recent studies conducted on
MTurk. In these studies, researchers have noticed that several low-quality
responses are pinned to the same geolocation. In this blog, we’d like to add
some context to the conversation, share the findings from our internal inquiry,
and inform researchers what TurkPrime is doing to address the issue.
Concern about Bots
The recent concern about bots appears to have begun on
Tuesday, August 7th, 2018, when a researcher asked the PscyhMap Facebook group
if anyone had experienced an increase in low quality data. In just the third
response to that thread, another researcher suggested, “maybe a machine?” Soon,
other researchers were reporting an increase in nonsense responses and
low-quality data, although at least a few reported no increase in junk
responses to their studies. The primary piece of evidence causing researchers
to suspect bots was that most of the low-quality responses were tagged to the
same geolocation and a few places in particular—Niagara Square in Buffalo, NY;
a lake in Kansas; and a forest in Venezuela. What’s more, many respondents from
these geolocations provided suspicious responses to open-ended questions, often
answering with “GOOD STUDY,” or “NICE.”
Although this activity raises concerns, the conversation, so
far, has overlooked some important points. Most critically, while it is clear
some researchers have unfortunately collected several bad responses, the
research community does not yet know how widespread this problem is. Diagnosing
the issue requires knowing how many studies don’t
fit the pattern, as well as how many do.
Scope of problem
At TurkPrime, we track the geolocation of all surveys
submitted in studies run on our platform. In the last 24 hours, we have worked
to determine whether there is a growing problem of multiple submissions from
the same geolocation. In reviewing over 100,000 studies that have been launched
on TurkPrime, we see that the rate of submissions from duplicate geolocations
typically bounced from less than 1% to 2.5% within a study—a number that could
be explained by people submitting surveys from the same building, office,
internet service provider, or even the same city. Geolocations are not precise,
an issue we will discuss in more detail in a future blog post.
Based on this analysis, we set 2.5% as the threshold for detecting
suspicious activity. Over 97% of studies have not reached this threshold, showing that the overwhelming majority
of studies have not been affected by data coming from the same geolocation.
However, when we look at the rate of duplicate submissions based on geolocation over time, we see that in March of this year the percentage of duplicate submissions began edging up. Clearly, this is a problem, but a problem that has emerged only recently.
However, when we look at the rate of duplicate submissions based on geolocation over time, we see that in March of this year the percentage of duplicate submissions began edging up. Clearly, this is a problem, but a problem that has emerged only recently.
What TurkPrime is Doing
At TurkPrime, we are developing tools that will help
researchers combat suspicious activity. We have identified that all suspicious
activity is coming from a relatively small number of sources. We have
additionally confirmed that blocking those sources completely eliminates the
problem. In fact, once the suspicious locations were removed, we saw that the
number of duplicate submissions had actually dropped over the summer to a rate
of just 1.7% in July 2018.
What you can do to eliminate the
problem
In the coming days, we will launch a Free feature that
allows researchers to block suspicious geolocations. This means researchers
will be able to block workers from suspicious geolocations, excluding
submissions from those locations in their data collection. We will also launch
a Pro feature that allows researchers to block multiple submissions from the
same geolocation within a study. This feature will cast a wider net and may
block well-intentioned workers using the same internet service provider, or
working in the same library. This tool will give researchers greater confidence
that they are not receiving submissions from anyone using the same location to
submit junk responses.
Conclusions
Our data, and the work of multiple researchers, show there
has been a recent increase in the number of low quality responses submitted on
Mechanical Turk. Data from the TurkPrime database show that the vast majority
of all studies, and the vast majority of recent studies, have never been
affected by the current concern of bots. What we still don’t know about the
recent issue, is whether these responses are coming from bots or foreign
workers using a VPN to disguise their location and submit surveys intended to
sample US workers. Either way, in the coming days TurkPrime will release tools
that allow researchers to block workers from suspicious locations and to decide
how narrowly they would like to set the exclusion criteria. Concerns about bots
and low quality data on MTurk are not new. But at TurkPrime we will continue to
look for ways to ensure quality data and to make conducting online research
easier for researchers.
