In this tutorial, we will demonstrate how to conduct behavioral research on Amazon's Mechanical Turk. We will begin by discussing the four main advantages to using Mechanical Turk as a platform for running online studies: access to a large pool of participants, diversity of the participants, low cost of running studies, and faster research cycle. We will outline the fundamental components of a job on Mechanical Turk and discuss the features of the marketplace, including who is doing the work. We will describe how to run three kinds of studies on Mechanical Turk: surveys, experiments with random assignment, and synchronous experiments. We will demonstrate the mechanics of putting a task on Mechanical Turk by creating a survey, posting the job to Mechanical Turk, reviewing the responses and paying the workers. Finally, we will discuss methods for quality assurance and ethical issues surrounding Mechanical Turk.
We anticipate the tutorial will last approximately four hours.
In this tutorial we will describe a new tool that has emerged in the last 5 years for conducting online behavioral research: crowdsourcing platforms. One of the main benefits of these platforms to behavioral researchers is that they provide access to a large set of people who are willing to do tasks---including participating in research studies---for relatively low pay. The crowdsourcing site with one of the largest subject pools is Amazon's Mechanical Turk (AMT), so it is the focus of this tutorial.
In this tutorial, we will begin by discussing some of the advantages of doing experiments on Mechanical Turk. Specifically, there are four main advantages to using Mechanical Turk as a platform for running online experiments:
We will then discuss how the behavior of workers compares to laboratory subjects, citing work by researchers from computer science and psychology. Then, we will walk through the mechanics of putting a task on Mechanical Turk including recruiting subjects, executing the task, and reviewing the work that was submitted. We will also provide solutions to common problems that a researcher might face when executing their research on this platform such as techniques for conducting synchronous experiments, methods to ensure high quality work, how to keep data private, and how to maintain code security.
At the end of this tutorial, we expect the audience members to:
The majority of the tutorial will be in a presentation format, utilizing slides to explain the process of building and running an experiment on Mechanical Turk. Ideally, if there is access to the internet, we will actually build and run a study on Mechanical Turk. This will allow us to not only demonstrate how to do research on Mechanical Turk, it will also demonstrate how quickly and efficiently research conducted on Mechanical Turk can be. We will send slides used in a previous version of this tutorial and a manuscript on which the tutorial is based to the tutorial chair.
We will begin by motivating the use of Mechanical Turk for behavioral research, focusing on the four advantages outlined in the introduction to this proposal. We will also discuss prior research that demonstrates the validity of using workers as participants.
We will then discuss the basic concepts associated with using Mechanical Turk, including who are the workers, who are the other requesters (i.e., employers), and what constitutes a Human Intelligence Task (HIT). This will include the information workers use to find HITs, where a HIT is stored, and the lifecycle of a HIT, from creation, to execution of the task, to approval of the work and payment of the worker. We will also discuss the typical cost of a HIT and how to assess the value of the work being requested.
We will then talk about three types of studies that can be conducted on Mechanical Turk: surveys, experiments with random assignment, and synchronous experiments. When introducing surveys, we will actually create a survey with the help of the audience and post the job to Mechanical Turk. This will demonstrate both how to do surveys specifically as well as how to generally create HITs on Mechanical Turk. When discussing experiments with random assignment and synchronous experiments, we will review work we have conducted in both of these categories.
We will also briefly discuss other tools for creating HITs, including the command-line tools (CLTs) and PHP scripts that we will make available to the audience. After discussing these tools, we will retrieve and present the results of the survey initiated earlier in the tutorial, which will demonstrate the methods for retrieving results and paying workers, as well as the speed and low cost of conducting research on Mechanical Turk.
Finally, we will discuss some of the issues specific to conducting research on Mechanical Turk, including quality assurance, data security, and ethical issues (many of which apply to online research generally).
Because Mechanical Turk is a tool for any researcher who does studies online, we expect this tutorial to have very broad appeal. The tutorial will be especially relevant to those who are unfamiliar with Mechanical Turk or want to know details about good practices when doing research on the site and ways to ensure reliable data collection. This tutorial will not be particularly useful to individuals who are already using Mechanical Turk for sophisticated research, who can only conduct their research in the laboratory, or are uninterested in conducting behavioral research.
We only expect audience members to have very basic familiarity with conducting behavioral research and the internet. No other knowledge will be required.
Attendees are invited to read our manuscript on which this paper is based, which can be found here. You may also download (.pptx) slides from an earlier (and shorter) version of this tutorial.