Jean-Michel Dalle, Univ. Pierre et Marie Curie and CRG-CNRS & Ecole Polytechnique
Thomas Lacroix, Supélec
Mathieu Lacage, Alcméon
Matthijs den Besten, Montpellier Business School & CRG
The part of the crowdsourcing phenomenon that relies on semi-structured labor markets, for which the “Amazon Mechanical Turk” (AMT) is a leading example, raises considerably many political, ethical and economic issues (e.g. Kittur et al., « The future of crowd work », 2013). However, a proper empirical understanding of the actual functioning of online labor markets based on crowdsourcing is still largely needed. Basic aspects, such as the determinants of workers’ choice among competing projects, are mainly unclear, even with respect to the propensity of online workers and therefore of online work supply to follow price (and other) signals. Online experiments, though a very promising area of research, are affected by selection biases due to the organization of online labor markets in sub-communities (Ipeirotis, 2011; den Besten et al., 2014). Furthermore, the findings of preliminary experiments are puzzling, if not slightly contradictory: workers would sometimes tend to respond to price signals in terms of speed or quantity or quality of work, sometimes not, depending notably on the nature of tasks (Mason & Watts, 2011); low-priced tasks would be addressed more rapidly (Yan et al., 2010); the size of projects (number of HITs in a HIT Group in AMT) would affect work supply, yet with mixed results in terms of elasticity, at least for low prices (Franklin et al, 2011); etc.
To make progress on these important issues for online labor markets, we have collected data from AMT from November 20th of 2013 until February 10th of 2014, scanning every 3 minutes the list of HIT Groups sorted by the number of HITs available, and recording associated data. We then focused on HIT groups which were associated with a standard AMT qualification called “Master has been granted” and specifically on a subset of about 100 HIT groups for which a significant dynamic evolution was directly observable during this period. For each of these HIT groups, we cleaned up its history (gaps in the data, such as when the requester temporarily removes the HIT Group, or jumps linked to the addition or removal of a high number of HITs) by automatically (and algorithmically) keeping only one period for each HIT Group for which data was continuously available (the red rectangles in the left hand side graphs of Figure 1 in the supplemental material, zoomed in in the right hand side graphs).
Removing a few outliers, we then computed the average speed at which individual HITs were treated by online workers for each HIT Group and tried to relate it several variables: Figure 2 present 3 different plots in this respect. Results from linear regression further show that the average speed at which HITs are solved – which is clearly directly related to the supply of online labor for a given set of tasks – seems independent from the price of individual tasks. This surprising result seems to holds under various specifications of the price variable, including dummy variables used to split the price distribution between lower-price and higher-price tasks, and also with various attempts to remove further outliers (very low average speed HIT groups, etc.). This finding is reinforced by the fact that, in line with previous literature, we find strong dependence of the average speed to the size of the HIT groups, measured by their maximum number of individual HITs. Table 1 reports the numerical results of a linear regression of average speed explained by the log of the initial (max) number of HITs in the HIT group and the task price.
However, we do not interpret these findings as reflecting a null price elasticity of online work supply in Amazon Mechanical Turk, but rather as suggesting more complex behavior from “turkers”, who could typically select the tasks and HIT groups they would work on depending on whether they would be able to focus on them for a few hours or days. We intend to inquire further these issues based on our original dataset and on other complementary sociological approaches.
