25 September 2014

Online labor markets and the failure to compete

Over the past several months I’ve been working on a project involving Amazon Mechanical Turk (henceforth AMT), a fascinating online labor market for “micro-labor”. Some researchers have done the whole crash course ethnographic thing, but for the most part research of online labor has taken a look from a step back (as Horton, Silberman et al., and others have done).

This project, resulting in a website called Dynamo, has been intense, but enlightening. It’s been a unique process going from iterating on a serious web project to thinking at a high level about the social significance of our design decisions and how to evoke the outcomes we want. In this case, the outcome we wanted was “collective action”. By this I mean that we wanted to bring people together specifically to accomplish something. In some sense, we had no agenda because of this. In another sense, we absolutely had an ulterior motive, and a forceful one at that.

Our initial fieldwork results suggested that there were serious problems with AMT that desperately called for some collective action of some sort. Whether that be a union of Turkers, a labor strike, a letter-writing campaign, or whatever, we got the impression that Turkers were calling for something. Before you get ahead of yourself, I’ll present a qualified view, not entirely dismissing this preconception but neither totally embracing it.

Our first round of fieldwork suggested really strongly that there was a single actor in the AMT marketplace who was poisoning the well in some sense by putting out what Turkers call penny HITs. By itself, in a totally naive environment, this really wouldn’t be that big a deal; he pays too little, some people aren’t willing to do that much work for $0.25, but some people are, so those that are willing will do his work and those that are not will go do other tasks. Everyone wins (except the Turkers who do his work, but they’re happy digging their grave. Or so the argument goes).

Not quite so. In practice, these HITs and others set a standard - a norm - in the marketplace, so that even academic researchers (whom we generally regard as being paragons of ethical behavior, justified or not) were paying workers poorly; the market had become one for lemons.

So we thought that a demonstration for this and other HIT requesters would give them the necessary push to show that

  1. Turkers are not to be trifled with
  2. Turkers are human beings
  3. Turkers ought to be treated with basic respect and decency

Our plan was to help Turkers stage a labor protest; they would do bad work intentionally, in a systematically defined way, so that all of the work a requester solicited would be unusable. This was the idea we came up with, bridging some gaps between online and offline labor markets, dynamics, and politics.

Turkers were not enthusiastic with our idea. In fact, more accurately, they hated it with a passion I haven’t seen in a while. So we went back to the drawing board for a bit.

Eventually, we realized that we didn’t know anything (one could argue that I should’ve known this for quite a while, but I didn’t know anything; also, I digress), so we figured that we should just let Turkers come up with their own actions. While this was on one hand more practical than designing our own perfect campaign, it was more importantly a truer representation of homegrown collective action, which was becoming the focal point of our research.

I won’t bore you with many more details - and frankly I find more and more that I don’t have time for them - but of note we have a few things which I think are worth patting ourselves on the back:

  • We came up with a guideline of ethical research protocols developed by the community to which these guidelines are relevant.
  • We got some serious buy-in from researchers in the HCI community (including, but not limited to, Don Norman of UCSD).
  • We learned a lot about online workers, online labor markets, and online communities in general. I genuinely feel more confident about community design and building now than I ever did before.

Now that we’ve submitted our paper to a conference (deadline was on my birthday, so that was fun…), I’m moving on to my own project, of my own design, with no conference deadline in immediate sight: I’m going to try and build a new type of online labor marketplace, where the workers set their own rules and norms, including things like minimum wage, what sorts of work they’ll do, and other factors. My hope is that this will return some of the leverage Amazon puts in the requester’s hands back to the workers themselves. Ideally, there should be as equal a playing field as possible, rather than one that’s exploitative of workers. If this all pans out, it just might be.


  • HIT: Human Intelligence Task (essentially, a work task)
  • penny HIT: A cheap HIT, often one not worth the time to complete it. For example, a HIT that pays $1 for 5 minutes of work does not seem to be considered a “penny HIT”, but a $1 HIT that takes 10 or 20 minutes to complete would fall squarely into “penny HIT” category
If you have something to say about this, contact me