How do Turkers search for tasks?

  • Read the paper (published at KDD-HCOMP Workshop June 2010)

Motivation: MTurk is a labor market and in order for a labor market to be efficient, workers have to sort through all the jobs and pick ones they want to do and employers have to find suitable workers out of all the possible workers available.  Unlike canonical market examples such as the corn market, or the stock market, all workers and jobs are unique, thus the matching problem requires special tools.  It’s really important for search to be efficient on MTurk.  For full-time employment, workers are willing to spend 6 months doing a job search.   For a 20-second task, workers probably aren’t willing to spend even an hour searching.  The future of online human computation depends on efficient worker-employer pairs.

Results: We published a paper studying how workers on MTurk currently.  MTurk has six ways workers can sort for tasks:

1. Title (e.g., “Choose the best category for this product”)

2. Requester (e.g., “Dolores Labs”)

3. HIT Expiration Date (e.g., “Jan 23, 2011 (38 weeks)”)

4. Time Allotted (e.g., “60 minutes”)

5. Reward (e.g., “$0.02″)

6. HITs Available (e.g., 17110)

7. Required quali cations (if any)

(Turkers can also search by keyword, but we didn’t study this).

We found strong evidence that Turkers sort by the most number of HITs available (so they can find one task, and then do 100 instances of them in a row) and the most recently posted HITS (so they get the latest and greatest HITs).

This second result is interesting.  It indicates that Turkers are interested in the newess of HITs.  New HITs are constantly coming in, so they can almost always find something interesting to do on the first N pages of the most recently posted HITs.  Also, it seems like Turkers enjoy getting things that are new – doing the newest tasks is more interesting than labeling images all day long.

Notably, Turkers don’t sort by the highest reward, we hypothesize that this is may be because:

1. high reward usually means a long task and Turkers either don’t like long tasks, or find it difficult to estimate how long they take, because Turkers presumably want to maximize their wage rate.

2. many high reward HITs are undesirable, and since you can’t delete items your not interested in from MTurk’s interface, it becomes very hard to revisit the “highest reward’ page to find new things, because you also see all the old things you didn’t want.

We feel that adding better search functionality to MTurk would substantially improve the service for both workers and requesters.

Employers! How to get your tasks done: If you’ve had trouble getting your tasks done, one strategy we saw some of the biggest requesters using is to use a script that constantly updates your HITs so that they stay on the first couple pages of most recently posted HITs.  You don’t actually have to change the HIT very much to be considered new again.

Abstract: In order to understand how a labor market for human computation functions, it is important to know how workers search for tasks. This paper uses two complementary methods to gain insight into how workers search for tasks on Mechanical Turk. First, we perform a high frequency scrape of 36 pages of search results and analyze it by looking at the rate of disappearance of tasks across key ways Mechanical Turk allows workers to sort tasks. Second, we present the results of a survey in which we paid workers for self-reported information about how they search for tasks. Our main findings are that on a large scale, workers sort by which tasks are most recently posted and which have the largest number of tasks available. Furthermore, we nd that workers look mostly at the rst page of the most recently posted tasks and the rst two pages of the tasks with the most available instances but in both categories the position on the result page is unimportant to workers. We observe that at least some employers try to manipulate the position of their task in the search results to exploit the tendency to search for recently posted tasks. On an individual level, we observed workers searching by almost all the possible categories and looking more than 10 pages deep. For a task we posted to Mechanical Turk, we con rmed that a favorable position in the search results do matter: our task with favorable positioning was completed 30 times faster and for less money than when its position was unfavorable.

You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

Comments are closed.