Archive for the ‘Uncategorized’ Category

How do Turkers search for tasks?

Read the paper (published at KDD-HCOMP Workshop June 2010)

Motivation: MTurk is a labor market and in order for a labor market to be efficient, workers have to sort through all the jobs and pick ones they want to do and employers have to find suitable workers out of all the possible workers available.  Unlike canonical [...]

Read the rest of this entry »

Dr. Suessify the News

Can we eke creativity out of Mechanical Turk? Clearly, lots of Turkers try to get away with as little thought as possible. But, if we frame the task correctly, we can get some very clever outcomes.
Here’s another stake in the ground: reinterpreting content “in the style of” an artist or writer. Let’s take as input [...]

Read the rest of this entry »

Sorites Paradox on Mechanical Turk

Sorites Paradox is something like this: Is this tile red? Sure. What about this tile ? No, it looks orange. Would you say that two sufficiently similar tiles are the same color? I suppose so, if they were so similar that I couldn’t tell them apart (if you can tell these particular [...]

Read the rest of this entry »

What Would You Turk for a Klondike Bar?

This is a guest post by Michael Bernstein, a computer science PhD student in our research group.
In all the discussion galvanizing Mechanical Turk workers to pursue worthwhile activities, we’ve overlooked one of the basic human social needs: humor. In a moment of fleeting fancy, I decided to ask 50 Mechanical Turk workers, “What would you [...]

Read the rest of this entry »

Trying to get 1000 people to pick a number from 1 to 10.

I said before that I had difficulty getting 500 people to do a task, but I had forgotten the details of whatever experiment that was, so I ran a fresh experiment. Turkers were offered 1 cent to pick a number from 1 to 10. I wanted 1000 people to do the task.
This chart shows the [...]

Read the rest of this entry »

The effect of instructions on a proofreading task

Total Cost: $0.94
15 iterative HITs
TurKit code for this experiment: proofread.js
TurKit Version: 0.1.42

Inspired by previous posts about proofreading research papers (1, 2), I used a proofreading task to explore how iterative tasks on MTurk converge or diverge.  There was an interesting effect caused by a small change in instructions.
The HITs used a paragraph drawn from one [...]

Read the rest of this entry »

Writing an Essay on MTurk: Part 3

In part 2, we had turkers vote on the best topic ideas. The top two ideas were:

How effective is recycling in saving the environment?
How important is a college education in todays job market?

I originally intended for the essay to be persuasive, so I wanted the topics to be statements rather than questions. Here is a [...]

Read the rest of this entry »

Writing an Essay on MTurk: Part 2

In part 1, turkers brainstormed some essay topic ideas. Now we want to choose the best topic. We’ve had turkers rate and compare items in the past. For this experiment, we tried something a little different: each turker saw all 18 essay topics, and selected the best 3.
We solicited 100 responses for 2 cents each. [...]

Read the rest of this entry »

Rating items on a scale of 1 to 10.

We have run a number of experiments that involve asking turkers to rate items on a scale of 1 to 10. This post explores data obtained from three prior experiments, including ratings of image descriptions, company name ideas, and photos.
Here is a histogram of all 7350 ratings obtained, from 314 different turkers:

The distribution looks somewhat [...]

Read the rest of this entry »

Turkers as Proofreaders – Pilot Results

For background, see my previous post.
The Turker proofreaders were helpful in place, e.g.:
line 28 – spelling mistake – should be “transcribing” not transcribe
line 33 – grammatical error – word “trivially” is not needed
line 38 – spelling mistake – “behvioral” should be “behavioral”
line 38 – spelling mistake – “idiosyncracies” should be “idiosyncrasies”
line 41 – spelling [...]

Read the rest of this entry »