conference paper

Start date01.01.2013
End date31.01.2013
PublisherIEEE Communications Society
VolumeVol. 8, No. 1, January 2013
Author(s)Luke Gottlieb, Jaeyoung Choi, Gerald Friedland, Pascal Kelm , Thomas Sikora
TitleOn Pushing the Limits of Mechanical Turk: Qualifying the Crowd for Video Geolocation
AbstractThis work was first appeared in Gottlieb et al. [1]. In this article we summarize the methods we took for finding skilled Mechanical Turk participants for our annotation task, which will be to determine the geolocation of random videos from the web. The task itself is unlike the standard setup for a Mechanical Turk task, in that it is difficult for both humans and machines, whereas a standard Mechanical Turk task is usually easy for humans and difficult or impossible for
machines. There are several notable challenges to finding skilled workers for this task: First, we must find what we termed “honest operators”, i.e., people who will seriously attempt to do the task and not just click quickly through it to collect the bounty. Second, we need to develop meaningful qualification test set(s) that are challenging enough to allow us to qualify people for the real task, but were also solvable by individuals regardless of their culture or location, although English language understanding was required for instructions.
Key wordsmultimedia analysis, Geotagging, Mechanical Turk, Crowdsourcing
NoteVolume 8, Number 1 - January 2013
EMERGING TOPICS: SPECIAL ISSUE ON "Multimedia and Cloud Computing"
INDUSTRIAL COLUMN: SPECIAL ISSUE ON "Crowdsourcing-based Multimedia Systems"