conference paper

Conference/ProceedingsACM Multimedia 2012 Workshop on Crowdsourcing for Multimedia
Start date29.10.2012
End date02.11.2012
OrganisationACM MM 2012
Author(s)Luke Gottlieb, Jaeyoung Choi, Gerald Friedland, Pascal Kelm, Thomas Sikora
TitlePushing the limits of Mechanical Turk; Qualifying the crowd for geolocation
AbstractIn this article we review the methods we have developed for
finding Mechanical Turk participants for the manual annota-
tion of the geo-location of random videos from the web. We
require high quality annotations for this project, as we are
attempting to establish a human baseline for future comparison to machine systems. This task is different from a
standard Mechanical Turk task in that it is difficult for both humans and machines, whereas a standard Mechanical Turk
task is usually easy for humans and difficult or impossible
for machines. This article discusses the varied difficulties we encountered while qualifying annotators and the steps that we took to select the individuals most likely to do well at our annotation task in the future.
Key wordscrowdsourcing, annotation, cheat detection, Mechanical Turk, qualification, multimodal