Blog #6: TurKit

Paper Title: TurKit: Human Computation Algorithms on Mechanical Turk

Authors: Greg Little, Lydia B. Chilton, Max Goldman, Robert C, Miller

Authors Bios:

Greg Little is a student at MIT. His personal website is glittle.org.

Lydia Chilton is a Computer Science grad student at University of Washington. She got undergraduate degrees in Economics and EECS and a Master’s degree in EECS from MIT.

Max Goldman is a graduate student at MIT CSAIL Labs. His areas of study are User Interfaces and Software Development.

Robert C. Miller is an Associate Professor in the EECS dept. at MIT. He is also the leader of the User Interface Design Group in the CSAIL Labs.

Presentation Venue: UIST '10 Proceedings of the 23nd annual ACM symposium on User interface software and technology


Summary:Paper Hypothesis: TurKit is a toolkit for prototyping and exploring algorithmic human computation. Mechanical Turk (MTurk) provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. 
General Summary: Amazon uses such a mechanical turk in which users are paid small amounts of money for completing the human intelligence tasks (HITS) like writing product reviews. However, various systems challenges make this difficult in practice, and most uses of MTurk post large numbers of independent tasks. TurKit provides an API written in JavaScript for accomplishing MTurk Tasks. TurKit is a toolkit for prototyping and exploring algorithmic human computation, while maintaining a straight-forward imperative programming style. The authors present the crash-and-rerun programming model that makes TurKit possible, along with a variety of applications for human computation algorithms. The authors also present case studies of TurKit used for real experiments across different fields. The crash-and-rerun model favors usability over efficiency, but does so at an inherent cost in scalability. So, one of the major limitations of this system is expensive space complexity.

Discussion:
Companies like Amazon.com, ebay, other e-commerce companies and companies that rely on customer ratings have been using mechanical turks for quite some time now. They have to stick to this procedure since these tasks require human intelligence and can not be achieved using computer algorithms. These systems have a drawback when it comes to scalability, but with modern processors and cheap storage solutions, that doesn't quite seem to be an issue.