Counting 3,834 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1

Last Commit
Apr. 26, 2019
Sep. 30, 2018

demo_pic demo_pic

turkey: an Amazon Mechanical Turk turn-key segment tool.


turkey lets you easily create a web UI on Amazon Mechanical Turk to crowd-source image annotation data. Its main functions include:

  • Customize the annotation modes and class labels on per-image basis
  • Import previous annotations generated by either another human or an algorithm
  • Zoom-in, zoom-out, delete, undo, reset

I'm aware that Amazon has recently released their own image segmentation tool. There are 3 advantages of my tool compared to theirs:

  • It seems to be a semantic segmentation tool, which is different from instance segmentation. (Edit: Amazon recently added a Polygon tool that essentially serves as instance segmentation, kudos to them!)
  • Their tool also seemed a little buggy. When I tried it, only 1 of the 2 images got loaded correctly, and the annotation results in the downloaded csv was confusing (it looked somewhat like "B8gzS96nSL7Xnp0").
  • The most import thing is that their tool is encapsulated into their internal API, which we have no direct access to. That means if you want to change something or add something that was not included in the official example, you would have to write a new tool from scratch. My tool is written in plain JavaScript, with a little help from jQuery and bootstrap. Feel free to adapt it to your needs.

Code is tested on Chrome and Microsoft Edge. Development is still in progress. Ideas and suggestions are welcome!

Configuring Amazon Turk to use turkey

Copy the contents of Mturk.html into the source code section when you create a custom HIT in the Amazon Turk Requester. After the HIT is created, simply Publish Batch with a CSV file of both image URIs and annotation modes that you provide. A sample CSV file may look like:

img_url,annotation_mode,classes,annotations, dot-polygon-link, person-dog-house, [{"class":"house","mode":"polygon","data":[[85,450],[41,524],[96,581],[163,531]]},{"class":"dog","mode":"polygon","data":[[246,461],[203,489],[268,500]]}], polygon, cat-person-sky-food, []

Here, we allow the worker to use Polygon, Link and Dot modes for annotating the first image, but only the Polygon mode for the second. The class labels that the user can choose from are house, person, car, and dog. The first item in the list will be the default option. You can also import previous annotations by putting the correctly formatted json string into column "annotations".

Testing without Turk

You can test the code before deploying it on MTurk by opening localDemo.html in your browser. This file is a lightweight wrapper that will load MTurk.html off GitHub, passing a sample image to it in the process. If you don't see anything here, make sure to start Chrome with the --allow-file-access-from-files flag (or the equivalent configuration for the browser of your choice). This will allow this wrapper page to load MTurk.html.

Unpacking data from .csv file

After the users annotate the images, Amazon Mechanical Turk provides a .csv file ready for downloading. getResults.m contains a sample MATLAB UI that displays the downloaded annotations for review. The approval results will be written into a csv file to be uploaded to MTurk for batch processing. The MATLAB UI is shown below:

demo_pic demo_pic