Sending tasks back for relabeling
Identifying tasks with mistakes and sending them back for relabeling
The easiest way to identify tasks that need to be relabeled is through the View Tasks page.
- Click on an image to view annotations in better detail.
- If you come across a task that requires relabeling, click on the Relabel button on the top right corner of your screen.
- Select a reason for why this task needs to be relabeled and add a description to let our labelers know what went wrong and how they should fix the mistake.
Relabeling period
Any tasks sent back for relabeling will be sent to the initial labeler for relabeling. However, if the initial labeler does not relabel the task in 3 hours, another labeler will be assiged to fix any issues on the task. If a relabeled task is sent back for another round of relabeling, the task will be assigned to a new labeler, and the initial labeler will not receive any payment.
Tasks can be sent for relabeling within 72 hours of the submission of a task. If a task is not sent for relabeling within 72 hours, it will be deemed as accepted and labelers will receive payment for any work done. The 72 hour period resets after each submission, including the submission of relabeled tasks.
Your feedback helps our labelers improve
To help ensure that mistakes are relabeled correctly, it is important to be as detailed as possible when writing a description of why a task was sent back for relabeling. Your feedback will be displayed to our labelers to help them identify the mistake and fix it accordingly.
Your feedback will also be a source of reference for labelers to improve subsequent iterations of your project. Through continuous iteration, you can ensure that your data labeling tasks are accurate and up to date, which is essential for the success of your project.
Here are some suggestions on how to give feedback:
Mislabeled annotations
| Issue | Reason | Description |
|---|---|---|
Green tomato has been mistakenly labeled with the "Rotten" label. | Mislabeled | The tomato labeled as "Rotten" should be labeled as "Green" instead. Rotten tomatoes are generally red in colour and usually have spots on their surfaces. Please refer to page 3 of the instructions for more image examples of rotten tomatoes. |
Bounding box does not cover the entire object of interest / contains too much empty space. | Bad overlap | The "Red" bounding box is not drawn tightly over the tomato. Bounding boxes should be drawn over the entire visible tomato as tightly as possible. Please refer to page 2 of the instructions for examples of tightly drawn bounding boxes. |
Object is labeled as "Red" even when there no visual cues to support that it is a tomato.  | Unwanted annotation(s) | Objects should not be labeled if they cannot be identified properly. You should only annotate a tomato if you can see a stem or sepal connected to it. |
No annotations drawn on task. | Other: Task not annotated. | The tomatoes in the image are not labeled. All tomatoes need to be annotated with the correct label. |
Charges for relabeling tasks
We charge based on the final number of annotations after the task has gone through relabeling. You will not incur any additional charges for existing annotations that require modifications.
Final number of annotations =
Initial number of annotations sent for relabeling – Number of removed annotations (false positives) + Number of new annotations (false negatives)
Here are a few scenarios to further illustrate how charging works:
| Scenario | Final number of annotations charged |
|---|---|
| - 10 bounding boxes sent for relabeling - Labeler edits 3 mislabeled annotations | 10 bounding boxes |
| - 10 bounding boxes sent for relabeling - Labeler removes 2 unwanted annotations | 8 bounding boxes |
| - 10 bounding boxes sent for relabeling - Labeler adds 1 missing annotation | 11 bounding boxes |
| - 10 bounding boxes sent for relabeling - Labeler edits 3 mislabeled annotations - Labeler adds 1 missing annotation - Labeler removes 2 unwanted annotations | 9 bounding boxes |
You can find more details on our rates here.
Updated over 1 year ago