We're making it easier for you to edit your taxonomy - and apply the changes you've made to your existing (or upcoming) tasks.
Instead of having to cancel existing tasks and create new ones from scratch that reflect your latest taxonomy, you can now make changes directly on your existing tasks. Follow these steps:
- Choose whether you want your taxonomy to apply to only newly created tasks, or existing tasks as well
- [If you chose existing tasks] Choose the specific batches you want to update with your new taxonomy
- [If you chose existing tasks] Choose whether you'd like to restart those tasks from scratch, or preserve the annotations that have already been added
To get started, head over to the Definition tab to edit your taxonomy
You need to be on the most up to date version of the taxonomy/instructions tool to use this feature
For any older projects you might have created, you will need to migrate to the new editor. All new projects created already utilize the new editor and do not require additional migration.
If you see a small popup in the bottom right hand corner, make sure you click on it to migrate to the new instructions editor. Otherwise, you will not have access to this new feature!
1 | Choose which tasks these changes should be applied to
Determine the types of tasks you want your taxonomy changes to affect
- Only new tasks: Only tasks created from this point onward will take on your new taxonomy. This is de facto how things work today
- If you choose this option, there are no more decisions to be made!
- New & unstarted tasks: In addition to new tasks, any tasks that have not yet been touched by an annotator will also take on your new taxonomy. Effectively, any tasks that have yet to be labeled will take on your new taxonomy
- If you choose this option, you will need to then choose which unlabeled tasks should get this change
- All tasks: Even completed tasks and those that have already started to be labeled will also get updated with the new taxonomy
- If you choose this option, you will need to choose not only which batches are impacted, but also how you want already labeled tasks to be handled
2 | Choose which tasks these changes should be applied to
If you chose "New & unstarted tasks" or "All tasks", you will then need to select which tasks to apply the new taxonomy to. You will make this choice at the batch level. All unstarted tasks within a selected batch will get the new taxonomy.
3 | Choose how you want already started tasks to be migrated
If you choose "All tasks", you will need to choose how you'd like to handle those tasks that already have annotations on them. For tasks that you have already finished labeling, or for tasks that annotators have started working on (e.g., finished the attempt, but waiting on review), you can persist the work that has already been done
- Review and update existing tasks
- This option would preserve any existing annotations that were already added to the task; but it would send the task back to an annotator to be validated, edited & resubmitted
- Annotators would receive a task in their queue that has labels corresponding to any taxonomy items that are still relevant. They would need to add any additional labels for newly added taxonomy items, and validate that other aspects of the response look correct before submitted
- For example, if I had a taxonomy of horse, cows, and pigs - and then I went and added "duck", my task of a barn scene will still have bounding boxes for any horses, cows, and pigs that were already added. But I would need to go and add additional bounding boxes for any ducks that might be in the picture
- For completed tasks:
- Tasks will get sent back to the latest stage in the pipeline
- For example, if I have a pipeline with one attempt and one review, my completed tasks will get sent back to the review layer
- For in progress tasks:
- Any actively in progress tasks would get restarted, but stay in the same pipeline layer
- For example, if I was in the middle of making edits as a reviewer, I would lose those edits I was making just now, but the task would still be in my possession
- Restart existing tasks from scratch
- This option would wipe any existing annotations that were previously added and it would completely start the task again from the beginning. That sends the task to the initial attempt layer and it is effectively the same as creating the task from scratch
- Update completed tasks without review
- This option will come up if you have only deleted labels from your taxomony
- Since nothing new is being added, we can automatically remove "excess" labels from your old taxonomy that are no longer needed, while preserving all annotations that are still relevant
A few things to keep in mind
- Taxonomy changes only work at the broadest level. Changes made to a label's specific settings will not get registered as a change
- Labels: If you change the settings of an individual label (e.g., min height, can rotate), those changes will not get registered & propagated. Only changing the name of a label will get registered as a change
- Attributes: If you have a category attribute, any labels you add / remove will not get registered as a change. Nor will any other specific settings that you would change within an attribute
- In these cases, we recommend deleting the existing labels and re-adding the labels with the settings you want
- We also recommend starting these tasks from scratch to be safe
- You must finish the update flow and save to kick off a migration. Once started, the migration will happen in the background and may take a few minutes to complete. During this time labelers will not be able to work on tasks in batches that are being migrated. You can check the notification bell icon to monitor the progress of your migration.
- We strongly recommend testing out your migration on a small set of tasks (1 batch) and confirming it works as you are expecting it to work, before applying changes to all tasks
Add a callout for your annotators & hit save!
To make sure your annotators are aware of the changes you have made, you can include a message for them. This message will pop up the next time they access a task.
If you have added or modified any taxonomy items, you can also updated your instructions to let your labelers know how they are expected to label it