Using Data Labeling with Human-in-the-Loop

Data Labeling enables you to upload raw data, annotate text data in the labeling tool (for classification or entity recognition), and use the labeled data to train ML models. Apart from this, you can use data labeling for human validation on model outputs.

A common scenario is when you train an extractor or classifier model. When the model prediction falls below a set confidence threshold, that data can be sent to Action Center for human validation. The validated data can be used to retrain the model in order to improve confidence on subsequent model predictions.

Enabling Human Validation

1. Use the Wait for External Task and Resume activity to create a task in Action Center from Studio.
2. Use the Create Labeling Task and Create External Task activities to convert the model output into a format compatible with Data Labeling.
   Data Labeling supports files using the JSON format. The JSON file should contain a data object that in turn contains the structure configured in the previous step.
3. Send the task to a human to review.
   Once a human has reviewed and completed the task, Task Object is updated with the output of the human review.
4. Convert Task Object into a format that the models can use as training data.
5. Send the validated data to an AI Center dataset as training data using the Upload File activity.
6. Start a pipeline run using the uploaded dataset.

The sample workflow completes the entire human-in-the-loop process going through the following steps: Raw data > Data Labeling > ML model training > Deploy ML Skill > Human-in-the-loop trigger on low confidence predictions > Retrain model with validated data to improve model performance.

You can use this sample workflow to test the human-in-the-loop sequences.