Validation

The Validation page shows users detailed information on the performance of their model, for both labels and general fields.

In the 'Labels' tab users can see their overall label Model Rating, including a detailed breakdown of the factors that make up their rating, and other metrics on their dataset and the performance of individual labels.

In the General fields tab, users can see statistics on the performance of general field predictions for all of the general fields enabled in the dataset.

Default Validation page for 'Labels'

The 'Model Version' dropdown, located above the model rating, lets you see all validation scores across past model versions on a given dataset. You can also prioritise or 'star' individual ones so that they appear at the top of the list in future. This tool can be useful for tracking and comparing progress as you build out your model.

The model version dropdown

Labels

The 'Factors' tab (as shown above) shows:

• The four key factors that contribute to the Model Rating: balance, coverage, average label performance, and the performance of the worst performing labels
• For each factor it provides a score, and a breakdown of the contributors to the factor's score
• Clickable recommended next best actions to improve the score of each factor

The 'Metrics' tab (as shown below) shows:

• The training set size – i.e. the number of messages on which the model was trained
• The test set size – i.e. the number of messages on which the model was evaluated
• Number of labels – i.e. the total number of labels in your taxonomy
• Mean precision at recall – a graph showing the average precision at a given recall value across all labels
• Mean average precision – a statistic showing the average precision across all labels
• A chart showing, across all labels, the average precision per label vs. training set size

Metrics tab within 'Labels' validation

The Validation page also allows users to select individual labels from their taxonomy to drill-down into their performance.

After selecting a label, users can see the average precision for that label, as well as the precision vs. recall for that label based on a given confidence threshold (which users can adjust themselves).

Label specific validation charts

To understand more about how Validation for labels actually works and how to use it, see here.

General Fields

Validation page for General Fields

The General Fields tab (as shown above) shows:

• The number of general fields in the train set – i.e. the number of annotated general fields on which the validation model was trained
• The number of general fields in the test set – i.e. the number of annotated general fields on which the validation model was evaluated
• The number of messages in the train set – i.e. the number of messages that have annotated general fields in the train set
• The number of messages in the test set – i.e. the number of messages that have annotated general fields in the test set
• Average precision - the average precision score across all general fields
• Average recall - the average recall score across all general fields
• Average F1 score - the average F1 score across all general fields (the F1 score is the harmonic mean of precision and recall, and weights them equally)
• The same statistics but for each individual general field

To understand more about how Validation for general fields actually works and how to use it, see here.