Understanding data requirements

Recommendations for lower data volumes

The following recommendations concern use cases with lower data volume, but high value and/or low complexity.

Generally, use cases should function as expected if their complexity aligns with the volume of message data. Very low volume use cases should typically be very simple, while high volume use cases can be more complex.

In some instances, synchronizing more than one year's worth of historical data can help in sourcing sufficient quality examples for training. This also provides the benefit of greater analytics in terms of trends and alerts.

Use cases with fewer than 20,000 messages (in terms of historical volumes or annual throughput) should be carefully considered in terms of complexity, ROI, and the effort required to support and enable the use case. While there is a chance that such use cases may be disqualified based on these considerations, they can still provide sufficient business value to proceed with.

Use case complexity guidelines

Every use case is unique, so there isnot a single guideline that fits all complexity scenarios. The labels and fields themselves can range from very simple to complex in terms of understanding and extraction.

The following table outlines rough guidelines for use case complexity.

Table 1. Use case complexity guidelines

Complexity	Labels	Extraction Fields	General Fields
Very Low	~ 2-5	N/A	1 - 2
Low	~ 5 - 15	1 - 2 for a few labels	1 - 3
Medium	15 - 50	1 - 5 for multiple labels	1 - 5 *
High	50+	1 - 8+ for high proportion of labels	1 - 5 *

* Use cases with extraction fields should rely on these rather than general fields. If you are not using extraction fields, you can expect more general fields, but they may not add equivalent value.

Summary

The following summary table outlines rough guidelines for low-data use cases:

Table 2.

# of Messages *	Limitations	Recommendation
Less than 2048	No clusters and label suggestions Insufficient data for some analytics to be meaningful Likely to be minimal ROI	Should only be: Testing
2048 - 20,000	Likely to have more limited value from active learning due to lower volumes Likely to be insufficient data to support complex use cases More likely to have low ROI	Should primarily be: POCs Low complexity use cases AI Centre migration use cases
20,000 - 50,000	Potentially insufficient data for very complex use cases (at least some complex fields/labels) Potentially lower ROI depending on complexity	Should primarily be: Low-Medium complexity use cases Some High complexity use cases AI Center migration use cases

Historical data volumes from which training examples will be sourced typically have only a small proportion of total volumes annotated. This proportion is usually higher on lower volume and higher complexity use cases.