Use Custom NER with continuous learning

Background information

This example is used to extract chemicals by the category mentioned in research papers. By following the procedure below you will extract the chemicals and categorize them as ABBREVIATION, FAMILY, FORMULA, IDENTIFIER, MULTIPLE, SYSTEMATIC, TRIVIAL and NO_CLASS.

Recommendations

When to use the Custom Named Entity Recognition (NER) model

Use the Custom NER model to extract:

• special information from the text. This information is called entity.

• the names of people, places, organisations, locations, dates, numerical values, etc. The extracted entities are mutually exclusive. Entities are at single or multi-word level, not at the sub-word level. For example, in the I live in New York sentence, an entity can be New York but not in the I read the New Yorker sentence.

You can use the extracted entities directly in the information extraction processes or as inputs to the downstream tasks like classification of the source text, sentiment analysis of the source text, PHI, etc.

Training dataset recommendations

• Have at least 200 samples per entity if the entities are dense in the samples, meaning that most of the samples (more then 75%) contain 3-5 of these entities.
• If the entities are sparse (every sample has less then three entities) i.e., only a few of all the entities appear in most of the documents, then it is recommended to have at least 400 samples per entity. This helps the model to better understand the discriminative features.
• If there are more than 10 entities, add 100 more samples in an incremental way until you reach the desired performance metric.

Best practices

1. Have meaningful entities; if a human cannot identify an entity, neither can a model.
2. Have simple entities. Instead of a single entity address, break it down into multiple entities: street name, state name, city name, or zip code etc.
3. Create both train and test datasets, and use a full pipeline for training.
4. Start with a minimum number of samples for annotation, covering all the entities.
5. Make sure all the entities are represented in both the train and test split.
6. Run a full pipeline and check the test metrics. If the test metric is not satisfactory, check the classification report and identify the ill-performing entities. Add more samples that cover the ill-performing entities and repeat the training process, until the desired metric is reached.

Prerequisites

This procedure uses the Custom Named Entity Recognition package. For more information on how this package works and what it can be used for, see Custom Named Entity Recognition.

For this procedure, we have provided sample files as follows:

• Pre-labeled training dataset in CoNLL format. You can download it from here.
• Pre-labeled test dataset. You can download it from here.
• Sample workflow for extracting categories of chemicals mentioned in research papers. You can download it from here.
  Note: Make sure that the following variables are filled in in the sample file:

  • in_emailAdress - the email address to which the Action Center task will be assigned to
  • in_MLSkillEndpoint - public endpoint of the ML Skill
  • in_MLSkillAPIKey - API key of the ML Skill
  • in_labelStudioEndpoint - optional, to enable continuous labeling: provide import URL of a label studio project

Procedure

Follow the procedure below to extract chemicals by their category from research papers.

1. Import the sample dataset in UiPath® AI Center. To do so, go to the Datasets menu and upload the train and test folder from the sample.
2. Select the desired Custom NER package from ML Packages > Out of the Box Packages > UiPath Language Analysis and create it.
3. Go to the Pipelines menu and create a new full pipeline run for the package created in the previous step. Point to the training and the test dataset provided in the sample file.

   Once complete, the results of the pipeline are available under Artifacts. You can download the classification report to evaluate the test results.

4. Create a new ML Skill using the package generated by the pipeline run from the previous step and deploy it.
5. Once the skill is deployed, leverage it in the UiPath® Studio workflow provided. To enable capturing data with weak predictions, deploy a Label Studio instance and provide the instance URL and API key in the Label Studio activity in the workflow.

Getting started with Label Studio

To get started with Label Studio and export data to AI Center, follow the instructions below.

1. Install Label Studio on your local machine or cloud instance. To do so, follow the instructions from here.
2. Create a new project from the Named Entity Recognition Template and define your Label Names.
   
   

   

   
   
3. Make sure that the label names have no special characters or spaces. For example, instead of Set Date, use SetDate.
4. Make sure that the value of the <Text> tag is "$text".
   
   

   

   
   
5. Upload the data using the API from here.

   cURL request example:

   curl --location --request POST 'https://<label-studio-instance>/api/projects/<id>/import' \)\)
   --header 'Content-Type: application/json' \)\)
   --header 'Authorization: Token <Token>' \)\)
   --data-raw '[
       {
         "data": {
           "text": "<Text1>"
         },
       },
       {
         "data": {
           "text": "<Text2>"
          }
       }
   ]'

6. Annotate your data.
7. Export the data in CoNLL 2003 format and upload it to AI Center.
8. Provided the Label Studio instance URL and API key in the provided sample workflow in order to capture incorrect and low confidence predictions.