🕷️ Crawler Inspector

  [Login](https://app.ubiai.tools/login) [Get a Demo](https://apiv2.ubiai.tools/widget/booking/rQdEzYzc0Mo3sA6EAeJt)  # Build An NLP Project From Zero To Hero (5): Model Training ###### Jan 18, 2022 Training an ML model is without a doubt the most interesting part for every data scientist and for every machine learning enthusiast. Model training refers simply to the model learning from its input data to generalize over a given phenomenon. With every training iteration, the model adjusts its weights to be able to make correct predictions as much as possible using a training algorithm like gradient descent. There are a lot of details that concern this phase: selecting the model, verifying the integrity of the input data, evaluating the model, training, and saving it. We will get to every detail and then we will show how we apply each one. ## Model Selection In general, since we have observed and have prepared our data through preprocessing and labeling, we should have a good idea of what model we will be using. Usually, there will be a list of models to choose from, and this will make the project far more complicated than it should be. A good intuition is to try the *simplest model* for your task and then proceed to improve the architecture of the existing model or to choose a more complex model that is compatible with the same task. However, the simplest model can fail from the start. So, you can identify certain characteristics and properties that will help you reduce the candidate model list. A very good example of this is from the Google Developers Guide of [Text Classification](https://developers.google.com/machine-learning/guides/text-classification/step-2-5). Through a lot of experimentation and testing, they identified a metric *S/W* or the number of samples/number of words per sample ratio. This metric will indicate wether you should choose **n-gram models** like logistic regression and support vector machines or **sequence models** like CNN or RNN for the text classification task. In practice, this is difficult to achieve on your own as you need to do a lot of experimentation and testing. This is why you should research industry-standard models. There are also the characteristics of the data itself that would help map to the correct model: [If your data has a large number of features but a significantly lower number of observations](https://medium.com/axum-labs/logistic-regression-vs-support-vector-machines-svm-c335610a3d16#:~:text=When%20To%20Use%20Logistic%20Regression%20vs%20Support%20Vector%20Machine), a support vector machine will perform better than logistic regression. For Named Entity Recognition, do you want to train a new model from scratch? Or use a pre-trained model and probably build upon it? For example, you can use a Spacy pre-trained NER model but it might not suit your need as in the case of this project. Or, you can train a Spacy Model from scratch using your own dataset, giving new vocabulary and labels for the model. This [article](https://medium.com/@b.terryjack/nlp-pretrained-named-entity-recognition-7caa5cd28d7b#:~:text=There%20are%20a%20good%20range,)%20API%20(e.g.%20GATE).) presents a great overview of pre-trained NER models, ranging from rule-based models like in NLTK to probabilistic models like Stanford Core NLP and Deep Learning Models like Flair. In the [Data Preprocessing Phase](https://medium.com/ubiai-nlp/build-an-nlp-project-from-zero-to-hero-3-data-preprocessing-9a7ef729d05?source=your_stories_page----------------------------------------), we have used the [Spacy pre-trained NER model](https://spacy.io/universe/project/video-spacys-ner-model) during the pre-annotation of the dataset. The model uses a sophisticated word embedding strategy using subword features and “Bloom” embeddings, a deep convolutional neural network with residual connections. Besides, we have trained a spacy-based model in the last article, using the Model Assisted Labeling feature within the [UBIAI tool](https://ubiai.tools/). The performance was not too bad as a start. We can download it and use it like any other spacy model by clicking the Download Button in the Action Column in the Models Tab of our current project:  UBIAI Model Training Dashboard For this article, we will feature the workflow for training two models: a **Probabilistic** Model, **CRF** or **Continuous Random Fields**, and a **Deep Learning** Model, **Spacy NER** model with **Transformers**. But before that, we need to talk about the Input Data Format and Model Evaluation. ## Input Data Format To assure that your model will actually work, you must identify clearly the format of its Input Data. This is necessary for both prediction and training. There exist many suitable formats for the NER task and among them: 1. **IOB format**: short for inside, outside, the beginning is **a common tagging format for tagging tokens in a chunking task** in NLP. Every document is separated into tokens. Each token will take a row and in front of every token, you will find its label. The Null label or ‘O’ is necessary in this case to mark unlabeled tokens. Since the labeling is practically word by word, there is an additional technique to label multiple token terms, I-notation (Inside of a labeled term) and B-notation (Beginning of a labeled term). Documents are separated between each other by a special separator (in our case ‘-DOCSTART- -X- O O’). 2. **JSON format**: In this format, your dataset is a list of JSON objects. Each object represents a document and a list of its annotations. An annotation is a labeled term represented by a dictionary containing its text, its label, its starting position, and its ending index in the text in the document string.   IOB left, Spacy JSON right If you recall, we have already used the JSON format with Spacy previously in the pre-annotation and the Data Labeling Process. In the UBIAI tool, you just need to open the Project list menu and click on the Download Button in the Actions’ Column. There is an important point to talk about, splitting your dataset into a training set, development (or validation set), and a test set. We can omit the dev set to simplify things as we are in a learning project. A ratio of 80/20 is good for our small dataset. ## Model Evaluation Since it is a classification task, we might begin with **Accuracy**. Accuracy is good if your dataset is balanced (every label has the same number of instances as everyone else). This is not our case. Traditionally, these three metrics are considered for the NER task: - **Precision**: Determines if your model predicts a real incorrect label as correct. In other words, your model predicts that ‘Google’ is a PERSON name while it is not correct in reality. The higher this metric is, the lesser your model makes this mistake. - **Recall**: Determines if your model predicts a real correct label as incorrect. For example, your model does not predict ‘Google’ as COMPANY even though it is in reality. The higher this metric is, the lesser your model will miss correct instances. - **F1-Score**: an overall indicator of the performance of the classifier that takes into account both Precision and Recall. You noticed that I explained these metrics in terms of intuition. To delve more theoretically, begin by checking out this [article](https://medium.com/analytics-vidhya/confusion-matrix-accuracy-precision-recall-f1-score-ade299cf63cd#:~:text=Recall%20is%20also%20known%20as,high) by [Harikrishnan](https://medium.com/@harikrishnannb). In the next two parts, I am supposing that you have a basic understanding of how a Machine Learning model train and what constitute a Model architecture. If you want to delve deeper into this topic, I recommend the [Coursera Deep Learning Specialization](https://www.coursera.org/specializations/deep-learning) by [Andrew Ng](https://twitter.com/andrewyng?lang=en). ## Training a CRF Model We have a sequence of tokens in every training example. These tokens are usually words if you decided to tokenize at the word level. We have talked about tokenization extensively in this episode. To predict the nature of a word (is it a PERSON, a COMPANY, etc), we cannot ignore the sequential nature of our data (tweets, sentences) as it is a significant loss of information. We have to select a model that can infer from previous positions for the prediction of the current position. Named Entity Recognition is of sequential nature after all. For example, using the IOB format and given the tweet “Facebook has a target price of \$10”, the labeling might be “Facebook (B-COMPANY) has (O) a (O) target (O) target (B-MONEY\_LABEL) price (I-MONEY\_LABEL) of (O) \$10 (MONEY)”. So to predict the word ‘price’ as an Inside MONEY\_LABEL, we need to know features of the previous word ‘target’. Knowing that it has the label Beginning MONEY\_LABEL, its features would serve very well in making a correct prediction for ‘price’. 1. **Model Architecture :** 2. CRF or Continuous Random Fields builds upon this intuition by building feature functions that take into account the sentence and arbitrary labels throughout it. As a simple example, let us consider this feature function input 3. Sentence **s** 4. The position **i** of a **word** in the sentence 5. The label **l(i)** of the current word 6. The label **l(i−1)** of the previous word 7. This feature function outputs a real-valued number which is usually binary. It is called **linear-chain CRF.** We then **\*\*assign each feature function a** weight **that is to be learned by the model. Lastly, we transform these functions into** probabilities\*\* by summing over every sentence for every feature function and by subsequent exponentiation and normalization.  Generic CRF Model I hope that you have an intuition of how CRF works. For more details, check out these two great articles by [Analyticsvidhya](https://www.analyticsvidhya.com/blog/2018/08/nlp-guide-conditional-random-fields-text-classification/#h2_5) and [Edwin Chen](https://blog.echen.me/2012/01/03/introduction-to-conditional-random-fields/). ## Workflow I am using Google Colaboratory as the working environment and Google Drive as where to store our data and our models. ``` ``` `` `` `` `` `` ``` Now, let us transform our data into useful features by collecting details about every token and its adjacent neighbors. You notice that I am commenting out part of speech tagging. You can include it if you have a Part Of Speech Model which can be a part of the feature engineering pipeline. ``` `` `` `` `` ``