🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 103 (from laksa196)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa103.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://www.geeksforgeeks.org/nlp/building-language-models-in-nlp/\')), getAhrefsUnparsedNoserviceFromURL(\'https://www.geeksforgeeks.org/nlp/building-language-models-in-nlp/\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/www.geeksforgeeks.org\/nlp\/building-language-models-in-nlp\/","crawl_time":1775857555,"first_indexed_time":1750093343,"http_code":200,"src_unparsed":"org,geeksforgeeks!www,\/nlp\/building-language-models-in-nlp\/ s443","src_root_hash":"12046344915360636903","history_drop_reason":null,"meta_title":"Building Language Models in NLP - GeeksforGeeks","meta_descriptions":["Your All-in-One Learning Portal: GeeksforGeeks is a comprehensive educational platform that empowers learners across domains-spanning computer science and programming, school education, upskilling, commerce, software tools, competitive exams, and more.","Your All-in-One Learning Portal. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice\/competitive programming\/company interview Questions."],"attrs_boilerpipe_text":"Last Updated :\n23 Jul, 2025\nBuilding language models is a fundamental task in natural language processing (NLP) that involves creating computational models capable of predicting the next word in a sequence of words. These models are essential for various NLP applications, such as machine translation, speech recognition, and text generation.\nIn this article, we will build a language model using NLP using LSTM.\nWhat is a Language Model?\nA language model is a statistical model that is used to predict the probability of a sequence of words.\nIt learns the structure and patterns of a language from a given text corpus and can be used to generate new text that is similar to the original text.\nLanguage models are a fundamental component of many natural language processing (\nNLP\n) tasks, such as machine translation, speech recognition, and text generation.\nSteps to Build a Language Model in NLP\nHere, we will implement these steps to build a language model in NLP.\nStep 1: Importing Necessary Libraries\nWe will, at first, import all the necessary libraries required for building our model.\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Embedding, LSTM, Dense\nfrom tensorflow.keras.models import Sequential\nStep 2: Generate Sample Data\nWe will at first take a sample text data.\ntext_data = \"Hello, how are you? I am doing well. Thank you for asking.\"\nStep 3: Preprocessing the Data\nThe preprocessing involves tokenizing the input text data, creates input sequences, and pads the sequences to make them equal in length.\n# Tokenize the text\ntokenizer = tf.keras.preprocessing.text.Tokenizer()\ntokenizer.fit_on_texts([text_data])\ntotal_words = len(tokenizer.word_index) + 1\n\n# Create input sequences and labels\ninput_sequences = []\nfor line in text_data.split('.'):\n    token_list = tokenizer.texts_to_sequences([line])[0]\n    for i in range(1, len(token_list)):\n        n_gram_sequence = token_list[:i+1]\n        input_sequences.append(n_gram_sequence)\n\n# Pad sequences for equal length\nmax_sequence_len = max([len(x) for x in input_sequences])\ninput_sequences = tf.keras.preprocessing.sequence.pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre')\nStep 4: One hot encoding\nThe input sequences are split into predictors (xs) and labels (ys). The labels are converted to one-hot encoding.\n# Create predictors and label\nxs, labels = input_sequences[:,:-1],input_sequences[:,-1]\n\n# Convert labels to one-hot encoding\nys = tf.keras.utils.to_categorical(labels, num_classes=total_words)\nStep 5: Defining and Compiling the Model\nThis code defines and compiles a simple\nLSTM\n-based language model using Keras\n# Define the model\nmodel = Sequential()\nmodel.add(Embedding(total_words, 64, input_length=max_sequence_len-1))\nmodel.add(LSTM(100))\nmodel.add(Dense(total_words, activation='softmax'))\n\n# Compile the model\nmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\n# Fit the model\nhistory = model.fit(xs, ys, epochs=100, verbose=1)\nStep 6: Generating Text\nThis\ngenerate_text\nfunction takes a\nseed_text\nas input and generates\nnext_words\nnumber of words using the provided\nmodel\nand\nmax_sequence_len\n.\ndef generate_text(seed_text, next_words, model, max_sequence_len):\n    for _ in range(next_words):\n        token_list = tokenizer.texts_to_sequences([seed_text])[0]\n        token_list = tf.keras.preprocessing.sequence.pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n        predicted_probs = model.predict(token_list, verbose=0)[0]\n        predicted_index = tf.argmax(predicted_probs, axis=-1).numpy()\n        output_word = \"\"\n        for word, index in tokenizer.word_index.items():\n            if index == predicted_index:\n                output_word = word\n                break\n        seed_text += \" \" + output_word\n    return seed_text\n\n# Generate text\nprint(generate_text(\"how\", 5, model, max_sequence_len))\nBelow is the complete Implemention:\nimport\ntensorflow\nas\ntf\nfrom\ntensorflow.keras.layers\nimport\nEmbedding\n,\nLSTM\n,\nDense\nfrom\ntensorflow.keras.models\nimport\nSequential\n# Sample text data\ntext_data\n=\n\"Hello, how are you? I am doing well. Thank you for asking.\"\n# Tokenize the text\ntokenizer\n=\ntf\n.\nkeras\n.\npreprocessing\n.\ntext\n.\nTokenizer\n()\ntokenizer\n.\nfit_on_texts\n([\ntext_data\n])\ntotal_words\n=\nlen\n(\ntokenizer\n.\nword_index\n)\n+\n1\n# Create input sequences and labels\ninput_sequences\n=\n[]\nfor\nline\nin\ntext_data\n.\nsplit\n(\n'.'\n):\ntoken_list\n=\ntokenizer\n.\ntexts_to_sequences\n([\nline\n])[\n0\n]\nfor\ni\nin\nrange\n(\n1\n,\nlen\n(\ntoken_list\n)):\nn_gram_sequence\n=\ntoken_list\n[:\ni\n+\n1\n]\ninput_sequences\n.\nappend\n(\nn_gram_sequence\n)\n# Pad sequences for equal length\nmax_sequence_len\n=\nmax\n([\nlen\n(\nx\n)\nfor\nx\nin\ninput_sequences\n])\ninput_sequences\n=\ntf\n.\nkeras\n.\npreprocessing\n.\nsequence\n.\npad_sequences\n(\ninput_sequences\n,\nmaxlen\n=\nmax_sequence_len\n,\npadding\n=\n'pre'\n)\n# Create predictors and label\nxs\n,\nlabels\n=\ninput_sequences\n[:,:\n-\n1\n],\ninput_sequences\n[:,\n-\n1\n]\n# Convert labels to one-hot encoding\nys\n=\ntf\n.\nkeras\n.\nutils\n.\nto_categorical\n(\nlabels\n,\nnum_classes\n=\ntotal_words\n)\n# Define the model\nmodel\n=\nSequential\n()\nmodel\n.\nadd\n(\nEmbedding\n(\ntotal_words\n,\n64\n,\ninput_length\n=\nmax_sequence_len\n-\n1\n))\nmodel\n.\nadd\n(\nLSTM\n(\n100\n))\nmodel\n.\nadd\n(\nDense\n(\ntotal_words\n,\nactivation\n=\n'softmax'\n))\n# Compile the model\nmodel\n.\ncompile\n(\nloss\n=\n'categorical_crossentropy'\n,\noptimizer\n=\n'adam'\n,\nmetrics\n=\n[\n'accuracy'\n])\n# Fit the model\nhistory\n=\nmodel\n.\nfit\n(\nxs\n,\nys\n,\nepochs\n=\n100\n,\nverbose\n=\n1\n)\ndef\ngenerate_text\n(\nseed_text\n,\nnext_words\n,\nmodel\n,\nmax_sequence_len\n):\nfor\n_\nin\nrange\n(\nnext_words\n):\ntoken_list\n=\ntokenizer\n.\ntexts_to_sequences\n([\nseed_text\n])[\n0\n]\ntoken_list\n=\ntf\n.\nkeras\n.\npreprocessing\n.\nsequence\n.\npad_sequences\n([\ntoken_list\n],\nmaxlen\n=\nmax_sequence_len\n-\n1\n,\npadding\n=\n'pre'\n)\npredicted_probs\n=\nmodel\n.\npredict\n(\ntoken_list\n,\nverbose\n=\n0\n)[\n0\n]\npredicted_index\n=\ntf\n.\nargmax\n(\npredicted_probs\n,\naxis\n=-\n1\n)\n.\nnumpy\n()\noutput_word\n=\n\"\"\nfor\nword\n,\nindex\nin\ntokenizer\n.\nword_index\n.\nitems\n():\nif\nindex\n==\npredicted_index\n:\noutput_word\n=\nword\nbreak\nseed_text\n+=\n\" \"\n+\noutput_word\nreturn\nseed_text\n# Generate text\nprint\n(\ngenerate_text\n(\n\"how\"\n,\n5\n,\nmodel\n,\nmax_sequence_len\n))\nOutput:\nhow are you i am doing\nIn summary, constructing language models for natural language processing (NLP) include various stages, including tokenization, sequence creation, model construction, training, and text generation. Tokenization transforms textual data into numerical representations, while sequence creation generates input-output pairs for model training. The model typically comprises layers like Embedding and LSTM, followed by a Dense layer for predictions. Training involves fitting the model to input sequences and their labels, while text generation utilizes the trained model to generate new text based on a provided seed text. Overall, language models are vital for NLP tasks such as text generation, machine translation, and sentiment analysis, among others.","attrs_markdown":"[![geeksforgeeks](https:\/\/media.geeksforgeeks.org\/gfg-gg-logo.svg)](https:\/\/www.geeksforgeeks.org\/)\n\n![search icon](https:\/\/media.geeksforgeeks.org\/auth-dashboard-uploads\/Property=Light---Default.svg)\n\n- Sign In\n- [Courses]()\n- [Tutorials]()\n- [Interview Prep]()\n\n- [NLP Tutorial](https:\/\/www.geeksforgeeks.org\/nlp\/natural-language-processing-nlp-tutorial\/)\n- [Libraries](https:\/\/www.geeksforgeeks.org\/nlp\/nlp-libraries-in-python\/)\n- [Phases](https:\/\/www.geeksforgeeks.org\/machine-learning\/phases-of-natural-language-processing-nlp\/)\n- [Text Preprosessing](https:\/\/www.geeksforgeeks.org\/nlp\/text-preprocessing-for-nlp-tasks\/)\n- [Tokenization](https:\/\/www.geeksforgeeks.org\/nlp\/nlp-how-tokenizing-text-sentence-words-works\/)\n- [Lemmatization](https:\/\/www.geeksforgeeks.org\/python\/python-lemmatization-with-nltk\/)\n- [Word Embeddings](https:\/\/www.geeksforgeeks.org\/nlp\/word-embeddings-in-nlp\/)\n- [Projects Ideas](https:\/\/www.geeksforgeeks.org\/nlp\/top-natural-language-processing-nlp-projects\/)\n- [Interview Question](https:\/\/www.geeksforgeeks.org\/nlp\/advanced-natural-language-processing-interview-question\/)\n- [NLP Quiz](https:\/\/www.geeksforgeeks.org\/quizzes\/natural-language-processing-quiz\/)\n\n# Building Language Models in NLP\nLast Updated : 23 Jul, 2025\n\nBuilding language models is a fundamental task in natural language processing (NLP) that involves creating computational models capable of predicting the next word in a sequence of words. These models are essential for various NLP applications, such as machine translation, speech recognition, and text generation.\n\n*****In this article, we will build a language model using NLP using LSTM.*****\n\n## What is a Language Model?\n- A language model is a statistical model that is used to predict the probability of a sequence of words.\n- It learns the structure and patterns of a language from a given text corpus and can be used to generate new text that is similar to the original text.\n- Language models are a fundamental component of many natural language processing ([NLP](https:\/\/www.geeksforgeeks.org\/nlp\/natural-language-processing-overview\/)) tasks, such as machine translation, speech recognition, and text generation.\n## Steps to Build a Language Model in NLP\nHere, we will implement these steps to build a language model in NLP.\n\n### Step 1: Importing Necessary Libraries\nWe will, at first, import all the necessary libraries required for building our model.\n```\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Embedding, LSTM, Dense\nfrom tensorflow.keras.models import Sequential\n```\n### Step 2: Generate Sample Data\nWe will at first take a sample text data.\n```\ntext_data = \"Hello, how are you? I am doing well. Thank you for asking.\"\n```\n### Step 3: Preprocessing the Data\nThe preprocessing involves tokenizing the input text data, creates input sequences, and pads the sequences to make them equal in length.\n```\n# Tokenize the text\ntokenizer = tf.keras.preprocessing.text.Tokenizer()\ntokenizer.fit_on_texts([text_data])\ntotal_words = len(tokenizer.word_index) + 1\n\n# Create input sequences and labels\ninput_sequences = []\nfor line in text_data.split('.'):\n    token_list = tokenizer.texts_to_sequences([line])[0]\n    for i in range(1, len(token_list)):\n        n_gram_sequence = token_list[:i+1]\n        input_sequences.append(n_gram_sequence)\n\n# Pad sequences for equal length\nmax_sequence_len = max([len(x) for x in input_sequences])\ninput_sequences = tf.keras.preprocessing.sequence.pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre')\n```\n### Step 4: One hot encoding\nThe input sequences are split into predictors (xs) and labels (ys). The labels are converted to one-hot encoding.\n```\n# Create predictors and label\nxs, labels = input_sequences[:,:-1],input_sequences[:,-1]\n\n# Convert labels to one-hot encoding\nys = tf.keras.utils.to_categorical(labels, num_classes=total_words)\n```\n### Step 5: Defining and Compiling the Model\nThis code defines and compiles a simple [LSTM](https:\/\/www.geeksforgeeks.org\/machine-learning\/understanding-of-lstm-networks\/)\\-based language model using Keras\n```\n# Define the model\nmodel = Sequential()\nmodel.add(Embedding(total_words, 64, input_length=max_sequence_len-1))\nmodel.add(LSTM(100))\nmodel.add(Dense(total_words, activation='softmax'))\n\n# Compile the model\nmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\n# Fit the model\nhistory = model.fit(xs, ys, epochs=100, verbose=1)\n```\n### Step 6: Generating Text\nThis `generate_text` function takes a `seed_text` as input and generates `next_words` number of words using the provided `model` and `max_sequence_len`.\n```\ndef generate_text(seed_text, next_words, model, max_sequence_len):\n    for _ in range(next_words):\n        token_list = tokenizer.texts_to_sequences([seed_text])[0]\n        token_list = tf.keras.preprocessing.sequence.pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n        predicted_probs = model.predict(token_list, verbose=0)[0]\n        predicted_index = tf.argmax(predicted_probs, axis=-1).numpy()\n        output_word = \"\"\n        for word, index in tokenizer.word_index.items():\n            if index == predicted_index:\n                output_word = word\n                break\n        seed_text += \" \" + output_word\n    return seed_text\n\n# Generate text\nprint(generate_text(\"how\", 5, model, max_sequence_len))\n```\n#### ****Below is the complete Implemention:****\nPython\n\n``\n\n****Output:****\n```\nhow are you i am doing\n```\nIn summary, constructing language models for natural language processing (NLP) include various stages, including tokenization, sequence creation, model construction, training, and text generation. Tokenization transforms textual data into numerical representations, while sequence creation generates input-output pairs for model training. The model typically comprises layers like Embedding and LSTM, followed by a Dense layer for predictions. Training involves fitting the model to input sequences and their labels, while text generation utilizes the trained model to generate new text based on a provided seed text. Overall, language models are vital for NLP tasks such as text generation, machine translation, and sentiment analysis, among others.\n\nComment\n\n[G](https:\/\/www.geeksforgeeks.org\/user\/gs7701ri2f\/)\n\n[gs7701ri2f](https:\/\/www.geeksforgeeks.org\/user\/gs7701ri2f\/)\n\n0\n\nArticle Tags:\n\nArticle Tags:\n\n[Blogathon](https:\/\/www.geeksforgeeks.org\/category\/blogathon\/)\n\n[NLP](https:\/\/www.geeksforgeeks.org\/category\/ai-ml-ds\/nlp\/)\n\n[AI-ML-DS](https:\/\/www.geeksforgeeks.org\/category\/ai-ml-ds\/)\n\n[Data Science Blogathon 2024](https:\/\/www.geeksforgeeks.org\/tag\/data-science-blogathon-2024\/)\n### Explore\n\n[![GeeksforGeeks](https:\/\/media.geeksforgeeks.org\/auth-dashboard-uploads\/gfgFooterLogo.png)](https:\/\/www.geeksforgeeks.org\/)\n\n![location](https:\/\/media.geeksforgeeks.org\/img-practice\/Location-1685004904.svg)\n\nCorporate & Communications Address:\n\nA-143, 7th Floor, Sovereign Corporate Tower, Sector- 136, Noida, Uttar Pradesh (201305)\n\n![location](https:\/\/media.geeksforgeeks.org\/img-practice\/Location-1685004904.svg)\n\nRegistered Address:\n\nK 061, Tower K, Gulshan Vivante Apartment, Sector 137, Noida, Gautam Buddh Nagar, Uttar Pradesh, 201305\n\n[![GFG App on Play Store](https:\/\/media.geeksforgeeks.org\/auth-dashboard-uploads\/googleplay-%281%29.png)](https:\/\/geeksforgeeksapp.page.link\/gfg-app)[![GFG App on App Store](https:\/\/media.geeksforgeeks.org\/auth-dashboard-uploads\/appstore-%281%29.png)](https:\/\/geeksforgeeksapp.page.link\/gfg-app)\n\n- Company\n- [About Us](https:\/\/www.geeksforgeeks.org\/about\/)\n- [Legal](https:\/\/www.geeksforgeeks.org\/legal\/)\n- [Privacy Policy](https:\/\/www.geeksforgeeks.org\/legal\/privacy-policy\/)\n- [Contact Us](https:\/\/www.geeksforgeeks.org\/about\/contact-us\/)\n- [Advertise with us](https:\/\/www.geeksforgeeks.org\/advertise-with-us\/)\n- [GFG Corporate Solution](https:\/\/www.geeksforgeeks.org\/gfg-corporate-solution\/)\n- [Campus Training Program](https:\/\/www.geeksforgeeks.org\/campus-training-program\/)\n\n- Explore\n- [POTD](https:\/\/www.geeksforgeeks.org\/problem-of-the-day)\n- [Job-A-Thon](https:\/\/practice.geeksforgeeks.org\/events\/rec\/job-a-thon\/)\n- [Blogs](https:\/\/www.geeksforgeeks.org\/category\/blogs\/?type=recent)\n- [Nation Skill Up](https:\/\/www.geeksforgeeks.org\/nation-skill-up\/)\n\n- Tutorials\n- [Programming Languages](https:\/\/www.geeksforgeeks.org\/computer-science-fundamentals\/programming-language-tutorials\/)\n- [DSA](https:\/\/www.geeksforgeeks.org\/dsa\/dsa-tutorial-learn-data-structures-and-algorithms\/)\n- [Web Technology](https:\/\/www.geeksforgeeks.org\/web-tech\/web-technology\/)\n- [AI, ML & Data Science](https:\/\/www.geeksforgeeks.org\/machine-learning\/ai-ml-and-data-science-tutorial-learn-ai-ml-and-data-science\/)\n- [DevOps](https:\/\/www.geeksforgeeks.org\/devops\/devops-tutorial\/)\n- [CS Core Subjects](https:\/\/www.geeksforgeeks.org\/gate\/gate-exam-tutorial\/)\n- [Interview Preparation](https:\/\/www.geeksforgeeks.org\/aptitude\/interview-corner\/)\n- [Software and Tools](https:\/\/www.geeksforgeeks.org\/websites-apps\/software-and-tools-a-to-z-list\/)\n\n- Courses\n- [ML and Data Science](https:\/\/www.geeksforgeeks.org\/courses\/category\/machine-learning-data-science)\n- [DSA and Placements](https:\/\/www.geeksforgeeks.org\/courses\/category\/dsa-placements)\n- [Web Development](https:\/\/www.geeksforgeeks.org\/courses\/category\/development-testing)\n- [Programming Languages](https:\/\/www.geeksforgeeks.org\/courses\/category\/programming-languages)\n- [DevOps & Cloud](https:\/\/www.geeksforgeeks.org\/courses\/category\/cloud-devops)\n- [GATE](https:\/\/www.geeksforgeeks.org\/courses\/category\/gate)\n- [Trending Technologies](https:\/\/www.geeksforgeeks.org\/courses\/category\/trending-technologies\/)\n\n- Videos\n- [DSA](https:\/\/www.geeksforgeeks.org\/videos\/category\/sde-sheet\/)\n- [Python](https:\/\/www.geeksforgeeks.org\/videos\/category\/python\/)\n- [Java](https:\/\/www.geeksforgeeks.org\/videos\/category\/java-w6y5f4\/)\n- [C++](https:\/\/www.geeksforgeeks.org\/videos\/category\/c\/)\n- [Web Development](https:\/\/www.geeksforgeeks.org\/videos\/category\/web-development\/)\n- [Data Science](https:\/\/www.geeksforgeeks.org\/videos\/category\/data-science\/)\n- [CS Subjects](https:\/\/www.geeksforgeeks.org\/videos\/category\/cs-subjects\/)\n\n- Preparation Corner\n- [Interview Corner](https:\/\/www.geeksforgeeks.org\/interview-prep\/interview-corner\/)\n- [Aptitude](https:\/\/www.geeksforgeeks.org\/aptitude\/aptitude-questions-and-answers\/)\n- [Puzzles](https:\/\/www.geeksforgeeks.org\/aptitude\/puzzles\/)\n- [GfG 160](https:\/\/www.geeksforgeeks.org\/courses\/gfg-160-series)\n- [System Design](https:\/\/www.geeksforgeeks.org\/system-design\/system-design-tutorial\/)\n\n[@GeeksforGeeks, Sanchhaya Education Private Limited](https:\/\/www.geeksforgeeks.org\/), [All rights reserved](https:\/\/www.geeksforgeeks.org\/copyright-information\/)\n\n![]()","attrs_readable_markdown":"Last Updated : 23 Jul, 2025\n\nBuilding language models is a fundamental task in natural language processing (NLP) that involves creating computational models capable of predicting the next word in a sequence of words. These models are essential for various NLP applications, such as machine translation, speech recognition, and text generation.\n\n*****In this article, we will build a language model using NLP using LSTM.*****\n\n## What is a Language Model?\n- A language model is a statistical model that is used to predict the probability of a sequence of words.\n- It learns the structure and patterns of a language from a given text corpus and can be used to generate new text that is similar to the original text.\n- Language models are a fundamental component of many natural language processing ([NLP](https:\/\/www.geeksforgeeks.org\/nlp\/natural-language-processing-overview\/)) tasks, such as machine translation, speech recognition, and text generation.\n## Steps to Build a Language Model in NLP\nHere, we will implement these steps to build a language model in NLP.\n\n### Step 1: Importing Necessary Libraries\nWe will, at first, import all the necessary libraries required for building our model.\n```\nimport tensorflow as tf\nfrom tensorflow.keras.layers import Embedding, LSTM, Dense\nfrom tensorflow.keras.models import Sequential\n```\n### Step 2: Generate Sample Data\nWe will at first take a sample text data.\n```\ntext_data = \"Hello, how are you? I am doing well. Thank you for asking.\"\n```\n### Step 3: Preprocessing the Data\nThe preprocessing involves tokenizing the input text data, creates input sequences, and pads the sequences to make them equal in length.\n```\n# Tokenize the text\ntokenizer = tf.keras.preprocessing.text.Tokenizer()\ntokenizer.fit_on_texts([text_data])\ntotal_words = len(tokenizer.word_index) + 1\n\n# Create input sequences and labels\ninput_sequences = []\nfor line in text_data.split('.'):\n    token_list = tokenizer.texts_to_sequences([line])[0]\n    for i in range(1, len(token_list)):\n        n_gram_sequence = token_list[:i+1]\n        input_sequences.append(n_gram_sequence)\n\n# Pad sequences for equal length\nmax_sequence_len = max([len(x) for x in input_sequences])\ninput_sequences = tf.keras.preprocessing.sequence.pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre')\n```\n### Step 4: One hot encoding\nThe input sequences are split into predictors (xs) and labels (ys). The labels are converted to one-hot encoding.\n```\n# Create predictors and label\nxs, labels = input_sequences[:,:-1],input_sequences[:,-1]\n\n# Convert labels to one-hot encoding\nys = tf.keras.utils.to_categorical(labels, num_classes=total_words)\n```\n### Step 5: Defining and Compiling the Model\nThis code defines and compiles a simple [LSTM](https:\/\/www.geeksforgeeks.org\/machine-learning\/understanding-of-lstm-networks\/)\\-based language model using Keras\n```\n# Define the model\nmodel = Sequential()\nmodel.add(Embedding(total_words, 64, input_length=max_sequence_len-1))\nmodel.add(LSTM(100))\nmodel.add(Dense(total_words, activation='softmax'))\n\n# Compile the model\nmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n\n# Fit the model\nhistory = model.fit(xs, ys, epochs=100, verbose=1)\n```\n### Step 6: Generating Text\nThis `generate_text` function takes a `seed_text` as input and generates `next_words` number of words using the provided `model` and `max_sequence_len`.\n```\ndef generate_text(seed_text, next_words, model, max_sequence_len):\n    for _ in range(next_words):\n        token_list = tokenizer.texts_to_sequences([seed_text])[0]\n        token_list = tf.keras.preprocessing.sequence.pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n        predicted_probs = model.predict(token_list, verbose=0)[0]\n        predicted_index = tf.argmax(predicted_probs, axis=-1).numpy()\n        output_word = \"\"\n        for word, index in tokenizer.word_index.items():\n            if index == predicted_index:\n                output_word = word\n                break\n        seed_text += \" \" + output_word\n    return seed_text\n\n# Generate text\nprint(generate_text(\"how\", 5, model, max_sequence_len))\n```\n#### ****Below is the complete Implemention:****\n``\n\n****Output:****\n```\nhow are you i am doing\n```\nIn summary, constructing language models for natural language processing (NLP) include various stages, including tokenization, sequence creation, model construction, training, and text generation. Tokenization transforms textual data into numerical representations, while sequence creation generates input-output pairs for model training. The model typically comprises layers like Embedding and LSTM, followed by a Dense layer for predictions. Training involves fitting the model to input sequences and their labels, while text generation utilizes the trained model to generate new text based on a provided seed text. Overall, language models are vital for NLP tasks such as text generation, machine translation, and sentiment analysis, among others.","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

1 day ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.1 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://www.geeksforgeeks.org/nlp/building-language-models-in-nlp/
Last Crawled	2026-04-10 21:45:55 (1 day ago)
First Indexed	2025-06-16 17:02:23 (9 months ago)
HTTP Status Code	200
Meta Title	Building Language Models in NLP - GeeksforGeeks
Meta Description	Your All-in-One Learning Portal: GeeksforGeeks is a comprehensive educational platform that empowers learners across domains-spanning computer science and programming, school education, upskilling, commerce, software tools, competitive exams, and more., Your All-in-One Learning Portal. It contains well written, well thought and well explained computer science and programming articles, quizzes and practice/competitive programming/company interview Questions.
Meta Canonical	null
Boilerpipe Text	Last Updated : 23 Jul, 2025 Building language models is a fundamental task in natural language processing (NLP) that involves creating computational models capable of predicting the next word in a sequence of words. These models are essential for various NLP applications, such as machine translation, speech recognition, and text generation. In this article, we will build a language model using NLP using LSTM. What is a Language Model? A language model is a statistical model that is used to predict the probability of a sequence of words. It learns the structure and patterns of a language from a given text corpus and can be used to generate new text that is similar to the original text. Language models are a fundamental component of many natural language processing ( NLP ) tasks, such as machine translation, speech recognition, and text generation. Steps to Build a Language Model in NLP Here, we will implement these steps to build a language model in NLP. Step 1: Importing Necessary Libraries We will, at first, import all the necessary libraries required for building our model. import tensorflow as tf from tensorflow.keras.layers import Embedding, LSTM, Dense from tensorflow.keras.models import Sequential Step 2: Generate Sample Data We will at first take a sample text data. text_data = "Hello, how are you? I am doing well. Thank you for asking." Step 3: Preprocessing the Data The preprocessing involves tokenizing the input text data, creates input sequences, and pads the sequences to make them equal in length. # Tokenize the text tokenizer = tf.keras.preprocessing.text.Tokenizer() tokenizer.fit_on_texts([text_data]) total_words = len(tokenizer.word_index) + 1 # Create input sequences and labels input_sequences = [] for line in text_data.split('.'): token_list = tokenizer.texts_to_sequences([line])[0] for i in range(1, len(token_list)): n_gram_sequence = token_list[:i+1] input_sequences.append(n_gram_sequence) # Pad sequences for equal length max_sequence_len = max([len(x) for x in input_sequences]) input_sequences = tf.keras.preprocessing.sequence.pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre') Step 4: One hot encoding The input sequences are split into predictors (xs) and labels (ys). The labels are converted to one-hot encoding. # Create predictors and label xs, labels = input_sequences[:,:-1],input_sequences[:,-1] # Convert labels to one-hot encoding ys = tf.keras.utils.to_categorical(labels, num_classes=total_words) Step 5: Defining and Compiling the Model This code defines and compiles a simple LSTM -based language model using Keras # Define the model model = Sequential() model.add(Embedding(total_words, 64, input_length=max_sequence_len-1)) model.add(LSTM(100)) model.add(Dense(total_words, activation='softmax')) # Compile the model model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) # Fit the model history = model.fit(xs, ys, epochs=100, verbose=1) Step 6: Generating Text This generate_text function takes a seed_text as input and generates next_words number of words using the provided model and max_sequence_len . def generate_text(seed_text, next_words, model, max_sequence_len): for _ in range(next_words): token_list = tokenizer.texts_to_sequences([seed_text])[0] token_list = tf.keras.preprocessing.sequence.pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre') predicted_probs = model.predict(token_list, verbose=0)[0] predicted_index = tf.argmax(predicted_probs, axis=-1).numpy() output_word = "" for word, index in tokenizer.word_index.items(): if index == predicted_index: output_word = word break seed_text += " " + output_word return seed_text # Generate text print(generate_text("how", 5, model, max_sequence_len)) Below is the complete Implemention: import tensorflow as tf from tensorflow.keras.layers import Embedding , LSTM , Dense from tensorflow.keras.models import Sequential # Sample text data text_data = "Hello, how are you? I am doing well. Thank you for asking." # Tokenize the text tokenizer = tf . keras . preprocessing . text . Tokenizer () tokenizer . fit_on_texts ([ text_data ]) total_words = len ( tokenizer . word_index ) + 1 # Create input sequences and labels input_sequences = [] for line in text_data . split ( '.' ): token_list = tokenizer . texts_to_sequences ([ line ])[ 0 ] for i in range ( 1 , len ( token_list )): n_gram_sequence = token_list [: i + 1 ] input_sequences . append ( n_gram_sequence ) # Pad sequences for equal length max_sequence_len = max ([ len ( x ) for x in input_sequences ]) input_sequences = tf . keras . preprocessing . sequence . pad_sequences ( input_sequences , maxlen = max_sequence_len , padding = 'pre' ) # Create predictors and label xs , labels = input_sequences [:,: - 1 ], input_sequences [:, - 1 ] # Convert labels to one-hot encoding ys = tf . keras . utils . to_categorical ( labels , num_classes = total_words ) # Define the model model = Sequential () model . add ( Embedding ( total_words , 64 , input_length = max_sequence_len - 1 )) model . add ( LSTM ( 100 )) model . add ( Dense ( total_words , activation = 'softmax' )) # Compile the model model . compile ( loss = 'categorical_crossentropy' , optimizer = 'adam' , metrics = [ 'accuracy' ]) # Fit the model history = model . fit ( xs , ys , epochs = 100 , verbose = 1 ) def generate_text ( seed_text , next_words , model , max_sequence_len ): for _ in range ( next_words ): token_list = tokenizer . texts_to_sequences ([ seed_text ])[ 0 ] token_list = tf . keras . preprocessing . sequence . pad_sequences ([ token_list ], maxlen = max_sequence_len - 1 , padding = 'pre' ) predicted_probs = model . predict ( token_list , verbose = 0 )[ 0 ] predicted_index = tf . argmax ( predicted_probs , axis =- 1 ) . numpy () output_word = "" for word , index in tokenizer . word_index . items (): if index == predicted_index : output_word = word break seed_text += " " + output_word return seed_text # Generate text print ( generate_text ( "how" , 5 , model , max_sequence_len )) Output: how are you i am doing In summary, constructing language models for natural language processing (NLP) include various stages, including tokenization, sequence creation, model construction, training, and text generation. Tokenization transforms textual data into numerical representations, while sequence creation generates input-output pairs for model training. The model typically comprises layers like Embedding and LSTM, followed by a Dense layer for predictions. Training involves fitting the model to input sequences and their labels, while text generation utilizes the trained model to generate new text based on a provided seed text. Overall, language models are vital for NLP tasks such as text generation, machine translation, and sentiment analysis, among others.
Markdown	[![geeksforgeeks](https://media.geeksforgeeks.org/gfg-gg-logo.svg)](https://www.geeksforgeeks.org/) ![search icon](https://media.geeksforgeeks.org/auth-dashboard-uploads/Property=Light---Default.svg) - Sign In - [Courses]() - [Tutorials]() - [Interview Prep]() - [NLP Tutorial](https://www.geeksforgeeks.org/nlp/natural-language-processing-nlp-tutorial/) - [Libraries](https://www.geeksforgeeks.org/nlp/nlp-libraries-in-python/) - [Phases](https://www.geeksforgeeks.org/machine-learning/phases-of-natural-language-processing-nlp/) - [Text Preprosessing](https://www.geeksforgeeks.org/nlp/text-preprocessing-for-nlp-tasks/) - [Tokenization](https://www.geeksforgeeks.org/nlp/nlp-how-tokenizing-text-sentence-words-works/) - [Lemmatization](https://www.geeksforgeeks.org/python/python-lemmatization-with-nltk/) - [Word Embeddings](https://www.geeksforgeeks.org/nlp/word-embeddings-in-nlp/) - [Projects Ideas](https://www.geeksforgeeks.org/nlp/top-natural-language-processing-nlp-projects/) - [Interview Question](https://www.geeksforgeeks.org/nlp/advanced-natural-language-processing-interview-question/) - [NLP Quiz](https://www.geeksforgeeks.org/quizzes/natural-language-processing-quiz/) # Building Language Models in NLP Last Updated : 23 Jul, 2025 Building language models is a fundamental task in natural language processing (NLP) that involves creating computational models capable of predicting the next word in a sequence of words. These models are essential for various NLP applications, such as machine translation, speech recognition, and text generation. ***In this article, we will build a language model using NLP using LSTM.* ## What is a Language Model? - A language model is a statistical model that is used to predict the probability of a sequence of words. - It learns the structure and patterns of a language from a given text corpus and can be used to generate new text that is similar to the original text. - Language models are a fundamental component of many natural language processing ([NLP](https://www.geeksforgeeks.org/nlp/natural-language-processing-overview/)) tasks, such as machine translation, speech recognition, and text generation. ## Steps to Build a Language Model in NLP Here, we will implement these steps to build a language model in NLP. ### Step 1: Importing Necessary Libraries We will, at first, import all the necessary libraries required for building our model. ``` import tensorflow as tf from tensorflow.keras.layers import Embedding, LSTM, Dense from tensorflow.keras.models import Sequential ``` ### Step 2: Generate Sample Data We will at first take a sample text data. ``` text_data = "Hello, how are you? I am doing well. Thank you for asking." ``` ### Step 3: Preprocessing the Data The preprocessing involves tokenizing the input text data, creates input sequences, and pads the sequences to make them equal in length. ``` # Tokenize the text tokenizer = tf.keras.preprocessing.text.Tokenizer() tokenizer.fit_on_texts([text_data]) total_words = len(tokenizer.word_index) + 1 # Create input sequences and labels input_sequences = [] for line in text_data.split('.'): token_list = tokenizer.texts_to_sequences([line])[0] for i in range(1, len(token_list)): n_gram_sequence = token_list[:i+1] input_sequences.append(n_gram_sequence) # Pad sequences for equal length max_sequence_len = max([len(x) for x in input_sequences]) input_sequences = tf.keras.preprocessing.sequence.pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre') ``` ### Step 4: One hot encoding The input sequences are split into predictors (xs) and labels (ys). The labels are converted to one-hot encoding. ``` # Create predictors and label xs, labels = input_sequences[:,:-1],input_sequences[:,-1] # Convert labels to one-hot encoding ys = tf.keras.utils.to_categorical(labels, num_classes=total_words) ``` ### Step 5: Defining and Compiling the Model This code defines and compiles a simple [LSTM](https://www.geeksforgeeks.org/machine-learning/understanding-of-lstm-networks/)\-based language model using Keras ``` # Define the model model = Sequential() model.add(Embedding(total_words, 64, input_length=max_sequence_len-1)) model.add(LSTM(100)) model.add(Dense(total_words, activation='softmax')) # Compile the model model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) # Fit the model history = model.fit(xs, ys, epochs=100, verbose=1) ``` ### Step 6: Generating Text This `generate_text` function takes a `seed_text` as input and generates `next_words` number of words using the provided `model` and `max_sequence_len`. ``` def generate_text(seed_text, next_words, model, max_sequence_len): for _ in range(next_words): token_list = tokenizer.texts_to_sequences([seed_text])[0] token_list = tf.keras.preprocessing.sequence.pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre') predicted_probs = model.predict(token_list, verbose=0)[0] predicted_index = tf.argmax(predicted_probs, axis=-1).numpy() output_word = "" for word, index in tokenizer.word_index.items(): if index == predicted_index: output_word = word break seed_text += " " + output_word return seed_text # Generate text print(generate_text("how", 5, model, max_sequence_len)) ``` #### Below is the complete Implemention: Python `` Output:** ``` how are you i am doing ``` In summary, constructing language models for natural language processing (NLP) include various stages, including tokenization, sequence creation, model construction, training, and text generation. Tokenization transforms textual data into numerical representations, while sequence creation generates input-output pairs for model training. The model typically comprises layers like Embedding and LSTM, followed by a Dense layer for predictions. Training involves fitting the model to input sequences and their labels, while text generation utilizes the trained model to generate new text based on a provided seed text. Overall, language models are vital for NLP tasks such as text generation, machine translation, and sentiment analysis, among others. Comment [G](https://www.geeksforgeeks.org/user/gs7701ri2f/) [gs7701ri2f](https://www.geeksforgeeks.org/user/gs7701ri2f/) 0 Article Tags: Article Tags: [Blogathon](https://www.geeksforgeeks.org/category/blogathon/) [NLP](https://www.geeksforgeeks.org/category/ai-ml-ds/nlp/) [AI-ML-DS](https://www.geeksforgeeks.org/category/ai-ml-ds/) [Data Science Blogathon 2024](https://www.geeksforgeeks.org/tag/data-science-blogathon-2024/) ### Explore [![GeeksforGeeks](https://media.geeksforgeeks.org/auth-dashboard-uploads/gfgFooterLogo.png)](https://www.geeksforgeeks.org/) ![location](https://media.geeksforgeeks.org/img-practice/Location-1685004904.svg) Corporate & Communications Address: A-143, 7th Floor, Sovereign Corporate Tower, Sector- 136, Noida, Uttar Pradesh (201305) ![location](https://media.geeksforgeeks.org/img-practice/Location-1685004904.svg) Registered Address: K 061, Tower K, Gulshan Vivante Apartment, Sector 137, Noida, Gautam Buddh Nagar, Uttar Pradesh, 201305 [![GFG App on Play Store](https://media.geeksforgeeks.org/auth-dashboard-uploads/googleplay-%281%29.png)](https://geeksforgeeksapp.page.link/gfg-app)[![GFG App on App Store](https://media.geeksforgeeks.org/auth-dashboard-uploads/appstore-%281%29.png)](https://geeksforgeeksapp.page.link/gfg-app) - Company - [About Us](https://www.geeksforgeeks.org/about/) - [Legal](https://www.geeksforgeeks.org/legal/) - [Privacy Policy](https://www.geeksforgeeks.org/legal/privacy-policy/) - [Contact Us](https://www.geeksforgeeks.org/about/contact-us/) - [Advertise with us](https://www.geeksforgeeks.org/advertise-with-us/) - [GFG Corporate Solution](https://www.geeksforgeeks.org/gfg-corporate-solution/) - [Campus Training Program](https://www.geeksforgeeks.org/campus-training-program/) - Explore - [POTD](https://www.geeksforgeeks.org/problem-of-the-day) - [Job-A-Thon](https://practice.geeksforgeeks.org/events/rec/job-a-thon/) - [Blogs](https://www.geeksforgeeks.org/category/blogs/?type=recent) - [Nation Skill Up](https://www.geeksforgeeks.org/nation-skill-up/) - Tutorials - [Programming Languages](https://www.geeksforgeeks.org/computer-science-fundamentals/programming-language-tutorials/) - [DSA](https://www.geeksforgeeks.org/dsa/dsa-tutorial-learn-data-structures-and-algorithms/) - [Web Technology](https://www.geeksforgeeks.org/web-tech/web-technology/) - [AI, ML & Data Science](https://www.geeksforgeeks.org/machine-learning/ai-ml-and-data-science-tutorial-learn-ai-ml-and-data-science/) - [DevOps](https://www.geeksforgeeks.org/devops/devops-tutorial/) - [CS Core Subjects](https://www.geeksforgeeks.org/gate/gate-exam-tutorial/) - [Interview Preparation](https://www.geeksforgeeks.org/aptitude/interview-corner/) - [Software and Tools](https://www.geeksforgeeks.org/websites-apps/software-and-tools-a-to-z-list/) - Courses - [ML and Data Science](https://www.geeksforgeeks.org/courses/category/machine-learning-data-science) - [DSA and Placements](https://www.geeksforgeeks.org/courses/category/dsa-placements) - [Web Development](https://www.geeksforgeeks.org/courses/category/development-testing) - [Programming Languages](https://www.geeksforgeeks.org/courses/category/programming-languages) - [DevOps & Cloud](https://www.geeksforgeeks.org/courses/category/cloud-devops) - [GATE](https://www.geeksforgeeks.org/courses/category/gate) - [Trending Technologies](https://www.geeksforgeeks.org/courses/category/trending-technologies/) - Videos - [DSA](https://www.geeksforgeeks.org/videos/category/sde-sheet/) - [Python](https://www.geeksforgeeks.org/videos/category/python/) - [Java](https://www.geeksforgeeks.org/videos/category/java-w6y5f4/) - [C++](https://www.geeksforgeeks.org/videos/category/c/) - [Web Development](https://www.geeksforgeeks.org/videos/category/web-development/) - [Data Science](https://www.geeksforgeeks.org/videos/category/data-science/) - [CS Subjects](https://www.geeksforgeeks.org/videos/category/cs-subjects/) - Preparation Corner - [Interview Corner](https://www.geeksforgeeks.org/interview-prep/interview-corner/) - [Aptitude](https://www.geeksforgeeks.org/aptitude/aptitude-questions-and-answers/) - [Puzzles](https://www.geeksforgeeks.org/aptitude/puzzles/) - [GfG 160](https://www.geeksforgeeks.org/courses/gfg-160-series) - [System Design](https://www.geeksforgeeks.org/system-design/system-design-tutorial/) [@GeeksforGeeks, Sanchhaya Education Private Limited](https://www.geeksforgeeks.org/), [All rights reserved](https://www.geeksforgeeks.org/copyright-information/) ![]()
Readable Markdown	Last Updated : 23 Jul, 2025 Building language models is a fundamental task in natural language processing (NLP) that involves creating computational models capable of predicting the next word in a sequence of words. These models are essential for various NLP applications, such as machine translation, speech recognition, and text generation. ***In this article, we will build a language model using NLP using LSTM.* ## What is a Language Model? - A language model is a statistical model that is used to predict the probability of a sequence of words. - It learns the structure and patterns of a language from a given text corpus and can be used to generate new text that is similar to the original text. - Language models are a fundamental component of many natural language processing ([NLP](https://www.geeksforgeeks.org/nlp/natural-language-processing-overview/)) tasks, such as machine translation, speech recognition, and text generation. ## Steps to Build a Language Model in NLP Here, we will implement these steps to build a language model in NLP. ### Step 1: Importing Necessary Libraries We will, at first, import all the necessary libraries required for building our model. ``` import tensorflow as tf from tensorflow.keras.layers import Embedding, LSTM, Dense from tensorflow.keras.models import Sequential ``` ### Step 2: Generate Sample Data We will at first take a sample text data. ``` text_data = "Hello, how are you? I am doing well. Thank you for asking." ``` ### Step 3: Preprocessing the Data The preprocessing involves tokenizing the input text data, creates input sequences, and pads the sequences to make them equal in length. ``` # Tokenize the text tokenizer = tf.keras.preprocessing.text.Tokenizer() tokenizer.fit_on_texts([text_data]) total_words = len(tokenizer.word_index) + 1 # Create input sequences and labels input_sequences = [] for line in text_data.split('.'): token_list = tokenizer.texts_to_sequences([line])[0] for i in range(1, len(token_list)): n_gram_sequence = token_list[:i+1] input_sequences.append(n_gram_sequence) # Pad sequences for equal length max_sequence_len = max([len(x) for x in input_sequences]) input_sequences = tf.keras.preprocessing.sequence.pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre') ``` ### Step 4: One hot encoding The input sequences are split into predictors (xs) and labels (ys). The labels are converted to one-hot encoding. ``` # Create predictors and label xs, labels = input_sequences[:,:-1],input_sequences[:,-1] # Convert labels to one-hot encoding ys = tf.keras.utils.to_categorical(labels, num_classes=total_words) ``` ### Step 5: Defining and Compiling the Model This code defines and compiles a simple [LSTM](https://www.geeksforgeeks.org/machine-learning/understanding-of-lstm-networks/)\-based language model using Keras ``` # Define the model model = Sequential() model.add(Embedding(total_words, 64, input_length=max_sequence_len-1)) model.add(LSTM(100)) model.add(Dense(total_words, activation='softmax')) # Compile the model model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) # Fit the model history = model.fit(xs, ys, epochs=100, verbose=1) ``` ### Step 6: Generating Text This `generate_text` function takes a `seed_text` as input and generates `next_words` number of words using the provided `model` and `max_sequence_len`. ``` def generate_text(seed_text, next_words, model, max_sequence_len): for _ in range(next_words): token_list = tokenizer.texts_to_sequences([seed_text])[0] token_list = tf.keras.preprocessing.sequence.pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre') predicted_probs = model.predict(token_list, verbose=0)[0] predicted_index = tf.argmax(predicted_probs, axis=-1).numpy() output_word = "" for word, index in tokenizer.word_index.items(): if index == predicted_index: output_word = word break seed_text += " " + output_word return seed_text # Generate text print(generate_text("how", 5, model, max_sequence_len)) ``` #### Below is the complete Implemention: `` Output:** ``` how are you i am doing ``` In summary, constructing language models for natural language processing (NLP) include various stages, including tokenization, sequence creation, model construction, training, and text generation. Tokenization transforms textual data into numerical representations, while sequence creation generates input-output pairs for model training. The model typically comprises layers like Embedding and LSTM, followed by a Dense layer for predictions. Training involves fitting the model to input sequences and their labels, while text generation utilizes the trained model to generate new text based on a provided seed text. Overall, language models are vital for NLP tasks such as text generation, machine translation, and sentiment analysis, among others.
Shard	103 (laksa)
Root Hash	12046344915360636903
Unparsed URL	org,geeksforgeeks!www,/nlp/building-language-models-in-nlp/ s443