🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 39 (from laksa182)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa039.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://www.ibm.com/think/topics/natural-language-processing\')), getAhrefsUnparsedNoserviceFromURL(\'https://www.ibm.com/think/topics/natural-language-processing\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/www.ibm.com\/think\/topics\/natural-language-processing","crawl_time":1774931269,"first_indexed_time":1734526981,"http_code":200,"src_unparsed":"com,ibm!www,\/think\/topics\/natural-language-processing s443","src_root_hash":"15319107097793762839","history_drop_reason":null,"meta_title":"What Is NLP (Natural Language Processing)? | IBM","meta_descriptions":["Natural language processing (NLP) is a subfield of artificial intelligence (AI) that uses machine learning to help computers communicate with human language."],"attrs_boilerpipe_text":"Natural language processing (NLP) is a subfield of computer science and\nartificial intelligence (AI)\nthat uses\nmachine learning\nto enable computers to understand and communicate with human language.\nNLP enables computers and digital devices to recognize, understand and generate text and speech by combining computational linguistics, the rule-based modeling of human language together with statistical modeling, machine learning and\ndeep learning\n.\nNLP research has helped enable the era of\ngenerative AI\n, from the communication skills of\nlarge language models\n(LLMs) to the ability of image generation models to understand requests. NLP is already part of everyday life for many, powering search engines, prompting\nchatbots\nfor customer service with spoken commands, voice-operated GPS systems and question-answering digital assistants on smartphones such as Amazon’s Alexa, Apple’s Siri and Microsoft’s Cortana.\nNLP also plays a growing role in enterprise solutions that help streamline and automate business operations, increase employee productivity and simplify business processes.\nNLP makes it easier for humans to communicate and collaborate with machines, by allowing them to do so in the natural human language they use every day. This offers benefits across many industries and applications.\nAutomation of repetitive tasks\nImproved data analysis and insights\nEnhanced search\nContent generation\nAutomation of repetitive tasks \nNLP is especially useful in fully or partially \nautomating tasks\n like customer support, data entry and document handling. For example, NLP-powered chatbots can handle routine customer queries, freeing up human agents for more complex issues. In \ndocument processing\n, NLP tools can automatically classify, extract key information and summarize content, reducing the time and errors associated with manual data handling. NLP facilitates language translation, converting text from one language to another while preserving meaning, context and nuances.\nImproved data analysis\nNLP enhances data analysis by enabling the extraction of insights from unstructured text data, such as customer reviews, social media posts and news articles. By using \ntext mining\n techniques, NLP can identify patterns, trends and sentiments that are not immediately obvious in large datasets. Sentiment analysis enables the \nextraction of subjective qualities\n, attitudes, emotions, sarcasm, confusion or suspicion from text. This is often used for routing communications to the system or the person most likely to make the next response.\nThis allows businesses to better understand customer preferences, market conditions and public opinion. NLP tools can also perform categorization and summarization of vast amounts of text, making it easier for analysts to identify key information and make data-driven decisions more efficiently.\nEnhanced search\nNLP benefits search by enabling systems to understand the intent behind user queries, providing more accurate and contextually relevant results. Instead of relying solely on keyword matching, NLP-powered search engines analyze the meaning of words and phrases, making it easier to find information even when queries are vague or complex. This improves user experience, whether in web searches, document retrieval or enterprise data systems.\nPowerful content generation\nNLP powers advanced language models to \ncreate human-like text\n for various purposes. Pre-trained models, such as GPT-4, can generate articles, reports, marketing copy, product descriptions and even creative writing based on prompts provided by users. NLP-powered tools can also assist in automating tasks like drafting emails, writing social media posts or legal documentation. By understanding context, tone and style, NLP sees to it that the generated content is coherent, relevant and aligned with the intended message, saving time and effort in content creation while maintaining quality.\nNLP combines the power of computational linguistics together with\nmachine learning algorithms\nand deep learning. Computational linguistics uses data science to analyze language and speech. It includes two main types of analysis: syntactical analysis and semantical analysis. Syntactical analysis determines the meaning of a word, phrase or sentence by parsing the syntax of the words and applying preprogrammed rules of grammar. Semantical analysis uses the syntactic output to draw meaning from the words and interpret their meaning within the sentence structure.\nThe parsing of words can take one of two forms. Dependency parsing looks at the relationships between words, such as identifying nouns and verbs, while constituency parsing then builds a parse tree (or syntax tree): a rooted and ordered representation of the syntactic structure of the sentence or string of words. The resulting parse trees underly the functions of language translators and speech recognition. Ideally, this analysis makes the output either text or speech understandable to both NLP models and people.\nSelf-supervised learning (SSL)\nin particular is useful for supporting NLP because NLP requires large amounts of labeled data to train AI models. Because these labeled datasets require time-consuming annotation, a process involving manual labeling by humans, gathering sufficient data can be prohibitively difficult. Self-supervised approaches can be more time-effective and cost-effective, as they replace some or all manually labeled training data.\n \nThree different approaches to NLP include:\nRules-based NLP\nThe earliest NLP applications were simple if-then decision trees, requiring preprogrammed rules. They are only able to provide answers in response to specific prompts, such as the original version of Moviefone, which had rudimentary natural language generation (NLG) capabilities. Because there is no machine learning or AI capability in rules-based NLP, this function is highly limited and not scalable.\nStatistical NLP\nDeveloped later, statistical NLP automatically extracts, classifies and labels elements of text and voice data and then assigns a statistical likelihood to each possible meaning of those elements. This relies on machine learning, enabling a sophisticated breakdown of linguistics such as part-of-speech tagging.\n \nStatistical NLP introduced the essential technique of mapping language elements, such as words and grammatical rules to a vector representation so that language can be modeled by using mathematical (statistical) methods, including regression or Markov models. This informed early NLP developments such as spellcheckers and T9 texting (Text on 9 keys, to be used on Touch-Tone telephones).\nDeep learning NLP\nRecently, deep learning models have become the dominant mode of NLP, by using huge volumes of raw, \nunstructured\n data both text and voice to become ever more accurate. Deep learning can be viewed as a further evolution of statistical NLP, with the difference that it uses \nneural network\n models. There are several subcategories of models:\nSequence-to-Sequence\n (seq2seq) models: Based \non recurrent neural networks (RNN)\n, they have mostly been used for machine translation by converting a phrase from one domain (such as the German language) into the phrase of another domain (such as English).\nTransformer models\n: They use \ntokenization\n of language (the position of each token words or subwords) and self-attention (capturing dependencies and relationships) to calculate the relation of different language parts to one another. \nTransformer models\n can be efficiently trained by using \nself-supervised learning\n on massive text databases. A landmark in \ntransformer models\n was Google’s bidirectional encoder representations from transformers (BERT), which became and remains the basis of how Google’s search engine works.\nAutoregressive models\n: This type of transformer model is trained specifically to predict the next word in a sequence, which represents a huge leap forward in the ability to generate text. Examples of autoregressive LLMs include GPT, \nLlama\n, Claude and the open-source Mistral.\nFoundation models\n: Prebuilt and curated foundation models can speed the launching of an NLP effort and boost trust in its operation. For example, the \nIBM® Granite™\n foundation models are widely applicable across industries. They support NLP tasks including content generation and insight extraction. Additionally, they facilitate retrieval-augmented generation, a framework for improving the quality of response by linking the model to external sources of knowledge. The models also perform named entity recognition which involves identifying and extracting key information in a text.\nSeveral NLP tasks typically help process human text and voice data in ways that help the computer make sense of what it’s ingesting. Some of these tasks include:\nCoreference resolution\nNamed entity recognition\nPart-of-speech tagging\nWord sense disambiguation\nCoreference resolution\nThis is the task of identifying if and when two words refer to the same entity. The most common example is determining the person or object to which a certain pronoun refers (such as “she” = “Mary”). But it can also identify a metaphor or an idiom in the text (such as an instance in which “bear” isn’t an animal, but a large and hairy person). \nNamed entity recognition (NER)\nNER\nidentifies words or phrases as useful entities. NER identifies “London” as a location or “Maria” as a person's name.\nPart-of-speech tagging\nAlso called grammatical tagging, this is the process of determining which part of speech a word or piece of text is, based on its use and context. For example, part-of-speech identifies “make” as a verb in “I can make a paper plane,” and as a noun in “What make of car do you own?”\nWord sense disambiguation\nThis is the selection of a word meaning for a word with multiple possible meanings. This uses a process of semantic\nanalysis\nto examine the word in context. For example, word sense disambiguation helps distinguish the meaning of the verb “make” in “make the grade” (to achieve) versus “make a bet” (to place). Sorting out “I will be merry when I marry Mary” requires a sophisticated NLP system.\nNLP works by combining various computational techniques to analyze, understand and generate human language in a way that machines can process. Here is an overview of a typical NLP pipeline and its steps:\nText preprocessing\nNLP text preprocessing prepares raw text for analysis by transforming it into a format that machines can more easily understand. It begins with tokenization, which involves splitting the text into smaller units like words, sentences or phrases. This helps break down complex text into manageable parts. Next, lowercasing is applied to standardize the text by converting all characters to lowercase, ensuring that words like \"Apple\" and \"apple\" are treated the same. Stop word removal is another common step, where frequently used words like \"is\" or \"the\" are filtered out because they don't add significant meaning to the text. \nStemming\n or \nlemmatization\n reduces words to their root form (e.g., \"running\" becomes \"run\"), making it easier to analyze language by grouping different forms of the same word. Additionally, text cleaning removes unwanted elements such as punctuation, special characters and numbers that may clutter the analysis.\nAfter preprocessing, the text is clean, standardized and ready for machine learning models to interpret effectively.\nFeature extraction\nFeature extraction is the process of converting raw text into numerical representations that machines can analyze and interpret. This involves transforming text into structured data by using NLP techniques like \nBag of Words\n and TF-IDF, which quantify the presence and importance of words in a document. More advanced methods include \nword embeddings\n like Word2Vec or GloVe, which represent words as dense vectors in a continuous space, capturing semantic relationships between words. Contextual embeddings further enhance this by considering the context in which words appear, allowing for richer, more nuanced representations.\nText analysis\nText analysis involves interpreting and extracting meaningful information from text data through various computational techniques. This process includes tasks such as part-of-speech (POS) tagging, which identifies grammatical roles of words and named entity recognition (NER), which detects specific entities like names, locations and dates. Dependency parsing analyzes grammatical relationships between words to understand sentence structure, while sentiment analysis determines the emotional tone of the text, assessing whether it is positive, negative or neutral. Topic modeling identifies underlying themes or topics within a text or across a corpus of documents. Natural language understanding (NLU) is a subset of NLP that focuses on analyzing the meaning behind sentences. NLU enables software to find similar meanings in different sentences or to process words that have different meanings. Through these techniques, NLP text analysis transforms unstructured text into insights.\nModel training\nProcessed data is then used to train machine learning models, which learn patterns and relationships within the data. During training, the model adjusts its parameters to minimize errors and improve its performance. Once trained, the model can be used to make predictions or generate outputs on new, unseen data. The effectiveness of NLP modeling is continually refined through evaluation, validation and fine-tuning to enhance accuracy and relevance in real-world applications.\nDifferent software environments are useful throughout the said processes. For example, the Natural Language Toolkit (NLTK) is a suite of libraries and programs for English that is written in the Python programming language. It supports text classification, tokenization, stemming, tagging, parsing and semantic reasoning functionalities. TensorFlow is a free and open-source software library for machine learning and AI that can be used to train models for NLP applications. Tutorials and certifications abound for those interested in familiarizing themselves with such tools.\nEven state-of-the-art NLP models are not perfect, just as human speech is prone to error. As with any AI technology, NLP comes with potential pitfalls. Human language is filled with ambiguities that make it difficult for programmers to write software that accurately determines the intended meaning of text or voice data. Human language might take years for humans to learn and many never stop learning. But then programmers must teach natural language-powered applications to recognize and understand irregularities so their applications can be accurate and useful. Associated risks might include:\nBiased training\nAs with any AI function,\nbiased data\nused in training will skew the answers. The more diverse the users of an NLP function, the more significant this risk becomes, such as in government services, healthcare and HR interactions. Training datasets scraped from the web, for example, are prone to bias.\nMisinterpretation\nAs in programming, there is a risk of garbage in, garbage out (GIGO).\nSpeech recognition\n, also known as speech-to-text, is the task of reliably converting voice data into text data. But NLP solutions can become confused if spoken input is in an obscure dialect, mumbled, too full of slang, homonyms, incorrect grammar, idioms, fragments, mispronunciations, contractions or recorded with too much background noise.\nNew vocabulary\nNew words are continually being invented or imported. The conventions of grammar can evolve or be intentionally broken. In these cases, NLP can either make a best guess or admit it’s unsure and either way, this creates a complication.\nTone of voice\nWhen people speak, their verbal delivery or even body language can give an entirely different meaning than the words alone. Exaggeration for effect, stressing words for importance or sarcasm can be confused by NLP, making the semantic analysis more difficult and less reliable.\nNLP applications can now be found across virtually every industry.\nFinance\nIn financial dealings, nanoseconds might make the difference between success and failure when accessing data, or making trades or deals. NLP can speed the mining of information from financial statements, annual and regulatory reports, news releases or even social media.\nHealthcare\nNew medical insights and breakthroughs can arrive faster than many healthcare professionals can keep up. NLP and AI-based tools can help speed the analysis of health records and medical research papers, making better-informed medical decisions possible, or assisting in the detection or even prevention of medical conditions.\nInsurance\nNLP can analyze claims to look for patterns that can identify areas of concern and find inefficiencies in claims processing, leading to greater optimization of processing and employee efforts.\nLegal\nAlmost any legal case might require reviewing mounds of paperwork, background information and legal precedent. NLP can help automate legal discovery, assisting in the organization of information, speeding review and making sure that all relevant details are captured for consideration.","attrs_markdown":"# What is NLP (natural language processing)?\nMachine learning\n\n- [Welcome](https:\/\/www.ibm.com\/think\/machine-learning#605511093)\n- Introduction\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/machine-learning#7281535)\n  - [Machine learning types](https:\/\/www.ibm.com\/think\/topics\/machine-learning-types#7281536)\n  - [Machine learning algorithms](https:\/\/www.ibm.com\/think\/topics\/machine-learning-algorithms#7281537)\n- Data science for machine learning\n  - [Statistical machine learning](https:\/\/www.ibm.com\/think\/topics\/statistical-machine-learning#498277090)\n  - [Linear algebra for machine learning](https:\/\/www.ibm.com\/think\/topics\/linear-algebra-for-machine-learning#498277089)\n  - [Uncertainty quantification](https:\/\/www.ibm.com\/think\/topics\/uncertainty-quantification#498277088)\n  - [Bias variance tradeoff](https:\/\/www.ibm.com\/think\/topics\/bias-variance-tradeoff#498277087)\n  - [Bayesian Statistics](https:\/\/www.ibm.com\/think\/topics\/bayesian-statistics#498277086)\n  - [Singular value decomposition](https:\/\/www.ibm.com\/think\/topics\/singular-value-decomposition#498277085)\n- Feature Engineering\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/feature-engineering#1003835715)\n  - [Feature selection](https:\/\/www.ibm.com\/think\/topics\/feature-selection#1003835714)\n  - [Feature extraction](https:\/\/www.ibm.com\/think\/topics\/feature-extraction#1003835713)\n  - [Vector embedding](https:\/\/www.ibm.com\/think\/topics\/vector-embedding#1003835712)\n  - [Latent space](https:\/\/www.ibm.com\/think\/topics\/latent-space#1003835711)\n  - Dimensionality reduction\n    - [Principal component analysis](https:\/\/www.ibm.com\/think\/topics\/principal-component-analysis#1793360183)\n    - [Linear discriminant analysis](https:\/\/www.ibm.com\/think\/topics\/linear-discriminant-analysis#1793360182)\n  - [Upsampling](https:\/\/www.ibm.com\/think\/topics\/upsampling#1003835709)\n  - [Downsampling](https:\/\/www.ibm.com\/think\/topics\/downsampling#1003835708)\n  - [Synthetic data](https:\/\/www.ibm.com\/think\/topics\/synthetic-data#1003835707)\n  - [Data leakage](https:\/\/www.ibm.com\/think\/topics\/data-leakage-machine-learning#1003835706)\n- Supervised learning\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/supervised-learning#1509394340)\n  - Regression\n    - [Linear regression](https:\/\/www.ibm.com\/think\/topics\/linear-regression#1190488334)\n    - [Lasso regression](https:\/\/www.ibm.com\/think\/topics\/lasso-regression#1190488335)\n    - [Ridge regression](https:\/\/www.ibm.com\/think\/topics\/ridge-regression#1190488336)\n    - [State space model](https:\/\/www.ibm.com\/think\/topics\/state-space-model#1190488337)\n    - [Time series](https:\/\/www.ibm.com\/think\/topics\/time-series-model#1190488338)\n    - [Autoregressive model](https:\/\/www.ibm.com\/think\/topics\/autoregressive-model#1190488339)\n  - Classification\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/classification-machine-learning#684929709)\n    - [Decision trees](https:\/\/www.ibm.com\/think\/topics\/decision-trees#684929710)\n    - [K-nearest neighbors (KNNs)](https:\/\/www.ibm.com\/think\/topics\/knn#684929711)\n    - [Naive bayes](https:\/\/www.ibm.com\/think\/topics\/naive-bayes#684929712)\n    - [Random forest](https:\/\/www.ibm.com\/think\/topics\/random-forest#684929713)\n    - [Support vector machine](https:\/\/www.ibm.com\/think\/topics\/support-vector-machine#684929714)\n    - [Logistic regression](https:\/\/www.ibm.com\/think\/topics\/logistic-regression#684929715)\n  - Ensemble learning\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/ensemble-learning#179371084)\n    - [Boosting](https:\/\/www.ibm.com\/think\/topics\/boosting#179371085)\n    - [Bagging](https:\/\/www.ibm.com\/think\/topics\/bagging#179371086)\n    - [Gradient boosting](https:\/\/www.ibm.com\/think\/topics\/gradient-boosting#179371087)\n    - [Gradient boosting classifier](https:\/\/www.ibm.com\/think\/tutorials\/gradient-boosting-classifier#179371088)\n  - Self-supervised learning\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/self-supervised-learning#326187541)\n    - [Transfer learning](https:\/\/www.ibm.com\/think\/topics\/transfer-learning#326187540)\n- Unsupervised learning\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/unsupervised-learning#2014952965)\n  - Clustering\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/clustering#2142864945)\n    - [K means clustering](https:\/\/www.ibm.com\/think\/topics\/k-means-clustering#2142864944)\n    - [Hierarchical clustering](https:\/\/www.ibm.com\/think\/topics\/hierarchical-clustering#2142864943)\n  - [A priori algorithm](https:\/\/www.ibm.com\/think\/topics\/apriori-algorithm#2014952963)\n  - [Gaussian mixture model](https:\/\/www.ibm.com\/think\/topics\/gaussian-mixture-model#2014952962)\n  - [Anomaly detection](https:\/\/www.ibm.com\/think\/topics\/machine-learning-for-anomaly-detection#2014952961)\n- Semi-supervised learning\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/semi-supervised-learning#1774455706)\n  - Recommendation engine\n    - [Collaborative filtering](https:\/\/www.ibm.com\/think\/topics\/collaborative-filtering#1181250928)\n    - [Content based filtering](https:\/\/www.ibm.com\/think\/topics\/content-based-filtering#1181250927)\n- Reinforcement learning\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/reinforcement-learning#1268897081)\n  - [Reinforcement learning human feedback](https:\/\/www.ibm.com\/think\/topics\/rlhf#1268897082)\n- Deep Learning\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/deep-learning#763338456)\n  - Neural networks\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/neural-networks#741977106)\n    - [Backpropagation](https:\/\/www.ibm.com\/think\/topics\/backpropagation#741977107)\n    - [Encoder-decoder model](https:\/\/www.ibm.com\/think\/topics\/encoder-decoder-model#741977108)\n  - [Recurrent neural networks](https:\/\/www.ibm.com\/think\/topics\/recurrent-neural-networks#763338458)\n  - [Long short-term memory (LSTM)](https:\/\/www.ibm.com\/think\/topics\/lstm#763338459)\n  - [Convolutional neural networks](https:\/\/www.ibm.com\/think\/topics\/convolutional-neural-networks#763338460)\n  - Transformer models\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/transformer-model#1280257394)\n    - [Attention mechanism](https:\/\/www.ibm.com\/think\/topics\/attention-mechanism#1280257393)\n    - [Grouped query attention](https:\/\/www.ibm.com\/think\/topics\/grouped-query-attention#1280257392)\n    - [Positional encoding](https:\/\/www.ibm.com\/think\/topics\/positional-encoding#1280257391)\n  - [Autoencoder](https:\/\/www.ibm.com\/think\/topics\/autoencoder#763338462)\n  - [Mamba model](https:\/\/www.ibm.com\/think\/topics\/mamba-model#763338463)\n  - [Graph neural network](https:\/\/www.ibm.com\/think\/topics\/graph-neural-network#763338464)\n- Generative AI\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/generative-ai#257779831)\n  - [Generative model](https:\/\/www.ibm.com\/think\/topics\/generative-model#257779832)\n  - [Generative AI vs. predictive AI](https:\/\/www.ibm.com\/think\/topics\/generative-ai-vs-predictive-ai-whats-the-difference#257779833)\n  - Large language models (LLMs)\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/large-language-models#692473873)\n    - [Reasoning models](https:\/\/www.ibm.com\/think\/topics\/reasoning-model#692473874)\n    - [Small language models](https:\/\/www.ibm.com\/think\/topics\/small-language-models#692473875)\n    - [Instruction tuning](https:\/\/www.ibm.com\/think\/topics\/instruction-tuning#692473876)\n    - [LLM parameters](https:\/\/www.ibm.com\/think\/topics\/llm-parameters#692473877)\n    - [LLM temperature](https:\/\/www.ibm.com\/think\/topics\/llm-temperature#692473878)\n    - [LLM benchmarks](https:\/\/www.ibm.com\/think\/topics\/llm-benchmarks#692473879)\n    - [LLM customization](https:\/\/www.ibm.com\/think\/topics\/llm-customization#692473880)\n    - [LLM alignment](https:\/\/www.ibm.com\/think\/topics\/llm-alignment#692473881)\n  - AI image generation\n    - [Diffusion models](https:\/\/www.ibm.com\/think\/topics\/diffusion-models#186915248)\n    - [Variational autoencoder (VAE)](https:\/\/www.ibm.com\/think\/topics\/variational-autoencoder#186915249)\n    - [Generative adversarial networks (GANs)](https:\/\/www.ibm.com\/think\/topics\/generative-adversarial-networks#186915250)\n  - Multimodal AI\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/multimodal-ai#318643377)\n    - [Vision language models](https:\/\/www.ibm.com\/think\/topics\/vision-language-models#318643376)\n    - [Tutorial: Build an AI stylist](https:\/\/www.ibm.com\/think\/tutorials\/build-ai-stylist-with-granite-python-watsonx-ai#318643375)\n    - [Tutorial: Multimodal AI queries using Llama](https:\/\/www.ibm.com\/think\/tutorials\/multimodal-ai-python-llama#318643374)\n    - [Tutorial: Multimodal AI queries using Pixtral](https:\/\/www.ibm.com\/think\/tutorials\/multimodal-ai-pixtral-12b-on-watsonx-ai#318643373)\n    - [Tutorial: Automatic podcast transcription with Granite](https:\/\/www.ibm.com\/think\/tutorials\/automatic-speech-recognition-podcast-transcript-granite-watsonx-ai#318643372)\n    - [Tutorial: PPT AI image analysis answering system](https:\/\/www.ibm.com\/think\/tutorials\/build-ppt-ai-image-analysis-question-answering-system-granite#318643371)\n  - Retrieval augmented generation (RAG)\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/retrieval-augmented-generation#824202002)\n    - [GraphRAG](https:\/\/www.ibm.com\/think\/topics\/graphrag#824202001)\n    - [Tutorial: Build a multimodal RAG system with Docling and Granite](https:\/\/www.ibm.com\/think\/tutorials\/build-multimodal-rag-langchain-with-docling-granite#824202000)\n    - [Tutorial: Evaluate RAG pipline using Ragas](https:\/\/www.ibm.com\/think\/tutorials\/evaluate-rag-pipeline-using-ragas-in-python-with-watsonx#824201999)\n    - [Tutorial: RAG chunking strategies](https:\/\/www.ibm.com\/think\/tutorials\/chunking-strategies-for-rag-with-langchain-watsonx-ai#824201998)\n    - [Tutorial: Graph RAG using knowledge graphs](https:\/\/www.ibm.com\/think\/tutorials\/knowledge-graph-rag#824201997)\n    - [Tutorial: Inference scaling to improve multimodal RAG](https:\/\/www.ibm.com\/think\/tutorials\/inference-scaling-improve-multimodal-rag-granite#824201996)\n  - AI code generation\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/ai-code-generation#1329760627)\n    - [Vibe coding](https:\/\/www.ibm.com\/think\/topics\/vibe-coding#1329760626)\n  - AI agents\n    - [Visit the 2025 Guide to AI Agents](https:\/\/www.ibm.com\/think\/ai-agents#1835319252)\n- Model training\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/model-training#1580786329)\n  - [LLM training](https:\/\/www.ibm.com\/think\/topics\/llm-training#1580786328)\n  - [Loss function](https:\/\/www.ibm.com\/think\/topics\/loss-function#1580786327)\n  - [Training data](https:\/\/www.ibm.com\/think\/topics\/training-data#1580786326)\n  - [Model parameters](https:\/\/www.ibm.com\/think\/topics\/model-parameters#1580786325)\n  - Optimization algorithm\n    - [Gradient descent](https:\/\/www.ibm.com\/think\/topics\/gradient-descent#18238369)\n    - [Stochastic gradient descent](https:\/\/www.ibm.com\/think\/topics\/stochastic-gradient-descent#18238368)\n  - Model hyperparameters\n    - [Hyperparameter tuning](https:\/\/www.ibm.com\/think\/topics\/hyperparameter-tuning#523796994)\n    - [Learning rate](https:\/\/www.ibm.com\/think\/topics\/learning-rate#523796993)\n  - Fine tuning\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/fine-tuning#1029355619)\n    - [Parameter efficient fine tuning (PEFT)](https:\/\/www.ibm.com\/think\/topics\/parameter-efficient-fine-tuning#1029355618)\n    - [LoRA](https:\/\/www.ibm.com\/think\/topics\/lora#1029355617)\n    - [Tutorial: Fine tuning Granite model with LoRA](https:\/\/www.ibm.com\/think\/tutorials\/lora-fine-tuning-granite-llm#1029355616)\n  - [Regularization](https:\/\/www.ibm.com\/think\/topics\/regularization#1580786321)\n  - [Foundation models](https:\/\/www.ibm.com\/think\/topics\/foundation-models#1580786320)\n  - [Overfitting](https:\/\/www.ibm.com\/think\/topics\/overfitting#1759735864)\n  - [Underfitting](https:\/\/www.ibm.com\/think\/topics\/underfitting#1759735863)\n  - N-shot learning\n    - [Few shot learning](https:\/\/www.ibm.com\/think\/topics\/few-shot-learning#823293247)\n    - [Zero shot learning](https:\/\/www.ibm.com\/think\/topics\/zero-shot-learning#823293246)\n  - [Knowledge distillation](https:\/\/www.ibm.com\/think\/topics\/knowledge-distillation#1759735861)\n  - [Meta learning](https:\/\/www.ibm.com\/think\/topics\/meta-learning#1759735860)\n  - [Data augmentation](https:\/\/www.ibm.com\/think\/topics\/data-augmentation#1759735859)\n  - Continual learning\n    - [Catastrophic forgetting](https:\/\/www.ibm.com\/think\/topics\/catastrophic-forgetting#1449439549)\n- Machine learning libraries\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/machine-learning-libraries#2086344954)\n  - [Scikit-learn](https:\/\/www.ibm.com\/think\/topics\/scikit-learn#2086344953)\n  - [XGboost](https:\/\/www.ibm.com\/think\/topics\/xgboost#2086344952)\n  - [PyTorch](https:\/\/www.ibm.com\/think\/topics\/pytorch#2086344951)\n- MLOps\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/mlops#1703063717)\n  - [AI lifecyle](https:\/\/www.ibm.com\/think\/topics\/ai-lifecycle#1703063718)\n  - [AI inference](https:\/\/www.ibm.com\/think\/topics\/ai-inference#1703063719)\n  - [Model deployment](https:\/\/www.ibm.com\/think\/topics\/model-deployment#1703063720)\n  - [Machine learning pipeline](https:\/\/www.ibm.com\/think\/topics\/machine-learning-pipeline#1703063721)\n  - [Data labeling](https:\/\/www.ibm.com\/think\/topics\/data-labeling#1703063722)\n  - Model governance\n    - [Model risk management](https:\/\/www.ibm.com\/think\/topics\/model-risk-management#1399431040)\n    - [Model drift](https:\/\/www.ibm.com\/think\/topics\/model-drift#1399431041)\n  - [AutoML](https:\/\/www.ibm.com\/think\/topics\/automl#1703063724)\n  - [Model selection](https:\/\/www.ibm.com\/think\/topics\/model-selection#1703063725)\n  - [Federated learning](https:\/\/www.ibm.com\/think\/topics\/federated-learning#1703063726)\n  - [Distributed machine learning](https:\/\/www.ibm.com\/think\/topics\/distributed-machine-learning#1255367754)\n  - [AI stack](https:\/\/www.ibm.com\/think\/topics\/ai-stack#1255367755)\n- Natural language processing\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/natural-language-processing#1197505092)\n  - [Natural language understanding](https:\/\/www.ibm.com\/think\/topics\/natural-language-understanding#1197505093)\n  - Text classification\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/text-classification#88312261)\n    - [Sentiment analysis](https:\/\/www.ibm.com\/think\/topics\/sentiment-analysis#88312262)\n    - [Tutorial: Spam text classifier with PyTorch](https:\/\/www.ibm.com\/think\/tutorials\/text-classification-pytorch#88312263)\n  - [Machine translation](https:\/\/www.ibm.com\/think\/topics\/machine-translation#1197505095)\n  - Text mining\n    - [Overview](https:\/\/www.ibm.com\/think\/topics\/text-mining#922804989)\n    - [Information retrieval](https:\/\/www.ibm.com\/think\/topics\/information-retrieval#922804988)\n    - [Information extraction](https:\/\/www.ibm.com\/think\/topics\/information-extraction#922804987)\n    - [Topic modeling](https:\/\/www.ibm.com\/think\/topics\/topic-modeling#922804986)\n    - [Latent semantic analysis](https:\/\/www.ibm.com\/think\/topics\/latent-semantic-analysis#922804985)\n    - [Latent Dirichlet Allocation](https:\/\/www.ibm.com\/think\/topics\/latent-dirichlet-allocation#922804984)\n    - [Named entity recognition](https:\/\/www.ibm.com\/think\/topics\/named-entity-recognition#922804983)\n  - [Word embeddings](https:\/\/www.ibm.com\/think\/topics\/word-embeddings#1197505097)\n  - [Bag of words](https:\/\/www.ibm.com\/think\/topics\/bag-of-words#1197505098)\n  - [Intelligent search](https:\/\/www.ibm.com\/think\/topics\/intelligent-search#1197505099)\n  - [Speech recognition](https:\/\/www.ibm.com\/think\/topics\/speech-recognition#1197505100)\n  - [Stemming and lemmatization](https:\/\/www.ibm.com\/think\/topics\/stemming-lemmatization#1197505101)\n  - [Text summarization](https:\/\/www.ibm.com\/think\/topics\/text-summarization#1532047733)\n  - [Conversational AI](https:\/\/www.ibm.com\/think\/topics\/conversational-ai#1532047732)\n  - [Conversational analytics](https:\/\/www.ibm.com\/think\/topics\/conversational-analytics#1532047731)\n  - [Natural language generation](https:\/\/www.ibm.com\/think\/topics\/natural-language-generation#1532047730)\n- Computer vision\n  - [Overview](https:\/\/www.ibm.com\/think\/topics\/computer-vision#691946467)\n  - [Image classification](https:\/\/www.ibm.com\/think\/topics\/image-classification#691946468)\n  - [Object detection](https:\/\/www.ibm.com\/think\/topics\/object-detection#691946469)\n  - Image segmentation\n    - [Instance segmentation](https:\/\/www.ibm.com\/think\/topics\/instance-segmentation#544367653)\n    - [Semantic segmentation](https:\/\/www.ibm.com\/think\/topics\/semantic-segmentation#544367654)\n  - [Optical character recognition](https:\/\/www.ibm.com\/think\/topics\/optical-character-recognition#691946471)\n  - [Image recognition](https:\/\/www.ibm.com\/think\/topics\/image-recognition#691946472)\n  - [Visual inspection](https:\/\/www.ibm.com\/think\/topics\/visual-inspection#691946473)\n\n## Authors\n[Cole Stryker](https:\/\/www.ibm.com\/think\/author\/cole-stryker.html)\n\nStaff Editor, AI Models\n\nIBM Think\n\n[Jim Holdsworth](https:\/\/www.ibm.com\/think\/author\/jim-holdsworth.html)\n\nStaff Writer\n\nIBM Think\n## What is NLP?\nNatural language processing (NLP) is a subfield of computer science and [artificial intelligence (AI)](https:\/\/www.ibm.com\/think\/topics\/artificial-intelligence) that uses [machine learning](https:\/\/www.ibm.com\/think\/topics\/machine-learning) to enable computers to understand and communicate with human language.\n\nNLP enables computers and digital devices to recognize, understand and generate text and speech by combining computational linguistics, the rule-based modeling of human language together with statistical modeling, machine learning and [deep learning](https:\/\/www.ibm.com\/think\/topics\/deep-learning).\n\nNLP research has helped enable the era of [generative AI](https:\/\/www.ibm.com\/think\/topics\/generative-ai), from the communication skills of [large language models](https:\/\/www.ibm.com\/think\/topics\/large-language-models) (LLMs) to the ability of image generation models to understand requests. NLP is already part of everyday life for many, powering search engines, prompting [chatbots](https:\/\/www.ibm.com\/think\/topics\/chatbots) for customer service with spoken commands, voice-operated GPS systems and question-answering digital assistants on smartphones such as Amazon’s Alexa, Apple’s Siri and Microsoft’s Cortana.\n\nNLP also plays a growing role in enterprise solutions that help streamline and automate business operations, increase employee productivity and simplify business processes.\n## Benefits of NLP\nNLP makes it easier for humans to communicate and collaborate with machines, by allowing them to do so in the natural human language they use every day. This offers benefits across many industries and applications.\n\n- Automation of repetitive tasks\n- Improved data analysis and insights\n- Enhanced search\n- Content generation\n### Automation of repetitive tasks\nNLP is especially useful in fully or partially [automating tasks](https:\/\/www.ibm.com\/think\/topics\/automation) like customer support, data entry and document handling. For example, NLP-powered chatbots can handle routine customer queries, freeing up human agents for more complex issues. In [document processing](https:\/\/www.ibm.com\/think\/topics\/document-processing), NLP tools can automatically classify, extract key information and summarize content, reducing the time and errors associated with manual data handling. NLP facilitates language translation, converting text from one language to another while preserving meaning, context and nuances.\n\n### Improved data analysis\nNLP enhances data analysis by enabling the extraction of insights from unstructured text data, such as customer reviews, social media posts and news articles. By using [text mining](https:\/\/www.ibm.com\/think\/topics\/text-mining-use-cases) techniques, NLP can identify patterns, trends and sentiments that are not immediately obvious in large datasets. Sentiment analysis enables the [extraction of subjective qualities](https:\/\/www.ibm.com\/think\/topics\/sentiment-analysis), attitudes, emotions, sarcasm, confusion or suspicion from text. This is often used for routing communications to the system or the person most likely to make the next response.\n\nThis allows businesses to better understand customer preferences, market conditions and public opinion. NLP tools can also perform categorization and summarization of vast amounts of text, making it easier for analysts to identify key information and make data-driven decisions more efficiently.\n### Enhanced search\nNLP benefits search by enabling systems to understand the intent behind user queries, providing more accurate and contextually relevant results. Instead of relying solely on keyword matching, NLP-powered search engines analyze the meaning of words and phrases, making it easier to find information even when queries are vague or complex. This improves user experience, whether in web searches, document retrieval or enterprise data systems.\n\n### Powerful content generation\nNLP powers advanced language models to [create human-like text](https:\/\/www.ibm.com\/think\/topics\/text-generation) for various purposes. Pre-trained models, such as GPT-4, can generate articles, reports, marketing copy, product descriptions and even creative writing based on prompts provided by users. NLP-powered tools can also assist in automating tasks like drafting emails, writing social media posts or legal documentation. By understanding context, tone and style, NLP sees to it that the generated content is coherent, relevant and aligned with the intended message, saving time and effort in content creation while maintaining quality.\n\n## Approaches to NLP\nNLP combines the power of computational linguistics together with [machine learning algorithms](https:\/\/www.ibm.com\/think\/topics\/machine-learning-algorithms) and deep learning. Computational linguistics uses data science to analyze language and speech. It includes two main types of analysis: syntactical analysis and semantical analysis. Syntactical analysis determines the meaning of a word, phrase or sentence by parsing the syntax of the words and applying preprogrammed rules of grammar. Semantical analysis uses the syntactic output to draw meaning from the words and interpret their meaning within the sentence structure.\n\nThe parsing of words can take one of two forms. Dependency parsing looks at the relationships between words, such as identifying nouns and verbs, while constituency parsing then builds a parse tree (or syntax tree): a rooted and ordered representation of the syntactic structure of the sentence or string of words. The resulting parse trees underly the functions of language translators and speech recognition. Ideally, this analysis makes the output either text or speech understandable to both NLP models and people.\n\n[Self-supervised learning (SSL)](https:\/\/www.ibm.com\/think\/topics\/self-supervised-learning) in particular is useful for supporting NLP because NLP requires large amounts of labeled data to train AI models. Because these labeled datasets require time-consuming annotation, a process involving manual labeling by humans, gathering sufficient data can be prohibitively difficult. Self-supervised approaches can be more time-effective and cost-effective, as they replace some or all manually labeled training data.  \n   \n Three different approaches to NLP include:\n### Rules-based NLP\nThe earliest NLP applications were simple if-then decision trees, requiring preprogrammed rules. They are only able to provide answers in response to specific prompts, such as the original version of Moviefone, which had rudimentary natural language generation (NLG) capabilities. Because there is no machine learning or AI capability in rules-based NLP, this function is highly limited and not scalable.\n\n### Statistical NLP\nDeveloped later, statistical NLP automatically extracts, classifies and labels elements of text and voice data and then assigns a statistical likelihood to each possible meaning of those elements. This relies on machine learning, enabling a sophisticated breakdown of linguistics such as part-of-speech tagging.  \n   \n Statistical NLP introduced the essential technique of mapping language elements, such as words and grammatical rules to a vector representation so that language can be modeled by using mathematical (statistical) methods, including regression or Markov models. This informed early NLP developments such as spellcheckers and T9 texting (Text on 9 keys, to be used on Touch-Tone telephones).\n\n### Deep learning NLP\nRecently, deep learning models have become the dominant mode of NLP, by using huge volumes of raw, [unstructured](https:\/\/www.ibm.com\/think\/topics\/text-mining) data both text and voice to become ever more accurate. Deep learning can be viewed as a further evolution of statistical NLP, with the difference that it uses [neural network](https:\/\/www.ibm.com\/think\/topics\/neural-networks) models. There are several subcategories of models:\n\n- *Sequence-to-Sequence* (seq2seq) models: Based [on recurrent neural networks (RNN)](https:\/\/www.ibm.com\/think\/topics\/recurrent-neural-networks), they have mostly been used for machine translation by converting a phrase from one domain (such as the German language) into the phrase of another domain (such as English).\n\n- *Transformer models*: They use [tokenization](https:\/\/www.ibm.com\/docs\/en\/watsonx\/saas?topic=solutions-tokens) of language (the position of each token words or subwords) and self-attention (capturing dependencies and relationships) to calculate the relation of different language parts to one another. [Transformer models](https:\/\/www.ibm.com\/think\/topics\/transformer-model) can be efficiently trained by using [self-supervised learning](https:\/\/www.ibm.com\/think\/topics\/self-supervised-learning) on massive text databases. A landmark in [transformer models](https:\/\/www.ibm.com\/think\/topics\/transformer-model) was Google’s bidirectional encoder representations from transformers (BERT), which became and remains the basis of how Google’s search engine works.\n\n- *Autoregressive models*: This type of transformer model is trained specifically to predict the next word in a sequence, which represents a huge leap forward in the ability to generate text. Examples of autoregressive LLMs include GPT, [Llama](https:\/\/www.ibm.com\/think\/topics\/llama-2), Claude and the open-source Mistral.\n\n- *Foundation models*: Prebuilt and curated foundation models can speed the launching of an NLP effort and boost trust in its operation. For example, the [IBM® Granite™](https:\/\/www.ibm.com\/think\/news\/granite-foundation-models) foundation models are widely applicable across industries. They support NLP tasks including content generation and insight extraction. Additionally, they facilitate retrieval-augmented generation, a framework for improving the quality of response by linking the model to external sources of knowledge. The models also perform named entity recognition which involves identifying and extracting key information in a text.\n## NLP Tasks\nSeveral NLP tasks typically help process human text and voice data in ways that help the computer make sense of what it’s ingesting. Some of these tasks include:\n\n- Coreference resolution\n- Named entity recognition\n- Part-of-speech tagging\n- Word sense disambiguation\n### Coreference resolution\nThis is the task of identifying if and when two words refer to the same entity. The most common example is determining the person or object to which a certain pronoun refers (such as “she” = “Mary”). But it can also identify a metaphor or an idiom in the text (such as an instance in which “bear” isn’t an animal, but a large and hairy person).\n\n### Named entity recognition (NER)\n[NER](https:\/\/www.ibm.com\/think\/topics\/named-entity-recognition) identifies words or phrases as useful entities. NER identifies “London” as a location or “Maria” as a person's name.\n\n### Part-of-speech tagging\nAlso called grammatical tagging, this is the process of determining which part of speech a word or piece of text is, based on its use and context. For example, part-of-speech identifies “make” as a verb in “I can make a paper plane,” and as a noun in “What make of car do you own?”\n\n### Word sense disambiguation\nThis is the selection of a word meaning for a word with multiple possible meanings. This uses a process of semantic [analysis](https:\/\/www.ibm.com\/consulting\/analytics) to examine the word in context. For example, word sense disambiguation helps distinguish the meaning of the verb “make” in “make the grade” (to achieve) versus “make a bet” (to place). Sorting out “I will be merry when I marry Mary” requires a sophisticated NLP system.\n\n## How NLP works\nNLP works by combining various computational techniques to analyze, understand and generate human language in a way that machines can process. Here is an overview of a typical NLP pipeline and its steps:\n\n### Text preprocessing\nNLP text preprocessing prepares raw text for analysis by transforming it into a format that machines can more easily understand. It begins with tokenization, which involves splitting the text into smaller units like words, sentences or phrases. This helps break down complex text into manageable parts. Next, lowercasing is applied to standardize the text by converting all characters to lowercase, ensuring that words like \"Apple\" and \"apple\" are treated the same. Stop word removal is another common step, where frequently used words like \"is\" or \"the\" are filtered out because they don't add significant meaning to the text. [Stemming](https:\/\/www.ibm.com\/think\/topics\/stemming) or [lemmatization](https:\/\/www.ibm.com\/think\/topics\/stemming-lemmatization) reduces words to their root form (e.g., \"running\" becomes \"run\"), making it easier to analyze language by grouping different forms of the same word. Additionally, text cleaning removes unwanted elements such as punctuation, special characters and numbers that may clutter the analysis.\n\nAfter preprocessing, the text is clean, standardized and ready for machine learning models to interpret effectively.\n### Feature extraction\nFeature extraction is the process of converting raw text into numerical representations that machines can analyze and interpret. This involves transforming text into structured data by using NLP techniques like [Bag of Words](https:\/\/www.ibm.com\/think\/topics\/bag-of-words) and TF-IDF, which quantify the presence and importance of words in a document. More advanced methods include [word embeddings](https:\/\/www.ibm.com\/think\/topics\/embedding) like Word2Vec or GloVe, which represent words as dense vectors in a continuous space, capturing semantic relationships between words. Contextual embeddings further enhance this by considering the context in which words appear, allowing for richer, more nuanced representations.\n\n### Text analysis\nText analysis involves interpreting and extracting meaningful information from text data through various computational techniques. This process includes tasks such as part-of-speech (POS) tagging, which identifies grammatical roles of words and named entity recognition (NER), which detects specific entities like names, locations and dates. Dependency parsing analyzes grammatical relationships between words to understand sentence structure, while sentiment analysis determines the emotional tone of the text, assessing whether it is positive, negative or neutral. Topic modeling identifies underlying themes or topics within a text or across a corpus of documents. Natural language understanding (NLU) is a subset of NLP that focuses on analyzing the meaning behind sentences. NLU enables software to find similar meanings in different sentences or to process words that have different meanings. Through these techniques, NLP text analysis transforms unstructured text into insights.\n\n### Model training\nProcessed data is then used to train machine learning models, which learn patterns and relationships within the data. During training, the model adjusts its parameters to minimize errors and improve its performance. Once trained, the model can be used to make predictions or generate outputs on new, unseen data. The effectiveness of NLP modeling is continually refined through evaluation, validation and fine-tuning to enhance accuracy and relevance in real-world applications.\n\nDifferent software environments are useful throughout the said processes. For example, the Natural Language Toolkit (NLTK) is a suite of libraries and programs for English that is written in the Python programming language. It supports text classification, tokenization, stemming, tagging, parsing and semantic reasoning functionalities. TensorFlow is a free and open-source software library for machine learning and AI that can be used to train models for NLP applications. Tutorials and certifications abound for those interested in familiarizing themselves with such tools.\n## Challenges of NLP\nEven state-of-the-art NLP models are not perfect, just as human speech is prone to error. As with any AI technology, NLP comes with potential pitfalls. Human language is filled with ambiguities that make it difficult for programmers to write software that accurately determines the intended meaning of text or voice data. Human language might take years for humans to learn and many never stop learning. But then programmers must teach natural language-powered applications to recognize and understand irregularities so their applications can be accurate and useful. Associated risks might include:\n\n### Biased training\nAs with any AI function, [biased data](https:\/\/www.ibm.com\/think\/topics\/ai-bias) used in training will skew the answers. The more diverse the users of an NLP function, the more significant this risk becomes, such as in government services, healthcare and HR interactions. Training datasets scraped from the web, for example, are prone to bias.\n\n### Misinterpretation\nAs in programming, there is a risk of garbage in, garbage out (GIGO). [Speech recognition](https:\/\/www.ibm.com\/think\/topics\/speech-recognition), also known as speech-to-text, is the task of reliably converting voice data into text data. But NLP solutions can become confused if spoken input is in an obscure dialect, mumbled, too full of slang, homonyms, incorrect grammar, idioms, fragments, mispronunciations, contractions or recorded with too much background noise.\n\n### New vocabulary\nNew words are continually being invented or imported. The conventions of grammar can evolve or be intentionally broken. In these cases, NLP can either make a best guess or admit it’s unsure and either way, this creates a complication.\n\n### Tone of voice\nWhen people speak, their verbal delivery or even body language can give an entirely different meaning than the words alone. Exaggeration for effect, stressing words for importance or sarcasm can be confused by NLP, making the semantic analysis more difficult and less reliable.\n\n## NLP use cases by industry\nNLP applications can now be found across virtually every industry.\n\n### Finance\nIn financial dealings, nanoseconds might make the difference between success and failure when accessing data, or making trades or deals. NLP can speed the mining of information from financial statements, annual and regulatory reports, news releases or even social media.\n\n### Healthcare\nNew medical insights and breakthroughs can arrive faster than many healthcare professionals can keep up. NLP and AI-based tools can help speed the analysis of health records and medical research papers, making better-informed medical decisions possible, or assisting in the detection or even prevention of medical conditions.\n\n### Insurance\nNLP can analyze claims to look for patterns that can identify areas of concern and find inefficiencies in claims processing, leading to greater optimization of processing and employee efforts.\n\n### Legal\nAlmost any legal case might require reviewing mounds of paperwork, background information and legal precedent. NLP can help automate legal discovery, assisting in the organization of information, speeding review and making sure that all relevant details are captured for consideration.\n\nLink copied\n\n[Report Beyond the hype - How AI assistants drive real business value Explore top use cases for leveraging AI assistants, understand the potential impact of Gen AI and automation technology on your business, and learn how to get started. Read the report](https:\/\/www.ibm.com\/account\/reg\/signup?formid=urx-53011)\n\n## Resources\n[AI models Explore IBM Granite® IBM Granite ® is our family of open, performant and trusted AI models, tailored for business and optimized to scale your AI applications. Explore language, code, time series and guardrail options. Meet Granite](https:\/\/www.ibm.com\/granite)\n\n[Explainer Beginner's guide to NLP Discover how natural language processing (NLP) can help you to converse more naturally with computers. Learn NLP](https:\/\/developer.ibm.com\/learningpaths\/get-started-artificial-intelligence\/next-steps\/beginner-natural-language-processing)\n\n[Report IBM is named leader in data science and machine learning Learn why IBM has been recognized as a Leader in the 2025 Gartner® Magic Quadrant™ for data science and machine learning platforms. Read the report](https:\/\/www.ibm.com\/forms\/mkt-53728)\n\n[Training Hands-on with generative AI Learn fundamental concepts and build your skills with hands-on labs, courses, guided projects, trials and more. Learn generative AI](https:\/\/developer.ibm.com\/technologies\/generative-ai\/?cm_sp=ibmdev-_-developer-_-getstarted-_-genai)\n\nRelated solutions\n\nIBM® watsonx Orchestrate™\n\nEasily design scalable AI assistants and agents, automate repetitive tasks and simplify complex processes with IBM® watsonx Orchestrate™.\n\n[Explore watsonx Orchestrate](https:\/\/www.ibm.com\/products\/watsonx-orchestrate)\n\nNatural language processing tools and APIs\n\nAccelerate the business value of artificial intelligence with a powerful and flexible portfolio of libraries, services and applications.\n\n[Explore NLP solutions](https:\/\/www.ibm.com\/solutions\/natural-language-processing)\n\nAI consulting and services\n\nReinvent critical workflows and operations by adding AI to maximize experiences, real-time decision-making and business value.\n\n[Explore AI services](https:\/\/www.ibm.com\/consulting\/artificial-intelligence)\n\nTake the next step\n\nEasily design scalable AI assistants and agents, automate repetitive tasks and simplify complex processes with IBM® watsonx Orchestrate™.\n\n1. [Explore watsonx Orchestrate](https:\/\/www.ibm.com\/products\/watsonx-orchestrate)\n2. [Explore NLP solutions](https:\/\/www.ibm.com\/solutions\/natural-language-processing)","attrs_readable_markdown":"Natural language processing (NLP) is a subfield of computer science and [artificial intelligence (AI)](https:\/\/www.ibm.com\/think\/topics\/artificial-intelligence) that uses [machine learning](https:\/\/www.ibm.com\/think\/topics\/machine-learning) to enable computers to understand and communicate with human language.\n\nNLP enables computers and digital devices to recognize, understand and generate text and speech by combining computational linguistics, the rule-based modeling of human language together with statistical modeling, machine learning and [deep learning](https:\/\/www.ibm.com\/think\/topics\/deep-learning).\n\nNLP research has helped enable the era of [generative AI](https:\/\/www.ibm.com\/think\/topics\/generative-ai), from the communication skills of [large language models](https:\/\/www.ibm.com\/think\/topics\/large-language-models) (LLMs) to the ability of image generation models to understand requests. NLP is already part of everyday life for many, powering search engines, prompting [chatbots](https:\/\/www.ibm.com\/think\/topics\/chatbots) for customer service with spoken commands, voice-operated GPS systems and question-answering digital assistants on smartphones such as Amazon’s Alexa, Apple’s Siri and Microsoft’s Cortana.\n\nNLP also plays a growing role in enterprise solutions that help streamline and automate business operations, increase employee productivity and simplify business processes.\n\nNLP makes it easier for humans to communicate and collaborate with machines, by allowing them to do so in the natural human language they use every day. This offers benefits across many industries and applications.\n\n- Automation of repetitive tasks\n- Improved data analysis and insights\n- Enhanced search\n- Content generation\n\nAutomation of repetitive tasks\n\nNLP is especially useful in fully or partially [automating tasks](https:\/\/www.ibm.com\/think\/topics\/automation) like customer support, data entry and document handling. For example, NLP-powered chatbots can handle routine customer queries, freeing up human agents for more complex issues. In [document processing](https:\/\/www.ibm.com\/think\/topics\/document-processing), NLP tools can automatically classify, extract key information and summarize content, reducing the time and errors associated with manual data handling. NLP facilitates language translation, converting text from one language to another while preserving meaning, context and nuances.\n\nImproved data analysis\n\nNLP enhances data analysis by enabling the extraction of insights from unstructured text data, such as customer reviews, social media posts and news articles. By using [text mining](https:\/\/www.ibm.com\/think\/topics\/text-mining-use-cases) techniques, NLP can identify patterns, trends and sentiments that are not immediately obvious in large datasets. Sentiment analysis enables the [extraction of subjective qualities](https:\/\/www.ibm.com\/think\/topics\/sentiment-analysis), attitudes, emotions, sarcasm, confusion or suspicion from text. This is often used for routing communications to the system or the person most likely to make the next response.\n\nThis allows businesses to better understand customer preferences, market conditions and public opinion. NLP tools can also perform categorization and summarization of vast amounts of text, making it easier for analysts to identify key information and make data-driven decisions more efficiently.\n\nEnhanced search\n\nNLP benefits search by enabling systems to understand the intent behind user queries, providing more accurate and contextually relevant results. Instead of relying solely on keyword matching, NLP-powered search engines analyze the meaning of words and phrases, making it easier to find information even when queries are vague or complex. This improves user experience, whether in web searches, document retrieval or enterprise data systems.\n\nPowerful content generation\n\nNLP powers advanced language models to [create human-like text](https:\/\/www.ibm.com\/think\/topics\/text-generation) for various purposes. Pre-trained models, such as GPT-4, can generate articles, reports, marketing copy, product descriptions and even creative writing based on prompts provided by users. NLP-powered tools can also assist in automating tasks like drafting emails, writing social media posts or legal documentation. By understanding context, tone and style, NLP sees to it that the generated content is coherent, relevant and aligned with the intended message, saving time and effort in content creation while maintaining quality.\n\nNLP combines the power of computational linguistics together with [machine learning algorithms](https:\/\/www.ibm.com\/think\/topics\/machine-learning-algorithms) and deep learning. Computational linguistics uses data science to analyze language and speech. It includes two main types of analysis: syntactical analysis and semantical analysis. Syntactical analysis determines the meaning of a word, phrase or sentence by parsing the syntax of the words and applying preprogrammed rules of grammar. Semantical analysis uses the syntactic output to draw meaning from the words and interpret their meaning within the sentence structure.\n\nThe parsing of words can take one of two forms. Dependency parsing looks at the relationships between words, such as identifying nouns and verbs, while constituency parsing then builds a parse tree (or syntax tree): a rooted and ordered representation of the syntactic structure of the sentence or string of words. The resulting parse trees underly the functions of language translators and speech recognition. Ideally, this analysis makes the output either text or speech understandable to both NLP models and people.\n\n[Self-supervised learning (SSL)](https:\/\/www.ibm.com\/think\/topics\/self-supervised-learning) in particular is useful for supporting NLP because NLP requires large amounts of labeled data to train AI models. Because these labeled datasets require time-consuming annotation, a process involving manual labeling by humans, gathering sufficient data can be prohibitively difficult. Self-supervised approaches can be more time-effective and cost-effective, as they replace some or all manually labeled training data.\n\nThree different approaches to NLP include:\n\nRules-based NLP\n\nThe earliest NLP applications were simple if-then decision trees, requiring preprogrammed rules. They are only able to provide answers in response to specific prompts, such as the original version of Moviefone, which had rudimentary natural language generation (NLG) capabilities. Because there is no machine learning or AI capability in rules-based NLP, this function is highly limited and not scalable.\n\nStatistical NLP\n\nDeveloped later, statistical NLP automatically extracts, classifies and labels elements of text and voice data and then assigns a statistical likelihood to each possible meaning of those elements. This relies on machine learning, enabling a sophisticated breakdown of linguistics such as part-of-speech tagging.\n\nStatistical NLP introduced the essential technique of mapping language elements, such as words and grammatical rules to a vector representation so that language can be modeled by using mathematical (statistical) methods, including regression or Markov models. This informed early NLP developments such as spellcheckers and T9 texting (Text on 9 keys, to be used on Touch-Tone telephones).\n\nDeep learning NLP\n\nRecently, deep learning models have become the dominant mode of NLP, by using huge volumes of raw, [unstructured](https:\/\/www.ibm.com\/think\/topics\/text-mining) data both text and voice to become ever more accurate. Deep learning can be viewed as a further evolution of statistical NLP, with the difference that it uses [neural network](https:\/\/www.ibm.com\/think\/topics\/neural-networks) models. There are several subcategories of models:\n\n- *Sequence-to-Sequence* (seq2seq) models: Based [on recurrent neural networks (RNN)](https:\/\/www.ibm.com\/think\/topics\/recurrent-neural-networks), they have mostly been used for machine translation by converting a phrase from one domain (such as the German language) into the phrase of another domain (such as English).\n\n- *Transformer models*: They use [tokenization](https:\/\/www.ibm.com\/docs\/en\/watsonx\/saas?topic=solutions-tokens) of language (the position of each token words or subwords) and self-attention (capturing dependencies and relationships) to calculate the relation of different language parts to one another. [Transformer models](https:\/\/www.ibm.com\/think\/topics\/transformer-model) can be efficiently trained by using [self-supervised learning](https:\/\/www.ibm.com\/think\/topics\/self-supervised-learning) on massive text databases. A landmark in [transformer models](https:\/\/www.ibm.com\/think\/topics\/transformer-model) was Google’s bidirectional encoder representations from transformers (BERT), which became and remains the basis of how Google’s search engine works.\n\n- *Autoregressive models*: This type of transformer model is trained specifically to predict the next word in a sequence, which represents a huge leap forward in the ability to generate text. Examples of autoregressive LLMs include GPT, [Llama](https:\/\/www.ibm.com\/think\/topics\/llama-2), Claude and the open-source Mistral.\n\n- *Foundation models*: Prebuilt and curated foundation models can speed the launching of an NLP effort and boost trust in its operation. For example, the [IBM® Granite™](https:\/\/www.ibm.com\/think\/news\/granite-foundation-models) foundation models are widely applicable across industries. They support NLP tasks including content generation and insight extraction. Additionally, they facilitate retrieval-augmented generation, a framework for improving the quality of response by linking the model to external sources of knowledge. The models also perform named entity recognition which involves identifying and extracting key information in a text.\n\nSeveral NLP tasks typically help process human text and voice data in ways that help the computer make sense of what it’s ingesting. Some of these tasks include:\n\n- Coreference resolution\n- Named entity recognition\n- Part-of-speech tagging\n- Word sense disambiguation\n\nCoreference resolution\n\nThis is the task of identifying if and when two words refer to the same entity. The most common example is determining the person or object to which a certain pronoun refers (such as “she” = “Mary”). But it can also identify a metaphor or an idiom in the text (such as an instance in which “bear” isn’t an animal, but a large and hairy person).\n\nNamed entity recognition (NER)\n\n[NER](https:\/\/www.ibm.com\/think\/topics\/named-entity-recognition) identifies words or phrases as useful entities. NER identifies “London” as a location or “Maria” as a person's name.\n\nPart-of-speech tagging\n\nAlso called grammatical tagging, this is the process of determining which part of speech a word or piece of text is, based on its use and context. For example, part-of-speech identifies “make” as a verb in “I can make a paper plane,” and as a noun in “What make of car do you own?”\n\nWord sense disambiguation\n\nThis is the selection of a word meaning for a word with multiple possible meanings. This uses a process of semantic [analysis](https:\/\/www.ibm.com\/consulting\/analytics) to examine the word in context. For example, word sense disambiguation helps distinguish the meaning of the verb “make” in “make the grade” (to achieve) versus “make a bet” (to place). Sorting out “I will be merry when I marry Mary” requires a sophisticated NLP system.\n\nNLP works by combining various computational techniques to analyze, understand and generate human language in a way that machines can process. Here is an overview of a typical NLP pipeline and its steps:\n\nText preprocessing\n\nNLP text preprocessing prepares raw text for analysis by transforming it into a format that machines can more easily understand. It begins with tokenization, which involves splitting the text into smaller units like words, sentences or phrases. This helps break down complex text into manageable parts. Next, lowercasing is applied to standardize the text by converting all characters to lowercase, ensuring that words like \"Apple\" and \"apple\" are treated the same. Stop word removal is another common step, where frequently used words like \"is\" or \"the\" are filtered out because they don't add significant meaning to the text. [Stemming](https:\/\/www.ibm.com\/think\/topics\/stemming) or [lemmatization](https:\/\/www.ibm.com\/think\/topics\/stemming-lemmatization) reduces words to their root form (e.g., \"running\" becomes \"run\"), making it easier to analyze language by grouping different forms of the same word. Additionally, text cleaning removes unwanted elements such as punctuation, special characters and numbers that may clutter the analysis.\n\nAfter preprocessing, the text is clean, standardized and ready for machine learning models to interpret effectively.\n\nFeature extraction\n\nFeature extraction is the process of converting raw text into numerical representations that machines can analyze and interpret. This involves transforming text into structured data by using NLP techniques like [Bag of Words](https:\/\/www.ibm.com\/think\/topics\/bag-of-words) and TF-IDF, which quantify the presence and importance of words in a document. More advanced methods include [word embeddings](https:\/\/www.ibm.com\/think\/topics\/embedding) like Word2Vec or GloVe, which represent words as dense vectors in a continuous space, capturing semantic relationships between words. Contextual embeddings further enhance this by considering the context in which words appear, allowing for richer, more nuanced representations.\n\nText analysis\n\nText analysis involves interpreting and extracting meaningful information from text data through various computational techniques. This process includes tasks such as part-of-speech (POS) tagging, which identifies grammatical roles of words and named entity recognition (NER), which detects specific entities like names, locations and dates. Dependency parsing analyzes grammatical relationships between words to understand sentence structure, while sentiment analysis determines the emotional tone of the text, assessing whether it is positive, negative or neutral. Topic modeling identifies underlying themes or topics within a text or across a corpus of documents. Natural language understanding (NLU) is a subset of NLP that focuses on analyzing the meaning behind sentences. NLU enables software to find similar meanings in different sentences or to process words that have different meanings. Through these techniques, NLP text analysis transforms unstructured text into insights.\n\nModel training\n\nProcessed data is then used to train machine learning models, which learn patterns and relationships within the data. During training, the model adjusts its parameters to minimize errors and improve its performance. Once trained, the model can be used to make predictions or generate outputs on new, unseen data. The effectiveness of NLP modeling is continually refined through evaluation, validation and fine-tuning to enhance accuracy and relevance in real-world applications.\n\nDifferent software environments are useful throughout the said processes. For example, the Natural Language Toolkit (NLTK) is a suite of libraries and programs for English that is written in the Python programming language. It supports text classification, tokenization, stemming, tagging, parsing and semantic reasoning functionalities. TensorFlow is a free and open-source software library for machine learning and AI that can be used to train models for NLP applications. Tutorials and certifications abound for those interested in familiarizing themselves with such tools.\n\nEven state-of-the-art NLP models are not perfect, just as human speech is prone to error. As with any AI technology, NLP comes with potential pitfalls. Human language is filled with ambiguities that make it difficult for programmers to write software that accurately determines the intended meaning of text or voice data. Human language might take years for humans to learn and many never stop learning. But then programmers must teach natural language-powered applications to recognize and understand irregularities so their applications can be accurate and useful. Associated risks might include:\n\nBiased training\n\nAs with any AI function, [biased data](https:\/\/www.ibm.com\/think\/topics\/ai-bias) used in training will skew the answers. The more diverse the users of an NLP function, the more significant this risk becomes, such as in government services, healthcare and HR interactions. Training datasets scraped from the web, for example, are prone to bias.\n\nMisinterpretation\n\nAs in programming, there is a risk of garbage in, garbage out (GIGO). [Speech recognition](https:\/\/www.ibm.com\/think\/topics\/speech-recognition), also known as speech-to-text, is the task of reliably converting voice data into text data. But NLP solutions can become confused if spoken input is in an obscure dialect, mumbled, too full of slang, homonyms, incorrect grammar, idioms, fragments, mispronunciations, contractions or recorded with too much background noise.\n\nNew vocabulary\n\nNew words are continually being invented or imported. The conventions of grammar can evolve or be intentionally broken. In these cases, NLP can either make a best guess or admit it’s unsure and either way, this creates a complication.\n\nTone of voice\n\nWhen people speak, their verbal delivery or even body language can give an entirely different meaning than the words alone. Exaggeration for effect, stressing words for importance or sarcasm can be confused by NLP, making the semantic analysis more difficult and less reliable.\n\nNLP applications can now be found across virtually every industry.\n\nFinance\n\nIn financial dealings, nanoseconds might make the difference between success and failure when accessing data, or making trades or deals. NLP can speed the mining of information from financial statements, annual and regulatory reports, news releases or even social media.\n\nHealthcare\n\nNew medical insights and breakthroughs can arrive faster than many healthcare professionals can keep up. NLP and AI-based tools can help speed the analysis of health records and medical research papers, making better-informed medical decisions possible, or assisting in the detection or even prevention of medical conditions.\n\nInsurance\n\nNLP can analyze claims to look for patterns that can identify areas of concern and find inefficiencies in claims processing, leading to greater optimization of processing and employee efforts.\n\nLegal\n\nAlmost any legal case might require reviewing mounds of paperwork, background information and legal precedent. NLP can help automate legal discovery, assisting in the organization of information, speeding review and making sure that all relevant details are captured for consideration.","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

6 days ago

🤖

ROBOTS SERVER UNREACHABLE

Failed to connect to robots server: Operation timed out after 2001 milliseconds with 0 bytes received

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.2 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://www.ibm.com/think/topics/natural-language-processing
Last Crawled	2026-03-31 04:27:49 (6 days ago)
First Indexed	2024-12-18 13:03:01 (1 year ago)
HTTP Status Code	200
Meta Title	What Is NLP (Natural Language Processing)? \| IBM
Meta Description	Natural language processing (NLP) is a subfield of artificial intelligence (AI) that uses machine learning to help computers communicate with human language.
Meta Canonical	null
Boilerpipe Text	Natural language processing (NLP) is a subfield of computer science and artificial intelligence (AI) that uses machine learning to enable computers to understand and communicate with human language. NLP enables computers and digital devices to recognize, understand and generate text and speech by combining computational linguistics, the rule-based modeling of human language together with statistical modeling, machine learning and deep learning . NLP research has helped enable the era of generative AI , from the communication skills of large language models (LLMs) to the ability of image generation models to understand requests. NLP is already part of everyday life for many, powering search engines, prompting chatbots for customer service with spoken commands, voice-operated GPS systems and question-answering digital assistants on smartphones such as Amazon’s Alexa, Apple’s Siri and Microsoft’s Cortana. NLP also plays a growing role in enterprise solutions that help streamline and automate business operations, increase employee productivity and simplify business processes. NLP makes it easier for humans to communicate and collaborate with machines, by allowing them to do so in the natural human language they use every day. This offers benefits across many industries and applications. Automation of repetitive tasks Improved data analysis and insights Enhanced search Content generation Automation of repetitive tasks NLP is especially useful in fully or partially automating tasks like customer support, data entry and document handling. For example, NLP-powered chatbots can handle routine customer queries, freeing up human agents for more complex issues. In document processing , NLP tools can automatically classify, extract key information and summarize content, reducing the time and errors associated with manual data handling. NLP facilitates language translation, converting text from one language to another while preserving meaning, context and nuances. Improved data analysis NLP enhances data analysis by enabling the extraction of insights from unstructured text data, such as customer reviews, social media posts and news articles. By using text mining techniques, NLP can identify patterns, trends and sentiments that are not immediately obvious in large datasets. Sentiment analysis enables the extraction of subjective qualities , attitudes, emotions, sarcasm, confusion or suspicion from text. This is often used for routing communications to the system or the person most likely to make the next response. This allows businesses to better understand customer preferences, market conditions and public opinion. NLP tools can also perform categorization and summarization of vast amounts of text, making it easier for analysts to identify key information and make data-driven decisions more efficiently. Enhanced search NLP benefits search by enabling systems to understand the intent behind user queries, providing more accurate and contextually relevant results. Instead of relying solely on keyword matching, NLP-powered search engines analyze the meaning of words and phrases, making it easier to find information even when queries are vague or complex. This improves user experience, whether in web searches, document retrieval or enterprise data systems. Powerful content generation NLP powers advanced language models to create human-like text for various purposes. Pre-trained models, such as GPT-4, can generate articles, reports, marketing copy, product descriptions and even creative writing based on prompts provided by users. NLP-powered tools can also assist in automating tasks like drafting emails, writing social media posts or legal documentation. By understanding context, tone and style, NLP sees to it that the generated content is coherent, relevant and aligned with the intended message, saving time and effort in content creation while maintaining quality. NLP combines the power of computational linguistics together with machine learning algorithms and deep learning. Computational linguistics uses data science to analyze language and speech. It includes two main types of analysis: syntactical analysis and semantical analysis. Syntactical analysis determines the meaning of a word, phrase or sentence by parsing the syntax of the words and applying preprogrammed rules of grammar. Semantical analysis uses the syntactic output to draw meaning from the words and interpret their meaning within the sentence structure. The parsing of words can take one of two forms. Dependency parsing looks at the relationships between words, such as identifying nouns and verbs, while constituency parsing then builds a parse tree (or syntax tree): a rooted and ordered representation of the syntactic structure of the sentence or string of words. The resulting parse trees underly the functions of language translators and speech recognition. Ideally, this analysis makes the output either text or speech understandable to both NLP models and people. Self-supervised learning (SSL) in particular is useful for supporting NLP because NLP requires large amounts of labeled data to train AI models. Because these labeled datasets require time-consuming annotation, a process involving manual labeling by humans, gathering sufficient data can be prohibitively difficult. Self-supervised approaches can be more time-effective and cost-effective, as they replace some or all manually labeled training data. Three different approaches to NLP include: Rules-based NLP The earliest NLP applications were simple if-then decision trees, requiring preprogrammed rules. They are only able to provide answers in response to specific prompts, such as the original version of Moviefone, which had rudimentary natural language generation (NLG) capabilities. Because there is no machine learning or AI capability in rules-based NLP, this function is highly limited and not scalable. Statistical NLP Developed later, statistical NLP automatically extracts, classifies and labels elements of text and voice data and then assigns a statistical likelihood to each possible meaning of those elements. This relies on machine learning, enabling a sophisticated breakdown of linguistics such as part-of-speech tagging. Statistical NLP introduced the essential technique of mapping language elements, such as words and grammatical rules to a vector representation so that language can be modeled by using mathematical (statistical) methods, including regression or Markov models. This informed early NLP developments such as spellcheckers and T9 texting (Text on 9 keys, to be used on Touch-Tone telephones). Deep learning NLP Recently, deep learning models have become the dominant mode of NLP, by using huge volumes of raw, unstructured data both text and voice to become ever more accurate. Deep learning can be viewed as a further evolution of statistical NLP, with the difference that it uses neural network models. There are several subcategories of models: Sequence-to-Sequence (seq2seq) models: Based on recurrent neural networks (RNN) , they have mostly been used for machine translation by converting a phrase from one domain (such as the German language) into the phrase of another domain (such as English). Transformer models : They use tokenization of language (the position of each token words or subwords) and self-attention (capturing dependencies and relationships) to calculate the relation of different language parts to one another. Transformer models can be efficiently trained by using self-supervised learning on massive text databases. A landmark in transformer models was Google’s bidirectional encoder representations from transformers (BERT), which became and remains the basis of how Google’s search engine works. Autoregressive models : This type of transformer model is trained specifically to predict the next word in a sequence, which represents a huge leap forward in the ability to generate text. Examples of autoregressive LLMs include GPT, Llama , Claude and the open-source Mistral. Foundation models : Prebuilt and curated foundation models can speed the launching of an NLP effort and boost trust in its operation. For example, the IBM® Granite™ foundation models are widely applicable across industries. They support NLP tasks including content generation and insight extraction. Additionally, they facilitate retrieval-augmented generation, a framework for improving the quality of response by linking the model to external sources of knowledge. The models also perform named entity recognition which involves identifying and extracting key information in a text. Several NLP tasks typically help process human text and voice data in ways that help the computer make sense of what it’s ingesting. Some of these tasks include: Coreference resolution Named entity recognition Part-of-speech tagging Word sense disambiguation Coreference resolution This is the task of identifying if and when two words refer to the same entity. The most common example is determining the person or object to which a certain pronoun refers (such as “she” = “Mary”). But it can also identify a metaphor or an idiom in the text (such as an instance in which “bear” isn’t an animal, but a large and hairy person). Named entity recognition (NER) NER identifies words or phrases as useful entities. NER identifies “London” as a location or “Maria” as a person's name. Part-of-speech tagging Also called grammatical tagging, this is the process of determining which part of speech a word or piece of text is, based on its use and context. For example, part-of-speech identifies “make” as a verb in “I can make a paper plane,” and as a noun in “What make of car do you own?” Word sense disambiguation This is the selection of a word meaning for a word with multiple possible meanings. This uses a process of semantic analysis to examine the word in context. For example, word sense disambiguation helps distinguish the meaning of the verb “make” in “make the grade” (to achieve) versus “make a bet” (to place). Sorting out “I will be merry when I marry Mary” requires a sophisticated NLP system. NLP works by combining various computational techniques to analyze, understand and generate human language in a way that machines can process. Here is an overview of a typical NLP pipeline and its steps: Text preprocessing NLP text preprocessing prepares raw text for analysis by transforming it into a format that machines can more easily understand. It begins with tokenization, which involves splitting the text into smaller units like words, sentences or phrases. This helps break down complex text into manageable parts. Next, lowercasing is applied to standardize the text by converting all characters to lowercase, ensuring that words like "Apple" and "apple" are treated the same. Stop word removal is another common step, where frequently used words like "is" or "the" are filtered out because they don't add significant meaning to the text. Stemming or lemmatization reduces words to their root form (e.g., "running" becomes "run"), making it easier to analyze language by grouping different forms of the same word. Additionally, text cleaning removes unwanted elements such as punctuation, special characters and numbers that may clutter the analysis. After preprocessing, the text is clean, standardized and ready for machine learning models to interpret effectively. Feature extraction Feature extraction is the process of converting raw text into numerical representations that machines can analyze and interpret. This involves transforming text into structured data by using NLP techniques like Bag of Words and TF-IDF, which quantify the presence and importance of words in a document. More advanced methods include word embeddings like Word2Vec or GloVe, which represent words as dense vectors in a continuous space, capturing semantic relationships between words. Contextual embeddings further enhance this by considering the context in which words appear, allowing for richer, more nuanced representations. Text analysis Text analysis involves interpreting and extracting meaningful information from text data through various computational techniques. This process includes tasks such as part-of-speech (POS) tagging, which identifies grammatical roles of words and named entity recognition (NER), which detects specific entities like names, locations and dates. Dependency parsing analyzes grammatical relationships between words to understand sentence structure, while sentiment analysis determines the emotional tone of the text, assessing whether it is positive, negative or neutral. Topic modeling identifies underlying themes or topics within a text or across a corpus of documents. Natural language understanding (NLU) is a subset of NLP that focuses on analyzing the meaning behind sentences. NLU enables software to find similar meanings in different sentences or to process words that have different meanings. Through these techniques, NLP text analysis transforms unstructured text into insights. Model training Processed data is then used to train machine learning models, which learn patterns and relationships within the data. During training, the model adjusts its parameters to minimize errors and improve its performance. Once trained, the model can be used to make predictions or generate outputs on new, unseen data. The effectiveness of NLP modeling is continually refined through evaluation, validation and fine-tuning to enhance accuracy and relevance in real-world applications. Different software environments are useful throughout the said processes. For example, the Natural Language Toolkit (NLTK) is a suite of libraries and programs for English that is written in the Python programming language. It supports text classification, tokenization, stemming, tagging, parsing and semantic reasoning functionalities. TensorFlow is a free and open-source software library for machine learning and AI that can be used to train models for NLP applications. Tutorials and certifications abound for those interested in familiarizing themselves with such tools. Even state-of-the-art NLP models are not perfect, just as human speech is prone to error. As with any AI technology, NLP comes with potential pitfalls. Human language is filled with ambiguities that make it difficult for programmers to write software that accurately determines the intended meaning of text or voice data. Human language might take years for humans to learn and many never stop learning. But then programmers must teach natural language-powered applications to recognize and understand irregularities so their applications can be accurate and useful. Associated risks might include: Biased training As with any AI function, biased data used in training will skew the answers. The more diverse the users of an NLP function, the more significant this risk becomes, such as in government services, healthcare and HR interactions. Training datasets scraped from the web, for example, are prone to bias. Misinterpretation As in programming, there is a risk of garbage in, garbage out (GIGO). Speech recognition , also known as speech-to-text, is the task of reliably converting voice data into text data. But NLP solutions can become confused if spoken input is in an obscure dialect, mumbled, too full of slang, homonyms, incorrect grammar, idioms, fragments, mispronunciations, contractions or recorded with too much background noise. New vocabulary New words are continually being invented or imported. The conventions of grammar can evolve or be intentionally broken. In these cases, NLP can either make a best guess or admit it’s unsure and either way, this creates a complication. Tone of voice When people speak, their verbal delivery or even body language can give an entirely different meaning than the words alone. Exaggeration for effect, stressing words for importance or sarcasm can be confused by NLP, making the semantic analysis more difficult and less reliable. NLP applications can now be found across virtually every industry. Finance In financial dealings, nanoseconds might make the difference between success and failure when accessing data, or making trades or deals. NLP can speed the mining of information from financial statements, annual and regulatory reports, news releases or even social media. Healthcare New medical insights and breakthroughs can arrive faster than many healthcare professionals can keep up. NLP and AI-based tools can help speed the analysis of health records and medical research papers, making better-informed medical decisions possible, or assisting in the detection or even prevention of medical conditions. Insurance NLP can analyze claims to look for patterns that can identify areas of concern and find inefficiencies in claims processing, leading to greater optimization of processing and employee efforts. Legal Almost any legal case might require reviewing mounds of paperwork, background information and legal precedent. NLP can help automate legal discovery, assisting in the organization of information, speeding review and making sure that all relevant details are captured for consideration.
Markdown	# What is NLP (natural language processing)? Machine learning - [Welcome](https://www.ibm.com/think/machine-learning#605511093) - Introduction - [Overview](https://www.ibm.com/think/topics/machine-learning#7281535) - [Machine learning types](https://www.ibm.com/think/topics/machine-learning-types#7281536) - [Machine learning algorithms](https://www.ibm.com/think/topics/machine-learning-algorithms#7281537) - Data science for machine learning - [Statistical machine learning](https://www.ibm.com/think/topics/statistical-machine-learning#498277090) - [Linear algebra for machine learning](https://www.ibm.com/think/topics/linear-algebra-for-machine-learning#498277089) - [Uncertainty quantification](https://www.ibm.com/think/topics/uncertainty-quantification#498277088) - [Bias variance tradeoff](https://www.ibm.com/think/topics/bias-variance-tradeoff#498277087) - [Bayesian Statistics](https://www.ibm.com/think/topics/bayesian-statistics#498277086) - [Singular value decomposition](https://www.ibm.com/think/topics/singular-value-decomposition#498277085) - Feature Engineering - [Overview](https://www.ibm.com/think/topics/feature-engineering#1003835715) - [Feature selection](https://www.ibm.com/think/topics/feature-selection#1003835714) - [Feature extraction](https://www.ibm.com/think/topics/feature-extraction#1003835713) - [Vector embedding](https://www.ibm.com/think/topics/vector-embedding#1003835712) - [Latent space](https://www.ibm.com/think/topics/latent-space#1003835711) - Dimensionality reduction - [Principal component analysis](https://www.ibm.com/think/topics/principal-component-analysis#1793360183) - [Linear discriminant analysis](https://www.ibm.com/think/topics/linear-discriminant-analysis#1793360182) - [Upsampling](https://www.ibm.com/think/topics/upsampling#1003835709) - [Downsampling](https://www.ibm.com/think/topics/downsampling#1003835708) - [Synthetic data](https://www.ibm.com/think/topics/synthetic-data#1003835707) - [Data leakage](https://www.ibm.com/think/topics/data-leakage-machine-learning#1003835706) - Supervised learning - [Overview](https://www.ibm.com/think/topics/supervised-learning#1509394340) - Regression - [Linear regression](https://www.ibm.com/think/topics/linear-regression#1190488334) - [Lasso regression](https://www.ibm.com/think/topics/lasso-regression#1190488335) - [Ridge regression](https://www.ibm.com/think/topics/ridge-regression#1190488336) - [State space model](https://www.ibm.com/think/topics/state-space-model#1190488337) - [Time series](https://www.ibm.com/think/topics/time-series-model#1190488338) - [Autoregressive model](https://www.ibm.com/think/topics/autoregressive-model#1190488339) - Classification - [Overview](https://www.ibm.com/think/topics/classification-machine-learning#684929709) - [Decision trees](https://www.ibm.com/think/topics/decision-trees#684929710) - [K-nearest neighbors (KNNs)](https://www.ibm.com/think/topics/knn#684929711) - [Naive bayes](https://www.ibm.com/think/topics/naive-bayes#684929712) - [Random forest](https://www.ibm.com/think/topics/random-forest#684929713) - [Support vector machine](https://www.ibm.com/think/topics/support-vector-machine#684929714) - [Logistic regression](https://www.ibm.com/think/topics/logistic-regression#684929715) - Ensemble learning - [Overview](https://www.ibm.com/think/topics/ensemble-learning#179371084) - [Boosting](https://www.ibm.com/think/topics/boosting#179371085) - [Bagging](https://www.ibm.com/think/topics/bagging#179371086) - [Gradient boosting](https://www.ibm.com/think/topics/gradient-boosting#179371087) - [Gradient boosting classifier](https://www.ibm.com/think/tutorials/gradient-boosting-classifier#179371088) - Self-supervised learning - [Overview](https://www.ibm.com/think/topics/self-supervised-learning#326187541) - [Transfer learning](https://www.ibm.com/think/topics/transfer-learning#326187540) - Unsupervised learning - [Overview](https://www.ibm.com/think/topics/unsupervised-learning#2014952965) - Clustering - [Overview](https://www.ibm.com/think/topics/clustering#2142864945) - [K means clustering](https://www.ibm.com/think/topics/k-means-clustering#2142864944) - [Hierarchical clustering](https://www.ibm.com/think/topics/hierarchical-clustering#2142864943) - [A priori algorithm](https://www.ibm.com/think/topics/apriori-algorithm#2014952963) - [Gaussian mixture model](https://www.ibm.com/think/topics/gaussian-mixture-model#2014952962) - [Anomaly detection](https://www.ibm.com/think/topics/machine-learning-for-anomaly-detection#2014952961) - Semi-supervised learning - [Overview](https://www.ibm.com/think/topics/semi-supervised-learning#1774455706) - Recommendation engine - [Collaborative filtering](https://www.ibm.com/think/topics/collaborative-filtering#1181250928) - [Content based filtering](https://www.ibm.com/think/topics/content-based-filtering#1181250927) - Reinforcement learning - [Overview](https://www.ibm.com/think/topics/reinforcement-learning#1268897081) - [Reinforcement learning human feedback](https://www.ibm.com/think/topics/rlhf#1268897082) - Deep Learning - [Overview](https://www.ibm.com/think/topics/deep-learning#763338456) - Neural networks - [Overview](https://www.ibm.com/think/topics/neural-networks#741977106) - [Backpropagation](https://www.ibm.com/think/topics/backpropagation#741977107) - [Encoder-decoder model](https://www.ibm.com/think/topics/encoder-decoder-model#741977108) - [Recurrent neural networks](https://www.ibm.com/think/topics/recurrent-neural-networks#763338458) - [Long short-term memory (LSTM)](https://www.ibm.com/think/topics/lstm#763338459) - [Convolutional neural networks](https://www.ibm.com/think/topics/convolutional-neural-networks#763338460) - Transformer models - [Overview](https://www.ibm.com/think/topics/transformer-model#1280257394) - [Attention mechanism](https://www.ibm.com/think/topics/attention-mechanism#1280257393) - [Grouped query attention](https://www.ibm.com/think/topics/grouped-query-attention#1280257392) - [Positional encoding](https://www.ibm.com/think/topics/positional-encoding#1280257391) - [Autoencoder](https://www.ibm.com/think/topics/autoencoder#763338462) - [Mamba model](https://www.ibm.com/think/topics/mamba-model#763338463) - [Graph neural network](https://www.ibm.com/think/topics/graph-neural-network#763338464) - Generative AI - [Overview](https://www.ibm.com/think/topics/generative-ai#257779831) - [Generative model](https://www.ibm.com/think/topics/generative-model#257779832) - [Generative AI vs. predictive AI](https://www.ibm.com/think/topics/generative-ai-vs-predictive-ai-whats-the-difference#257779833) - Large language models (LLMs) - [Overview](https://www.ibm.com/think/topics/large-language-models#692473873) - [Reasoning models](https://www.ibm.com/think/topics/reasoning-model#692473874) - [Small language models](https://www.ibm.com/think/topics/small-language-models#692473875) - [Instruction tuning](https://www.ibm.com/think/topics/instruction-tuning#692473876) - [LLM parameters](https://www.ibm.com/think/topics/llm-parameters#692473877) - [LLM temperature](https://www.ibm.com/think/topics/llm-temperature#692473878) - [LLM benchmarks](https://www.ibm.com/think/topics/llm-benchmarks#692473879) - [LLM customization](https://www.ibm.com/think/topics/llm-customization#692473880) - [LLM alignment](https://www.ibm.com/think/topics/llm-alignment#692473881) - AI image generation - [Diffusion models](https://www.ibm.com/think/topics/diffusion-models#186915248) - [Variational autoencoder (VAE)](https://www.ibm.com/think/topics/variational-autoencoder#186915249) - [Generative adversarial networks (GANs)](https://www.ibm.com/think/topics/generative-adversarial-networks#186915250) - Multimodal AI - [Overview](https://www.ibm.com/think/topics/multimodal-ai#318643377) - [Vision language models](https://www.ibm.com/think/topics/vision-language-models#318643376) - [Tutorial: Build an AI stylist](https://www.ibm.com/think/tutorials/build-ai-stylist-with-granite-python-watsonx-ai#318643375) - [Tutorial: Multimodal AI queries using Llama](https://www.ibm.com/think/tutorials/multimodal-ai-python-llama#318643374) - [Tutorial: Multimodal AI queries using Pixtral](https://www.ibm.com/think/tutorials/multimodal-ai-pixtral-12b-on-watsonx-ai#318643373) - [Tutorial: Automatic podcast transcription with Granite](https://www.ibm.com/think/tutorials/automatic-speech-recognition-podcast-transcript-granite-watsonx-ai#318643372) - [Tutorial: PPT AI image analysis answering system](https://www.ibm.com/think/tutorials/build-ppt-ai-image-analysis-question-answering-system-granite#318643371) - Retrieval augmented generation (RAG) - [Overview](https://www.ibm.com/think/topics/retrieval-augmented-generation#824202002) - [GraphRAG](https://www.ibm.com/think/topics/graphrag#824202001) - [Tutorial: Build a multimodal RAG system with Docling and Granite](https://www.ibm.com/think/tutorials/build-multimodal-rag-langchain-with-docling-granite#824202000) - [Tutorial: Evaluate RAG pipline using Ragas](https://www.ibm.com/think/tutorials/evaluate-rag-pipeline-using-ragas-in-python-with-watsonx#824201999) - [Tutorial: RAG chunking strategies](https://www.ibm.com/think/tutorials/chunking-strategies-for-rag-with-langchain-watsonx-ai#824201998) - [Tutorial: Graph RAG using knowledge graphs](https://www.ibm.com/think/tutorials/knowledge-graph-rag#824201997) - [Tutorial: Inference scaling to improve multimodal RAG](https://www.ibm.com/think/tutorials/inference-scaling-improve-multimodal-rag-granite#824201996) - AI code generation - [Overview](https://www.ibm.com/think/topics/ai-code-generation#1329760627) - [Vibe coding](https://www.ibm.com/think/topics/vibe-coding#1329760626) - AI agents - [Visit the 2025 Guide to AI Agents](https://www.ibm.com/think/ai-agents#1835319252) - Model training - [Overview](https://www.ibm.com/think/topics/model-training#1580786329) - [LLM training](https://www.ibm.com/think/topics/llm-training#1580786328) - [Loss function](https://www.ibm.com/think/topics/loss-function#1580786327) - [Training data](https://www.ibm.com/think/topics/training-data#1580786326) - [Model parameters](https://www.ibm.com/think/topics/model-parameters#1580786325) - Optimization algorithm - [Gradient descent](https://www.ibm.com/think/topics/gradient-descent#18238369) - [Stochastic gradient descent](https://www.ibm.com/think/topics/stochastic-gradient-descent#18238368) - Model hyperparameters - [Hyperparameter tuning](https://www.ibm.com/think/topics/hyperparameter-tuning#523796994) - [Learning rate](https://www.ibm.com/think/topics/learning-rate#523796993) - Fine tuning - [Overview](https://www.ibm.com/think/topics/fine-tuning#1029355619) - [Parameter efficient fine tuning (PEFT)](https://www.ibm.com/think/topics/parameter-efficient-fine-tuning#1029355618) - [LoRA](https://www.ibm.com/think/topics/lora#1029355617) - [Tutorial: Fine tuning Granite model with LoRA](https://www.ibm.com/think/tutorials/lora-fine-tuning-granite-llm#1029355616) - [Regularization](https://www.ibm.com/think/topics/regularization#1580786321) - [Foundation models](https://www.ibm.com/think/topics/foundation-models#1580786320) - [Overfitting](https://www.ibm.com/think/topics/overfitting#1759735864) - [Underfitting](https://www.ibm.com/think/topics/underfitting#1759735863) - N-shot learning - [Few shot learning](https://www.ibm.com/think/topics/few-shot-learning#823293247) - [Zero shot learning](https://www.ibm.com/think/topics/zero-shot-learning#823293246) - [Knowledge distillation](https://www.ibm.com/think/topics/knowledge-distillation#1759735861) - [Meta learning](https://www.ibm.com/think/topics/meta-learning#1759735860) - [Data augmentation](https://www.ibm.com/think/topics/data-augmentation#1759735859) - Continual learning - [Catastrophic forgetting](https://www.ibm.com/think/topics/catastrophic-forgetting#1449439549) - Machine learning libraries - [Overview](https://www.ibm.com/think/topics/machine-learning-libraries#2086344954) - [Scikit-learn](https://www.ibm.com/think/topics/scikit-learn#2086344953) - [XGboost](https://www.ibm.com/think/topics/xgboost#2086344952) - [PyTorch](https://www.ibm.com/think/topics/pytorch#2086344951) - MLOps - [Overview](https://www.ibm.com/think/topics/mlops#1703063717) - [AI lifecyle](https://www.ibm.com/think/topics/ai-lifecycle#1703063718) - [AI inference](https://www.ibm.com/think/topics/ai-inference#1703063719) - [Model deployment](https://www.ibm.com/think/topics/model-deployment#1703063720) - [Machine learning pipeline](https://www.ibm.com/think/topics/machine-learning-pipeline#1703063721) - [Data labeling](https://www.ibm.com/think/topics/data-labeling#1703063722) - Model governance - [Model risk management](https://www.ibm.com/think/topics/model-risk-management#1399431040) - [Model drift](https://www.ibm.com/think/topics/model-drift#1399431041) - [AutoML](https://www.ibm.com/think/topics/automl#1703063724) - [Model selection](https://www.ibm.com/think/topics/model-selection#1703063725) - [Federated learning](https://www.ibm.com/think/topics/federated-learning#1703063726) - [Distributed machine learning](https://www.ibm.com/think/topics/distributed-machine-learning#1255367754) - [AI stack](https://www.ibm.com/think/topics/ai-stack#1255367755) - Natural language processing - [Overview](https://www.ibm.com/think/topics/natural-language-processing#1197505092) - [Natural language understanding](https://www.ibm.com/think/topics/natural-language-understanding#1197505093) - Text classification - [Overview](https://www.ibm.com/think/topics/text-classification#88312261) - [Sentiment analysis](https://www.ibm.com/think/topics/sentiment-analysis#88312262) - [Tutorial: Spam text classifier with PyTorch](https://www.ibm.com/think/tutorials/text-classification-pytorch#88312263) - [Machine translation](https://www.ibm.com/think/topics/machine-translation#1197505095) - Text mining - [Overview](https://www.ibm.com/think/topics/text-mining#922804989) - [Information retrieval](https://www.ibm.com/think/topics/information-retrieval#922804988) - [Information extraction](https://www.ibm.com/think/topics/information-extraction#922804987) - [Topic modeling](https://www.ibm.com/think/topics/topic-modeling#922804986) - [Latent semantic analysis](https://www.ibm.com/think/topics/latent-semantic-analysis#922804985) - [Latent Dirichlet Allocation](https://www.ibm.com/think/topics/latent-dirichlet-allocation#922804984) - [Named entity recognition](https://www.ibm.com/think/topics/named-entity-recognition#922804983) - [Word embeddings](https://www.ibm.com/think/topics/word-embeddings#1197505097) - [Bag of words](https://www.ibm.com/think/topics/bag-of-words#1197505098) - [Intelligent search](https://www.ibm.com/think/topics/intelligent-search#1197505099) - [Speech recognition](https://www.ibm.com/think/topics/speech-recognition#1197505100) - [Stemming and lemmatization](https://www.ibm.com/think/topics/stemming-lemmatization#1197505101) - [Text summarization](https://www.ibm.com/think/topics/text-summarization#1532047733) - [Conversational AI](https://www.ibm.com/think/topics/conversational-ai#1532047732) - [Conversational analytics](https://www.ibm.com/think/topics/conversational-analytics#1532047731) - [Natural language generation](https://www.ibm.com/think/topics/natural-language-generation#1532047730) - Computer vision - [Overview](https://www.ibm.com/think/topics/computer-vision#691946467) - [Image classification](https://www.ibm.com/think/topics/image-classification#691946468) - [Object detection](https://www.ibm.com/think/topics/object-detection#691946469) - Image segmentation - [Instance segmentation](https://www.ibm.com/think/topics/instance-segmentation#544367653) - [Semantic segmentation](https://www.ibm.com/think/topics/semantic-segmentation#544367654) - [Optical character recognition](https://www.ibm.com/think/topics/optical-character-recognition#691946471) - [Image recognition](https://www.ibm.com/think/topics/image-recognition#691946472) - [Visual inspection](https://www.ibm.com/think/topics/visual-inspection#691946473) ## Authors [Cole Stryker](https://www.ibm.com/think/author/cole-stryker.html) Staff Editor, AI Models IBM Think [Jim Holdsworth](https://www.ibm.com/think/author/jim-holdsworth.html) Staff Writer IBM Think ## What is NLP? Natural language processing (NLP) is a subfield of computer science and [artificial intelligence (AI)](https://www.ibm.com/think/topics/artificial-intelligence) that uses [machine learning](https://www.ibm.com/think/topics/machine-learning) to enable computers to understand and communicate with human language. NLP enables computers and digital devices to recognize, understand and generate text and speech by combining computational linguistics, the rule-based modeling of human language together with statistical modeling, machine learning and [deep learning](https://www.ibm.com/think/topics/deep-learning). NLP research has helped enable the era of [generative AI](https://www.ibm.com/think/topics/generative-ai), from the communication skills of [large language models](https://www.ibm.com/think/topics/large-language-models) (LLMs) to the ability of image generation models to understand requests. NLP is already part of everyday life for many, powering search engines, prompting [chatbots](https://www.ibm.com/think/topics/chatbots) for customer service with spoken commands, voice-operated GPS systems and question-answering digital assistants on smartphones such as Amazon’s Alexa, Apple’s Siri and Microsoft’s Cortana. NLP also plays a growing role in enterprise solutions that help streamline and automate business operations, increase employee productivity and simplify business processes. ## Benefits of NLP NLP makes it easier for humans to communicate and collaborate with machines, by allowing them to do so in the natural human language they use every day. This offers benefits across many industries and applications. - Automation of repetitive tasks - Improved data analysis and insights - Enhanced search - Content generation ### Automation of repetitive tasks NLP is especially useful in fully or partially [automating tasks](https://www.ibm.com/think/topics/automation) like customer support, data entry and document handling. For example, NLP-powered chatbots can handle routine customer queries, freeing up human agents for more complex issues. In [document processing](https://www.ibm.com/think/topics/document-processing), NLP tools can automatically classify, extract key information and summarize content, reducing the time and errors associated with manual data handling. NLP facilitates language translation, converting text from one language to another while preserving meaning, context and nuances. ### Improved data analysis NLP enhances data analysis by enabling the extraction of insights from unstructured text data, such as customer reviews, social media posts and news articles. By using [text mining](https://www.ibm.com/think/topics/text-mining-use-cases) techniques, NLP can identify patterns, trends and sentiments that are not immediately obvious in large datasets. Sentiment analysis enables the [extraction of subjective qualities](https://www.ibm.com/think/topics/sentiment-analysis), attitudes, emotions, sarcasm, confusion or suspicion from text. This is often used for routing communications to the system or the person most likely to make the next response. This allows businesses to better understand customer preferences, market conditions and public opinion. NLP tools can also perform categorization and summarization of vast amounts of text, making it easier for analysts to identify key information and make data-driven decisions more efficiently. ### Enhanced search NLP benefits search by enabling systems to understand the intent behind user queries, providing more accurate and contextually relevant results. Instead of relying solely on keyword matching, NLP-powered search engines analyze the meaning of words and phrases, making it easier to find information even when queries are vague or complex. This improves user experience, whether in web searches, document retrieval or enterprise data systems. ### Powerful content generation NLP powers advanced language models to [create human-like text](https://www.ibm.com/think/topics/text-generation) for various purposes. Pre-trained models, such as GPT-4, can generate articles, reports, marketing copy, product descriptions and even creative writing based on prompts provided by users. NLP-powered tools can also assist in automating tasks like drafting emails, writing social media posts or legal documentation. By understanding context, tone and style, NLP sees to it that the generated content is coherent, relevant and aligned with the intended message, saving time and effort in content creation while maintaining quality. ## Approaches to NLP NLP combines the power of computational linguistics together with [machine learning algorithms](https://www.ibm.com/think/topics/machine-learning-algorithms) and deep learning. Computational linguistics uses data science to analyze language and speech. It includes two main types of analysis: syntactical analysis and semantical analysis. Syntactical analysis determines the meaning of a word, phrase or sentence by parsing the syntax of the words and applying preprogrammed rules of grammar. Semantical analysis uses the syntactic output to draw meaning from the words and interpret their meaning within the sentence structure. The parsing of words can take one of two forms. Dependency parsing looks at the relationships between words, such as identifying nouns and verbs, while constituency parsing then builds a parse tree (or syntax tree): a rooted and ordered representation of the syntactic structure of the sentence or string of words. The resulting parse trees underly the functions of language translators and speech recognition. Ideally, this analysis makes the output either text or speech understandable to both NLP models and people. [Self-supervised learning (SSL)](https://www.ibm.com/think/topics/self-supervised-learning) in particular is useful for supporting NLP because NLP requires large amounts of labeled data to train AI models. Because these labeled datasets require time-consuming annotation, a process involving manual labeling by humans, gathering sufficient data can be prohibitively difficult. Self-supervised approaches can be more time-effective and cost-effective, as they replace some or all manually labeled training data. Three different approaches to NLP include: ### Rules-based NLP The earliest NLP applications were simple if-then decision trees, requiring preprogrammed rules. They are only able to provide answers in response to specific prompts, such as the original version of Moviefone, which had rudimentary natural language generation (NLG) capabilities. Because there is no machine learning or AI capability in rules-based NLP, this function is highly limited and not scalable. ### Statistical NLP Developed later, statistical NLP automatically extracts, classifies and labels elements of text and voice data and then assigns a statistical likelihood to each possible meaning of those elements. This relies on machine learning, enabling a sophisticated breakdown of linguistics such as part-of-speech tagging. Statistical NLP introduced the essential technique of mapping language elements, such as words and grammatical rules to a vector representation so that language can be modeled by using mathematical (statistical) methods, including regression or Markov models. This informed early NLP developments such as spellcheckers and T9 texting (Text on 9 keys, to be used on Touch-Tone telephones). ### Deep learning NLP Recently, deep learning models have become the dominant mode of NLP, by using huge volumes of raw, [unstructured](https://www.ibm.com/think/topics/text-mining) data both text and voice to become ever more accurate. Deep learning can be viewed as a further evolution of statistical NLP, with the difference that it uses [neural network](https://www.ibm.com/think/topics/neural-networks) models. There are several subcategories of models: - Sequence-to-Sequence (seq2seq) models: Based [on recurrent neural networks (RNN)](https://www.ibm.com/think/topics/recurrent-neural-networks), they have mostly been used for machine translation by converting a phrase from one domain (such as the German language) into the phrase of another domain (such as English). - Transformer models: They use [tokenization](https://www.ibm.com/docs/en/watsonx/saas?topic=solutions-tokens) of language (the position of each token words or subwords) and self-attention (capturing dependencies and relationships) to calculate the relation of different language parts to one another. [Transformer models](https://www.ibm.com/think/topics/transformer-model) can be efficiently trained by using [self-supervised learning](https://www.ibm.com/think/topics/self-supervised-learning) on massive text databases. A landmark in [transformer models](https://www.ibm.com/think/topics/transformer-model) was Google’s bidirectional encoder representations from transformers (BERT), which became and remains the basis of how Google’s search engine works. - Autoregressive models: This type of transformer model is trained specifically to predict the next word in a sequence, which represents a huge leap forward in the ability to generate text. Examples of autoregressive LLMs include GPT, [Llama](https://www.ibm.com/think/topics/llama-2), Claude and the open-source Mistral. - Foundation models: Prebuilt and curated foundation models can speed the launching of an NLP effort and boost trust in its operation. For example, the [IBM® Granite™](https://www.ibm.com/think/news/granite-foundation-models) foundation models are widely applicable across industries. They support NLP tasks including content generation and insight extraction. Additionally, they facilitate retrieval-augmented generation, a framework for improving the quality of response by linking the model to external sources of knowledge. The models also perform named entity recognition which involves identifying and extracting key information in a text. ## NLP Tasks Several NLP tasks typically help process human text and voice data in ways that help the computer make sense of what it’s ingesting. Some of these tasks include: - Coreference resolution - Named entity recognition - Part-of-speech tagging - Word sense disambiguation ### Coreference resolution This is the task of identifying if and when two words refer to the same entity. The most common example is determining the person or object to which a certain pronoun refers (such as “she” = “Mary”). But it can also identify a metaphor or an idiom in the text (such as an instance in which “bear” isn’t an animal, but a large and hairy person). ### Named entity recognition (NER) [NER](https://www.ibm.com/think/topics/named-entity-recognition) identifies words or phrases as useful entities. NER identifies “London” as a location or “Maria” as a person's name. ### Part-of-speech tagging Also called grammatical tagging, this is the process of determining which part of speech a word or piece of text is, based on its use and context. For example, part-of-speech identifies “make” as a verb in “I can make a paper plane,” and as a noun in “What make of car do you own?” ### Word sense disambiguation This is the selection of a word meaning for a word with multiple possible meanings. This uses a process of semantic [analysis](https://www.ibm.com/consulting/analytics) to examine the word in context. For example, word sense disambiguation helps distinguish the meaning of the verb “make” in “make the grade” (to achieve) versus “make a bet” (to place). Sorting out “I will be merry when I marry Mary” requires a sophisticated NLP system. ## How NLP works NLP works by combining various computational techniques to analyze, understand and generate human language in a way that machines can process. Here is an overview of a typical NLP pipeline and its steps: ### Text preprocessing NLP text preprocessing prepares raw text for analysis by transforming it into a format that machines can more easily understand. It begins with tokenization, which involves splitting the text into smaller units like words, sentences or phrases. This helps break down complex text into manageable parts. Next, lowercasing is applied to standardize the text by converting all characters to lowercase, ensuring that words like "Apple" and "apple" are treated the same. Stop word removal is another common step, where frequently used words like "is" or "the" are filtered out because they don't add significant meaning to the text. [Stemming](https://www.ibm.com/think/topics/stemming) or [lemmatization](https://www.ibm.com/think/topics/stemming-lemmatization) reduces words to their root form (e.g., "running" becomes "run"), making it easier to analyze language by grouping different forms of the same word. Additionally, text cleaning removes unwanted elements such as punctuation, special characters and numbers that may clutter the analysis. After preprocessing, the text is clean, standardized and ready for machine learning models to interpret effectively. ### Feature extraction Feature extraction is the process of converting raw text into numerical representations that machines can analyze and interpret. This involves transforming text into structured data by using NLP techniques like [Bag of Words](https://www.ibm.com/think/topics/bag-of-words) and TF-IDF, which quantify the presence and importance of words in a document. More advanced methods include [word embeddings](https://www.ibm.com/think/topics/embedding) like Word2Vec or GloVe, which represent words as dense vectors in a continuous space, capturing semantic relationships between words. Contextual embeddings further enhance this by considering the context in which words appear, allowing for richer, more nuanced representations. ### Text analysis Text analysis involves interpreting and extracting meaningful information from text data through various computational techniques. This process includes tasks such as part-of-speech (POS) tagging, which identifies grammatical roles of words and named entity recognition (NER), which detects specific entities like names, locations and dates. Dependency parsing analyzes grammatical relationships between words to understand sentence structure, while sentiment analysis determines the emotional tone of the text, assessing whether it is positive, negative or neutral. Topic modeling identifies underlying themes or topics within a text or across a corpus of documents. Natural language understanding (NLU) is a subset of NLP that focuses on analyzing the meaning behind sentences. NLU enables software to find similar meanings in different sentences or to process words that have different meanings. Through these techniques, NLP text analysis transforms unstructured text into insights. ### Model training Processed data is then used to train machine learning models, which learn patterns and relationships within the data. During training, the model adjusts its parameters to minimize errors and improve its performance. Once trained, the model can be used to make predictions or generate outputs on new, unseen data. The effectiveness of NLP modeling is continually refined through evaluation, validation and fine-tuning to enhance accuracy and relevance in real-world applications. Different software environments are useful throughout the said processes. For example, the Natural Language Toolkit (NLTK) is a suite of libraries and programs for English that is written in the Python programming language. It supports text classification, tokenization, stemming, tagging, parsing and semantic reasoning functionalities. TensorFlow is a free and open-source software library for machine learning and AI that can be used to train models for NLP applications. Tutorials and certifications abound for those interested in familiarizing themselves with such tools. ## Challenges of NLP Even state-of-the-art NLP models are not perfect, just as human speech is prone to error. As with any AI technology, NLP comes with potential pitfalls. Human language is filled with ambiguities that make it difficult for programmers to write software that accurately determines the intended meaning of text or voice data. Human language might take years for humans to learn and many never stop learning. But then programmers must teach natural language-powered applications to recognize and understand irregularities so their applications can be accurate and useful. Associated risks might include: ### Biased training As with any AI function, [biased data](https://www.ibm.com/think/topics/ai-bias) used in training will skew the answers. The more diverse the users of an NLP function, the more significant this risk becomes, such as in government services, healthcare and HR interactions. Training datasets scraped from the web, for example, are prone to bias. ### Misinterpretation As in programming, there is a risk of garbage in, garbage out (GIGO). [Speech recognition](https://www.ibm.com/think/topics/speech-recognition), also known as speech-to-text, is the task of reliably converting voice data into text data. But NLP solutions can become confused if spoken input is in an obscure dialect, mumbled, too full of slang, homonyms, incorrect grammar, idioms, fragments, mispronunciations, contractions or recorded with too much background noise. ### New vocabulary New words are continually being invented or imported. The conventions of grammar can evolve or be intentionally broken. In these cases, NLP can either make a best guess or admit it’s unsure and either way, this creates a complication. ### Tone of voice When people speak, their verbal delivery or even body language can give an entirely different meaning than the words alone. Exaggeration for effect, stressing words for importance or sarcasm can be confused by NLP, making the semantic analysis more difficult and less reliable. ## NLP use cases by industry NLP applications can now be found across virtually every industry. ### Finance In financial dealings, nanoseconds might make the difference between success and failure when accessing data, or making trades or deals. NLP can speed the mining of information from financial statements, annual and regulatory reports, news releases or even social media. ### Healthcare New medical insights and breakthroughs can arrive faster than many healthcare professionals can keep up. NLP and AI-based tools can help speed the analysis of health records and medical research papers, making better-informed medical decisions possible, or assisting in the detection or even prevention of medical conditions. ### Insurance NLP can analyze claims to look for patterns that can identify areas of concern and find inefficiencies in claims processing, leading to greater optimization of processing and employee efforts. ### Legal Almost any legal case might require reviewing mounds of paperwork, background information and legal precedent. NLP can help automate legal discovery, assisting in the organization of information, speeding review and making sure that all relevant details are captured for consideration. Link copied [Report Beyond the hype - How AI assistants drive real business value Explore top use cases for leveraging AI assistants, understand the potential impact of Gen AI and automation technology on your business, and learn how to get started. Read the report](https://www.ibm.com/account/reg/signup?formid=urx-53011) ## Resources [AI models Explore IBM Granite® IBM Granite ® is our family of open, performant and trusted AI models, tailored for business and optimized to scale your AI applications. Explore language, code, time series and guardrail options. Meet Granite](https://www.ibm.com/granite) [Explainer Beginner's guide to NLP Discover how natural language processing (NLP) can help you to converse more naturally with computers. Learn NLP](https://developer.ibm.com/learningpaths/get-started-artificial-intelligence/next-steps/beginner-natural-language-processing) [Report IBM is named leader in data science and machine learning Learn why IBM has been recognized as a Leader in the 2025 Gartner® Magic Quadrant™ for data science and machine learning platforms. Read the report](https://www.ibm.com/forms/mkt-53728) [Training Hands-on with generative AI Learn fundamental concepts and build your skills with hands-on labs, courses, guided projects, trials and more. Learn generative AI](https://developer.ibm.com/technologies/generative-ai/?cm_sp=ibmdev-_-developer-_-getstarted-_-genai) Related solutions IBM® watsonx Orchestrate™ Easily design scalable AI assistants and agents, automate repetitive tasks and simplify complex processes with IBM® watsonx Orchestrate™. [Explore watsonx Orchestrate](https://www.ibm.com/products/watsonx-orchestrate) Natural language processing tools and APIs Accelerate the business value of artificial intelligence with a powerful and flexible portfolio of libraries, services and applications. [Explore NLP solutions](https://www.ibm.com/solutions/natural-language-processing) AI consulting and services Reinvent critical workflows and operations by adding AI to maximize experiences, real-time decision-making and business value. [Explore AI services](https://www.ibm.com/consulting/artificial-intelligence) Take the next step Easily design scalable AI assistants and agents, automate repetitive tasks and simplify complex processes with IBM® watsonx Orchestrate™. 1. [Explore watsonx Orchestrate](https://www.ibm.com/products/watsonx-orchestrate) 2. [Explore NLP solutions](https://www.ibm.com/solutions/natural-language-processing)
Readable Markdown	Natural language processing (NLP) is a subfield of computer science and [artificial intelligence (AI)](https://www.ibm.com/think/topics/artificial-intelligence) that uses [machine learning](https://www.ibm.com/think/topics/machine-learning) to enable computers to understand and communicate with human language. NLP enables computers and digital devices to recognize, understand and generate text and speech by combining computational linguistics, the rule-based modeling of human language together with statistical modeling, machine learning and [deep learning](https://www.ibm.com/think/topics/deep-learning). NLP research has helped enable the era of [generative AI](https://www.ibm.com/think/topics/generative-ai), from the communication skills of [large language models](https://www.ibm.com/think/topics/large-language-models) (LLMs) to the ability of image generation models to understand requests. NLP is already part of everyday life for many, powering search engines, prompting [chatbots](https://www.ibm.com/think/topics/chatbots) for customer service with spoken commands, voice-operated GPS systems and question-answering digital assistants on smartphones such as Amazon’s Alexa, Apple’s Siri and Microsoft’s Cortana. NLP also plays a growing role in enterprise solutions that help streamline and automate business operations, increase employee productivity and simplify business processes. NLP makes it easier for humans to communicate and collaborate with machines, by allowing them to do so in the natural human language they use every day. This offers benefits across many industries and applications. - Automation of repetitive tasks - Improved data analysis and insights - Enhanced search - Content generation Automation of repetitive tasks NLP is especially useful in fully or partially [automating tasks](https://www.ibm.com/think/topics/automation) like customer support, data entry and document handling. For example, NLP-powered chatbots can handle routine customer queries, freeing up human agents for more complex issues. In [document processing](https://www.ibm.com/think/topics/document-processing), NLP tools can automatically classify, extract key information and summarize content, reducing the time and errors associated with manual data handling. NLP facilitates language translation, converting text from one language to another while preserving meaning, context and nuances. Improved data analysis NLP enhances data analysis by enabling the extraction of insights from unstructured text data, such as customer reviews, social media posts and news articles. By using [text mining](https://www.ibm.com/think/topics/text-mining-use-cases) techniques, NLP can identify patterns, trends and sentiments that are not immediately obvious in large datasets. Sentiment analysis enables the [extraction of subjective qualities](https://www.ibm.com/think/topics/sentiment-analysis), attitudes, emotions, sarcasm, confusion or suspicion from text. This is often used for routing communications to the system or the person most likely to make the next response. This allows businesses to better understand customer preferences, market conditions and public opinion. NLP tools can also perform categorization and summarization of vast amounts of text, making it easier for analysts to identify key information and make data-driven decisions more efficiently. Enhanced search NLP benefits search by enabling systems to understand the intent behind user queries, providing more accurate and contextually relevant results. Instead of relying solely on keyword matching, NLP-powered search engines analyze the meaning of words and phrases, making it easier to find information even when queries are vague or complex. This improves user experience, whether in web searches, document retrieval or enterprise data systems. Powerful content generation NLP powers advanced language models to [create human-like text](https://www.ibm.com/think/topics/text-generation) for various purposes. Pre-trained models, such as GPT-4, can generate articles, reports, marketing copy, product descriptions and even creative writing based on prompts provided by users. NLP-powered tools can also assist in automating tasks like drafting emails, writing social media posts or legal documentation. By understanding context, tone and style, NLP sees to it that the generated content is coherent, relevant and aligned with the intended message, saving time and effort in content creation while maintaining quality. NLP combines the power of computational linguistics together with [machine learning algorithms](https://www.ibm.com/think/topics/machine-learning-algorithms) and deep learning. Computational linguistics uses data science to analyze language and speech. It includes two main types of analysis: syntactical analysis and semantical analysis. Syntactical analysis determines the meaning of a word, phrase or sentence by parsing the syntax of the words and applying preprogrammed rules of grammar. Semantical analysis uses the syntactic output to draw meaning from the words and interpret their meaning within the sentence structure. The parsing of words can take one of two forms. Dependency parsing looks at the relationships between words, such as identifying nouns and verbs, while constituency parsing then builds a parse tree (or syntax tree): a rooted and ordered representation of the syntactic structure of the sentence or string of words. The resulting parse trees underly the functions of language translators and speech recognition. Ideally, this analysis makes the output either text or speech understandable to both NLP models and people. [Self-supervised learning (SSL)](https://www.ibm.com/think/topics/self-supervised-learning) in particular is useful for supporting NLP because NLP requires large amounts of labeled data to train AI models. Because these labeled datasets require time-consuming annotation, a process involving manual labeling by humans, gathering sufficient data can be prohibitively difficult. Self-supervised approaches can be more time-effective and cost-effective, as they replace some or all manually labeled training data. Three different approaches to NLP include: Rules-based NLP The earliest NLP applications were simple if-then decision trees, requiring preprogrammed rules. They are only able to provide answers in response to specific prompts, such as the original version of Moviefone, which had rudimentary natural language generation (NLG) capabilities. Because there is no machine learning or AI capability in rules-based NLP, this function is highly limited and not scalable. Statistical NLP Developed later, statistical NLP automatically extracts, classifies and labels elements of text and voice data and then assigns a statistical likelihood to each possible meaning of those elements. This relies on machine learning, enabling a sophisticated breakdown of linguistics such as part-of-speech tagging. Statistical NLP introduced the essential technique of mapping language elements, such as words and grammatical rules to a vector representation so that language can be modeled by using mathematical (statistical) methods, including regression or Markov models. This informed early NLP developments such as spellcheckers and T9 texting (Text on 9 keys, to be used on Touch-Tone telephones). Deep learning NLP Recently, deep learning models have become the dominant mode of NLP, by using huge volumes of raw, [unstructured](https://www.ibm.com/think/topics/text-mining) data both text and voice to become ever more accurate. Deep learning can be viewed as a further evolution of statistical NLP, with the difference that it uses [neural network](https://www.ibm.com/think/topics/neural-networks) models. There are several subcategories of models: - Sequence-to-Sequence (seq2seq) models: Based [on recurrent neural networks (RNN)](https://www.ibm.com/think/topics/recurrent-neural-networks), they have mostly been used for machine translation by converting a phrase from one domain (such as the German language) into the phrase of another domain (such as English). - Transformer models: They use [tokenization](https://www.ibm.com/docs/en/watsonx/saas?topic=solutions-tokens) of language (the position of each token words or subwords) and self-attention (capturing dependencies and relationships) to calculate the relation of different language parts to one another. [Transformer models](https://www.ibm.com/think/topics/transformer-model) can be efficiently trained by using [self-supervised learning](https://www.ibm.com/think/topics/self-supervised-learning) on massive text databases. A landmark in [transformer models](https://www.ibm.com/think/topics/transformer-model) was Google’s bidirectional encoder representations from transformers (BERT), which became and remains the basis of how Google’s search engine works. - Autoregressive models: This type of transformer model is trained specifically to predict the next word in a sequence, which represents a huge leap forward in the ability to generate text. Examples of autoregressive LLMs include GPT, [Llama](https://www.ibm.com/think/topics/llama-2), Claude and the open-source Mistral. - Foundation models: Prebuilt and curated foundation models can speed the launching of an NLP effort and boost trust in its operation. For example, the [IBM® Granite™](https://www.ibm.com/think/news/granite-foundation-models) foundation models are widely applicable across industries. They support NLP tasks including content generation and insight extraction. Additionally, they facilitate retrieval-augmented generation, a framework for improving the quality of response by linking the model to external sources of knowledge. The models also perform named entity recognition which involves identifying and extracting key information in a text. Several NLP tasks typically help process human text and voice data in ways that help the computer make sense of what it’s ingesting. Some of these tasks include: - Coreference resolution - Named entity recognition - Part-of-speech tagging - Word sense disambiguation Coreference resolution This is the task of identifying if and when two words refer to the same entity. The most common example is determining the person or object to which a certain pronoun refers (such as “she” = “Mary”). But it can also identify a metaphor or an idiom in the text (such as an instance in which “bear” isn’t an animal, but a large and hairy person). Named entity recognition (NER) [NER](https://www.ibm.com/think/topics/named-entity-recognition) identifies words or phrases as useful entities. NER identifies “London” as a location or “Maria” as a person's name. Part-of-speech tagging Also called grammatical tagging, this is the process of determining which part of speech a word or piece of text is, based on its use and context. For example, part-of-speech identifies “make” as a verb in “I can make a paper plane,” and as a noun in “What make of car do you own?” Word sense disambiguation This is the selection of a word meaning for a word with multiple possible meanings. This uses a process of semantic [analysis](https://www.ibm.com/consulting/analytics) to examine the word in context. For example, word sense disambiguation helps distinguish the meaning of the verb “make” in “make the grade” (to achieve) versus “make a bet” (to place). Sorting out “I will be merry when I marry Mary” requires a sophisticated NLP system. NLP works by combining various computational techniques to analyze, understand and generate human language in a way that machines can process. Here is an overview of a typical NLP pipeline and its steps: Text preprocessing NLP text preprocessing prepares raw text for analysis by transforming it into a format that machines can more easily understand. It begins with tokenization, which involves splitting the text into smaller units like words, sentences or phrases. This helps break down complex text into manageable parts. Next, lowercasing is applied to standardize the text by converting all characters to lowercase, ensuring that words like "Apple" and "apple" are treated the same. Stop word removal is another common step, where frequently used words like "is" or "the" are filtered out because they don't add significant meaning to the text. [Stemming](https://www.ibm.com/think/topics/stemming) or [lemmatization](https://www.ibm.com/think/topics/stemming-lemmatization) reduces words to their root form (e.g., "running" becomes "run"), making it easier to analyze language by grouping different forms of the same word. Additionally, text cleaning removes unwanted elements such as punctuation, special characters and numbers that may clutter the analysis. After preprocessing, the text is clean, standardized and ready for machine learning models to interpret effectively. Feature extraction Feature extraction is the process of converting raw text into numerical representations that machines can analyze and interpret. This involves transforming text into structured data by using NLP techniques like [Bag of Words](https://www.ibm.com/think/topics/bag-of-words) and TF-IDF, which quantify the presence and importance of words in a document. More advanced methods include [word embeddings](https://www.ibm.com/think/topics/embedding) like Word2Vec or GloVe, which represent words as dense vectors in a continuous space, capturing semantic relationships between words. Contextual embeddings further enhance this by considering the context in which words appear, allowing for richer, more nuanced representations. Text analysis Text analysis involves interpreting and extracting meaningful information from text data through various computational techniques. This process includes tasks such as part-of-speech (POS) tagging, which identifies grammatical roles of words and named entity recognition (NER), which detects specific entities like names, locations and dates. Dependency parsing analyzes grammatical relationships between words to understand sentence structure, while sentiment analysis determines the emotional tone of the text, assessing whether it is positive, negative or neutral. Topic modeling identifies underlying themes or topics within a text or across a corpus of documents. Natural language understanding (NLU) is a subset of NLP that focuses on analyzing the meaning behind sentences. NLU enables software to find similar meanings in different sentences or to process words that have different meanings. Through these techniques, NLP text analysis transforms unstructured text into insights. Model training Processed data is then used to train machine learning models, which learn patterns and relationships within the data. During training, the model adjusts its parameters to minimize errors and improve its performance. Once trained, the model can be used to make predictions or generate outputs on new, unseen data. The effectiveness of NLP modeling is continually refined through evaluation, validation and fine-tuning to enhance accuracy and relevance in real-world applications. Different software environments are useful throughout the said processes. For example, the Natural Language Toolkit (NLTK) is a suite of libraries and programs for English that is written in the Python programming language. It supports text classification, tokenization, stemming, tagging, parsing and semantic reasoning functionalities. TensorFlow is a free and open-source software library for machine learning and AI that can be used to train models for NLP applications. Tutorials and certifications abound for those interested in familiarizing themselves with such tools. Even state-of-the-art NLP models are not perfect, just as human speech is prone to error. As with any AI technology, NLP comes with potential pitfalls. Human language is filled with ambiguities that make it difficult for programmers to write software that accurately determines the intended meaning of text or voice data. Human language might take years for humans to learn and many never stop learning. But then programmers must teach natural language-powered applications to recognize and understand irregularities so their applications can be accurate and useful. Associated risks might include: Biased training As with any AI function, [biased data](https://www.ibm.com/think/topics/ai-bias) used in training will skew the answers. The more diverse the users of an NLP function, the more significant this risk becomes, such as in government services, healthcare and HR interactions. Training datasets scraped from the web, for example, are prone to bias. Misinterpretation As in programming, there is a risk of garbage in, garbage out (GIGO). [Speech recognition](https://www.ibm.com/think/topics/speech-recognition), also known as speech-to-text, is the task of reliably converting voice data into text data. But NLP solutions can become confused if spoken input is in an obscure dialect, mumbled, too full of slang, homonyms, incorrect grammar, idioms, fragments, mispronunciations, contractions or recorded with too much background noise. New vocabulary New words are continually being invented or imported. The conventions of grammar can evolve or be intentionally broken. In these cases, NLP can either make a best guess or admit it’s unsure and either way, this creates a complication. Tone of voice When people speak, their verbal delivery or even body language can give an entirely different meaning than the words alone. Exaggeration for effect, stressing words for importance or sarcasm can be confused by NLP, making the semantic analysis more difficult and less reliable. NLP applications can now be found across virtually every industry. Finance In financial dealings, nanoseconds might make the difference between success and failure when accessing data, or making trades or deals. NLP can speed the mining of information from financial statements, annual and regulatory reports, news releases or even social media. Healthcare New medical insights and breakthroughs can arrive faster than many healthcare professionals can keep up. NLP and AI-based tools can help speed the analysis of health records and medical research papers, making better-informed medical decisions possible, or assisting in the detection or even prevention of medical conditions. Insurance NLP can analyze claims to look for patterns that can identify areas of concern and find inefficiencies in claims processing, leading to greater optimization of processing and employee efforts. Legal Almost any legal case might require reviewing mounds of paperwork, background information and legal precedent. NLP can help automate legal discovery, assisting in the organization of information, speeding review and making sure that all relevant details are captured for consideration.
Shard	39 (laksa)
Root Hash	15319107097793762839
Unparsed URL	com,ibm!www,/think/topics/natural-language-processing s443