What is natural language processing (NLP)?

How does natural language processing work?

NLP uses many different techniques to enable computers to understand natural language as humans do. Whether the language is spoken or written, natural language processing can use AI to take real-world input, process it and make sense of it in a way a computer can understand. Just as humans have different sensors -- such as ears to hear and eyes to see -- computers have programs to read and microphones to collect audio. And just as humans have a brain to process that input, computers have a program to process their respective inputs. At some point in processing, the input is converted to code that the computer can understand.

There are two main phases to natural language processing: data preprocessing and algorithm development.

Data preprocessing involves preparing and cleaning text data so that machines can analyze it. Preprocessing puts data in a workable form and highlights features in the text that an algorithm can work with. There are several ways this can be done, including the following:

• Tokenization. Tokenization substitutes sensitive information with nonsensitive information, or a token. Tokenization is often used in payment transactions to protect credit card data.
• Stop word removal. Common words are removed from the text, so unique words that offer the most information about the text remain.
• Lemmatization and stemming. Lemmatization groups together different inflected versions of the same word. For example, the word "walking" would be reduced to its root form, or stem, "walk" to process.
• Part-of-speech tagging. Words are tagged based on which part of speech they correspond to -- such as nouns, verbs or adjectives.

Once the data has been preprocessed, an algorithm is developed to process it. There are many different natural language processing algorithms, but the following two main types are commonly used:

• Rule-based system. This system uses carefully designed linguistic rules. This approach was used early in the development of natural language processing and is still used.
• Machine learning-based system. Machine learning algorithms use statistical methods. They learn to perform tasks based on training data they're fed and adjust their methods as more data is processed. Using a combination of machine learning, deep learning and neural networks, natural language processing algorithms hone their own rules through repeated processing and learning.

Why is natural language processing important?

Businesses use large amounts of unstructured, text-heavy data and need a way to efficiently process it. Much of the information created online and stored in databases is natural human language, and until recently, businesses couldn't effectively analyze this data. This is where natural language processing is useful.

The advantages of natural language processing can be seen when considering the following two statements: "Cloud computing insurance should be part of every service-level agreement" and "A good SLA ensures an easier night's sleep -- even in the cloud." If a user relies on natural language processing for search, the program will recognize that cloud computing is an entity, that cloud is an abbreviated form of cloud computing, and that SLA is an industry acronym for service-level agreement.

These are the types of vague elements that frequently appear in human language and that machine learning algorithms have historically been bad at interpreting. Now, with improvements in deep learning and machine learning methods, algorithms can effectively interpret them. These improvements expand the breadth and depth of data that can be analyzed.

Likewise, NLP is useful for the same reasons as when a person interacts with a generative AI chatbot or AI voice assistant. Instead of needing to use specific predefined language, a user could interact with a voice assistant like Siri on their phone using their regular diction, and their voice assistant will still be able to understand them.

Techniques and methods of natural language processing

Syntax and semantic analysis are two main techniques used in natural language processing.

Syntax is the arrangement of words in a sentence to make grammatical sense. NLP uses syntax to assess meaning from a language based on grammatical rules. Syntax NLP techniques include the following:

Parsing

This is the grammatical analysis of a sentence. For example, a natural language processing algorithm is fed the sentence, "The dog barked." Parsing involves breaking this sentence into parts of speech -- i.e., dog = noun, barked = verb. This is useful for more complex downstream processing tasks.

Word segmentation

This is the act of taking a string of text and deriving word forms from it. For example, a person scans a handwritten document into a computer. The algorithm can analyze the page and recognize that the words are divided by white spaces.

Sentence breaking

This places sentence boundaries in large texts. For example, a natural language processing algorithm is fed the text, "The dog barked. I woke up." The algorithm can use sentence breaking to recognize the period that splits up the sentences.

Morphological segmentation

This divides words into smaller parts called morphemes. For example, the word untestably would be broken into [[un[[test]able]]ly], where the algorithm recognizes "un," "test," "able" and "ly" as morphemes. This is especially useful in machine translation and speech recognition.

Stemming

This divides words with inflection in them into root forms. For example, in the sentence, "The dog barked," the algorithm would recognize the root of the word "barked" is "bark." This is useful if a user is analyzing text for all instances of the word bark, as well as all its conjugations. The algorithm can see that they're essentially the same word even though the letters are different.

Semantics involves the use of and meaning behind words. Natural language processing applies algorithms to understand the meaning and structure of sentences. Semantic techniques include the following:

Word sense disambiguation

This derives the meaning of a word based on context. For example, consider the sentence, "The pig is in the pen." The word pen has different meanings. An algorithm using this method can understand that the use of the word here refers to a fenced-in area, not a writing instrument.

Named entity recognition (NER)

NER determines words that can be categorized into groups. For example, an algorithm using this method could analyze a news article and identify all mentions of a certain company or product. Using the semantics of the text, it could differentiate between entities that are visually the same. For instance, in the sentence, "Daniel McDonald's son went to McDonald's and ordered a Happy Meal," the algorithm could recognize the two instances of "McDonald's" as two separate entities -- one a restaurant and one a person.

Natural language generation (NLG)

NLG uses a database to determine the semantics behind words and generate new text. For example, an algorithm could automatically write a summary of findings from a business intelligence (BI) platform, mapping certain words and phrases to features of the data in the BI platform. Another example would be automatically generating news articles or tweets based on a certain body of text used for training.

Current approaches to natural language processing are based on deep learning, a type of AI that examines and uses patterns in data to improve a program's understanding. Deep learning models require massive amounts of labeled data for the natural language processing algorithm to train on and identify relevant correlations, and assembling this kind of big data set is one of the main hurdles to natural language processing.

Earlier approaches to natural language processing involved a more rule-based approach, where simpler machine learning algorithms were told what words and phrases to look for in text and given specific responses when those phrases appeared. But deep learning is a more flexible, intuitive approach in which algorithms learn to identify speakers' intent from many examples -- almost like how a child would learn human language.

Three open source tools commonly used for natural language processing include Natural Language Toolkit (NLTK), Gensim and NLP Architect by Intel. NLTK is a Python module with data sets and tutorials. Gensim is a Python library for topic modeling and document indexing. NLP Architect by Intel is a Python library for deep learning topologies and techniques.

What is natural language processing used for?

Some of the main functions and NLP tasks that natural language processing algorithms perform include the following:

• Text classification. This function assigns tags to texts to put them in categories. This can be useful for sentiment analysis, which helps the natural language processing algorithm determine the sentiment, or emotion, behind a text. For example, when brand A is mentioned in X number of texts, the algorithm can determine how many of those mentions were positive and how many were negative. It can also be useful for intent detection, which helps predict what the speaker or writer might do based on the text they're producing.
• Text extraction. This function automatically summarizes text and finds important pieces of data. One example of this is keyword extraction, which pulls the most important words from the text, which can be useful for search engine optimization. Doing this with natural language processing requires some programming -- it isn't completely automated. However, there are plenty of simple keyword extraction tools that automate most of the process -- the user just sets parameters within the program. For example, a tool might pull out the most frequently used words in the text. Another example is entity recognition, which extracts the names of people, places and other entities from text.
• Machine translation. In this process, a computer translates text from one language, such as English, to another language, such as French, without human intervention.
• Natural language generation. This process uses natural language processing algorithms to analyze unstructured data and automatically produce content based on that data. One example of this is in language models like the third-generation Generative Pre-trained Transformer (GPT-3), which can analyze unstructured text and then generate believable articles based on that text.

The functions listed above are used in a variety of real-world applications, including the following:

• Customer feedback analysis. Tools using AI can analyze social media reviews and filter out comments and queries for a company.
• Customer service automation. Voice assistants on a customer service phone line can use speech recognition to understand what the customer is saying, so that it can direct their call correctly.
• Automatic translation. Tools such as Google Translate, Bing Translator and Translate Me can translate text, audio and documents into another language.
• Academic research and analysis. Tools using AI can analyze huge amounts of academic material and research papers based on the metadata of the text as well as the text itself.
• Analysis and categorization of healthcare records. AI-based tools can use insights to predict and, ideally, prevent disease.
• Plagiarism detection. Tools such as Copyleaks and Grammarly use AI technology to scan documents and detect text matches and plagiarism.
• Stock forecasting and insights into financial trading. NLP tools can analyze market history and annual reports that contain comprehensive summaries of a company's financial performance.
• Talent recruitment in human resources. Organizations can use AI-based tools to reduce hiring time by automating the candidate sourcing and screening process.
• Automation of routine litigation. AI-powered tools can do research, identify possible issues and summarize cases faster than human attorneys.
• Spam detection. NLP-enabled tools can be used to classify text for language that's often used in spam or phishing attempts. For example, AI-enabled tools can detect bad grammar, misspelled names, urgent calls to action and threatening terms.

Benefits of natural language processing

The main benefit of NLP is that it improves the way humans and computers communicate with each other. The most direct way to manipulate a computer is through code -- the computer's language. Enabling computers to understand human language makes interacting with computers much more intuitive for humans.

Other benefits include the following:

• Offers improved accuracy and efficiency of documentation.
• Enables an organization to use chatbots for customer support.
• Provides an organization with the ability to automatically make a readable summary of a larger, more complex original text.
• Lets organizations analyze structured and unstructured data.
• Enables personal assistants such as Alexa to understand the spoken word.
• Makes it easier for organizations to perform sentiment analysis.
• Organizations can use NLP to better understand lead generation, social media posts, surveys and reviews.
• Provides advanced insights from analytics that were previously unreachable due to data volume.

Challenges of natural language processing

There are numerous challenges in natural language processing, and most of them boil down to the fact that natural language is ever-evolving and somewhat ambiguous. They include the following:

• Precision. Computers traditionally require humans to speak to them in a programming language that's precise, unambiguous and highly structured -- or through a limited number of clearly enunciated voice commands. Human speech, however, isn't always precise; it's often ambiguous and the linguistic structure can depend on many complex variables, including slang, regional dialects and social context.
• Tone of voice and inflection. Natural language processing hasn't yet been perfected. For example, semantic analysis can still be a challenge. Other difficulties include the fact that the abstract use of language is typically tricky and complex for programs to understand. For instance, natural language processing doesn't pick up sarcasm easily. These topics usually require understanding the words being used and their context in a conversation. Also, a sentence can change meaning depending on which word or syllable the speaker puts stress on. NLP algorithms can miss the subtle but important tone changes in a person's voice when performing speech recognition. The tone and inflection of speech can also vary among different accents, which can be challenging for an algorithm to parse.
• Evolving use of language. Natural language processing is also challenged by the fact that language -- and the way people use it -- is continually changing. Although there are rules to language, none are written in stone, and they're subject to change over time. Hard computational rules that work now might become obsolete, as the characteristics of real-world language change over time.
• Bias. NLP systems can be biased when their processes reflect the biases that appear in their training data. This is an issue in medical fields and hiring positions, where a person might be discriminated against.

The evolution of natural language processing

NLP draws from a variety of disciplines, including computer science and computational linguistics developments dating back to the mid-20th century. Its evolution included the following major milestones:

1950s

Natural language processing has its roots in this decade, when Alan Turing developed the Turing Test to determine whether or not a computer is truly intelligent. The test involves automated interpretation and the generation of natural language as a criterion of intelligence.

1950s-1990s

 NLP was largely rules-based, using handcrafted rules developed by linguists to determine how computers would process language. The Georgetown-IBM experiment in 1954 became a notable demonstration of machine translation, automatically translating more than 60 sentences from Russian to English. The 1980s and 1990s saw the development of rule-based parsing, morphology, semantics and other forms of natural language understanding.

1990s

The top-down, language-first approach to natural language processing was replaced with a more statistical approach because advancements in computing made this a more efficient way of developing NLP technology. Computers were becoming faster and could be used to develop rules based on linguistic statistics without a linguist creating all the rules. Data-driven natural language processing became mainstream during this decade. Natural language processing shifted from a linguist-based approach to an engineer-based approach, drawing on a wider variety of scientific disciplines instead of delving into linguistics.

2000-2020s

Natural language processing saw dramatic growth in popularity as a term. NLP processes using unsupervised and semi-supervised machine learning algorithms were also explored. With advances in computing power, natural language processing has also gained numerous real-world applications. NLP also began powering other applications like chatbots and virtual assistants. Today, approaches to NLP involve a combination of classical linguistics and statistical methods.

Natural language processing plays a vital part in technology and the way humans interact with it. Though it has its challenges, NLP is expected to become more accurate with more sophisticated models, more accessible and more relevant in numerous industries. NLP will continue to be an important part of both industry and everyday life.

As natural language processing is making significant strides in new fields, it's becoming more important for developers to learn how it works. Learn how to develop your skills in creating NLP programs.