NLP Guide 101: NLP Evolution — Past, Present and Future.…

Let’s check out how NLP works and the major evolution in NLP..

What is Natural Language Processing (NLP)?

As per Wikipedia:

Natural Language Processing (NLP) is an area of computer science and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to fruitfully process large amounts of natural language (text and speech) data.

In simple words, Natural Language Processing or NLP is a technique or program which makes computers or machines enable to understand and process human language. Means it’s a communication tool between human and machine.

Before diving deeper into NLP, Let’s understand how communication happens in the human language and how NLP originated?

In this process one generates the message in his natural language and another person understands the same message and then another person generates the message in his natural language and the first person understands the message. To make this communication complete or successful, there should be a common language from both ends. If it’s not there then what?

Then we have to involve a third person in this communication process who can translate the message for both of them. But there are some limitations with this third person in terms of communication and his availability. It means a lot of dependency on him to make successful communication. To remove this dependency to translate the message, NLP came into existence which helped to replace third person (human) with machines not only to perform translator’s job but also helps to analyze and derive meaning information (knowledge) from the communication data (text and speech) in a smart and useful way.

Now we know, NLP originated from the idea of Machine Translation (MT) which came to existence during the Second World War.

The primary idea was to convert one human language to another human language, for example, turning the Russian language to English language using the brain of the Computers but after that, the thought of conversion of human language to computer language and vice-versa emerged, so that communication with the machine became easy.

Computer Science and Artificial Intelligence both came together and made it possible for us to use machines to do successful communication. In simple words, NLP is basically teaching computers to process human language.

It is primarily concerned with designing and building applications and systems that enable interaction between machines and natural languages that have been evolved for use by humans.

Let’s check out how NLP works and the major evolution in NLP....

Levels of NLP: NLP includes a wide set of syntax, semantics, discourse, and speech tasks. The following chart broadly shows these points.

Don’t get confused by these new terms such as Phonology, Pragmatics, Morphology, Syntax, and Semantics. Let’s explore these in a very brief manner -

Phonology — This science helps to deal with patterns present in the sound and speeches related to the sound as a physical entity. It’s the smallest unit of sound in a language.

Morphology — This science deals with the structure of the words and the systematic relations between them. It’s the smallest unit of meaning in language.

Syntax — This science deals with the structure of the sentences where we know how words and sentences are constructed from these two building blocks.

Semantics — This science deals with the literal meaning of the words, phrases as well as sentences.

Discourse — Semantics in context. Conversation, persuasive writing, etc.

Natural Language Processing (NLP): Text Vs Speech

Natural Language can refer to Text or Speech. Goal of both is the same: translate raw data (text or speech) into underlying concepts (NLU) then possibly into the other form (NLG).

The area of Natural Language Processing is divided into sub-areas, i.e., Natural Language Generation and Natural Language Understanding which are as the name suggests associated with the generation and understanding the message (text or speech).

Natural Language Understanding

• NLU is an artificial intelligence language that uses computer software to recognize text or speech sentences.
• The NLU provides a direct human-computer interaction. The NLU allows human languages to be understood statically by the computer without the use of if / else.
• Goal is to derive meaning from natural language.
• Imagine a Concept (aka Semantic or Representation) space
• In it, an idea/word/concept has a unique computer representation
• Usually via a vector space
• NLU Mapping language into this space

NLU Applications:

• ML on Text (Classification, Regression, Clustering)
• Document Recommendation
• Language Identification
• Natural Language Search
• Sentiment Analysis
• Text Summarization
• Extracting Word/Document Meaning (vectors)
• Relationship Extraction
• Topic Modelling

Relationship between NLP and NLU

NLU Service Provider

Natural Language Generation (NLG) :

• Natural-language generation (NLG) is a software process that transforms structured data into natural language (or written narrative).
• Mapping from computer representation space to language space
• Opposite direction of NLU
• Usually need NLU to perform NLG!

NLG Applications:

• Image Captioning
• (Better) Text Summarization
• Machine Translation
• Question Answering (QA)/Chatbots

Major evolutions in NLP…

The following is a list of some of the most commonly researched tasks in NLP.

Real-world applications that make use of NLP techniques:

• Sentimental Analysis: By implementing NLP Tech giants such as Amazon and Netflix, gain insights on their customers to enhance their products and make better recommendations.
• Speech Recognition: NLP has been used widely in speech recognition, we’re all aware of Alexa, Siri, Google assistant, and Cortana. They’re all applications of NLP.
• Machine Translation: The popular Google translator uses Natural Language Processing to process and translate one language to the other. It is also used in spell checkers, keyword search, information extraction.
• Advertisement Matching: NLP is also used in advertisement matching to recommend ads based on search history.
• Chatbot: Chatbots are becoming popular in the field of customer service. A popular example is Eva the HDFC chatbot who has addressed over 3 million customer queries, interacted with over half a million unique users, and held over a million conversations.

Basic Structure of NLP Application (Chatbot)

Knowledge Base — It contains the database of information that is used to equip chatbots with the information needed to respond to queries of customers’ requests.

Data Store — It contains an interaction history of chatbot with users.

NLP Layer — It translates users queries (free form) into information that can be used for appropriate responses.

Application Layer — It is the application interface that is used to interact with the user.

Future of NLP

• Human readable natural language processing is the biggest Al- problem. It is almost the same as solving the central artificial intelligence problem and making computers as intelligent as people.
• Future computers or machines will be smarter than today with the help of NLP. It can be learned from the information online and applied in the real world, however, lots of work needs to be done in this regard.
• Combined with natural language generation, computers will become more capable of receiving and giving useful and resourceful information.

Special Thanks : Intel Corporation for Intel AI Program which help me a lot to write this article.