1. Tokenization

Tokenization is the process of breaking a string of text into smaller pieces or tokens. The most common form of tokenization is word tokenization, which splits a string of text into individual words. However, other forms of tokenization are also possible, such as sentence tokenization, which splits a string of text into individual sentences.

2. Stemming and Lemmatization

Stemming and lemmatization are two related methods for reducing a word to its base form. Stemming typically involves removing suffixes, such as “ing” or “ly”, while lemmatization typically involves converting a word to its base form, such as converting “cats” to “cat”.

3. Part-of-Speech Tagging

Part-of-speech tagging is the process of assigning a part of speech to each token in a string of text. The most common parts of speech are nouns, verbs, adjectives, and adverbs. Assigning a part of speech to each word can be helpful for downstream tasks such as parsing and machine translation.

4. Named Entity Recognition

Named entity recognition is the task of identifying named entities in a string of text. Named entities can be people, places, organizations, or other things. Identifying named entities can be helpful for tasks such as information extraction and question answering.

5. Sentiment Analysis

Sentiment analysis is the task of determining the sentiment of a string of text. The most common form of sentiment analysis is binary sentiment analysis, which classifies a string of text as either positive or negative. However, other forms of sentiment analysis are also possible, such as multi-class sentiment analysis, which classifies a string of text into multiple categories (such as positive, negative, and neutral).

6. Topic Modelling

Topic modelling is the task of identifying the topics present in a collection of documents. Each document can be thought of as a mixture of topics, and topic modelling can be used to identify the underlying topics present in each document. This can be helpful for tasks such as information retrieval and document classification.

7. Document Classification

Document classification is the task of assigning a class label to a document. The most common form of document classification is binary document classification, which assigns one of two class labels to each document (such as “positive” or “negative”). However, other forms of document classification are also possible, such as multi-class document classification, which assigns multiple class labels to each document (such as “sports”, “politics”, or “technology”).

8 . Text Clustering

Text clustering is the task of grouping a set of documents into clusters based on their similarity. This can be helpful for tasks such as information retrieval and topic modelling.

9 . Natural Language Generation

Natural language generation is the task of generating natural language text from structured data. This can be helpful for tasks such as summarization and report generation.

10 . Machine Translation

Machine translation is the task of translating text from one language to another. This can be helpful for tasks such as information retrieval and machine reading comprehension.