Natural Language Processing and Understanding (NLU)

Tokenizationnlpsplitting text into words and subword units Text Classificationnlpassigning labels to documents Named Entity Recognition (NER)nlpfinding and typing entities Summarizationnlpextractive and abstractive summaries Machine Translationnlptranslating between languages Speech (ASR, Text-to-Speech)nlpconverting between audio and text

Natural language processing (NLP) covers the techniques that let software read, classify, transform and generate human language. Natural language understanding (NLU) is the subset concerned with meaning: intent, entities and relations. This section collects pages on the core NLP tasks; the taxonomy diagram further down places them within the wider AI and machine learning landscape.

Natural language processing and natural language understanding compared
Artificial Intelligence Taxonomy Artificial Intelligence (AI) Expert Systems Rule-based Systems Knowledge-based Systems Natural Language Processing (NLP) Text Classification Named Entity Recognition (NER) Sentiment Analysis Machine Translation Question Answering Computer Vision Image Classification Object Detection Image Segmentation Facial Recognition Robotics Planning & Scheduling Machine Learning (ML) Supervised Learning Classification Logistic Regression Decision Trees Random Forest Support Vector Machines (SVM) Naive Bayes K-Nearest Neighbors (KNN) Machine Learning (ML) (cont.) Supervised Learning (cont.) Regression Linear Regression Polynomial Regression Ridge/Lasso Regression Unsupervised Learning Clustering K-Means DBSCAN Hierarchical Clustering Dimensionality Reduction PCA (Principal Component Analysis) t-SNE UMAP Reinforcement Learning Q-Learning Policy Gradient Actor-Critic Deep Q-Networks (DQN) Deep Learning Feedforward Neural Networks (FNN) Perceptron Multi-Layer Perceptron (MLP) Convolutional Neural Networks (CNN) LeNet AlexNet VGGNet ResNet EfficientNet Machine Learning (ML) (cont.) Deep Learning (cont.) Recurrent Neural Networks (RNN) Vanilla RNN LSTM (Long Short-Term Memory) Bidirectional LSTM Stacked LSTM GRU (Gated Recurrent Unit) Transformer Architecture Encoder-Only (BERT, RoBERTa) Embedding Models ← Your RAG embeddings! MiniLM (384 dims) multi-qa-mpnet (768 dims) OpenAI ada-002 (1536 dims) Decoder-Only (GPT, Claude, LLaMA) Large Language Models (LLMs) Encoder-Decoder (T5, BART) Generative Models Variational Autoencoders (VAE) Generative Adversarial Networks (GAN) DCGAN StyleGAN CycleGAN Diffusion Models Stable Diffusion DALL-E Midjourney Graph Neural Networks (GNN) Graph Convolutional Networks (GCN) Graph Attention Networks (GAT)

1. How the NLP tasks fit together

A typical NLP pipeline Raw text or audio documents, chat, calls Speech-to-text if audio Tokenization subwords, sentences Classification labels NER entities Summarization shorter text Translation other language Search, analytics, agents, RAG

Almost every pipeline starts by turning input into tokens: audio is first transcribed by a speech recognition model, and text is split into words or subword units that a model can embed. From there, tasks branch. Classification and NER extract structured signals (a label per document, spans per entity) that feed dashboards, routing rules and search filters. Summarization and translation are generation tasks that produce new text. Modern transformer models can perform all of these, but smaller task-specific models remain cheaper and more predictable at high volume.

2. Choosing an approach

The table below summarises what each task produces and the usual tooling, from lightweight libraries to large language models.

TaskOutputTypical toolsUse when
TokenizationTokens, idsspaCy, Hugging Face tokenizers, tiktokenAlways; it precedes every model
Text classificationLabel(s) per documentscikit-learn, fine-tuned BERT, LLM zero-shotRouting, moderation, sentiment
NEREntity spans and typesspaCy, Presidio, fine-tuned transformersExtracting people, orgs, PII
SummarizationShorter textBART, T5, LLMsLong documents, meeting notes
Machine translationText in another languageMarianMT, NLLB, cloud APIs, LLMsMultilingual content
SpeechTranscript or audioWhisper, cloud ASR/TTSCalls, voice interfaces

A practical rule: use a small supervised model when you have labelled data and need low latency and stable outputs, and an LLM when labels are scarce, the task changes often, or the output is free-form text.

Search and RetrievalSectionusing NLP output for retrieval LLM EngineeringSectionbuilding with large language models