Text classification assigns one or more labels from a fixed set to a piece of text: spam or not, sentiment, topic, intent, toxicity, support-ticket routing, document type, language. It is the most widely deployed NLP task, and the right approach depends far more on label count, data volume, latency budget and need for explanation than on chasing the newest model. A well-tuned TF-IDF plus logistic regression baseline is often surprisingly hard to beat, and it is always the first thing to build.
Part of Natural Language Processing. Prerequisite: Tokenization. For token-level labels see Named Entity Recognition.
Getting this framing right matters more than the model: training a softmax on a genuinely multi-label problem forces labels to compete and silently loses information.
Every approach is a representation followed by a decision function. Sparse lexical features (bag of words, TF-IDF, character n-grams) with a linear model train in seconds on CPU, are easy to inspect, and remain strong when labels depend on keywords. Frozen embeddings from a sentence encoder add semantic generalization with minimal training. Fine-tuned encoders such as BERT-family models are usually the most accurate option once you have a few thousand labeled examples. Prompted LLMs need no training data at all and handle long-tail labels and changing label definitions, at higher per-item cost and latency.
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline, make_union
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
cats = ["sci.med", "sci.space", "rec.autos", "comp.graphics"]
train = fetch_20newsgroups(subset="train", categories=cats, remove=("headers", "footers", "quotes"))
test = fetch_20newsgroups(subset="test", categories=cats, remove=("headers", "footers", "quotes"))
features = make_union(
TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True, stop_words="english"),
TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5), min_df=3, sublinear_tf=True),
)
pipe = make_pipeline(features, LogisticRegression(max_iter=2000, class_weight="balanced"))
search = GridSearchCV(pipe, {"logisticregression__C": [1, 4, 16]}, cv=3, scoring="f1_macro", n_jobs=-1)
search.fit(train.data, train.target)
pred = search.predict(test.data)
print(search.best_params_)
print(classification_report(test.target, pred, target_names=test.target_names))
# Most indicative words per class (word-level vectorizer only)
vec = search.best_estimator_[0].transformer_list[0][1]
clf = search.best_estimator_[-1]
vocab = vec.get_feature_names_out()
n_word = len(vocab)
for k, name in enumerate(test.target_names):
top = clf.coef_[k][:n_word].argsort()[-8:][::-1]
print(name, [vocab[i] for i in top])
TF-IDF weights a term by how often it appears in the document and down-weights terms common across the corpus: tfidf(t,d) = tf(t,d) · log(N / df(t)) (scikit-learn uses a smoothed variant). Removing headers and quotes from 20 Newsgroups matters: without it the model learns e-mail addresses and signatures, a textbook case of leakage. Character n-grams add robustness to typos and morphology.
A pretrained encoder (see Encoder-Only models) produces a contextual vector per token; a linear layer on the pooled or [CLS] representation maps it to class logits. Fine-tuning updates all weights with cross-entropy loss, typically with a learning rate in the low 1e-5 range, a few epochs, warmup and early stopping on a validation set. With Hugging Face this is AutoModelForSequenceClassification.from_pretrained(name, num_labels=K) plus Trainer; set problem_type="multi_label_classification" for multi-label. Distilled models (DistilBERT, MiniLM) cut latency substantially at a small accuracy cost, and parameter-efficient methods (LoRA, SetFit for few-shot) reduce training cost.
Accuracy is misleading under class imbalance: a fraud model that predicts "legit" for everything can be 99 % accurate and useless. Report per-class precision, recall and F1, macro-averaged when rare classes matter, plus PR-AUC for imbalanced binary tasks. The default 0.5 threshold is rarely optimal; pick thresholds on a validation set according to the business cost of false positives versus false negatives, and check calibration (reliability curves) if downstream logic uses the probabilities. Always inspect the confusion matrix and a sample of errors; they often reveal label noise or ambiguous label definitions rather than model weakness.
| Situation | Start with |
|---|---|
| No labels yet, need something this week | Prompted LLM with a clear label schema and structured output; log predictions for later training |
| Hundreds of labeled examples, CPU only | TF-IDF + logistic regression; or sentence embeddings + logistic regression |
| Thousands of examples, accuracy matters | Fine-tuned encoder (DeBERTa/RoBERTa class) |
| Very high volume, tight latency | Distilled encoder or linear model, possibly trained on LLM-generated labels |
| Labels change often | Embeddings + nearest-neighbor or LLM with label descriptions |
Suppose a support-ticket classifier is evaluated on 1,000 tickets: 900 "billing" (F1 0.95), 80 "outage" (F1 0.60) and 20 "security" (F1 0.20). The weighted F1 averages per-class scores by support: (900·0.95 + 80·0.60 + 20·0.20) / 1000 = 0.907. The macro F1 gives each class equal weight: (0.95 + 0.60 + 0.20) / 3 ≈ 0.58. Both are correct; they answer different questions. Weighted F1 says the typical ticket is handled well. Macro F1 says the rare classes, here the security tickets that probably matter most, are handled badly. When the business cost sits in the minority classes, report macro F1 and the per-class table, not just a single headline number.
The same logic applies to multi-label tasks, where micro-averaging pools every (document, label) decision and is dominated by frequent tags. Pick the averaging scheme before you look at results, so the choice is not driven by which number looks best.
Most of the effort in a deployed classifier goes into the loop around the model. Labeling guidelines with worked examples and edge-case rulings are the real specification; when two annotators disagree, fix the guideline, not just the label. Active learning sends the examples the model is least sure about (probabilities near the threshold, or disagreement between models) to human reviewers, which usually improves the model faster per label than random sampling. Calibration (temperature scaling or isotonic regression on a held-out set) makes probabilities meaningful, so thresholds can be set from cost: if a missed security ticket costs ten times a false alarm, the threshold should sit well below 0.5.
Once deployed, monitor the share of each predicted class, the mean confidence, and the rate of out-of-vocabulary or very long inputs. A sudden change in any of them is a cheaper early warning than waiting for labeled data to reveal an accuracy drop. Keep a small, stratified, human-labeled audit sample every week or month; it is the only reliable way to measure live accuracy. For high-throughput systems, a common pattern is to use an LLM to bootstrap labels, then train a small encoder or linear model that serves cheaply, routing low-confidence cases back to the LLM or to a human queue.