Naive Bayes

Naive Bayes is a family of probabilistic classifiers that apply Bayes' theorem with one strong simplification: every feature is assumed independent of every other feature once the class is known. The assumption is almost never literally true, yet the resulting models train in a single pass over the data, handle tens of thousands of sparse features, need very little data, and are surprisingly hard to beat on text tasks such as spam filtering, document categorization and quick sentiment baselines.

Part of Classification within Supervised Learning.

1. Bayes' Rule as a Classifier

Bayes' theorem gives the posterior probability of class y given features x: P(y | x) = P(y) · P(x | y) / P(x). The denominator is the same for every class, so for classification we only need argmaxy P(y) · P(x | y). The prior P(y) is estimated from class frequencies; the hard part is the likelihood P(x | y), a joint distribution over all features that would need exponentially many parameters to estimate directly.

Naive Bayes is a generative model: it models how each class generates features, then inverts with Bayes' rule. Logistic regression is the matching discriminative model, estimating P(y | x) directly. A classic result (Ng & Jordan, 2001) is that the generative model tends to reach its (higher) asymptotic error with fewer samples, while the discriminative one wins as data grows — a useful rule of thumb when choosing between them.

2. The Naive Independence Assumption

Assuming conditional independence factorizes the likelihood: P(x | y) = Πj P(xj | y). Each factor is a one-dimensional distribution that can be estimated from counts, so the number of parameters grows linearly with the number of features instead of exponentially. Training is a single pass that tallies statistics per class: O(n · d).

Why does it work when features are clearly correlated (e.g. "New" and "York")? Classification only needs the ranking of posteriors to be right, not their values. Correlated evidence gets double-counted, which pushes probabilities toward 0 or 1, but often leaves the argmax unchanged. The practical consequence: Naive Bayes labels are frequently good while its probabilities are poorly calibrated.

3. Variants: Gaussian, Multinomial, Bernoulli, Complement

The variants differ only in the distribution chosen for P(xj | y):

In Gaussian NB, each class gets its own variance per feature, so the decision boundary between two classes is quadratic in general. For Multinomial NB, the log-posterior is linear in the word counts, so it is a linear classifier in disguise.

4. Smoothing and Log Space

If a word never appeared with class y in training, its estimated probability is zero and the whole product collapses to zero regardless of other evidence. Additive (Laplace/Lidstone) smoothing fixes this:

θyj = (Nyj + α) / (Ny + α · d)

where Nyj is the count of feature j in class y, Ny the total count for class y, and d the vocabulary size. α = 1 is Laplace smoothing; tuning α on validation data (often well below 1 for large vocabularies) is one of the few hyperparameters worth searching. Multiplying thousands of small probabilities underflows floating point, so implementations sum log-probabilities instead: log P(y) + Σ log P(xj | y).

5. Python Example (scikit-learn)

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import ComplementNB
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report

cats = ["sci.space", "rec.autos", "comp.graphics", "talk.politics.misc"]
train = fetch_20newsgroups(subset="train", categories=cats, remove=("headers", "footers", "quotes"))
test = fetch_20newsgroups(subset="test", categories=cats, remove=("headers", "footers", "quotes"))

pipe = make_pipeline(TfidfVectorizer(sublinear_tf=True, min_df=2, stop_words="english"), ComplementNB())
grid = GridSearchCV(pipe, {"complementnb__alpha": [0.01, 0.1, 0.3, 1.0]}, cv=5).fit(train.data, train.target)

pred = grid.predict(test.data)
print("best alpha:", grid.best_params_)
print(classification_report(test.target, pred, target_names=test.target_names))

# Most indicative words per class
vec, nb = grid.best_estimator_[0], grid.best_estimator_[-1]
vocab = vec.get_feature_names_out()
for i, name in enumerate(test.target_names):
    top = nb.feature_log_prob_[i].argsort()[::-1][:8]   # highest weight = most indicative
    print(f"{name:<20}", ", ".join(vocab[top]))

Training on a few thousand documents takes well under a second. This pipeline is a reasonable baseline to run before reaching for transformer embeddings.

6. When to Use It — Trade-offs

Use it for text baselines, spam/abuse filters that must retrain constantly, very small labeled sets, and as a fast feature in a larger system. Avoid it when you need reliable probabilities without calibration or when features interact strongly.

7. Common Pitfalls

• Feeding negative values to MultinomialNB (e.g. standardized features) — it expects non-negative counts.
• Using predict_proba as a calibrated risk score. Wrap with CalibratedClassifierCV if probabilities matter.
• Duplicated or highly correlated features are counted multiple times and dominate the decision.
• Leaving alpha at 1.0 on large vocabularies; it often over-smooths.
• Skewed priors. If the deployment class mix differs from training, override class_prior.

8. Worked Example: A Tiny Spam Filter

Suppose the training data has equal numbers of spam and ham messages over a four-word vocabulary. Word counts per class are: spam free 3, win 2, meeting 0, lunch 0; ham free 1, win 0, meeting 2, lunch 2. Each class has 5 word tokens. With Laplace smoothing (α = 1, d = 4), every estimate is (count + 1) / (5 + 4):

For the message "free lunch", the class scores with equal priors are proportional to 4/9 · 1/9 for spam and 2/9 · 3/9 for ham, that is 4 versus 6, so P(spam) = 4 / 10 = 0.40. In log space the same calculation is a sum of per-word log-ratios: log(4/2) + log(1/3) ≈ 0.69 − 1.10 = −0.41, and the logistic function of −0.41 is 0.40. For "free win", the scores are 12 versus 2, giving P(spam) ≈ 0.86. Without smoothing, "lunch" would have probability 0 under spam and any message containing it could never be classified as spam, however many spam words it also contained.

The log-ratio view also explains why Naive Bayes is easy to inspect: each word contributes a fixed weight log P(w | spam) − log P(w | ham), and the prediction is the sum of the weights of the words present plus the prior log-odds. Listing the words with the largest weights is a quick sanity check for leakage, such as a signature line that appears only in one class.

9. Generative vs Discriminative: Naive Bayes and Logistic Regression

For binary text classification with multinomial features, Naive Bayes and logistic regression produce the same form of model: a linear function of word counts passed through a sigmoid. They differ in how the weights are chosen. Naive Bayes sets each weight independently from class-conditional counts; logistic regression fits all weights jointly to maximize the conditional likelihood P(y | x), so correlated words share credit instead of double counting. Ng and Jordan (2001) analysed this pair and showed that the generative model can approach its (higher) asymptotic error with far fewer examples, while the discriminative model usually wins once data is plentiful. The diagram is a schematic of that pattern, not a measurement.

Practical consequences: with tens or hundreds of labeled examples, or when the model must be retrained every few minutes on a stream, Naive Bayes is a strong choice. With thousands of examples, try logistic regression or a linear SVM on the same features; the difference is usually measurable. A well-known hybrid, NB-SVM (Wang and Manning, 2012), scales features by Naive Bayes log-count ratios before fitting a linear classifier, and is a strong baseline for sentiment and topic tasks. For modern pipelines, see Text Classification.

10. Further Reading

• Andrew Ng and Michael Jordan, "On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes" (NIPS 2001).
• Pedro Domingos and Michael Pazzani, "On the Optimality of the Simple Bayesian Classifier under Zero-One Loss" (1997).
• Andrew McCallum and Kamal Nigam, "A Comparison of Event Models for Naive Bayes Text Classification" (1998).
• Rennie, Shih, Teevan and Karger, "Tackling the Poor Assumptions of Naive Bayes Text Classifiers" (2003), the origin of Complement NB.
• Sida Wang and Christopher Manning, "Baselines and Bigrams: Simple, Good Sentiment and Topic Classification" (2012).

11. Related Topics

• Classification — family overview.
• Logistic Regression — the discriminative counterpart.
• Support Vector Machines — the other classic text classifier.
• Supervised Learning — the parent paradigm.
• scikit-learn — the sklearn.naive_bayes module.