Naive Bayes
Start with a prior guess, update it when you see "free" — that's Bayesian thinking in a nutshell
01 Core Concept (Plain English)
You receive an email. Before opening it, you can already guess there's a 30% chance it's spam — that's the prior probability, based on historical experience.
You open it and the first line says "Free claim." You immediately update: spam probability jumps to 90% — that's the posterior probability. Bayes' theorem is exactly this update.
"Naive" refers to a simplifying assumption: each word's occurrence is independent of all others. In reality "free" and "prize" often appear together — they're not independent. But this naive assumption makes computation trivially simple, and works surprisingly well in practice.
Bayes' Theorem
Before reading the email — what fraction of emails are spam historically? Say 40%.
If this were spam, how likely would these words appear? Naive assumption: multiply word probabilities independently.
Combining both, what's the probability this email is spam? Predict the class with the highest posterior.
Step 1: Bayes' theorem — how seeing "free" updates the probability
Step 2: Multiple words — the naive independence assumption
With multiple words, multiply the individual likelihoods (naive: assume independence):
Step 3: Why use log-probabilities
More words → smaller products → floating-point underflow to zero. Use log-sum instead of product:
Step 4: Full classifier test
Learn word frequencies from training data and output spam probability for test emails:
Laplace smoothing: words unseen in training would get probability 0, collapsing the whole product to 0. Fix: add 1 to every word count (α=1) and add vocab size V to the denominator, giving every word a small non-zero probability.
02 Code
Edit the tests array (space-tokenized text) and observe how the spam probability changes in real time.
03 Deep Dive
Why does "naive" work?
The conditional independence assumption almost never holds ("free" and "prize" are correlated), yet Naive Bayes works well because:
- It only needs to predict the correct class, not accurate probabilities. As long as the spam score is higher than the ham score, it's correct — even if both numbers are off.
- With little data, the independence assumption acts as strong regularization against overfitting.
- With more features, independence errors tend to cancel out (law of large numbers effect).
Multinomial vs Bernoulli vs Gaussian
Multinomial NB
Models word counts. Best for text classification. This page implements this variant.
Bernoulli NB
Models word presence/absence (0/1), ignores frequency. Works well for short texts.
Gaussian NB
Uses Gaussian distribution for continuous features. Suitable for numerical data (height, weight, etc.).
Naive Bayes vs Logistic Regression
Prefer Naive Bayes
Very little training data (works with dozens of examples), need online updates as new data arrives, need a fast baseline model.
Prefer Logistic Regression
Features are highly correlated, sufficient data available, need well-calibrated probability outputs.