Autoencoder

Compress data to its smallest possible form, then reconstruct it — forcing the network to learn what information truly matters

01 Core Concept (Plain English)

Imagine summarising a painting in one sentence, then asking someone to repaint it from that sentence alone. The sentence is the "latent code" — it must capture the most important information, or the reconstruction will fail.

The encoder compresses: input → bottleneck (low-dim). The decoder reconstructs: bottleneck → output. Training minimises the reconstruction loss so the output matches the input as closely as possible.

A narrower bottleneck forces more aggressive compression, making the network learn only the most essential features. No labels needed — autoencoders are unsupervised.

Architecture

Input x → [Encoder] → Latent z → [Decoder] → Reconstruction x̂
Training goal: minimise MSE(x, x̂) = ‖x − x̂‖²
1
Encoder

Compresses high-dimensional input (e.g. 784 pixels) down to a low-dimensional bottleneck (e.g. 32 dims), extracting key features.

2
Bottleneck (Latent Space)

The narrowest hidden layer — stores only the information most critical for reconstruction. Smaller dimension = more aggressive compression.

3
Decoder

Reconstructs from the latent vector back to the original dimension. The goal is for the output to match the input as closely as possible.

Step 1: Encoder-decoder structure and information loss

Step 2: Reconstruction loss (MSE) vs bottleneck size

The narrower the bottleneck, the less information retained, the higher the reconstruction error:

Step 3: Latent space visualization

A well-trained encoder clusters same-class data together in latent space, revealing semantic structure:

Step 4: Hand-written autoencoder training (gradient descent)

Implement a linear autoencoder from scratch and watch the loss decrease over epochs:

Linear autoencoder ≈ PCA: When both encoder and decoder are linear, the learned directions are equivalent to PCA principal components. Nonlinear activations let the autoencoder capture more complex structure beyond linear subspaces.

02 Code

TF.js nonlinear autoencoder. Try changing LATENT to see how bottleneck size affects reconstruction quality.

03 Deep Dive

Autoencoder variants

Denoising AE (DAE)

Input is corrupted with noise; target is still the clean data. Forces the model to learn robust features rather than memorising noise.

Variational AE (VAE)

Latent space follows a Gaussian distribution, enabling sampling to generate new data. A generative model, not just a compression tool.

Sparse AE

L1 regularisation on activations forces only a few neurons to fire at a time — analogous to sparse coding in the brain.

Autoencoder vs PCA vs VAE

PCA

Linear, globally optimal, no training needed, interpretable. Cannot capture nonlinear structure.

Autoencoder

Nonlinear, learns complex manifolds, but latent space is not interpretable and not continuous (can't sample freely).

VAE

Continuous, probabilistic latent space enables generation. More complex training (ELBO objective).

Applications

  • Anomaly detection: Normal data has low reconstruction error; anomalies have high error — use reconstruction loss as an anomaly score.
  • Denoising: DAEs remove random noise from images or signals.
  • Pre-training (historical): Early deep learning used layer-wise autoencoder pre-training before fine-tuning. Now superseded by random init + BatchNorm.
  • Data compression: Learn domain-specific compression schemes that outperform general-purpose codecs.