❮ Previous

Autoencoders

  • This lesson introduces autoencoders and explains how they learn efficient representations of data using neural networks.

  • Encoder & Decoder

    Autoencoder has two main parts:

    Input → Encoder → Latent Space → Decoder → Reconstructed Output

    Encoder

    • Compresses input into lower dimension

    • Extracts important features

    • Output is called Latent Vector (Bottleneck)

    Example:

    Input image: 784 features (28×28)
    Encoder → 32 features

    Decoder

    • Reconstructs original input from compressed vector

    • Expands latent representation back to original size

    Goal:

    Output≈Input\text{Output} \approx \text{Input}Output≈Input


    Dimensionality Reduction

    Autoencoders reduce dimensions similar to PCA.

    Example:

    Original features: 100
    Compressed features: 10

    It keeps most important information.

    Why Better than PCA?

    PCA

    Autoencoder

    Linear

    Non-linear

    Fixed transformation

    Learns complex patterns

    No deep layers

    Can be deep

    Autoencoders can capture non-linear relationships.


    Reconstruction Loss

    Autoencoder tries to minimize difference between:

    Original Input (X)
     Reconstructed Output (X̂)

    Loss Function:

    For numeric data:

    Loss=MSE=(X−X^)2Loss = MSE = (X - \hat{X})^2Loss=MSE=(X−X^)2

    For images:

    • Mean Squared Error

    • Binary Crossentropy

    Lower reconstruction loss = Better compression.


    Simple Autoencoder Architecture

    Example:

    Input: 784
     Encoder:

    • Dense(128)

    • Dense(64)

    • Dense(32) ← Bottleneck

    Decoder:

    • Dense(64)

    • Dense(128)

    • Dense(784)

    Autoencoder Code Example (MNIST)

Autoencoder Example in Python using TensorFlow Keras

This Python example demonstrates how to build and train an Autoencoder using TensorFlow Keras for dimensionality reduction and reconstruction. The code loads and normalizes the MNIST dataset, flattens the images, and creates a neural network with an encoder (compressing to a 32-dimensional bottleneck) and a decoder (reconstructing the original input). The model is compiled with the Adam optimizer and MSE loss, and trained to reconstruct the input images. 

import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow.keras.datasets import mnist

# Load dataset
(x_train, _), (x_test, _) = mnist.load_data()

# Normalize
x_train = x_train / 255.0
x_test = x_test / 255.0

# Flatten
x_train = x_train.reshape(-1, 784)
x_test = x_test.reshape(-1, 784)

# Build Autoencoder
model = models.Sequential([
    layers.Dense(128, activation='relu', input_shape=(784,)),
    layers.Dense(32, activation='relu'),   # Bottleneck
    layers.Dense(128, activation='relu'),
    layers.Dense(784, activation='sigmoid')
])

model.compile(optimizer='adam', loss='mse')

model.fit(x_train, x_train,
          epochs=5,
          batch_size=256,
          validation_data=(x_test, x_test))

  • Notice:
    Target = Input
    Because goal is reconstruction.


    Applications of Autoencoders

    1. Dimensionality Reduction

    Feature extraction before classification.

    2. Image Denoising

    Remove noise from images.

    Input: Noisy Image
    Output: Clean Image

    3. Anomaly Detection

    Train on normal data only.

    If reconstruction error is high → anomaly.

    Used in:

    • Fraud detection

    • Network security

    • Manufacturing defect detection

    4. Image Compression

    Reduce storage size.

    5. Pretraining Deep Networks

    Before modern techniques, used for unsupervised pretraining.

    Types of Autoencoders

    Type

    Purpose

    Vanilla Autoencoder

    Basic compression

    Sparse Autoencoder

    Sparse representation

    Denoising Autoencoder

    Noise removal

    Convolutional Autoencoder

    Image-based

    Variational Autoencoder (VAE)

    Generative modeling

❮ Previous