Guus Bouwens
← Writing & notes

FROM THE NOTEBOOK

Deep Learning Algorithms 3: Sequence Models

Sequential data, where the order of elements is crucial, is a common data type in many domains, including natural language, time series, and audio. This document covers the major deep learning architectures designed specifically to model and understand the temporal dependencies in such data.

Table of Contents

  1. Recurrent Neural Networks (RNNs): The Classic Approach
  2. Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU)
  3. Temporal Convolutional Networks (TCNs): A Convolutional Approach
  4. State Space Models (SSMs): The New Wave

1. Recurrent Neural Networks (RNNs): The Classic Approach

Recurrent Neural Networks (RNNs) were the first architecture specifically designed to handle sequential data. Unlike feedforward networks, RNNs have a "memory" that allows them to persist information across time steps.

How it Works: The Recurrent Loop

An RNN processes a sequence element by element. At each time step t, the network takes the input x_t and the hidden state h_{t-1} from the previous time step to produce an output y_t and a new hidden state h_t. h_t = f(W_hh * h_{t-1} + W_xh * x_t + b_h)

This recurrent loop allows the hidden state to act as a memory, carrying a summary of the past sequence. The same set of weights (W_hh, W_xh) is used at every time step, making the network efficient.

The Vanishing and Exploding Gradient Problem

When training RNNs using Backpropagation Through Time (BPTT), the gradients have to flow backward through the recurrent connections. For long sequences, these gradients can either shrink exponentially until they become zero (vanishing gradients) or grow exponentially until they overflow (exploding gradients).

The vanishing gradient problem severely limited the practical application of simple RNNs and led to the development of more sophisticated architectures.

2. Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU)

LSTMs and GRUs are advanced types of RNNs specifically designed to solve the vanishing gradient problem and effectively learn long-term dependencies. They achieve this through a mechanism of gates.

Long Short-Term Memory (LSTM)

An LSTM introduces a separate cell state C_t in addition to the hidden state h_t. The cell state acts as an information highway, allowing information to flow down the sequence with minimal modification. This flow is controlled by three sophisticated gates:

  1. Forget Gate: Decides what information from the previous cell state C_{t-1} should be thrown away.
  2. Input Gate: Decides what new information from the current input x_t and previous hidden state h_{t-1} should be stored in the current cell state C_t.
  3. Output Gate: Decides what part of the cell state C_t should be output as the new hidden state h_t.

Each gate is a small neural network with a sigmoid activation function, which outputs a value between 0 (let nothing through) and 1 (let everything through). This gating mechanism allows the LSTM to selectively remember or forget information over very long sequences.

Gated Recurrent Unit (GRU)

A GRU is a simplified version of the LSTM that often achieves comparable performance. It combines the forget and input gates into a single update gate and also merges the cell state and hidden state. It has two gates:

  1. Update Gate: Determines how much of the previous hidden state to keep and how much of the new candidate state to incorporate.
  2. Reset Gate: Determines how to combine the new input with the previous hidden state to create a new candidate state.

Because of its simpler architecture, a GRU has fewer parameters than an LSTM and can be slightly faster to train.

3. Temporal Convolutional Networks (TCNs): A Convolutional Approach

Temporal Convolutional Networks (TCNs) offer a powerful alternative to RNNs for sequence modeling, using a convolutional architecture instead of a recurrent one.

Core Components

  1. Causal Convolutions: TCNs use 1D convolutional layers, but with a crucial modification: the convolutions are causal. This means that the output at time step t can only depend on inputs from time t and earlier. This ensures that the model cannot look into the future, preserving the temporal ordering of the data.
  2. Dilated Convolutions: To capture long-range dependencies, TCNs use dilated convolutions. This involves applying the filter over a wider area by skipping input values with a certain step (the dilation rate). By stacking layers with exponentially increasing dilation factors, the network can have an extremely large receptive field, allowing it to model very long-term dependencies efficiently.

Advantages over RNNs

4. State Space Models (SSMs): The New Wave

State Space Models (SSMs) are a class of models that have recently emerged as a highly efficient and powerful alternative to both RNNs and Transformers for long-sequence modeling.

How it Works

SSMs map an input sequence x(t) to a latent state h(t) through a system of differential equations, from which an output y(t) is generated. The core innovation of modern SSMs like Mamba is a selection mechanism.

Mamba

Mamba is a selective State Space Model. Its key features are:

Vision Mamba (Vim)

Vision Mamba adapts the Mamba architecture for computer vision. It treats an image as a sequence of flattened patches and uses a bidirectional Mamba architecture to model relationships between them. Vim achieves performance competitive with Vision Transformers while being significantly more efficient, especially for high-resolution images.

All writing