Deep Learning Algorithms 1: Fundamentals

This document covers the core concepts that form the foundation of all artificial neural networks. We will build from the simplest computational unit, the Perceptron, to the mechanics of training complex, multi-layered networks.

Table of Contents

1. The Perceptron: The Simplest Neural Network
2. Multi-Layer Perceptron (MLP): Introducing Depth
3. The Training Engine: Feedforward and Backpropagation
4. Activation Functions: Introducing Non-Linearity
5. Loss Functions: Measuring Error
6. Optimization: Learning from Error

1. The Perceptron: The Simplest Neural Network

The Perceptron is the original and simplest form of a neural network, conceptualized as a single neuron. It serves as a linear classifier for binary classification tasks.

How it works

1. Inputs and Weights: It takes a set of binary inputs (x₁, x₂, ...) and multiplies them by a corresponding set of weights (w₁, w₂, ...). These weights signify the importance of each input.
2. Weighted Sum: It computes the weighted sum of the inputs. A bias term is often added to this sum.
3. Activation: It applies a simple step function to the sum. If the sum exceeds a certain threshold, the Perceptron "fires" and outputs 1; otherwise, it outputs 0.

Limitations

A single-layer Perceptron is only capable of learning linearly separable patterns. It famously cannot solve the XOR problem, a limitation that highlighted the need for more complex, multi-layered networks.

2. Multi-Layer Perceptron (MLP): Introducing Depth

A Multi-Layer Perceptron (MLP) is a class of feedforward neural network that overcomes the limitations of the single-layer Perceptron. It consists of an input layer, one or more hidden layers, and an output layer.

Key Features

• Hidden Layers: These layers between the input and output allow the network to learn complex, non-linear relationships in the data. The presence of hidden layers enables an MLP to solve problems like XOR.
• Non-linear Activation Functions: Unlike the simple step function in a Perceptron, each neuron in an MLP's hidden layers uses a non-linear activation function (e.g., Sigmoid, Tanh, ReLU). This is crucial for the network to learn hierarchical features and model complex patterns.

3. The Training Engine: Feedforward and Backpropagation

Feedforward and Backpropagation are the two fundamental processes that allow an MLP to learn from data.

Feedforward Propagation

This is the process of passing input data through the network to get a prediction.

1. The input data is fed to the input layer.
2. Each layer of neurons receives inputs from the previous layer, computes a weighted sum, and applies its activation function.
3. This process continues layer by layer until the output layer produces a final prediction.

Backpropagation (Backward Propagation of Errors)

This is the algorithm used to train the network by adjusting its weights.

1. Calculate Loss: A loss function is used to calculate the error, which is the difference between the network's prediction and the actual target value.
2. Compute Gradients: Backpropagation uses the chain rule of calculus to efficiently compute the gradient (the partial derivative) of the loss function with respect to each weight and bias in the network. It starts from the output layer and works its way backward.
3. Update Weights: An optimizer uses these gradients to update the weights and biases in the direction that minimizes the loss. This is typically done via a method like Gradient Descent.

This two-step cycle of feedforward and backpropagation is repeated for many epochs (passes through the entire dataset) until the network's performance converges.

4. Activation Functions: Introducing Non-Linearity

Activation functions are a critical component of a neural network, applied to the output of each neuron. They introduce the non-linearity that allows the network to learn anything more complex than a simple linear transformation.

Common Activation Functions

• Sigmoid: σ(x) = 1 / (1 + e⁻ˣ). It squashes values to a range between 0 and 1. Historically important but less used in hidden layers today due to the vanishing gradient problem.
• Tanh (Hyperbolic Tangent): tanh(x). Similar to sigmoid but squashes values between -1 and 1. It is zero-centered, which can help in training.
• ReLU (Rectified Linear Unit): f(x) = max(0, x). It outputs the input directly if positive, otherwise, it outputs zero. It is the most widely used activation function due to its computational efficiency and ability to mitigate the vanishing gradient problem.
• Leaky ReLU: A variant of ReLU that allows a small, non-zero gradient for negative inputs (f(x) = max(0.01x, x)), preventing "dying ReLU" issues.
• Softmax: Used in the output layer of multi-class classification networks. It converts a vector of raw scores (logits) into a probability distribution, where each value is between 0 and 1 and the sum of all values is 1.

5. Loss Functions: Measuring Error

A loss function (or cost function) quantifies how well the model is performing. It calculates a single number representing the difference between the model's predictions and the ground truth labels. The goal of training is to minimize this value.

Common Loss Functions

• For Regression (predicting continuous values):
  • Mean Squared Error (MSE / L2 Loss): Calculates the average of the squared differences between predicted and actual values. It is sensitive to outliers.
  • Mean Absolute Error (MAE / L1 Loss): Calculates the average of the absolute differences. It is more robust to outliers than MSE.
• For Classification (predicting discrete labels):
  • Binary Cross-Entropy: Used for binary (two-class) classification. It measures the distance between two probability distributions (the predicted and the actual).
  • Categorical Cross-Entropy: The extension of binary cross-entropy for multi-class classification tasks.

6. Optimization: Learning from Error

Optimizers are the algorithms that implement the backpropagation process. They take the gradients calculated by backpropagation and use them to update the network's weights and biases in order to minimize the loss function.

Gradient Descent

The foundational optimization algorithm is Gradient Descent. It updates parameters by taking a step in the direction of the negative gradient of the loss function. new_weight = old_weight - learning_rate * gradient

Types of Gradient Descent

• Batch Gradient Descent: Calculates the gradient using the entire training dataset. It is accurate but very slow on large datasets.
• Stochastic Gradient Descent (SGD): Calculates the gradient and updates the weights for each individual training example. It is much faster but the updates are noisy, leading to a less stable convergence.
• Mini-Batch Gradient Descent: A compromise that is used in practice. It updates the weights based on a small batch of training examples. This balances the stability of batch GD with the efficiency of SGD.

Advanced Optimizers

While SGD is the basis for most training, more advanced optimizers have been developed to improve the speed and reliability of convergence.

• Momentum: Helps SGD accelerate in the correct direction and dampens oscillations by adding a fraction of the previous update vector to the current one.
• RMSprop: Adapts the learning rate for each parameter, dividing by a moving average of recent gradient magnitudes.
• Adam (Adaptive Moment Estimation): The most popular and often default optimizer. It combines the ideas of Momentum and RMSprop, maintaining both an adaptive learning rate for each parameter and a momentum term.