Computer Vision is a field of artificial intelligence that trains computers to interpret and understand the visual world. Deep learning has revolutionized this field, moving from handcrafted feature extractors to architectures that can learn hierarchical representations of visual data automatically. This document covers the foundational and advanced neural network architectures that power modern computer vision.
Table of Contents
- Convolutional Neural Networks (CNNs): The Foundation
- Advanced CNN Architectures: Building Deeper and More Efficient Models
- Vision Transformers (ViT): A New Paradigm
- Specialized Vision Architectures
1. Convolutional Neural Networks (CNNs): The Foundation
Convolutional Neural Networks (CNNs or ConvNets) are the cornerstone of modern computer vision. Inspired by the human visual cortex, they are specifically designed to process pixel data and are exceptional at tasks like image classification, object detection, and image segmentation.
Core Components and Mathematical Principles
CNNs introduce two key concepts that distinguish them from standard Multi-Layer Perceptrons:
Local Receptive Fields & Parameter Sharing: Instead of connecting every neuron to every neuron in the previous layer, a CNN uses filters (or kernels). Each filter is a small matrix of weights that slides (convolves) over the input image. This filter is responsible for detecting a specific feature (like an edge or a texture). The same filter is used across the entire image (parameter sharing), making the network efficient and allowing it to detect features regardless of their position (translation invariance).
Pooling Layers: After a convolutional layer, a pooling layer (typically Max Pooling) is often used to downsample the feature map. It reduces the spatial dimensions of the data, which decreases the number of parameters and computational complexity, and also helps to make the feature detection more robust to small translations.
A typical CNN architecture consists of a stack of convolutional layers and pooling layers, which act as feature extractors, followed by one or more fully connected layers for classification.
Key Historical Architectures
- LeNet-5 (1998): One of the earliest successful CNNs, used for handwritten digit recognition.
- AlexNet (2012): A deep CNN that won the ImageNet competition, proving the power of deep learning for image tasks. It introduced the use of the ReLU activation function and dropout in CNNs.
- VGGNet (2014): Showed that a simple architecture with very small (3x3) convolutional filters, stacked to a great depth (16-19 layers), could achieve excellent performance.
2. Advanced CNN Architectures
To train even deeper networks and improve efficiency, several advanced architectural innovations have been developed.
ResNet (Residual Network)
- Problem: As networks get deeper, they become harder to train due to the vanishing gradient problem.
- Innovation: ResNet introduced the "identity shortcut connection" or skip connection. This allows the gradient to be directly backpropagated to earlier layers, bypassing some layers in between. This makes it possible to train networks that are hundreds or even thousands of layers deep. ResNet is one of the most influential architectures in deep learning.
ConvNeXt
ConvNeXt is a modern, pure CNN that was designed by "modernizing" a standard ResNet with architectural principles borrowed from Vision Transformers. By systematically adapting components like using depthwise separable convolutions and different activation functions, ConvNeXt achieves the performance of Transformers while maintaining the simplicity and efficiency of CNNs.
Capsule Networks (CapsNets)
Capsule Networks aim to overcome a key limitation of CNNs: their lack of understanding of hierarchical relationships between object parts (e.g., a face has a nose and two eyes in a specific spatial relationship).
- Innovation: Instead of individual neurons, CapsNets use capsules, which are groups of neurons that output a vector. This vector can represent various properties of a feature, such as its pose, texture, and position. A "dynamic routing" mechanism is used to model the part-to-whole relationships, making the network more robust to changes in viewpoint.
3. Vision Transformers (ViT): A New Paradigm
The Vision Transformer (ViT) represents a significant shift in computer vision, applying the highly successful Transformer architecture, originally designed for Natural Language Processing (NLP), directly to images.
How it Works
- Image Patching: The ViT model splits an input image into a sequence of fixed-size, non-overlapping patches (e.g., 16x16 pixels).
- Patch Embedding: Each patch is flattened into a vector and linearly projected into an "embedding".
- Positional Encoding: To retain spatial information, positional encodings are added to the patch embeddings.
- Transformer Encoder: The resulting sequence of vectors is fed into a standard Transformer encoder. The encoder uses the self-attention mechanism to weigh the importance of different patches relative to each other, allowing it to model global relationships across the entire image from the very first layer.
- Classification Head: A final classification head (a simple MLP) is attached to the output to make the final prediction.
Significance
ViT demonstrated that a pure Transformer architecture, with no convolutions, can achieve state-of-the-art results on image classification, particularly when pre-trained on very large datasets. This has opened up a new direction for research in computer vision.
4. Specialized Vision Architectures
U-Net: For Image Segmentation
The U-Net architecture is a specialized CNN designed for biomedical image segmentation, a task that requires precise localization (predicting a class for every pixel).
How it Works
U-Net's architecture is symmetric and U-shaped:
- Contracting Path (Encoder): A standard CNN stack that downsamples the image and captures context, learning the "what" of the image.
- Expansive Path (Decoder): This path uses transposed convolutions to upsample the feature maps back to the original image resolution, localizing the features and learning the "where".
- Skip Connections: The key innovation of U-Net is the use of skip connections that concatenate feature maps from the encoder with the corresponding feature maps in the decoder. This combines the high-level semantic information from the decoder with the low-level, high-resolution feature information from the encoder, allowing for very precise segmentation.
The U-Net design has become the de-facto standard for many image segmentation tasks and is also the core network used in modern Diffusion Models.
Object Detection Models
Object detection is a task that involves both classifying and locating objects within an image using bounding boxes.
Key Architectures
Two-Stage Detectors (e.g., Faster R-CNN):
- Region Proposal: A Region Proposal Network (RPN) first identifies a set of candidate regions in the image that might contain an object.
- Classification: These proposed regions are then passed to a second network that classifies the object and refines the bounding box. These models are generally very accurate but can be slower.
- Mask R-CNN: An extension of Faster R-CNN that adds a parallel branch for predicting a pixel-level segmentation mask for each object, enabling instance segmentation.
Single-Stage Detectors (e.g., YOLO - You Only Look Once): These models treat object detection as a single regression problem. They look at the image only once and directly predict a set of bounding boxes and class probabilities for those boxes. This makes them extremely fast and suitable for real-time applications.
Neural Radiance Fields (NeRF): For 3D Scene Synthesis
Neural Radiance Fields (NeRF) are a groundbreaking method for creating novel, photorealistic 3D views of a scene from a sparse set of 2D images.
How it Works
A NeRF model represents a 3D scene as a continuous function, implemented as a simple MLP. This network takes a 5D coordinate (3D location (x, y, z) and 2D viewing direction (θ, φ)) as input and outputs a volume density and an RGB color.
To render an image from a new viewpoint, virtual camera rays are cast through the scene. The network is queried at many points along each ray to get the color and density. These values are then integrated using classical volume rendering techniques to compute the final color of the pixel. The result is a stunningly realistic 3D reconstruction.