Guus Bouwens
← Writing & notes

FROM THE NOTEBOOK

Deep Learning for Computer Vision - Stanford CS231N

Stanford CS231N Deep Learning for Computer Vision I 2025 - YouTube

This course provides a definitive, end-to-end journey into the principles and practice of deep learning for computer vision. The course builds from the ground up, starting with the foundational challenge of image classification and the limitations of simple linear classifiers. It then constructs the core machinery of modern AI, detailing the architecture of neural networks, the mathematics of optimization (SGD, Adam), and the pivotal backpropagation algorithm that enables learning. The curriculum dedicates significant time to Convolutional Neural Networks (CNNs), tracing their evolution from foundational architectures like VGGNet to the revolutionary ResNet with its critical skip connections.
With these tools established, the course expands its scope to a wide array of core vision tasks, including object detection, semantic and instance segmentation, video understanding, and 3D vision, covering both classic and modern algorithmic approaches from R-CNN to Transformers and NeRFs. The latter half of the course pivots to the state-of-the-art, exploring the paradigm shift towards training on massive, unlabeled datasets through self-supervised learning (e.g., contrastive methods, masked autoencoders). This sets the stage for a deep dive into generative modeling, tracing the evolution from Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) to the dominant paradigm of Diffusion Models. The architectural focus then shifts to the attention mechanism and the Transformer, the highly parallelizable and scalable architecture that now underpins not only vision but the entire field of AI.
The final lectures synthesize these concepts into the modern paradigm of multimodal foundation models (e.g., CLIP, LLaVA), which integrate vision and language, and address the massive systems challenge of large-scale distributed training. The course culminates by exploring the application of these powerful models in the physical world through robot learning and concludes with a critical discussion on human-centered AI, examining the ethical implications and societal impact of the technology. Ultimately, CS231n equips students not just with the technical skills to build vision systems, but with a deep, historical, and principled understanding of the field's past, present, and future.

L1: Introduction

The course is introduced by Professor Fei-Fei Li, who will be co-teaching with Professor Ehsan Adeli and graduate student Zane.

Part 1: A Brief History of Computer Vision (Presented by Fei-Fei Li)
The history is framed as a long evolutionary and intellectual journey to understand and replicate sight.

  1. The Evolutionary Beginning (540 Million Years Ago): The lecture starts with the Cambrian Explosion, a period of rapid animal speciation. A leading theory attributes this explosion to the evolution of the first eyes. Vision transformed life from passive metabolism into an active, intelligent process of navigating an environment, finding food, and avoiding predators. Vision, she argues, is the primary driver behind the evolution of the nervous system and intelligence itself.
  2. Early Human Innovations: The lecture touches on early human attempts to understand and manipulate images, from ancient Greek and Chinese philosophers studying pinhole cameras to Leonardo da Vinci's work on the camera obscura.
  3. The Neuroscience Foundation (1950s): The modern scientific study of vision began with the seminal work of Hubel and Wiesel, who won a Nobel Prize for their experiments on the visual cortex of cats. Their key discoveries were:
    • Neurons in the primary visual cortex respond to simple, oriented edges within specific receptive fields.
    • The visual pathway is hierarchical: neurons in deeper layers of the brain respond to increasingly complex patterns by combining the inputs from simpler ones. This hierarchical structure would become a core inspiration for neural network architectures.
  4. The Birth of Computer Vision as a Field (1960s-1970s):
    • 1963: Larry Roberts' PhD thesis on understanding 3D shapes from 2D images is considered the first in the field.
    • 1966: An MIT summer project famously (and optimistically) aimed to "solve vision."
    • 1970s: David Marr's influential book proposed a systematic pipeline for vision, moving from a "primal sketch" (edges) to a "2.5D sketch" (surfaces and depth) and finally to a full 3D model. This highlighted the core, ill-posed problem of vision: recovering a 3D world from 2D projections.
  5. The AI Winter and Continued Progress (1980s-1990s): While the broader AI field entered a "winter" due to unfulfilled promises, foundational research continued. Cognitive science revealed the importance of context in object recognition and the incredible speed of human visual processing (categorization within 150ms). This research pointed towards a crucial North Star for the field: object recognition in natural scenes. Early successes included face detection algorithms that were eventually integrated into consumer digital cameras.
  6. The Deep Learning Thread (Parallel History):
    • 1980s-1990s: While computer vision focused on features and geometry, a parallel track of research was developing neural networks. Key milestones included Fukushima's Neocognitron (a hand-designed, hierarchical network inspired by the visual cortex) and the invention of the backpropagation learning algorithm by Rumelhart and Hinton, which was a watershed moment. Yann LeCun applied this to create a convolutional neural network (CNN) that could recognize handwritten digits for banks and postal services.
  7. The Inflection Point (2000s-2012): The field realized that a major bottleneck was the lack of large-scale data.
    • ImageNet: To address this, Professor Li's team created ImageNet, a massive dataset of millions of labeled images. They launched the ImageNet Large Scale Visual Recognition Challenge (ILSVRC).
    • 2012: The "AlexNet" moment. A deep convolutional neural network, created by Jeff Hinton and his students, dramatically outperformed all other approaches in the ILSVRC, cutting the error rate nearly in half. This event is widely considered the "Big Bang" of the modern deep learning revolution, demonstrating the power of combining deep neural networks, the backpropagation algorithm, large datasets, and powerful GPU hardware.
  8. The Modern Era (Post-2012): The AlexNet breakthrough triggered an explosion of progress. The field rapidly developed models for more complex tasks beyond classification, including object detection, image segmentation, video understanding, medical imaging, and generative AI (style transfer, DALL-E). The lecture concludes by acknowledging that while progress has been immense, vision is not "solved," and the field faces significant challenges, particularly regarding the societal impact of AI, such as inherited biases and ethical applications.

Part 2: Course Overview (Presented by Ehsan Adeli)
Professor Adeli outlines the structure and content of the course, which is divided into four main topics.

  1. Deep Learning Basics: The course begins with the fundamentals.
    • Starting with image classification as the core task.
    • Moving from simple linear classifiers to the need for non-linear models.
    • Covering foundational concepts like overfitting/underfitting, regularization, and optimization.
    • A deep dive into neural networks, how they are built, trained, and debugged.
  2. Perceiving and Understanding the Visual World: This section covers core computer vision tasks and the models used to solve them.
    • Tasks: Moving beyond classification to semantic segmentation (labeling every pixel), object detection (locating objects with bounding boxes), and instance segmentation (identifying individual object instances). It also covers tasks involving time, such as video classification and multimodal understanding.
    • Models: The focus will be on Convolutional Neural Networks (CNNs), but the course will also cover Recurrent Neural Networks (RNNs) and modern architectures like Transformers and attention.
  3. Generative and Interactive Visual Intelligence: This unit explores more advanced and recent topics.
    • Self-Supervised Learning: Learning representations from vast amounts of unlabeled data.
    • Generative Models: Creating new visual content, including style transfer and diffusion models. Students will implement a text-to-emoji generator.
    • Vision-Language Models: Connecting images and text for tasks like image captioning and retrieval.
    • 3D Vision and Embodied AI (robotics).
  4. Human-Centered Applications and Implications: The final part of the course will address the broader societal impact of the technology, building on the themes introduced by Professor Li. The course will conclude by looking ahead at the next topics: image classification and linear classifiers.

L2: Image Classification with Linear Classifiers

The lecture begins by recapping the course structure and sets the agenda: to formally define the image classification task and introduce two foundational data-driven approaches: the k-Nearest Neighbor (k-NN) algorithm and Linear Classifiers.
Part 1: The Image Classification Problem
The core task is to assign a single, predefined label (e.g., "cat," "dog") to an input image.

Part 2: The Data-Driven Approach
Instead of trying to hard-code a set of rules to identify objects, the modern approach is data-driven, following a three-step pipeline:

  1. Collect Data: Gather a large dataset of images with their corresponding labels.
  2. Train a Classifier: Use a machine learning algorithm to learn a model that maps images to labels from the training data.
  3. Evaluate: Test the trained model on a new, unseen set of images to assess its performance.

Part 3: The k-Nearest Neighbor (k-NN) Classifier
This section introduces the first, and simplest, data-driven classifier.

Part 4: Hyperparameter Tuning
This section covers the crucial process of selecting the best hyperparameters for a given problem.

Part 5: Linear Classifiers
The lecture then moves to the most important building block of modern deep learning: the linear classifier.

The lecture concludes by noting that the next step is to figure out the optimization process—how to efficiently find the weights W that minimize this loss function, which will be the topic of the next session.

L3: Regularization and Optimization

The lecture begins with a recap of the previous session, reinforcing the core concepts of image classification.

Part 1: Regularization
The first major topic is regularization, a technique to combat overfitting. The core idea is to intentionally make the model perform worse on the training data to help it generalize better to new, unseen test data.

Part 2: Optimization
With a loss function defined, the next step is optimization: the process of finding the set of weights W that minimizes the total loss.

Part 3: Challenges and Improvements in Optimization
This section covers the common problems with vanilla SGD and the advanced optimizers designed to solve them.

Part 4: Learning Rate Schedules
The lecture concludes by discussing learning rate schedules, the practice of changing the learning rate during training.

The lecture ends with a preview of the next topic: neural networks, which are built by stacking these linear classifiers together with non-linear activation functions in between.

L4: Neural Networks and Backpropagation

The lecture begins by positioning itself as a foundational session for the rest of the course. The topics of Neural Networks and Backpropagation are central to nearly all modern computer vision algorithms. The session starts with a recap of the previous lecture:

Part 1: Neural Networks
This section introduces the concept of multi-layer neural networks as a way to overcome the limitations of linear classifiers.

Part 2: Backpropagation and Computational Graphs
The core of the lecture is a detailed, step-by-step explanation of backpropagation, the algorithm used to efficiently compute the gradients needed for optimization.

Part 3: Backpropagation with Vectors and Matrices
The final section generalizes the concept from scalar values to the vectors and matrices used in real neural networks.

The lecture concludes by summarizing that backpropagation provides a modular and efficient way to compute gradients for any complex, differentiable function represented as a computational graph. This powerful algorithm is the engine that drives the learning process in virtually all deep neural networks, a topic that will be foundational for the upcoming lectures on convolutional neural networks.

L5: Image Classification with CNNs

The lecture begins with an introduction from a new instructor, Justin Johnson, and transitions into a recap of the first four lectures, framing them as the complete, foundational pipeline for any deep learning system.

  1. The Problem (Image Classification): Formulating a task in terms of tensor inputs (raw pixel values) and tensor outputs (class scores), using a linear classifier (f(x) = Wx + b) as the initial model.
  2. The Loss Function: Defining a way to measure how well a given set of weights W performs on a dataset. The lecture reviewed the softmax loss, a standard for classification.
  3. Optimization: The process of finding the optimal W that minimizes the loss. This is analogized to a blindfolded person navigating a "loss landscape." The core algorithm is gradient descent, with practical implementations like SGD with Momentum and Adam.
  4. From Linear Models to Neural Networks: The limitations of linear classifiers were highlighted—they can only learn a single "template" per class and can only solve linearly separable problems. This motivated the move to multi-layer neural networks, which stack linear layers with non-linear activation functions (like ReLU) in between.
  5. Backpropagation: The "magical" algorithm that makes training deep networks possible. It uses the chain rule on a computational graph to efficiently and automatically compute the gradients of the loss with respect to all model parameters, enabling optimization.

Part 1: Convolutional Neural Networks (CNNs) - A Better Architecture for Images
This section introduces CNNs as a specialized type of neural network designed to better handle the unique structure of images.

Part 2: The Convolution Layer
This is the core building block of a CNN and the main focus of the lecture.

Part 3: The Pooling Layer
The second key primitive for CNNs is the pooling layer.

The lecture ends by setting the stage for the next session, which will explore how these convolutional and pooling layers are stitched together to form powerful CNN architectures.

L6: CNN Architectures

The lecture begins by positioning itself at an inflection point in the course, moving from the fundamentals of deep learning to more specialized topics in computer vision. It starts with a comprehensive recap of the core deep learning pipeline covered in previous lectures:

  1. The Problem: Formulating tasks like image classification in terms of tensor inputs and outputs.
  2. The Model: Using a function, like a linear classifier, to map inputs to outputs.
  3. The Loss Function: A mathematical way to measure how "bad" a model's predictions are.
  4. Optimization: The process of finding the model parameters that minimize the loss, typically using gradient descent and optimizers like Adam.
  5. Neural Networks: Overcoming the limitations of linear models by stacking layers with non-linear activations.
  6. Backpropagation: The core algorithm that enables efficient gradient computation for these complex models.

Part 1: How to Build CNNs - Architectures and Layers
This section focuses on the components and design principles of Convolutional Neural Networks (CNNs).

Part 2: How to Train CNNs - Practical Steps and Tricks
This section shifts from architecture design to the practical process of training a model effectively.

  1. Data Pre-processing: The standard first step is to normalize the input images by subtracting the mean and dividing by the standard deviation of the entire dataset, calculated on a per-channel (R, G, B) basis.
  2. Data Augmentation: A critical regularization technique. During training, random transformations are applied to each image to create new, plausible training examples. This makes it harder for the model to overfit and improves its generalization. Common augmentations include:
    • Horizontal flipping
    • Random resized cropping
    • Color jittering (adjusting brightness, contrast, etc.)
  3. Transfer Learning: A cornerstone of modern computer vision practice, especially for smaller datasets.
    • Concept: Instead of training a model from scratch, start with a model that has been pre-trained on a massive dataset like ImageNet.
    • Methods:
      • Feature Extractor: Freeze the pre-trained convolutional layers and only train a new, small classifier head on top. This is effective for very small or similar datasets.
      • Fine-tuning: Initialize the entire network with the pre-trained weights but allow all layers to be updated during training on the new dataset, typically with a smaller learning rate. This is the most common and effective approach.
  4. Hyperparameter Tuning: The lecture concludes with a practical guide to finding good hyperparameters.
    • Sanity Check: The most important debugging step is to first try to overfit on a tiny subset of the data (e.g., one or a few examples). If the model can't achieve zero loss on this small sample, there is likely a bug in the code.
    • Search Strategy: Use random search over a defined range of hyperparameters rather than a grid search. Random search is more efficient at exploring the parameter space, especially when some hyperparameters are more important than others.
    • Monitoring: Track both training and validation accuracy/loss curves. A large gap between them indicates overfitting, which can be addressed with more data or stronger regularization. If both are still improving, the model can likely benefit from more training.

L7: Recurrent Neural Networks

The lecture begins with a few clarifications from the previous session:

The lecture then introduces its main topic: sequence modeling, focusing on Recurrent Neural Networks (RNNs) as a foundational architecture before the era of Transformers.
Part 1: Sequence Modeling and Recurrent Neural Networks (RNNs)
This section defines the task of sequence modeling and introduces the core concepts of RNNs.

Part 2: Training RNNs - Backpropagation Through Time (BPTT)
This section explains how RNNs are trained.

Part 3: Challenges of RNNs and the Rise of LSTMs
This section details the fundamental problems that made vanilla RNNs difficult to train and motivated more complex architectures.

Part 4: Applications and the Modern Resurgence of RNN-like Models

The lecture ends by setting the stage for the next session, which will cover the Transformer architecture itself.

L8: Attention and Transformers

The lecture begins by recapping the previous session on Recurrent Neural Networks (RNNs), which introduced architectures for processing variable-length sequences. This capability opened up new problem domains beyond the fixed-size inputs of CNNs, such as image captioning (one-to-many) and video classification (many-to-one).
Today's lecture will build on this foundation to introduce two pivotal concepts:

  1. Attention: A powerful neural network primitive that operates on sets of vectors.
  2. The Transformer: A revolutionary neural network architecture built around attention, which has become the dominant architecture for nearly all state-of-the-art deep learning applications today.

The lecture follows the historical development of these ideas, starting with the problems in RNNs that motivated the invention of attention.
Part 1: The Bottleneck in Sequence-to-Sequence RNNs
The lecture uses machine translation (e.g., English to Italian) as a motivating example for sequence-to-sequence (seq2seq) models.

Part 2: The Attention Mechanism
Attention was developed to solve this bottleneck. The core idea is to allow the decoder to look back at the entire input sequence at every step of its generation process.

Part 3: Generalizing Attention as a Standalone Operator
The lecture then abstracts the attention mechanism away from the RNN, defining it as a general and powerful computational primitive.

Part 4: The Transformer Architecture
The Transformer is a neural network architecture that completely discards recurrence and convolution, relying almost entirely on self-attention.

The lecture concludes by positioning the Transformer as the dominant, state-of-the-art architecture used across virtually all of deep learning today, thanks to its scalability, power, and efficiency. The next lecture will explore how to apply these powerful architectures to new computer vision tasks.

L9: Object Detention, Image Segmentation, Visualizing

The lecture begins by positioning itself as the bridge between the foundational concepts of neural networks and their practical application in building and training state-of-the-art Convolutional Neural Networks (CNNs). It starts with a recap of key topics from previous lectures:

Part 1: Building CNNs - Layers and Architectures
This section details the specific layers used to construct CNNs and explores classic architectural patterns.

Part 2: Training CNNs - Practical Tricks of the Trade
This section moves from architectural design to the practical process of training a model effectively.

Part 3: From Classification to Detection and Segmentation
The lecture expands from the core task of image classification to more complex Computer Vision problems.

Part 4: Visualization and Understanding
The final section covers techniques for interpreting what a trained CNN has learned.

The lecture concludes by setting the stage for the next session, which will cover video understanding.

L10: Video Understanding

The lecture, delivered by guest speaker Dr. Ruohan Gao, begins by transitioning from the course's focus on 2D image understanding to the more complex domain of video understanding. While students are now familiar with tasks like image classification, object detection, and segmentation on static images, video introduces a crucial new dimension: time.

Part 1: Architectures for Video Understanding
This section explores different architectural approaches for extracting features from video clips.

  1. Single-Frame CNN: The simplest baseline. Treat each frame of a video as an independent image, run a standard 2D CNN (image classifier) on each, and then average the predictions. This often works surprisingly well, especially for actions with little temporal variation.
  2. Fusion Strategies: These methods aim to combine information across frames.
    • Late Fusion: Run a 2D CNN on each frame independently to extract feature vectors. Then, at a very "late" stage, aggregate these feature vectors across time, either by concatenating them or, more efficiently, by using a temporal pooling operation (e.g., max or average pooling). The main drawback is that important low-level motion cues may be lost by the time the features are fused.
    • Early Fusion: The opposite approach. Stack the input frames along the channel dimension at the very "early" stage, and use a single, wide 2D convolution in the first layer to collapse all temporal information at once. The rest of the network is a standard 2D CNN. This is computationally cheap but may be too aggressive in collapsing temporal information.
    • Slow Fusion (3D CNNs): The most principled approach. It replaces 2D convolutions and pooling with their 3D counterparts. The filters are now 3D kernels (e.g., 3x3x3) that slide through the video volume in both space and time. This allows the network to gradually and hierarchically learn spatio-temporal features. A key advantage is that it learns temporal shift-invariant features, meaning it can recognize a motion pattern regardless of when it occurs in the clip.
  3. Two-Stream Networks (Appearance + Motion): This approach is motivated by the observation that human perception is highly sensitive to motion.
    • The Idea: Decompose the video into two separate streams:
      1. Appearance Stream: A standard 2D CNN that processes individual RGB frames.
      2. Motion Stream: A separate 2D CNN that operates on optical flow fields, which explicitly represent the pixel-wise motion between adjacent frames.
    • The predictions from both streams are fused at the end. This method often outperforms single-stream models because optical flow provides a very strong, explicit signal for action recognition.

Part 2: Modeling Long-Term Temporal Structure
While CNNs are good at processing short clips, videos often have long-range dependencies.

Part 3: Beyond Basic Classification - Multimodality and Efficiency
The lecture concludes by looking at more advanced topics and future directions in video understanding.

L11: Large Scale Distributed Training

The lecture begins by establishing that large-scale distributed training is no longer a niche topic but the standard for building state-of-the-art neural networks. While models were once trained on single GPUs, the norm today is to use thousands, or even tens of thousands, of devices concurrently. The lecture uses Llama 3 405B as a running example, not because it's necessarily the best model, but because Meta shared a rare and detailed technical report on its training process, providing a valuable window into how frontier models are built. This is contrasted with the secrecy of other top labs like OpenAI, whose GPT-4 paper famously disclosed no details about its architecture, hardware, or training methods.
Part 1: GPU Hardware and the Data Center as a Computer
This section provides a deep dive into the hardware that powers large-scale training, moving from a single GPU to the entire data center.

Part 2: The Five Degrees of Parallelism
This section details the five main algorithmic strategies for splitting up the computation of a single neural network across thousands of GPUs. The goal is to keep all devices busy by overlapping communication with computation.

  1. Data Parallelism (DP):
    • Concept: The simplest form. Each GPU holds a full copy of the model but processes a different slice of the data batch. After the backward pass, gradients are synchronized across all GPUs using an all_reduce operation.
    • Problem (Memory): It's memory-inefficient because every GPU must store the entire model, its gradients, and the bulky optimizer states.
    • Solution (FSDP - Fully Sharded Data Parallelism): To solve the memory issue, FSDP shards the model parameters, gradients, and optimizer states across the GPUs. During the forward/backward pass, each GPU fetches the parameters for the current layer on-demand from its peers, computes, and then discards them. This is made efficient by prefetching the next layer's weights while computing the current one.
  2. Pipeline Parallelism:
    • Concept: The model is split by layers (depth-wise). GPU 0 computes layers 1-8, GPU 1 computes layers 9-16, and so on, passing activations sequentially.
    • Problem (The "Bubble"): This is inherently inefficient, creating large bubbles of idle time where most GPUs are waiting.
    • Solution: The bubble can be shrunk by splitting the batch into smaller "microbatches" and pipelining them through the stages concurrently.
  3. Tensor Parallelism:
    • Concept: The model is split within layers by sharding the large weight matrices (width-wise). Each GPU computes a portion of a single matrix multiplication, and the results are synchronized.
    • Use Case: It's extremely communication-intensive and is therefore almost exclusively used for the fastest, intra-node connections (e.g., across 8 GPUs in a single server).
  4. Context Parallelism (or Sequence Parallelism):
    • Concept: Used for very long sequences. The sequence itself is sharded across GPUs. This is straightforward for most of the Transformer (MLPs, LayerNorm), but tricky for the attention mechanism, which requires all-to-all communication.
    • Solutions: Specialized algorithms like Ring Attention or Ulysses Attention are used to parallelize the attention computation efficiently.
  5. Hybrid Sharded Data Parallelism (HSDP): A 2D approach that combines FSDP and DP, designed to map efficiently onto the data center's network topology. GPUs are arranged in a grid; FSDP is used along one axis (e.g., within a fast server), and standard DP is used along the other axis (e.g., across slower racks).

Part 3: Practical Scaling and the Guiding Light of MFU

The lecture concludes by emphasizing that modern large-scale training is a complex, multi-dimensional optimization problem that requires a deep understanding of both the algorithms and the underlying hardware to solve effectively.

L12: Self-Supervised Learning

The lecture begins by recapping the journey so far, emphasizing that while supervised learning on labeled data produces meaningful feature representations, the need for large-scale manual labeling is a major bottleneck. This sets the stage for the central question of the lecture: Can we train neural networks to learn good visual features without requiring huge, manually labeled datasets?
Part 1: The Concept of Self-Supervised Learning (SSL)
The lecture introduces Self-Supervised Learning as the paradigm that answers this question.

Part 2: SSL via Image Transformations (Pretext Tasks)
This section explores several classic pretext tasks based on applying transformations to images.

  1. Rotation Prediction:
    • Task: An image is randomly rotated by one of four angles (0°, 90°, 180°, 270°). The model's task is to predict which of the four rotations was applied.
    • Hypothesis: To know that a car is upside down (rotated 180°), the model must first learn what a car looks like right-side up. This forces it to learn about object orientation and structure.
  2. Relative Patch Prediction / Jigsaw Puzzles:
    • Task: An image is divided into a 3x3 grid of patches. The model is given the center patch and one other random patch and must predict the relative position of the second patch (e.g., "top-left," "bottom-right"). A more complex version requires the model to reassemble a shuffled set of all nine patches into the correct order.
    • Hypothesis: To solve the puzzle, the model must learn about the spatial relationships between object parts (e.g., an eye is typically above a mouth).
  3. Image Inpainting / Completion:
    • Task: A portion of the image is masked out (e.g., a black square is placed over it). The model must predict the content of the missing patch.
    • Hypothesis: To realistically fill in the missing region, the model needs to understand the context provided by the surrounding pixels.
  4. Image Colorization:
    • Task: The model is given a grayscale version of an image (the L channel in LAB color space) and must predict the corresponding color channels (the A and B channels).
    • Hypothesis: To correctly color an object (e.g., know that grass is green and the sky is blue), the model must first be able to recognize the object itself. The lecture also shows how this idea can be extended to video to learn to track objects over time.

Part 3: Masked Autoencoders (MAE) - A Modern Approach
The lecture then transitions to a state-of-the-art SSL method that builds upon the inpainting idea.

Part 4: Contrastive Learning
The final section introduces a different, and highly influential, family of SSL methods: contrastive learning.

The lecture concludes by noting that these self-supervised methods, particularly contrastive learning and masked autoencoders, have been instrumental in enabling the training of large-scale foundation models that power many of today's vision applications.

L13: Generative Models 1

The lecture begins by recapping the previous session on Self-Supervised Learning (SSL). The core idea of SSL is to train a model on a large, unlabeled dataset using a pretext task (e.g., rotation prediction, image inpainting) for which "labels" can be automatically generated. This pre-training phase learns a powerful feature encoder, which can then be transferred and fine-tuned on a smaller, labeled dataset for a specific downstream task. The lecture briefly revisits contrastive learning methods like SimCLR and MoCo, highlighting their goal of pulling similar (augmented) samples together and pushing dissimilar ones apart in the feature space.
Part 1: The Taxonomy of Generative Models
The main topic of the lecture is generative models. The session starts by establishing a clear taxonomy to navigate this complex field.

Part 2: Autoregressive Models
This section covers the first leaf on the taxonomy tree: autoregressive models, a type of tractable, explicit density model.

Part 3: Variational Autoencoders (VAEs)
The lecture then moves to the second branch of explicit models: VAEs, which provide an approximate density.

The lecture concludes by setting the stage for the next session, which will cover the implicit density models: GANs and Diffusion Models.

L14: Generative Models 2

The lecture begins by recapping the previous session on generative models, reinforcing the taxonomy that divides them into two main families. Last time, the focus was on explicit density models, which provide a way to directly compute or approximate the probability density p(x) of a data point x.

Part 1: Generative Adversarial Networks (GANs)
The lecture then moves to the first category of implicit density models: Generative Adversarial Networks (GANs). These models can sample from a learned distribution but cannot explicitly compute its density.

Part 2: Diffusion Models
The second half of the lecture covers diffusion models, the implicit density models that have largely superseded GANs as the state-of-the-art for image and video generation.

Part 3: Latent Diffusion Models - The Modern Pipeline
The lecture concludes by revealing that state-of-the-art models like Stable Diffusion are not standard diffusion models but Latent Diffusion Models, which combine all the generative modeling paradigms discussed.

L15: Vision

The lecture, delivered by guest speaker Professor Jiajun Wu, begins by transitioning the course's focus from 2D images to the more complex domain of 3D vision. The core challenge is immediately identified: while 2D images have a standard representation (a grid of pixels), 3D objects can be represented in many different, non-standard ways. The lecture is structured to first introduce these diverse 3D representations and then explore how deep learning methods have been adapted to work with them.
Part 1: A Taxonomy of 3D Representations
The first part of the lecture provides a comprehensive overview of how 3D object geometry can be represented, dividing the methods into two main families.
1. Explicit Representations: These methods directly describe the points on an object's surface.

The Trade-off of Explicit Representations: They make it very easy to sample points on the object's surface but make it very difficult to test if an arbitrary point in space is inside or outside the object.
2. Implicit Representations: These methods represent an object's surface implicitly, as the level set of a function.

The Trade-off of Implicit Representations: The inverse of explicit methods. They make it very easy to test if a point is inside or outside the object but make it very difficult to directly sample points on the surface.
Part 2: The Data Landscape for 3D Vision
Before diving into deep learning methods, the lecture covers the crucial topic of 3D data. A major challenge for 3D vision has been the relative scarcity of data compared to the billions of images available for 2D vision.

Part 3: Deep Learning on 3D Data
This section details the evolution of deep learning methods designed to work with these different 3D representations.

  1. Early Approaches (View-Based Methods): The very first attempts avoided the complexity of 3D data altogether. They rendered a 3D object from multiple 2D viewpoints and then fed these 2D images into a standard, pre-trained 2D CNN, pooling the results for a final classification. This approach is still relevant today, given the power of modern 2D foundation models.
  2. Volumetric CNNs on Voxels: The most direct extension of 2D deep learning. The 3D object is represented as a voxel grid, and standard 2D convolutions are replaced with 3D convolutions. This was the approach used in the first deep generative models for 3D shapes. However, voxels are computationally expensive and memory-intensive due to their uniform sampling of space.
  3. Deep Learning on Point Clouds (PointNet): A groundbreaking architecture from Stanford that processes raw point clouds directly. The key challenge is permutation invariance (the output should not depend on the order of the points). PointNet solves this by applying a shared function (an MLP) to each point independently and then using a symmetric aggregation function (like max pooling) to combine the point features into a global shape representation.
  4. Deep Implicit Functions: This paradigm shift, occurring around 2019, embraced implicit representations. Instead of trying to predict an explicit structure like a voxel grid or point cloud, the neural network itself becomes the implicit function.
    • The Model: A simple MLP is trained to take a 3D coordinate (x, y, z) as input and output a value indicating, for example, whether that point is inside or outside the object (occupancy) or its signed distance to the surface (SDF).
    • Advantages: This approach produces continuous, high-resolution surfaces and is very memory efficient.
  5. Neural Radiance Fields (NeRF): The revolutionary method that combined deep implicit functions with principles from computer graphics.
    • The Model: A NeRF is an MLP that learns an implicit function mapping a 3D coordinate (x, y, z) and a viewing direction (θ, φ) to a color (RGB) and a volume density (σ).
    • Differentiable Rendering: The key innovation is coupling this neural representation with a differentiable volume rendering equation. This allows the model to be trained end-to-end directly from a set of 2D images of a scene, without needing any 3D ground truth data.
  6. 3D Gaussian Splatting: A very recent and highly efficient alternative to NeRF. It represents a scene with a set of explicit 3D Gaussians (like a point cloud of "blobs"). By combining this explicit representation with a differentiable rasterization pipeline, it achieves real-time rendering speeds that are orders of magnitude faster than NeRF while maintaining comparable quality.

Part 4: Beyond Elemental Geometry - Modeling Structure
The lecture concludes by looking beyond the raw geometry of surfaces to higher-level structural representations.

L16: Vision and Language

The lecture, delivered by guest speaker Professor Ranjay Krishna, begins by framing a major shift in the field: moving away from training specialized models for individual tasks (e.g., a dedicated image classifier) towards building foundation models.

Part 1: Foundation Models for Image Classification (CLIP & CoCa)
This section explains how the principles of self-supervised learning were extended to create the first true foundation model for vision.

Part 2: Multimodal Language Models (LLaVA & Flamingo)
This section covers the next evolution: integrating vision directly into Large Language Models (LLMs) to create powerful, conversational visual assistants.

Part 3: The Frontier of Vision-Language Models (MoLMo)
This section discusses the speaker's own recent work on closing the performance gap between open-source and proprietary models like GPT-4V.

Part 4: Generalizing Outputs and Chaining Models
The lecture concludes by looking at how foundation models are being extended beyond text outputs and how they can be combined into more powerful systems.

L17: Robot Learning

The lecture, delivered by guest speaker Professor Yunzhu Li, distinguishes robot learning from the traditional computer vision tasks covered so far. While supervised and self-supervised learning focus on perception and representation (mapping an input x to an output y), robot learning is about interaction.

Part 1: Problem Formulation - The Language of Interaction
The lecture formalizes the robot learning problem using the framework of sequential decision-making.

Part 2: The Uniqueness of Robot Perception
Robot perception differs significantly from standard computer vision.

Part 3: Algorithms for Robot Learning
This section provides a high-level overview of the major algorithmic paradigms in robot learning.
1. Reinforcement Learning (RL):

2. Model-Based Learning and Planning:

3. Imitation Learning:

Part 4: The Frontier - Robotic Foundation Models
The final section covers the most recent and exciting trend: building robotic foundation models.

The lecture concludes with a vision of a future where these powerful foundation models enable robots to robustly perceive and interact with the complex, unstructured environments of the real world.

L18: Human-Centered AI

The lecture, delivered by guest speaker Professor Yunzhu Li, distinguishes robot learning from the passive perception tasks that have dominated the course so far.

Part 1: Problem Formulation and Robot Perception

Part 2: A Survey of Robot Learning Algorithms
This section provides a high-level overview of the major algorithmic paradigms in robot learning.

  1. Reinforcement Learning (RL):
    • Concept: Learning through extensive trial and error. By interacting with an environment (often a simulation) millions or billions of times, the agent learns a policy that maps states to actions that maximize long-term reward.
    • Successes: RL has been phenomenally successful in solving complex games (Atari, Go, StarCraft) and is now considered the dominant solution for robot locomotion. The lecture highlights that robust locomotion policies can be learned in highly randomized simulations and then successfully transferred to the real world (sim-to-real).
    • Limitations: Model-free RL is extremely sample-inefficient and can be unsafe to train directly in the real world, which has limited its success in complex manipulation tasks.
  2. Model-Based Learning and Planning:
    • Concept: To improve sample efficiency, this approach first learns a model of the world (a "dynamics model"). This learned model acts as an internal simulator, predicting how the state of the world will change in response to an action: s_{t+1} = f(s_t, a_t).
    • Planning: Once this forward model is learned, the robot can use it to plan by "imagining" the outcomes of different action sequences and optimizing for the sequence that best achieves its goal.
    • Case Study (Dumpling Robot): The speaker's work on a dumpling-making robot is presented. The system uses a learned, particle-based dynamics model to predict how dough will deform. This model enables the robot to plan a complex, multi-tool sequence of actions and robustly recover from human-introduced perturbations.
  4. Imitation Learning:
    • Concept: Learning by mimicking a dataset of expert (usually human) demonstrations. This is a form of supervised learning that maps observed states to expert actions (a = π(s)).
    • Modern Approaches (Diffusion Policies): Recent breakthroughs have applied powerful generative models, like diffusion models, to imitation learning. Diffusion policies have shown remarkable success in learning complex, dexterous manipulation tasks from a relatively small number of human demonstrations, representing a highly efficient way to get a working policy.
  5. Robotic Foundation Models:
    • Concept: The latest trend, which aims to build a single, general-purpose policy that can perform a wide variety of tasks, often guided by high-level language instructions (Vision-Language-Action models).
    • The Recipe (e.g., Google's RT series, Pi-Zero): The typical approach is to start with a powerful, pre-trained Vision-Language Model (VLM) and then co-fine-tune it on a large, diverse dataset of robot demonstration data.

Part 3: Challenges and the Human Perspective
The lecture concludes by focusing on the broader challenges and the human-centered aspects of AI and robotics.

The ultimate take-home message is that the goal of AI should not just be to create intelligent systems, but to build tools that help people and are aligned with human values.

All writing