Guus Bouwens
← Writing & notes

FROM THE NOTEBOOK

DL 2018-2021 Research Papers

This comprehensive analysis covers the pivotal AI research papers from 2018-2021 that were not included in Ilya Sutskever's legendary 30u30 reading list, yet fundamentally shaped the trajectory of modern artificial intelligence. These works represent breakthrough moments that established new paradigms, solved long-standing challenges, and laid the foundation for today's AI systems.

Overview: A Transformative Period

The years 2018-2021 witnessed unprecedented advances in AI, particularly in natural language processing, computer vision, and scientific applications. This period saw the emergence of large language models, the adaptation of transformers to computer vision, breakthrough self-supervised learning methods, and AI systems solving complex scientific problems. These developments established the foundation for modern AI applications and influenced the scaling strategies that drive today's systems.

Key Themes and Developments

  1. The Rise of Large Language Models: From BERT's bidirectional training to GPT-3's emergent capabilities
  2. Transformer Universality: Expansion beyond NLP to computer vision and multimodal applications
  3. Self-Supervised Learning Revolution: Learning powerful representations without labeled data
  4. Scientific AI Breakthroughs: AI solving fundamental scientific problems like protein folding
  5. Efficient Architecture Design: Balancing performance with computational constraints
  6. Multimodal AI: Connecting vision and language understanding

2018: Foundation Year

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Authors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (Google AI)
Publication: arXiv:1810.04805
Impact: Revolutionized NLP with bidirectional pretraining

Revolutionary Approach

BERT introduced bidirectional training for language models, fundamentally changing how machines understand text. Unlike previous models that processed text left-to-right (like GPT) or used separate left-to-right and right-to-left models (like ELMo), BERT jointly conditioned on both directions simultaneously.

Technical Innovation

Masked Language Modeling (MLM): Randomly mask 15% of input tokens and predict them using bidirectional context

Input: The cat sat on the [MASK]
Target: mat

Next Sentence Prediction (NSP): Train model to understand relationships between sentences

Sentence A: The cat sat on the mat.
Sentence B: It was very comfortable.
Label: IsNext

Architecture Details

Breakthrough Results

Training Methodology

Pre-training Phase: Large-scale unsupervised training on BookCorpus (800M words) and English Wikipedia (2.5B words) Fine-tuning Phase: Task-specific adaptation with single additional output layer

Long-term Impact

BERT established the pre-train-then-fine-tune paradigm that dominated NLP for years. It demonstrated that bidirectional context significantly improves language understanding and inspired numerous follow-up works (RoBERTa, ALBERT, DistilBERT, etc.).

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Authors: Jonathan Frankle, Michael Carbin (MIT)
Publication: ICLR 2019
Impact: Fundamental insights into neural network training and pruning

Core Hypothesis

"Dense, randomly-initialized, feed-forward networks contain subnetworks ('winning tickets') that—when trained in isolation—reach test accuracy comparable to the original network in a similar number of iterations."

Methodology

Iterative Magnitude Pruning Algorithm:

  1. Train network to convergence
  2. Prune lowest-magnitude weights (typically 20% per iteration)
  3. Reset remaining weights to their initial values
  4. Retrain pruned network

Key Findings

Implications for AI

Training Efficiency: Suggests that much of neural network capacity is unnecessary Architecture Design: Influences development of more efficient architectures Understanding Learning: Provides insights into what makes neural networks trainable

Subsequent Developments

2019: Scaling and Efficiency

GPT-2: Language Models are Unsupervised Multitask Learners

Authors: Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever (OpenAI)
Publication: OpenAI Technical Report
Impact: Demonstrated scaling effects and emergent capabilities

Scaling Breakthrough

GPT-2 showed that simply scaling up language models leads to qualitative improvements in capability. The 1.5B parameter model demonstrated unprecedented zero-shot performance across diverse tasks.

Technical Architecture

Zero-Shot Capabilities

Reading Comprehension: 55 F1 on CoQA without training examples Translation: Competitive with unsupervised methods Summarization: Coherent summaries of articles Question Answering: Reasonable performance across diverse domains

Emergent Behaviors

As model size increased, GPT-2 exhibited qualitatively new behaviors:

Social Impact

GPT-2's capabilities raised concerns about misuse, leading OpenAI to initially withhold the full model. This highlighted the need for responsible AI development practices.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Authors: Mingxing Tan, Quoc V. Le (Google)
Publication: ICML 2019
Impact: Systematic approach to scaling CNN architectures

Compound Scaling Innovation

Instead of arbitrarily scaling depth, width, or resolution independently, EfficientNet proposed scaling all dimensions uniformly with a compound coefficient.

Mathematical Framework

Scaling Formula:

depth: d = α^φ
width: w = β^φ  
resolution: r = γ^φ

Subject to: α · β² · γ² ≈ 2, where α ≥ 1, β ≥ 1, γ ≥ 1

Key Insights

Balance is Critical: Scaling all dimensions together more effective than scaling individually Diminishing Returns: Each dimension has saturation points when scaled alone Resource Efficiency: Compound scaling achieves better accuracy per FLOP

Neural Architecture Search

Used automated search to find optimal baseline architecture (EfficientNet-B0), then scaled systematically to create family of models (B1-B7).

Performance Achievements

Broad Impact

EfficientNet's scaling principles influenced architecture design across computer vision and beyond, establishing systematic approaches to model scaling.

T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Authors: Colin Raffel, Noam Shazeer, Adam Roberts, et al. (Google)
Publication: JMLR 2020
Impact: Unified framework for all NLP tasks

Text-to-Text Paradigm

T5 reformulated all NLP tasks into a unified text-to-text format:

Translation: "translate English to German: That is good." → "Das ist gut."
Classification: "cola sentence: The course is jumping well." → "not acceptable"
Summarization: "summarize: [article text]" → "[summary]"

Comprehensive Empirical Study

T5 systematically studied transfer learning components:

C4 Dataset

Created Colossal Clean Crawled Corpus (C4):

Key Findings

Denoising Objectives: Span corruption worked best for pre-training Architecture Choice: Encoder-decoder performs well across all tasks
Scale Benefits: Larger models and datasets consistently improve performance Multi-task Learning: Joint training helpful but task-specific fine-tuning often better

Performance Results

T5 achieved SOTA on multiple benchmarks:

StyleGAN: A Style-Based Generator Architecture for Generative Adversarial Networks

Authors: Tero Karras, Samuli Laine, Timo Aila (NVIDIA)
Publication: CVPR 2019
Impact: Revolutionary GAN architecture enabling unprecedented control over image generation

Style-Based Generation

StyleGAN introduced a novel generator architecture that separated high-level attributes (pose, identity) from stochastic variation (hair, freckles):

Mapping Network: z → W latent space transformation Synthesis Network: Progressive generation with style injection at each layer Adaptive Instance Normalization (AdaIN): Style control mechanism

Technical Innovation

Style Mixing: Different styles applied at different resolutions

Coarse styles (4×4-8×8): Pose, face shape, glasses
Middle styles (16×16-32×32): Hair style, eyes, face structure  
Fine styles (64×64-1024×1024): Color scheme, micro-features

Noise Injection: Stochastic features controlled independently of style Progressive Growing: Training starts at low resolution and gradually increases

Breakthrough Results

Impact on Computer Vision

StyleGAN established new standards for image generation quality and control, influencing research in:

StyleGAN2 Improvements (2020)

Follow-up work addressed artifacts and improved quality:

2020: The Year of Scale and Emergence

GPT-3: Language Models are Few-Shot Learners

Authors: Tom B. Brown, Benjamin Mann, Nick Ryder, et al. (OpenAI)
Publication: NeurIPS 2020
Impact: Demonstrated emergent capabilities at scale

Unprecedented Scale

GPT-3's 175 billion parameters represented a 100x increase over GPT-2, demonstrating that scale alone could drive qualitative improvements in AI capabilities.

Few-Shot Learning Paradigm

Zero-Shot: Task specified by text prompt alone One-Shot: Single example provided in context
Few-Shot: Multiple examples provided in context

Few-shot arithmetic example:
Q: What is 15 + 25?
A: 15 + 25 = 40
Q: What is 83 + 29?
A: 83 + 29 = 112  
Q: What is 55 + 67?
A: [model completes: 55 + 67 = 122]

Emergent Capabilities

Natural Language Understanding: Strong performance on reading comprehension, language inference Code Generation: Ability to write functional code from descriptions Creative Writing: Coherent stories, poems, and creative text Reasoning: Basic arithmetic, word puzzles, and logical reasoning Domain Adaptation: Quick adaptation to specialized domains

Performance Highlights

Societal Impact

GPT-3's capabilities sparked widespread discussion about:

Technical Architecture

Vision Transformer (ViT): An Image is Worth 16x16 Words

Authors: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. (Google Research)
Publication: ICLR 2021
Impact: Applied Transformers to computer vision, challenging CNN dominance

Paradigm Shift

ViT demonstrated that pure Transformer architectures could match or exceed CNN performance on image classification, without convolutional inductive biases.

Architecture Innovation

Patch Tokenization: Images divided into fixed-size patches (16×16 pixels) Linear Embedding: Patches linearly projected to token embeddings Position Embeddings: Learnable position information added to patch embeddings Classification Token: Special [CLS] token for classification output

Mathematical Framework

Image patches: x_p ∈ ℝ^(N×(P²×C))
Linear projection: z_0 = [x_class; x_p¹E; x_p²E; ...; x_pᴺE] + E_pos
Transformer layers: z_ℓ = MSA(LN(z_ℓ₋₁)) + z_ℓ₋₁
                    z_ℓ = MLP(LN(z_ℓ)) + z_ℓ
Classification: y = LN(z_L⁰)

Training Requirements

Large-Scale Pre-training: Required JFT-300M (300M images) for best results Transfer Learning: Pre-train on large dataset, fine-tune on target tasks Data Efficiency: Performance scaled with pre-training dataset size

Performance Results

Impact on Computer Vision

ViT's success initiated the "Vision Transformer Revolution":

SimCLR: A Simple Framework for Contrastive Learning of Visual Representations

Authors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton (Google Research)
Publication: ICML 2020
Impact: Revolutionized self-supervised learning for computer vision

Contrastive Learning Framework

SimCLR learned visual representations by maximizing agreement between differently augmented views of the same image:

Data Augmentation: Random cropping, color distortion, Gaussian blur Encoder Network: ResNet backbone for feature extraction
Projection Head: Nonlinear transformation for contrastive learning Contrastive Loss: NT-Xent (normalized temperature-scaled cross-entropy)

Key Technical Insights

Data Augmentation is Critical: Composition of multiple augmentations essential

Projection Head Benefits: Nonlinear projection between representation and loss

Large Batch Sizes: Contrastive learning benefits from more negative examples

Temperature Parameter: Properly tuned temperature crucial for performance

Mathematical Framework

Contrastive Loss: ℓ(i,j) = -log(exp(sim(z_i, z_j)/τ) / Σ(k=1 to 2N, k≠i) exp(sim(z_i, z_k)/τ))
Where: sim(u,v) = u^T v / (||u|| ||v||)

Performance Achievements

Impact on Self-Supervised Learning

SimCLR's success led to explosion of self-supervised methods:

AlphaFold 2: Highly Accurate Protein Structure Prediction

Authors: John Jumper, Richard Evans, Alexander Pritzel, et al. (DeepMind)
Publication: Nature 2021
Impact: Solved the 50-year-old protein folding problem

The Protein Folding Challenge

Predicting 3D protein structure from amino acid sequence was considered one of biology's greatest challenges, with implications for:

Revolutionary Approach

AlphaFold 2 combined multiple AI techniques in a novel end-to-end architecture:

Multiple Sequence Alignment (MSA) Processing:

Pair Representation Learning:

Structure Module:

Technical Architecture

Evoformer Blocks: Novel attention mechanism jointly processing:

Structure Module: Converts pair representations to 3D coordinates:

Unprecedented Accuracy

CASP14 Results: Median GDT_TS score of 92.4 (previous best: ~70) Atomic Precision: Typically within 1.5Å RMSD of experimental structures Difficult Targets: Successful on proteins without close homologs Speed: Minutes vs. months/years for experimental methods

Global Impact

Scientific Community:

Drug Discovery:

Biotechnology:

Technical Innovations

End-to-End Learning: Single model trained on structure prediction task Geometric Deep Learning: Incorporation of 3D geometric priors Multi-Scale Attention: Processing information at different biological scales Iterative Refinement: Progressive improvement of structure predictions

2021: Multimodal and Efficient AI

CLIP: Learning Transferable Visual Models from Natural Language Supervision

Authors: Alec Radford, Jong Wook Kim, Chris Hallacy, et al. (OpenAI)
Publication: ICML 2021
Impact: Connected vision and language, enabling zero-shot visual understanding

Multimodal Breakthrough

CLIP learned visual concepts directly from natural language descriptions, creating a shared embedding space for images and text.

Architecture and Training

Dual Encoder Design:

Massive Scale Training:

Key Innovations

Natural Language Supervision: Learning from descriptive text rather than fixed labels Scale Effects: Performance improved dramatically with larger datasets Robust Representations: Zero-shot performance on diverse visual tasks Flexible Interface: Natural language queries for image classification

Zero-Shot Capabilities

Image Classification: Competitive with supervised ResNet-50 Object Recognition: Understanding of objects, scenes, activities Optical Character Recognition: Reading text in images Action Recognition: Understanding of human activities Geo-localization: Identifying locations from images

Performance Highlights

Impact on Multimodal AI

CLIP's success catalyzed development of:

DeiT: Training Data-Efficient Image Transformers & Distillation Through Attention

Authors: Hugo Touvron, Matthieu Cord, Matthijs Douze, et al. (Facebook AI)
Publication: ICML 2021
Impact: Made Vision Transformers practical without massive datasets

Data Efficiency Challenge

Original ViT required JFT-300M dataset (300 million images) for competitive performance. DeiT achieved similar results with only ImageNet (1.2 million images).

Knowledge Distillation Innovation

Teacher-Student Framework:

Distillation Token Mechanism:

Input tokens: [CLS] [DISTILL] [patch_1] [patch_2] ... [patch_N]
- CLS token: Traditional classification
- DISTILL token: Learns from teacher predictions  
- Final prediction: Ensemble of both tokens

Training Strategy

Strong Data Augmentation:

Optimization Techniques:

Performance Results

Contributions to Vision Transformers

Democratization: Made ViTs accessible without massive computational resources Distillation Techniques: Showed effective knowledge transfer from CNNs to Transformers Training Recipes: Established best practices for ViT training Practical Impact: Enabled widespread adoption of Vision Transformers

Cross-Cutting Themes and Impact

The Scaling Paradigm

Multiple papers demonstrated that scale alone could drive qualitative improvements:

Self-Supervised Learning Revolution

Several breakthrough methods showed that learning without labels could match or exceed supervised approaches:

Transformer Architecture Universality

The Transformer architecture proved effective far beyond natural language processing:

Efficiency and Practical Deployment

Several works focused on making powerful AI systems more practical:

Multimodal and Scientific AI

AI systems began connecting different modalities and solving real-world scientific problems:

Legacy and Influence

Foundation for Modern AI

These papers established the foundation for many of today's AI systems:

Methodological Contributions

Key methodological insights that continue to influence research:

Practical Impact

Real-world applications enabled by these breakthroughs:

Research Directions Established

These papers opened entire new research areas:

Conclusion: A Transformative Era

The period from 2018-2021 represented a transformative era in AI research, establishing many of the principles and architectures that drive today's systems. These papers demonstrated that:

  1. Scale Matters: Systematic scaling of models and data leads to qualitative improvements
  2. Architecture Transfers: Successful architectures often generalize across domains
  3. Self-Supervision Works: Learning from data structure can match supervised approaches
  4. Efficiency is Crucial: Practical deployment requires attention to computational constraints
  5. Multimodal is Powerful: Connecting different modalities enhances AI capabilities
  6. AI Can Solve Science: Advanced AI systems can tackle fundamental scientific problems

The influence of these works extends far beyond their immediate contributions, shaping the research agenda and development priorities that continue to drive AI progress today. Together with the foundational papers in Ilya's 30u30 reading list, they represent the essential knowledge base for understanding modern artificial intelligence and its trajectory toward even more capable and beneficial systems.

The papers covered in this analysis demonstrate the rapid pace of AI advancement and the importance of foundational research in driving practical applications. They serve as both historical artifacts of a remarkable period of progress and continuing sources of insight for current and future AI development.

All writing