Guus Bouwens
← Writing & notes

FROM THE NOTEBOOK

DL 2021-2025 Research Papers

This comprehensive analysis covers the pivotal AI research papers from 2021-2025 that defined the modern era of artificial intelligence. This period witnessed the emergence of foundation models, multimodal AI systems, advanced reasoning capabilities, and the democratization of AI through open source initiatives. These developments established the technological foundation for today's AI applications and set the trajectory for future AI development.

Overview: The Foundation Model Era

The years 2021-2025 represent the most transformative period in AI history, characterized by the transition from specialized models to general-purpose foundation models. This era saw the emergence of models with unprecedented capabilities across text, images, code, and reasoning tasks, fundamentally changing how we think about AI systems and their applications.

Key Themes and Developments

  1. Compute-Optimal Training: Understanding how to efficiently scale models with available computational resources
  2. Human Alignment: Training AI systems to be helpful, honest, and harmless through human feedback
  3. Multimodal Foundation Models: Systems that can process and generate across multiple modalities
  4. Advanced Reasoning: Chain-of-thought prompting and reasoning capabilities
  5. Tool Use and Agency: AI systems that can interact with external tools and environments
  6. Open Source Revolution: Democratization of powerful AI models through open source releases
  7. Efficient Generation: Breakthrough text-to-image and multimodal generation capabilities

2022: The Alignment and Reasoning Revolution

Chinchilla: Training Compute-Optimal Large Language Models

Authors: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. (DeepMind)
Publication: arXiv:2203.15556
Impact: Revolutionized understanding of optimal model scaling

The Compute-Optimal Breakthrough

Chinchilla fundamentally changed how the AI community thinks about scaling language models by demonstrating that previous models were significantly undertrained relative to their parameter count.

Key Findings

Optimal Scaling Law: For every doubling of model size, training data should also double

Empirical Validation

Chinchilla vs. Gopher: Same compute budget, 70B vs. 280B parameters, 4x more training data

Impact on Model Development

Industry Transformation:

Theoretical Implications:

InstructGPT: Training Language Models to Follow Instructions with Human Feedback

Authors: Long Ouyang, Jeff Wu, Xu Jiang, et al. (OpenAI)
Publication: arXiv:2203.02155
Impact: Established RLHF as the standard alignment technique

The Alignment Problem

Traditional language modeling objectives don't align with what users actually want from AI systems:

Three-Stage Training Process

Stage 1: Supervised Fine-Tuning (SFT)

Data Collection: Human labelers write high-quality responses to prompts Training: Fine-tune GPT-3 on these demonstrations Result: Improved instruction following but still suboptimal

Stage 2: Reward Model Training

Comparison Data: Labelers rank multiple model outputs for quality Reward Model: Train a separate model to predict human preferences Objective: Learn what humans consider "good" responses

Stage 3: Reinforcement Learning from Human Feedback (RLHF)

PPO Training: Use Proximal Policy Optimization with the reward model Optimization: Maximize reward while staying close to SFT model Balance: Prevent optimization hacking and maintain capabilities

Breakthrough Results

Performance: 1.3B parameter InstructGPT preferred over 175B GPT-3

Technical Innovations

Reward Modeling: First large-scale application of learned reward functions RLHF Pipeline: Established the standard training pipeline for aligned models Human Preference Learning: Systematic approach to capturing human values

Long-term Impact

Industry Standard: RLHF became the default alignment technique ChatGPT Foundation: Direct precursor to ChatGPT and GPT-4 Safety Research: Established AI safety as a practical, not just theoretical, concern

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Authors: Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. (Google Research)
Publication: arXiv:2201.11903
Impact: Unlocked reasoning capabilities in large language models

The Reasoning Challenge

Large language models struggled with multi-step reasoning tasks despite their impressive capabilities on simpler problems.

Chain-of-Thought Innovation

Core Insight: Include step-by-step reasoning in prompts to elicit reasoning behavior

Standard Prompting:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
A: 17

Chain-of-Thought Prompting:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.

Empirical Findings

Scale Dependence: Chain-of-thought only emerges at sufficient scale (~100B parameters) Task Breadth: Improvements across arithmetic, commonsense, and symbolic reasoning GSM8K: State-of-the-art 58.1% accuracy with PaLM-540B Robustness: Works across different models and prompt variations

Technical Analysis

Emergence: Reasoning ability appears suddenly at scale rather than gradually Few-Shot Learning: Only requires a few examples in the prompt Interpretability: Provides insight into model's reasoning process Error Analysis: Enables identification of reasoning failures

Broader Impact

Reasoning Research: Sparked extensive research into LLM reasoning capabilities Prompt Engineering: Established chain-of-thought as fundamental technique AI Capabilities: Demonstrated that reasoning could emerge from scale and prompting Applications: Enabled complex problem-solving in production systems

Flamingo: A Visual Language Model for Few-Shot Learning

Authors: Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. (DeepMind)
Publication: arXiv:2204.14198
Impact: Established few-shot learning for vision-language tasks

Multimodal Few-Shot Challenge

Building models that can rapidly adapt to novel vision-language tasks using only a handful of examples.

Architectural Innovations

Perceiver Resampler

Function: Convert variable-size vision inputs to fixed number of visual tokens Architecture: Cross-attention mechanism that queries visual features Benefit: Handles images and videos with consistent interface

Gated Cross-Attention Layers

Integration: Interleave with frozen language model layers Mechanism: Allow language model to incorporate visual information Training: Only new cross-attention layers are trained, preserving LM capabilities

Vision Encoder

Foundation: Use powerful pre-trained vision models (CLIP ViT) Frozen Weights: Preserve visual understanding capabilities Feature Extraction: Process images/videos into rich visual representations

Training Methodology

Multi-Modal Web Data: Train on large-scale interleaved text and image data Few-Shot Setup: Model learns to adapt from context examples Task Diversity: Single model handles captioning, VQA, classification, etc.

Breakthrough Results

Few-Shot Performance: SOTA on 16 vision-language benchmarks Task Transfer: Single model excels across diverse task types Data Efficiency: Outperforms models fine-tuned on 1000x more data Generalization: Strong performance on held-out evaluation benchmarks

Technical Contributions

Architecture Design: Efficient way to combine vision and language models Training Strategy: Interleaved multimodal data for few-shot capabilities Evaluation Framework: Comprehensive assessment across diverse tasks

Impact on Multimodal AI

Foundation Models: Influenced GPT-4V, Gemini, and other multimodal systems Few-Shot Learning: Established paradigm for rapid task adaptation Architecture Patterns: Cross-attention integration became standard approach

High-Resolution Image Synthesis with Latent Diffusion Models (Stable Diffusion)

Authors: Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer
Publication: arXiv:2112.10752
Impact: Democratized high-quality image generation

The Computational Challenge

Previous diffusion models operated in pixel space, requiring enormous computational resources:

Latent Space Innovation

Core Insight: Perform diffusion in a compressed latent space rather than pixel space

Two-Stage Process:

  1. Autoencoder: Learn perceptually equivalent but compact representation
  2. Diffusion Model: Train diffusion process in the latent space

Technical Architecture

Variational Autoencoder

Encoder: Compress images to lower-dimensional latent space Decoder: Reconstruct high-quality images from latent representations Training: Optimize for perceptual similarity, not pixel-level accuracy

Diffusion U-Net

Architecture: U-Net with attention layers for text conditioning Cross-Attention: Enable text-to-image generation capabilities Efficiency: Much faster than pixel-space diffusion

Text Encoder

CLIP Integration: Use CLIP text encoder for text conditioning Flexibility: Support for various conditioning signals (text, class labels, etc.)

Breakthrough Results

Efficiency: 10x reduction in training time compared to pixel-space methods Quality: Competitive with state-of-the-art pixel-space models Versatility: Single architecture for multiple tasks (text-to-image, inpainting, super-resolution) Accessibility: Trainable on single GPU, runnable on consumer hardware

Impact and Applications

Open Source: Released weights and code, democratizing image generation Stable Diffusion: Became foundation for numerous commercial and research applications Creative Tools: Enabled new forms of digital art and content creation Research: Spawned extensive research into diffusion model improvements

Technical Innovations

Latent Diffusion: Established latent space as preferred approach for diffusion models Cross-Attention Conditioning: Standard method for controllable generation Efficient Training: Demonstrated how to make powerful generative models accessible

Constitutional AI: Harmlessness from AI Feedback

Authors: Yuntao Bai, Andy Jones, Kamal Ndousse, et al. (Anthropic)
Publication: arXiv:2212.08073
Impact: Advanced AI alignment through constitutional principles

The Human Feedback Problem

Traditional RLHF has limitations:

Constitutional AI Framework

Phase 1: Supervised Learning

Self-Critique and Revision:

  1. Sample initial responses from base model
  2. Model critiques its own responses using constitutional principles
  3. Model revises responses based on critique
  4. Fine-tune on revised responses

Constitutional Principles: Set of rules guiding model behavior

Phase 2: Reinforcement Learning from AI Feedback (RLAIF)

AI Preference Model:

  1. Generate multiple responses to prompts
  2. AI evaluates responses according to constitutional principles
  3. Train preference model on AI evaluations
  4. Use preference model for RLHF training

Experimental Results

Performance: Constitutional AI models outperform RLHF models

Efficiency: Dramatically reduced human oversight requirements Transparency: Explicit constitutional principles make values interpretable

Technical Innovations

Self-Supervision: AI system improves itself through self-critique Constitutional Framework: Explicit encoding of desired behavior RLAIF: Alternative to human feedback that scales better

Philosophical Implications

Value Alignment: How to encode human values in AI systems Scalable Oversight: Enabling AI systems to supervise other AI systems Transparency: Making AI decision-making more interpretable and accountable

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Authors: Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, et al.
Publication: arXiv:2212.10560
Impact: Automated instruction tuning without human annotation

The Data Bottleneck Problem

Instruction-tuned models require large amounts of high-quality human-written instruction data, which is:

Self-Instruct Bootstrap Process

Stage 1: Seed Instructions

Initial Set: Start with small set of manually written instructions (175 tasks) Diversity: Cover different task types and formats Quality: High-quality examples to guide generation

Stage 2: Instruction Generation

Model Prompting: Use language model to generate new instructions In-Context Learning: Show existing instructions as examples Creativity: Model creates novel task variations and types

Stage 3: Instance Generation

Input-Output Creation: Generate input-output examples for new instructions Consistency: Ensure examples match instruction requirements Quality Control: Filter low-quality or inconsistent examples

Stage 4: Quality Filtering

Automatic Filtering:

Data Augmentation: Add filtered data back to instruction pool for next iteration

Experimental Results

Performance Gains: 33% absolute improvement over vanilla GPT-3 on SuperNaturalInstructions Human Evaluation: Near parity with InstructGPT-001 (only 5% gap) Cost Efficiency: Dramatic reduction in human annotation requirements Scalability: Process can be repeated indefinitely to grow instruction dataset

Technical Contributions

Bootstrapping Framework: Self-supervised approach to instruction tuning Quality Control: Automated methods for filtering generated instructions Scalability: Demonstrated how AI can improve itself without human intervention

Broader Impact

Democratization: Made instruction tuning accessible without expensive human labor Open Source: Released large synthetic instruction dataset Research Direction: Influenced subsequent work on self-improving AI systems

Toolformer: Language Models Can Teach Themselves to Use Tools

Authors: Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, et al. (Meta AI)
Publication: arXiv:2302.04761
Impact: Enabled language models to use external tools

The Tool Use Challenge

Language models excel at reasoning but struggle with tasks where specialized tools outperform:

Self-Supervised Tool Learning

Step 1: Tool-Augmented Dataset Creation

API Demonstrations: Provide few examples of each tool's usage Model Sampling: Generate many potential API calls for each text Filtering: Keep only API calls that improve next-token prediction

Example Process:

Input: "The population of Paris is about 2.2 million, and Tokyo is about"
Model generates: [Search("Tokyo population")]
API returns: "Tokyo population is 13.96 million"
Final: "The population of Paris is about 2.2 million, and Tokyo is about 13.96 million"
Step 2: Model Training

Fine-tuning: Train language model on filtered tool-augmented data Decision Learning: Model learns when and how to call tools Integration: Learn to incorporate tool results into generation

Tool Integration

Supported Tools:

API Format: Text-to-text interfaces for all tools Seamless Integration: Tools called naturally during text generation

Experimental Results

Zero-Shot Performance: Substantial improvements across diverse tasks Tool Usage: Model learned appropriate tool usage patterns

Benchmark Results:

Technical Innovations

Self-Supervised Learning: No manual annotation of tool usage required Filtering Mechanism: Automatic quality control for generated tool calls Multi-Tool Framework: Single model capable of using diverse tools

Impact on AI Systems

Agent Capabilities: Foundation for AI agents that interact with external systems Practical Applications: Enabled more reliable and accurate AI assistants Research Direction: Influenced development of tool-using AI systems

ReAct: Synergizing Reasoning and Acting in Language Models

Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu, et al. (Google Research, Princeton)
Publication: arXiv:2210.03629
Impact: Combined reasoning and action for better problem-solving

The Reasoning-Acting Divide

Previous approaches treated reasoning and acting as separate capabilities:

ReAct Framework

Core Innovation: Interleave reasoning traces and actions in a single sequence

Pattern:

Thought: I need to search for information about X
Action: Search[X]
Observation: [Search results]
Thought: Based on the results, I should look for Y
Action: Search[Y]
Observation: [More results]
Thought: Now I have enough information to answer
Action: Finish[Final answer]

Technical Implementation

Prompting Strategy: Few-shot examples showing reasoning-action interleaving Action Space: Define available actions for each task domain Observation Integration: Incorporate external feedback into reasoning process

Task Applications

Question Answering (HotpotQA)

Challenge: Multi-hop questions requiring information from multiple sources ReAct Approach: Search for information, reason about findings, search for additional details Results: Overcame hallucination issues common in pure reasoning approaches

Fact Verification (Fever)

Challenge: Verify claims against evidence from Wikipedia ReAct Approach: Search for relevant evidence, reason about claim validity Performance: More accurate than chain-of-thought due to grounding in real information

Interactive Decision Making (ALFWorld, WebShop)

Challenge: Navigate environments and complete tasks through actions ReAct Approach: Reason about goals, take actions, adapt based on observations Results: 34% and 10% improvement over imitation/RL methods respectively

Key Advantages

Grounding: Actions provide external feedback to correct reasoning errors Interpretability: Clear reasoning traces show decision-making process Adaptability: Can handle unexpected situations through reasoning Generalization: Same framework applies across diverse task types

Technical Contributions

Unified Framework: Single approach for both reasoning and acting tasks Prompting Methodology: Effective way to elicit reasoning-acting behavior Error Correction: External actions help correct internal reasoning errors

Impact on AI Research

Agent Development: Foundation for more capable AI agents Interactive AI: Better handling of tasks requiring environmental interaction Reasoning Enhancement: Showed how external grounding improves reasoning

2023: The Multimodal and Open Source Revolution

GPT-4 Technical Report

Authors: OpenAI
Publication: arXiv:2303.08774
Impact: Established multimodal capabilities and human-level performance

Multimodal Breakthrough

GPT-4 represented the first widely-deployed multimodal foundation model capable of processing both text and images while generating text outputs.

Capability Achievements

Professional Benchmarks: Human-level performance on many professional exams

Academic Benchmarks: State-of-the-art on many NLP tasks

Technical Innovations

Predictable Scaling

Infrastructure Development: Training systems that behave predictably across scales Performance Prediction: Accurately predicted GPT-4 capabilities from smaller models Loss Scaling: Validated scaling laws across 6 orders of magnitude

Safety and Alignment

Post-Training: Extensive alignment process using RLHF Safety Evaluation: Comprehensive testing for potential harms Red Teaming: Adversarial testing by domain experts

Architectural Details

Transformer Base: Decoder-only transformer architecture Scale: Significantly larger than GPT-3 but exact parameters not disclosed Training: Pre-training on diverse internet text and image data Multimodal: Joint training on text and image inputs

Limitations and Risks

Hallucination: Still generates false information Limited Context: Fixed context window limitations Bias: Inherits biases from training data Safety Concerns: Potential for misuse in various domains

Impact and Applications

Industry Standard: Established capabilities benchmark for foundation models Multimodal AI: Accelerated development of vision-language models Commercial Success: Powered ChatGPT Plus and numerous applications

LLaMA: Open and Efficient Foundation Language Models

Authors: Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. (Meta AI)
Publication: arXiv:2302.13971
Impact: Democratized access to state-of-the-art language models

The Open Source Revolution

LLaMA challenged the closed-source dominance of large language models by demonstrating that competitive models could be trained on publicly available data.

Model Family

Size Range: 7B to 65B parameters Efficiency Focus: Smaller models trained longer for better inference efficiency Data: 1.0-1.4 trillion tokens from publicly available sources

Training Data

Publicly Available Only: No proprietary or private datasets Sources:

Key Results

LLaMA-13B vs GPT-3 (175B): Outperformed GPT-3 on most benchmarks despite being 13x smaller LLaMA-65B: Competitive with Chinchilla-70B and PaLM-540B Efficiency: Much lower inference costs due to smaller size

Performance Highlights

Zero-Shot Performance:

Few-Shot Performance:

Technical Contributions

Efficient Scaling: Demonstrated optimal training strategies for compute budgets Data Quality: Showed importance of high-quality, diverse training data Open Research: Enabled widespread research on large language models

Broader Impact

Research Acceleration: Enabled thousands of researchers to work with state-of-the-art models Innovation: Led to numerous fine-tuned variants (Alpaca, Vicuna, WizardLM) Competition: Forced closed-source providers to improve their offerings Democratization: Made powerful AI capabilities accessible globally

PaLM 2 Technical Report

Authors: Google Research
Publication: arXiv:2305.10403
Impact: Advanced multilingual and reasoning capabilities

Architectural Improvements

PaLM 2 demonstrated that careful architectural choices and training procedures could achieve better performance than simple parameter scaling.

Key Innovations

Improved Dataset Mixture

Multilingual Focus: Enhanced training on 100+ languages Diverse Sources: Better balance of web text, books, code, mathematics, and multilingual data Quality Control: More aggressive filtering and deduplication

Compute-Optimal Scaling

Following Chinchilla: Applied compute-optimal principles Smaller But Better: Fewer parameters than PaLM but trained on more data Efficiency: Better performance per FLOP than predecessor

Enhanced Training Objectives

Mixture of Objectives: Combined different training objectives Multilingual Training: Specialized objectives for multilingual capabilities Code Training: Enhanced programming capabilities through targeted training

Performance Results

Multilingual: State-of-the-art on translation and multilingual understanding Reasoning: Significant improvements on mathematical and logical reasoning Coding: Enhanced programming capabilities across multiple languages

Specific Benchmarks:

Specialized Variants

Med-PaLM 2: Medical domain specialization achieving expert-level performance Sec-PaLM: Cybersecurity applications for threat analysis Product Integration: Powered Bard, Gmail, Google Docs, and other products

Technical Contributions

Training Efficiency: More efficient training procedures Multilingual Architecture: Better handling of diverse languages Domain Specialization: Framework for creating domain-specific variants

2024-2025: Reasoning and Multimodal Integration

Advanced Reasoning Models

The period 2024-2025 saw the emergence of models specifically designed for complex reasoning tasks, with systems like OpenAI's o1 series demonstrating breakthrough capabilities in mathematics, science, and programming.

Mixture of Experts Evolution

Models like Mixtral and others advanced the mixture of experts architecture, enabling larger model capacity with constant computational cost during inference.

Long Context Models

Development of models with dramatically extended context windows (Gemini 1.5 with 1M+ tokens) enabling new applications in document analysis and complex reasoning.

Multimodal Integration

Further advancement in vision-language models with improved image understanding, video processing, and cross-modal reasoning capabilities.

Cross-Cutting Themes and Impact

The Compute-Optimal Revolution

Multiple works established new understanding of optimal resource allocation:

Human Alignment Advancement

Revolutionary progress in aligning AI systems with human values:

Multimodal Foundation Models

Breakthrough in systems that process multiple modalities:

Reasoning and Agency

Major advances in AI reasoning and interaction capabilities:

Open Source Transformation

Democratization of AI through open source releases:

Methodological Breakthroughs

Training Innovations

RLHF Pipeline: Standardized approach to human preference learning Constitutional Training: Self-supervised alignment through principles Tool Integration: Methods for incorporating external tools and APIs

Architectural Advances

Cross-Attention Integration: Standard method for multimodal fusion Efficient Attention: Techniques for handling longer sequences Modular Architectures: Composable systems for different capabilities

Scaling Insights

Data-Centric Approach: Quality and quantity of data matters as much as parameters Compute Allocation: Optimal distribution of resources across model size and training Emergent Capabilities: Understanding when and how capabilities emerge

Practical Impact and Applications

Commercial Transformation

ChatGPT Era: Mass adoption of conversational AI Creative Tools: AI-powered content creation and artistic applications Productivity: Integration into workflows and professional tools Code Generation: Revolutionary impact on software development

Research Acceleration

Open Models: Enabled global research participation Benchmarking: Established comprehensive evaluation frameworks Safety Research: Practical approaches to AI alignment and safety

Societal Implications

Democratization: Made powerful AI capabilities widely accessible Economic Impact: Transformed industries and created new markets Education: Changed how people learn and access information Creative Expression: New forms of human-AI creative collaboration

Technical Challenges Addressed

Scalability Solutions

Efficient Training: Methods for training larger models with limited resources Inference Optimization: Techniques for deploying large models efficiently Data Processing: Handling massive datasets for training

Safety and Alignment

Value Learning: Methods for learning human preferences and values Robustness: Making models more reliable and less prone to harmful outputs Interpretability: Understanding and explaining model behavior

Capability Enhancement

Reasoning: Unlocking complex reasoning and problem-solving abilities Multimodality: Processing and generating across different modalities Tool Use: Enabling AI systems to interact with external environments

Looking Forward: Implications for Future Development

Established Principles

Compute-Optimal Training: Data and parameters should scale together Human Feedback: Critical for aligning systems with human values Multimodal Foundation: Future systems will be inherently multimodal Open Development: Combination of open and closed source driving innovation

Emerging Directions

Reasoning Enhancement: Continued progress in complex reasoning capabilities Agent Development: More capable AI agents that can act in the world Scientific Applications: AI systems advancing scientific discovery Efficiency Improvements: Better performance with lower computational costs

Challenges Ahead

Scaling Sustainably: Managing computational and environmental costs Safety at Scale: Ensuring advanced systems remain aligned and safe Equitable Access: Distributing AI benefits fairly across global populations Regulatory Frameworks: Developing appropriate governance for AI systems

Conclusion: A Transformative Era

The period from 2021-2025 represents the most transformative era in artificial intelligence history. The papers covered in this analysis collectively:

  1. Established Foundation Models: Created the paradigm of general-purpose models that can be adapted to many tasks
  2. Solved Alignment Challenges: Developed practical methods for aligning AI systems with human values
  3. Enabled Multimodal AI: Demonstrated how to effectively combine vision, language, and other modalities
  4. Unlocked Reasoning: Showed that large models can perform complex reasoning when properly prompted and trained
  5. Democratized Access: Made powerful AI capabilities available to researchers and developers globally
  6. Created New Industries: Enabled entirely new categories of applications and businesses

These developments established the technological foundation for modern AI applications and set the trajectory for continued advancement. The combination of technical breakthroughs, methodological insights, and open source initiatives created an environment of rapid innovation that continues to drive AI progress.

The papers represent not just individual contributions but a collective transformation of how we build, train, and deploy AI systems. They demonstrate the importance of combining technical innovation with responsible development practices, ensuring that advanced AI capabilities benefit humanity broadly rather than being concentrated in the hands of a few.

Looking forward, the principles and techniques established during this period continue to guide AI development, while new challenges around safety, efficiency, and equitable access shape the research agenda for the future. The era of foundation models that began in 2021 continues to evolve, with each new development building upon the foundational work documented in these pivotal papers.

All writing