VAEs to Diffusion Models with Live Code

VAEs to Diffusion Models with Live Code#

About Me Publications My Teaching Research Notes

This Spring 2026 guest lecture series is designed as a coherent four-part journey from first principles to state-of-the-art diffusion model design.

Each notebook combines theory, derivation, and runnable NumPy-based code. The goal is not just to show formulas, but to connect every equation to implementation choices.

Notebook Sequence#

  1. Lecture 1: The Essence of Generative AI

  2. Lecture 2: Variational Autoencoders in Depth (ELBO + from-scratch implementation)

  3. Lecture 3: Diffusion Models in Depth (ELBO + training loss derivation)

  4. Lecture 4: Latest Architectures in Diffusion (comprehensive overview)

How to Use These Notes#

  • Read in order: each lecture assumes ideas from previous ones.

  • Run cells actively: the code is intentionally educational and readable.

  • Track objective terms numerically: reconstruction/KL/noise-prediction terms are visualized and interpreted.

  • Compare modeling paradigms: we repeatedly map latent-variable and diffusion viewpoints to one another.

Global Learning Outcomes#

By the end of the full sequence, you should be able to:

  1. Formalize generative modeling as density learning and sampling from \(p_\theta(x)\).

  2. Derive the VAE ELBO step-by-step and interpret each term geometrically and statistically.

  3. Derive diffusion training objectives from the variational bound and connect them to practical denoising losses.

  4. Explain modern diffusion variants (latent, deterministic, consistency, flow-matching, few-step, video) in one unified conceptual map.

  5. Implement minimal but complete training/sampling pipelines without deep-learning frameworks.

Per-Lecture Deliverables#

Lecture 1

  • Probabilistic foundations for generation.

  • Data density, likelihood, MLE, KL intuition, sampling mechanics.

Lecture 2

  • Full ELBO derivation and decomposition.

  • From-scratch VAE implementation on 1D and 2D toy datasets.

  • Hidden utility cells for optimizer/housekeeping; visible core VAE logic.

Lecture 3

  • Forward diffusion process, posterior structure, variational objective decomposition.

  • Why the practical objective is an MSE on noise prediction.

  • NumPy denoiser training plus DDPM and DDIM-style sampling.

Lecture 4

  • Comprehensive architecture survey for current diffusion ecosystem.

  • Conceptual and practical differences among major accelerated and generalized variants.

Prerequisites and Scope#

  • Basic probability (Gaussian distributions, conditional probability, KL divergence).

  • Linear algebra (matrix products, covariance intuition).

  • Comfort with Python + NumPy.

The notebooks are designed to be mathematically explicit while staying implementation-centered.