Literature Review of Diffusion Models

Interests

• Flow-based generative models (flow matching, diffusion/score models, and related formulations)
• Sampling and learning algorithms
• One-step and few-step models; distillation and consistency-style training
• Optimal transport perspectives
• Connections with other generative paradigms
• Discrete flow and diffusion models, and applications to language modeling, protein modeling
• Applications including vision, text, multimodal modeling, and scientific settings
• what’s after diffusion (markov-process based iterative noising and denoising), score (navigating the data manifold through score approximation), and flow matching (approximating the optimal transport map)? Energy based matching?

Questions

Gaussians

• Why high-dimensional gaussians’ density is concentrated on a sphere?
• What’s fisher divergence, and what’s the geometry like? Compare it to more general bregman divergence?

Diffusion & sampling

• What’s tweedie’s identity? How to derive it? Explain it in plain words.
• Write the continuous time version of langevin dynamics? Why is there a square-root 2 in the diffusion term? Why is it useful for diffusion models?

Score Matching

• What’s the difference between score matching (Hyvärinen and Dayan, 2005) and denoising score matching (Vincent, 2011)?
• What’s the issue with Score matching?
• How does sliced score matching sidestep the problems in score matching? And what’s the problem with sliced score matching?
• Write the loss function for denoising score matching (DSM). Express denoiser. How does it connect with tweedie’s identity?
• How does NCSN improves upon previous iteration of score matching? Write training objective of NCSN.
• can i explain the difference between VE-SDE, and VP-SDE?
• Why is NCSN VE-SDE, and DDPM VP-SDE?
• How are NCSN and DDPM losses connected?
• What’s the difference between denoising score matching and NCSN?
• Illustrate DDPM, NCSN.
• Proof of affine-drift conditional forward kernel closed-form analytical formulation as gaussian.
• Why does forward marginal density converge to prior distribution?
• Write reverse SDE dynamics equation. Explain the reason for diffusion coefficient in the drift coefficient term.
• How does f,g in forward and reverse SDE vary with time?
• What’s PF-ODE? How to convert between other representations of the same thing? i.e. going from SDE to ODE to discretization.
• What’s the algorithm/pseudocode for annealed langevin dynamics? How is it different from Unadjusted langevin algorithm?
• Proof of fokker-planck.
• How does $\sigma$ vary from $t=0\to t=T$? Is it more at the start or the end?

Flow matching

• NF, continuous NF
• NODE: Instantaneous change-of-variables formula. How does $\log p(x)=\log p(z)-\log \left|\det \frac{\partial f}{\partial z}\right|$ for discrete case equal $\frac{d\log p_t(x(t))}{dt}=-\mathrm{Tr}\left(\frac{\partial f}{\partial x}\right)$ for continuous case?
• How does normalizing flows enable exact log-likelihood computation?
• How is backpropagation happen for continuous NF models?
• Illustrate flow matching models.

Efficient solvers and samplers

• What are the different samplers for ODEs and SDEs that are used?
• What are the different ODE solvers used for sampling the diffusion models?
• Write the equation for euler-maruyama?
• What’s the sde solver beside euler-maruyama?
• What are the main takeaways from EDM paper?
• why is DDIM an integrator for PF-ODE?
• Derive PF-ODE from SDE? Why does there exist an ODE that preserves the marginal density evolution of the SDE?

Distillation and one-step diffusion

• Progressive Distillation
• Consistency models
  • What are the issues with CTM? Why do we lose the ability to compute exact log-likelihood?
• Flow maps
  • Shortcut models
  • Mean flows

Guided diffusion

• How will you explain DPS really quickly?
• guidance: classifier-based, classifier free

Multimodal diffusion

• What’s the problem with CLIP? What are other better multimodal encoders?

Architecture

• Lay out the architecture for U-Net and DiTs. How are they different? Which to prefer? What’s the pitfalls?
• Write the architecture for Image generators: SD2, SD3, Flux 1-2, Nano Banana
• Illustrate the architecture for multimodal DiT
• Write the architecture for Video Gen models. Meta Movie gen, google omni
• What’s the current SOTA architecture for Any-to-Any generative model?

Discrete diffusion or flow matching

• Illustrate discrete flow matching
• What’s the difference between MDLM
• Block diffusion modeling
• CTMC theory. Why is it useful?

Training

• What is the design space over which diffusion models can be categorized?

Diffusion

Tutorials

• Diffusion Models From Scratch
• [2510.21890] The Principles of Diffusion Models
• [2403.18103] Tutorial on Diffusion Models for Imaging and Vision
• [2406.08929] Step-by-Step Diffusion: An Elementary Tutorial
• [2512.05092] Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction
• All Posts – Sander Dieleman
• [Simon Coste's notes for a graduate-level course on flow-based generative models]()
• Notes — Antonio Franca: detailed notes on flow matching, optimal transport, Generator matching, Symmetry
• Topics in High-Dimensional Sampling and Generative Diffusions – Yifan Chen, UCLA

DDPM

• Deep Unsupervised Learning using Nonequilibrium Thermodynamics: introduced iterative Markov process based noising and denoising.
• [2006.11239] Denoising Diffusion Probabilistic Models: Introduced ELBO training and epsilon-prediction objective formulation.

Score Matching and SDE

• [1907.05600] Generative Modeling by Estimating Gradients of the Data Distribution: Reintroduced score matching as viable objective
• [2011.13456] Score-Based Generative Modeling through Stochastic Differential Equations: Introduced diffusion models as SDE that can be reversed and sampled using langevin samplers.

Drifting: [2602.04770] Generative Modeling via Drifting

Design space and solvers

• [2206.00364] Elucidating the Design Space of Diffusion-Based Generative Models
  • Main ODE equation that arises from solving the diffusion ODE
  • Solution to ODE can now be done using different ODE solvers, particularly better approximator like RK2/4
  • Reparameterization of loss function.
  • Analyzing different noise (sigma) schedules.
• [2312.02696] Analyzing and Improving the Training Dynamics of Diffusion Models
• [2010.02502] Denoising Diffusion Implicit Models: DDIM
• [2206.00927] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps: DPM-Solver
• [2102.09672] Improved Denoising Diffusion Probabilistic Models: iDDPM
• [2107.00630] Variational Diffusion Models: VDM
• Distillation, one-step sampling
  • [2202.00512] Progressive Distillation for Fast Sampling of Diffusion Models: Introduced v-prediction parametrization and progressive distillation by distilling a student model from teacher trajectories that match 2 DDIM teacher steps in one step. Progressively applying this K times leads to a student model that directly denoise
  • [2303.01469] Consistency Models
  • [2505.18825] How to build a consistency model: Learning flow maps via self-distillation
  • [2410.12557] One Step Diffusion via Shortcut Models

Guidance

• [2105.05233] Diffusion Models Beat GANs on Image Synthesis: Introduces Classifier-based guidance
• [2207.12598] Classifier-Free Diffusion Guidance: Introduced classifier-free guidance

Latent diffusion

• [2112.10752] High-Resolution Image Synthesis with Latent Diffusion Models
• Latent Diffusion Models: A Survey on Foundations, Variants, and Web-scale Deployments | Journal of Web Engineering

Discrete

• [2107.03006] Structured Denoising Diffusion Models in Discrete State-Spaces: D3PM
• [2205.14987] A Continuous Time Framework for Discrete Denoising Models
• [2402.04997] Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design
• [2406.07524] Simple and Effective Masked Diffusion Language Models: MDLM
• [2503.09573] Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
• [2510.22852] Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
• [2506.10892] The Diffusion Duality
• [2412.10193] Simple Guidance Mechanisms for Discrete Diffusion Models
• [Beyond Single Tokens: Distilling discrete diffusion models | Emiel Hoogeboom](https://ehoogeboom.github.io/post/discrete_mmd_diffusion_language_models
• [2407.15595] Discrete Flow Matching
• Flow Map Language Models: One-step Language Modeling via Continuous Denoising

Diffusion x RL

• [2208.06193] Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
• [2606.17551] Reversal Q-Learning
• [2407.13734] Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review
• [2505.05470] Flow-GRPO: Training Flow Matching Models via Online RL
• [2505.07818] DanceGRPO: Unleashing GRPO on Visual Generation
• [2603.18991] CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think
• [2605.26491] Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models

Geometry x diffusion

• [2505.17517] The Spacetime of Diffusion Models: An Information Geometry Perspective
• Riemannian Diffusion Models
• Scaling Riemannian Diffusion Models
• Riemannian Score-Based Generative Modelling
• Flow Matching on General Geometries
• [2605.31106] Riemannian Diffusion Models on General Manifolds via Physics-Informed Neural Networks

Diffusion x Interp

• [2606.20560] How Transparent is DiffusionGemma?
• DiffusionGemma model card  |  Google AI for Developers
• [2408.13256] How Diffusion Models Learn to Factorize and Compose

Generalization

• [2506.03719] On the Closed-Form of Flow Matching: Generalization Does Not Arise from Target Stochasticity
• [2408.13256] How Diffusion Models Learn to Factorize and Compose

Flow Matching

Tutorials

• [2412.06264] Flow Matching Guide and Code

Flow matching

• [2209.03003] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow: Introduced first flavor of flow matching models called Rectified Flows,
• [2303.08797] Stochastic Interpolants: A Unifying Framework for Flows and Diffusions
• [2602.16813] Flow Map Language Models: One-step Language Modeling via Continuous Denoising
• [2510.21608] Generalised Flow Maps for Few-Step Generative Modelling on Riemannian Manifolds

Evaluation

• [2606.20536] The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation [Website]:Evaluated lack of FID reproduction for generative models, and why it exists. More here.

Foundational Models

• [2403.03206] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis: SD3. Scaled Rectified flows with QK-normalization, logit-normal noise scheduler. Introduced MM-DiT architecture.
• [2506.15742] FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space
• Lance: Unified Multimodal Modeling by Multi-Task Synergy
• [2603.06507] Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis: Flux-3 backbone

Miscellaneous

• [2504.10612] Energy Matching: Unifying Flow Matching and Energy-Based Models for Generative Modeling
• [2511.13720] Back to Basics: Let Denoising Generative Models Denoise: Debunked myths surrounding diffusion model training, particularly, showing low FID scores with x-pred and v-loss due to predicting in high dimension without preconditioning introduced in EDM.
• [2502.13129] Is Noise Conditioning Necessary for Denoising Generative Models?: Another Kaiming He’s paper that debunks old myths in diffusion models. Removing noise conditioning as popularized by EDM paper, doesn’t offer much degradation to FID values except DDIM. Flow models are particularly interesting to notice as they marginally show improvements in FID scores.
• [2602.18428] The Geometry of Noise: Why Diffusion Models Don’t Need Noise Conditioning

Inverse Problems

• [2209.14687] Diffusion Posterior Sampling for General Noisy Inverse Problems
• Pseudoinverse-Guided Diffusion Models for Inverse Problems | ICLR 2023
• [2509.26489] Contrastive Diffusion Guidance for Spatial Inverse Problems
• [2505.05657] ArrayDPS: Unsupervised Blind Speech Separation with a Diffusion Prior
• Zero-shot Human Pose Estimation using Diffusion-based Inverse solvers

Implementations

• GitHub - yuanchenyang/smalldiffusion: Simple and readable code for training and sampling from diffusion models · GitHub

• add difflow here.

More Tutorials

• Diffusion Models: A Comprehensive Survey of Methods and Applications | alphaXiv
• Statistical Analysis of Markovian Generative Modeling | alphaXiv
• Diffusion Models: A Mathematical Introduction | alphaXiv
• Score-based Diffusion Models via Stochastic Differential Equations — a Technical Tutorial | alphaXiv
• A Tutorial on Diffusion Theory: From Differential Equations to Diffusion Models | alphaXiv
• [2412.11024v2] Exploring Diffusion and Flow Matching Under Generator Matching
• CVPR 2026 Tutorials: CVPR tutorials is cool. what other conferences has these tutorials track?
  • CVPR’26 Tutorial on Diffusion Models
  • CVPR 2026 Tutorial on Accelerated Diffusion Models
  • Analytic Understanding of Diffusion Models — CVPR 2026 Tutorial
• Latest Posts – Sander Dieleman: Sander’s posts are a delight to read.
• Simon Coste’s notes for a graduate-level course on flow-based generative models: Transport and Generation