概念

Reparameterization Trick

— 把"随机采样"改写成"确定函数 + 外部噪声",梯度才能流过去。

很多人以为 reparameterization 只是 Kingma & Welling 在 2014 年那篇 VAE 论文里设计的一个小技巧,专门给 VAE 用、用完即弃。其实这是任何"想对随机采样反向传播"的场景都必须用到的通用范式——只要你的损失函数形如 $\mathbb{E}_{z \sim q_\phi}[f(z)]$,且分布 $q_\phi$ 的参数 $\phi$ 是可学习的,你就会撞到同一面墙:梯度无法穿过 z = sample(q_\phi) 这一步。后面会看到,diffusion 模型的 forward process、flow matching 的插值轨迹、SDE-based 生成模型的扰动核,本质上全都在用同一个 trick。

问题的本质用一句话讲清楚:$z$ 通过分布参数依赖于 $\phi$,但"采样"这个操作本身不可微。Vizuara 的 Dr. Raj 在 Lecture 2 讲 VAE 时反复强调,VAE 之所以能用普通的 SGD + autograd 训练出来,全靠这一招把随机性"挤"到一个跟 $\phi$ 完全无关的外部噪声 $\epsilon$ 里——这样 encoder 的梯度通路就不会被采样事件切断。

「once you capture the secret recipe for a student you can determine their handwriting...there is a 1:1 mapping from the secret recipe to the handwriting.」—— Vizuara L2 @ 9:02(Dr. Raj 把 latent z 比作"secret recipe",而 reparameterization 就是让这台"按 recipe 出字"的机器可以被梯度反推、被 SGD 调节的关键工程改造。)

在训练里,这件事对应的具体动作是这样的:encoder 网络读入 $x$、输出两个张量 $\mu_\phi(x)$ 和 $\sigma_\phi(x)$,然后我们不直接写 z = np.random.normal(mu, sigma),而是改成两行代码 eps = np.random.normal(0, 1); z = mu + sigma * eps。这两种写法在数值分布上完全等价——$z$ 都服从 $\mathcal{N}(\mu, \sigma^2)$,但反向传播时第二种写法让 $\partial z / \partial \mu = 1$、$\partial z / \partial \sigma = \epsilon$ 都成为良定义的确定函数偏导,梯度因此能一路回流到 encoder 的权重上。这一个微小的代码改动,是 VAE 与 diffusion 整个家族能"端到端可训练"的工程基石。

问题的形式化:为什么直接采样不可导

我们要优化的目标长这样——在 VAE 里就是 ELBO 的重建项 $\mathbb{E}_{z \sim q_\phi(z|x)}[\log p_\theta(x|z)]$,在 diffusion 里就是去噪损失 $\mathbb{E}_{x_t \sim q(x_t|x_0)}[\|\epsilon - \epsilon_\theta(x_t, t)\|^2]$——抽象一下都是同一个模板:

$$\mathcal{L}(\phi) = \mathbb{E}_{z \sim q_\phi(z|x)}\big[f(z)\big]$$

我们想要的是 $\nabla_\phi \mathcal{L}$,把它代入 SGD 更新 $\phi$。展开期望:

$$\nabla_\phi \mathcal{L} = \nabla_\phi \int q_\phi(z|x)\, f(z)\, dz$$

右边的 $q_\phi(z|x)$ 也依赖 $\phi$——这意味着你不能把 $\nabla_\phi$ 直接搬进期望里(如果分布跟 $\phi$ 无关你就可以,但这里不行)。从代码层面看:torch.normal(mu, sigma) 返回的张量虽然数值上由 $\mu, \sigma$ 决定,但 PyTorch 的 autograd 不会沿"采样算子"反向传播——这是和"discrete operation breaks the graph"同类的问题。如果你硬把这一行写在 forward 里,PyTorch 报的不是错,而是悄悄给你一个错误的、与 $\mu, \sigma$ 无关的梯度,这才是真正危险的。

Why(要点):梯度的数学定义需要"确定函数对输入的偏导数"。$z \sim \mathcal{N}(\mu, \sigma^2)$ 是一个采样事件,它没有"$z$ 对 $\mu$ 的偏导"这种东西——$z$ 是随机变量的一次实现(realization),不是 $\mu$ 的函数。autograd 需要的是"输出张量是输入张量的可微函数",而采样恰恰不是。

The Trick:把随机源剥离出去

Kingma & Welling 2014 的洞察非常优雅:与其在分布内部采样,不如在分布外部找一个跟参数 $\phi$ 无关的"基础噪声分布" $p(\epsilon)$,再找一个确定的、可微的变换 $g_\phi(\epsilon, x)$,使得

$$z = g_\phi(\epsilon, x), \quad \epsilon \sim p(\epsilon) \quad \Longleftrightarrow \quad z \sim q_\phi(z|x)$$

两者在分布意义上完全等价(数学上叫 measure-preserving transform,测度保持变换)。代入到期望里:

$$\mathcal{L}(\phi) = \mathbb{E}_{\epsilon \sim p(\epsilon)}\big[f(g_\phi(\epsilon, x))\big]$$

注意这一步的关键:期望符号下的分布 $p(\epsilon)$ 不再依赖 $\phi$——$\phi$ 只出现在 $g_\phi$ 这个被积函数里,而 $g_\phi$ 是个普通的可微函数(神经网络 + 加乘运算)。这下我们可以放心把 $\nabla_\phi$ 搬进期望:

$$\nabla_\phi \mathcal{L} = \mathbb{E}_{\epsilon}\big[\nabla_\phi f(g_\phi(\epsilon, x))\big] \approx \frac{1}{N}\sum_{i=1}^N \nabla_\phi f(g_\phi(\epsilon^{(i)}, x))$$

用 Monte Carlo 估计——实践中 $N = 1$ 就够用,每个 mini-batch 抽一次 $\epsilon$,配合 batch size 自然就摊平了方差。这就是 VAE 训练能用普通 SGD 流程跑起来的全部秘密。

Why(要点):随机性还在,但它被推到了计算图的外部输入端——梯度不需要穿过它,只需要穿过 $g_\phi$ 这个确定函数。这就是 "trick" 两个字的真正含义:不是消除随机性,而是重新摆放它在计算图里的位置。这种思路在数学里叫"变量替换"(change of variables),在物理里叫"参数化族",在统计里叫"location-scale family"——同一思想在不同领域反复出现。

高斯情形:最常见的具体形式

当 $q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \mathrm{diag}(\sigma_\phi(x)^2))$ 是对角高斯(VAE 的默认选择),标准的 reparameterization 就是:

$$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

其中 $\odot$ 是逐元素乘(Hadamard product)。验证一下分布等价:$\mu + \sigma \cdot \epsilon$ 是 $\epsilon$ 的仿射变换,根据高斯的性质 $\mathbb{E}[z] = \mu + \sigma \cdot \mathbb{E}[\epsilon] = \mu$、$\mathrm{Var}[z] = \sigma^2 \cdot \mathrm{Var}[\epsilon] = \sigma^2$——所以 $z$ 的边际分布确实就是 $\mathcal{N}(\mu, \sigma^2)$,跟直接采样在概率分布意义上完全相同。梯度也直接可读:$\partial z / \partial \mu = 1$,$\partial z / \partial \sigma = \epsilon$。两者都是良定义、可数值计算的张量,autograd 可以无障碍处理。

Why(工程细节):实际代码里通常输出 $\log \sigma^2$ 而不是 $\sigma$ 本身,再 sigma = exp(0.5 * log_var) 转换。原因有两层——第一,神经网络输出可正可负,但 $\sigma$ 必须为正;用 $\log \sigma^2$ 做参数化天然满足正性约束,省去 softplus 或 abs 这类非光滑操作。第二,数值上更稳定:当模型预测 $\sigma \to 0$(确定性增加)时,直接输出 $\sigma$ 会面临除零和梯度爆炸;输出 $\log \sigma^2$ 则把这个区域映到 $-\infty$,被 exp 自然 clip 住。

第二种视角:测度变换的常识例子

如果上面的形式化让你晕了,可以用一个生活类比来理解 reparam 的思路。想象你要研究"班上同学的体重 $W$"——同学体重服从某个分布,你想统计 $W$ 的平均、方差。但你设备只能称"行李 $\epsilon$"(一个标准化的、跟同学无关的随机量)。怎么办?你建一个变换 $W = a + b \cdot \epsilon$($a, b$ 跟班级有关),先称 $\epsilon$,再用变换算出 $W$。本质上你把"对 $W$ 的统计"翻译成了"对 $\epsilon$ 的统计 + 一次变换"。神经网络这里的 $a, b$ 就是 $\mu_\phi, \sigma_\phi$,差别只在于这是高维的、变换由神经网络给出的版本。重点在于:随机性的"源头" $\epsilon$ 跟我们要调节的参数 $\phi$ 解耦了,所以调节 $\phi$ 不会影响"采样这件事本身怎么发生",只会影响"采样结果怎么被映射"。这就是它能跟 SGD 兼容的关键。

diffusion 里它无处不在

Vizuara 后续讲 forward process 时(Lecture 7 起)会写出 DDPM 最核心的那一行公式:

$$x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

这一行公式本身就是一次 reparameterization。$x_t$ 本应"从 $q(x_t | x_0) = \mathcal{N}(\sqrt{\bar\alpha_t}x_0, (1-\bar\alpha_t)I)$ 这个分布里采"——如果你照字面写,autograd 在这里就断了。但 DDPM 通过这一行公式,把整个 $t$ 步加噪过程改写成了"输入 $x_0$ + 一次外部噪声 $\epsilon$ 的确定算术运算"。这个改写带来的好处是巨大的,远超"能反向传播"这一点:

  • closed-form 跳步采样:训练时你可以直接 t = randint(1, T); x_t = sqrt(alpha_bar[t]) * x_0 + sqrt(1 - alpha_bar[t]) * eps,不用真的迭代 $t$ 次马尔可夫链。这把训练成本从 $O(T)$ 砍到 $O(1)$——没有这一点,$T = 1000$ 的 DDPM 训练在算力上根本不可行。
  • $\epsilon$-prediction 训练目标合法:DDPM 让网络去预测噪声本身,损失是 $\|\epsilon - \epsilon_\theta(x_t, t)\|^2$。这个目标能成立的前提,就是 $x_t$ 必须是 $\epsilon$ 的可微函数——只有这样,从重构误差出发,梯度才能倒推到"应该输出哪个噪声"。
  • 变量替换 ELBO 化简:所有 $\mathbb{E}_{q(x_t|x_0)}[\cdot]$ 都能换成 $\mathbb{E}_{\epsilon \sim \mathcal{N}(0,I)}[\cdot]$,原本的 KL 散度也变成对 $\epsilon$ 的简单期望。DDPM 论文里 simple loss 的整个推导链都建立在这一步上。
🟡 教材外补充:DDPM 论文(Ho et al. 2020)里 Algorithm 1(training)的第 4 行 x_t = sqrt(alpha_bar) * x_0 + sqrt(1 - alpha_bar) * eps 和 Algorithm 2(sampling)的第 4 行 x_{t-1} = ... + sigma_t * z,全都在显式使用 reparameterization。可以说没有这一招,整个 $\epsilon$-prediction + simple loss + 单步加噪的工程范式都不成立。Score-based SDE(Yang Song 2021)把这个想法推得更远——SDE 形式 $dx = f(x,t)dt + g(t)dw$ 里,Wiener process $dw$ 就是"reparam 的外部噪声"在连续时间下的对应物。Flow matching(Lipman et al. 2023)的 $x_t = (1-t)x_0 + t x_1$ 是 reparam 在确定性轨迹上的版本。EDM(Karras et al. 2022)把整个去噪框架重新写成 $x = D_\theta(x_0 + n\sigma)$,本质也是 reparam。

梯度估计器的对比:reparam vs REINFORCE

如果你不用 reparameterization,还可以用另一种估计器叫 score function estimator(也叫 REINFORCE 估计器、似然比估计器):

$$\nabla_\phi \mathbb{E}_{z \sim q_\phi}[f(z)] = \mathbb{E}_{z \sim q_\phi}\big[f(z) \cdot \nabla_\phi \log q_\phi(z|x)\big]$$

这个估计器的好处是不需要 $f$ 或 $g_\phi$ 可微(所以离散变量也能用,强化学习里的 policy gradient 就是它),但代价是方差极高——常常需要 baseline、控制变量、上千次采样才能让估计收敛。Reparameterization 的方差通常比它低 1 到 2 个数量级,这才是 reparam 能让 VAE 和 diffusion 用 mini-batch + 单次采样就训练得动的真正工程根因。如果当年 Kingma & Welling 只会用 REINFORCE,VAE 大概率根本训不出来。

Why(直觉):reparam 估计器之所以方差低,是因为它利用了 $f$ 的局部信息——通过 $\nabla f$ 知道"$z$ 在这个方向上能让 loss 下降多少"。REINFORCE 只看 $f$ 的标量值,完全不利用"loss 在 $z$ 周围怎么变化"的结构,相当于盲人摸象,所以方差自然高。可以把它想成两种导航:reparam 像有 GPS 加陀螺仪知道每一步该往哪偏;REINFORCE 像只能问"你现在离目的地近还是远",然后靠多次试错猜方向。

更深一层:把 reparam 看作分布的"参数化族"

从测度论的视角,reparameterization 其实是在做一件优雅的事:把一族分布 $\{q_\phi\}_{\phi}$ 表示成"一个固定的基础测度 $p(\epsilon)$ 经过参数化变换 $g_\phi$ 推前(pushforward)"得到的像测度。这个视角下,$\phi$ 的角色从"分布的参数"变成了"变换的参数"——而对变换的微分是数学上完全合法的(流形上的切空间),对采样事件的微分则不是。这也是为什么 normalizing flow 整个研究方向能从 reparam 自然生长出来:把 $g_\phi$ 从仿射变换扩展到任意可逆光滑映射,就得到了 NF;让 $g_\phi$ 随时间演化、由 ODE 定义,就得到了 continuous normalizing flow;让噪声本身随时间演化、由 SDE 定义,就得到了 score-based 扩散模型。同一个数学思想,分叉出整个深度生成模型谱系。

什么时候 reparam 不工作

除了离散分布之外,还有几种实际情形会让 reparam 失灵或退化:

  • 分布族不容许 location-scale 形式:例如 mixture of Gaussians(混合高斯),各分量的混合权重是离散选择,需要 Gumbel-Softmax 或单独处理每个分量。又如 Dirichlet、Beta 这些分布,参数 $\phi$ 不是简单的位置-尺度结构,要用 implicit reparam(隐式重参数化,Figurnov et al. 2018)通过 CDF 的逆来构造可微采样。
  • $g_\phi$ 几乎不可微:比如 $z = \mathrm{ReLU}(\mu + \sigma \epsilon)$ 在 0 处不光滑,会带来次梯度问题——但实践中 ReLU 的次梯度(在 0 处取 0 或 0.5)足够好用,不算真正障碍。真正麻烦的是 round / sign / Heaviside 这类阶梯函数,它们处处梯度为 0,整个回路就废了。
  • 截断分布:truncated normal 在边界处的 reparam 需要小心处理边界 mass,常见做法是用 inverse CDF reparam(也叫 implicit reparam),通过 $z = F^{-1}_\phi(\Phi(\epsilon))$ 来构造,其中 $F^{-1}_\phi$ 是目标分布的分位数函数。

历史小注:trick 这个名字

Kingma 在 2014 年 ICLR 的论文里直接把这一节标题命名为 "The Reparameterization Trick"——"trick" 这个词其实有点低估它的重要性。它看起来像一个工程小聪明,但其实是所有现代连续型生成模型可训练性的数学基础。Rezende, Mohamed & Wierstra 几乎同时(DeepMind 团队)独立提出了等价的想法,叫 "stochastic backpropagation"——这个名字其实更准确,因为它强调了"梯度穿越随机性"这件事的本质。两个团队在差不多的时间点撞上同一个 idea,说明这是一个"水到渠成"的发现——一旦你想用神经网络参数化连续分布、想用 SGD 优化,就必然要走到这一步。后来的 normalizing flow、diffusion、flow matching、neural ODE,全是在这个发现之上的不同方向延展。

双计算图对照:直接采样 vs reparam

下面这张图是这个概念最核心的可视化。左边是"直接采样"版本——computation graph 在采样节点处断开,梯度无法从 loss 倒推到 encoder 参数 $\phi$。右边是"reparam"版本——通过引入外部噪声 $\epsilon$,graph 变成一条连续的可微路径,梯度可以沿橘色虚线箭头一路回流到 $f_\phi$。读图的关键看两点:(1) $\epsilon$ 节点在哪——左图没有它,右图它独立悬挂;(2) 从 $\mathcal{L}$ 到 $\phi$ 有没有一条不经过虚线(dashed = 不可微)的回路。

直接采样:梯度断在采样节点 x f_φ(x)→μ,σ z ~ N(μ,σ²) ✗ 采样 不可微 L=f(z) ∇φ L 无法计算 (autograd 在虚线处停止) Reparam:梯度沿 g_φ 流回 x f_φ(x)→μ,σ ε ~ N(0,I) 外部噪声 z = μ + σ·ε (确定运算) L=f(z) ∇φ L 沿确定路径回流 ✓

点击下面四个步骤按钮,看 reparameterization 是怎么把"采样事件"一步步替换为"确定函数 + 外部噪声"的。不变量:$z$ 的边际分布始终是 $\mathcal{N}(\mu, \sigma^2)$;改变的只是 $\phi$ 出现在计算图里的位置——从"分布内部"挪到"确定函数内部"。

优化 $\mathcal{L}(\phi) = \mathbb{E}_{z \sim q_\phi(z|x)}[f(z)]$。我们想要 $\nabla_\phi \mathcal{L}$ 来更新 encoder 参数。

手算一次前向 + 反向

设 encoder 在某个样本上输出 $\mu = 0.5$、$\sigma = 1.2$,本次 mini-batch 采样 $\epsilon = 0.3$,下游损失为 $f(z) = z^2$(一个最简化的代理损失)。

前向计算:

$$z = \mu + \sigma \cdot \epsilon = 0.5 + 1.2 \cdot 0.3 = 0.5 + 0.36 = 0.86$$

$$\mathcal{L} = f(z) = z^2 = 0.86^2 = 0.7396$$

反向计算(链式法则一步步代入,不要跳):

$$\frac{\partial \mathcal{L}}{\partial \mu} = \frac{\partial f}{\partial z} \cdot \frac{\partial z}{\partial \mu} = 2z \cdot 1 = 2 \cdot 0.86 \cdot 1 = 1.72$$

$$\frac{\partial \mathcal{L}}{\partial \sigma} = \frac{\partial f}{\partial z} \cdot \frac{\partial z}{\partial \sigma} = 2z \cdot \epsilon = 2 \cdot 0.86 \cdot 0.3 = 0.516$$

两个梯度都是具体的数——可以拿去更新 encoder 的 $\mu, \sigma$ 输出层权重了。如果你用直接采样、不用 reparam,这两个偏导根本写不出来,autograd 给你的会是错误的零或 NaN。

三个常见误区

  1. "reparam 是 VAE 的专属技巧"。反例:DDPM 训练算法第 4 行 x_t = √ᾱ_t · x_0 + √(1−ᾱ_t) · ε 就是 reparameterization 的直接应用;flow matching 里 $x_t = (1-t)x_0 + t x_1$ 同样是它($\epsilon$ 这里是 $x_1$);score SDE 里 $dx = f(x,t)dt + g(t)dw$ 的 Wiener 增量 $dw$ 是它的连续时间版本;甚至 GAN 里 $z \sim \mathcal{N}(0, I); G(z)$ 也是。它是梯度穿越随机性的通用钥匙,不是 VAE 的私有财产。
  2. "reparam 消除了采样的随机性,所以训练是确定的"。反例:随机性还在 $\epsilon$ 里——每个 mini-batch 抽不同的 $\epsilon$,loss 仍然是随机变量。同一个 $x$ 跑两次前向,得到的 $z$ 不一样,loss 也不一样。reparam 只是把随机源挪了位置(从计算图的"内部不可微节点"挪到"外部输入端"),并没有让模型变成确定模型。具体表现:相同 batch 跑两次,loss 差异通常在 1% 量级,正是采样方差的体现。
  3. "任何分布都能 reparam"。反例:离散分布做不到——比如 $z \sim \mathrm{Categorical}(\pi_\phi)$,没有一个可微的 $g_\phi(\epsilon)$ 能把均匀噪声映射成 one-hot 类别向量,因为 argmax 这一步本质不可微(输出是阶梯函数,梯度处处为 0)。Reparam 严格要求"连续分布 + 可微变换"这两个条件同时成立。
    🟡 教材外补充:离散情形要用 Gumbel-Softmax(Jang & Gu 2017;Maddison et al. 2017)——它通过 $z = \mathrm{softmax}((\log \pi + g)/\tau)$ 其中 $g$ 是 Gumbel 噪声,做出一个"可微的近似 argmax",温度 $\tau \to 0$ 时退化为真正的 categorical 采样。这是 reparam 在离散世界里的对应物,被 VQ-VAE 的训练、discrete diffusion language model 等场景广泛使用。Diffusion language model(如 D3PM、SEDD)里离散态扩散也面临这个问题,常见解法是把 token embed 到连续空间再做 reparam,或者直接用 score matching for discrete domains。

相关概念

  • VAE Encoder —— reparam 第一次登场的舞台,encoder 输出 $\mu, \sigma$ 然后 reparam 采 $z$。
  • Variational Lower Bound —— ELBO 的重建项之所以能展开成 $\mathbb{E}_\epsilon[\cdot]$、能用 SGD 优化,全靠 reparam。

来源

  1. diffusion-principles-vizuara · L2 · Variational Autoencoders Explained From Scratch · (6:45) —— Dr. Raj 引出 latent variable 概念,把它比作"secret recipe",为 reparameterization 的需求埋下伏笔。
  2. diffusion-principles-vizuara · L2 · "1:1 mapping from secret recipe to handwriting" · (9:02) —— Dr. Raj 强调"采样后的 z 必须能反推回参数 μ, σ"——这正是 reparam 要解决的可微性问题。
查看 91 条课程原文出处
CourseLectureTimeExcerpt
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 00: 05 Hello everyone and welcome to the second Hello everyone and welcome to the second lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 01: 17 from scratch. from scratch. And this lecture is And this lecture is And this lecture is going to be very important for the going to be very important for the going to be very important for the subsequ…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 02: 19 with the understanding of VAS. with the understanding of VAS. So first we'll take a simple example. So first we'll take a simple example. So first we'll take a simple example. Imagine that you have co…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 03: 19 in cursive. Some of us u don't write in cursive. So there are a u don't write in cursive. So there are a u don't write in cursive. So there are a lot of different ways in which people lot of different…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 04: 26 Then another time you press a button. Another time you get another hello. This Another time you get another hello. This Another time you get another hello. This time it might be some different student…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 05: 35 such that or generate a machine such that the samples drawn from that machine that the samples drawn from that machine that the samples drawn from that machine are from the probability distribution ar…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 06: 45 factors that determine the style of the factors that determine the style of the handwriting. handwriting. handwriting. So each student handwriting depends on So each student handwriting depends on So …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 08: 03 So my first intuition is that okay fine So my first intuition is that okay fine my first job is to figure out all these my first job is to figure out all these my first job is to figure out all these …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 09: 02 final shape of the handwriting. final shape of the handwriting. So once you capture the secret recipe So once you capture the secret recipe So once you capture the secret recipe for for for a student …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 09: 57 machine looks as follows. machine looks as follows. Secret recipe is given as an input to Secret recipe is given as an input to Secret recipe is given as an input to the machine. The machine takes thi…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 10: 52 interesting, right? It it it looks like okay fine this this it it makes a lot of okay fine this this it it makes a lot of okay fine this this it it makes a lot of sense. [snorts] sense. [snorts] sense…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 11: 53 now um in the rest of this lecture we We now um in the rest of this lecture we We are going to assume that are going to assume that are going to assume that for the handwriting example that we took fo…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 12: 54 not know these factors beforehand. That is why they are called as latent is why they are called as latent is why they are called as latent variables. variables. variables. So So intuitively we think t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 13: 51 y-axis is represented by the symbol zed 2. Zed1 stands for slantness of the 2. Zed1 stands for slantness of the 2. Zed1 stands for slantness of the handwriting and zed2 stands for neatness handwriting…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 14: 48 below it's it's not neat. So I have made some spelling mistake So I have made some spelling mistake So I have made some spelling mistake here. But what I want to show is I want here. But what I want t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 15: 33 mapping which I was talking about. For mapping which I was talking about. For example, if you pick a point here, you example, if you pick a point here, you example, if you pick a point here, you know …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 16: 31 in your class. in your class. So this is also called as the So this is also called as the So this is also called as the distribution in the latent space of the distribution in the latent space of the …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 17: 37 typical handwriting typical handwriting let's say it is divided into 28x 28 let's say it is divided into 28x 28 let's say it is divided into 28x 28 pixels. pixels. pixels. So there are 784 numbers rig…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 18: 39 latent space is has a mean of zero and a latent space is has a mean of zero and a standard deviation of one or any other standard deviation of one or any other standard deviation of one or any other v…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 19: 27 So for now what we have seen is that we have this general architecture for the have this general architecture for the have this general architecture for the machine which takes the secret recipe as ma…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 20: 34 another word for this machine which is called as decoder. called as decoder. called as decoder. Let us quickly summarize what all we Let us quickly summarize what all we Let us quickly summarize what …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 21: 44 vary. So here you can see that as the yellow dot moves around the Latin space, yellow dot moves around the Latin space, yellow dot moves around the Latin space, the handwriting on the right hand side …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 22: 44 next. So far we have used the decoder to generate samples from the latent generate samples from the latent generate samples from the latent variables. But what is this decoder variables. But what is t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 23: 55 we create a latent space with the latent variables. So just as before we assume variables. So just as before we assume variables. So just as before we assume that there are two latent variables zed th…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 24: 54 know. [snorts] Now let's say we have chosen a point in Now let's say we have chosen a point in Now let's say we have chosen a point in the Latin space which corresponds to the the Latin space which co…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 25: 56 we can represent digit 5 as a bunch of we can represent digit 5 as a bunch of pixels pixels pixels which are divided as 28 into 28 pixels where some pixels have a value of where some pixels have a val…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 27: 08 five, if you closely look at it, it it does look like a five. It's it's it's does look like a five. It's it's it's does look like a five. It's it's it's almost something like this. Okay. So it looks l…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 28: 08 is that with this approach we will get a fixed shape of five every time but we fixed shape of five every time but we fixed shape of five every time but we will not get variations of it and what will n…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 29: 10 restricted to zero and one. You are open to the possibility of to the possibility of to the possibility of a color which appears in between white a color which appears in between white a color which a…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 30: 08 the decoder does not give a single value but for every pixel the decoder gives but for every pixel the decoder gives but for every pixel the decoder gives the mean the mean the mean and the standard d…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 31: 13 deterministic approach only. So now let us So now let us So now let us have a simple example to understand how have a simple example to understand how have a simple example to understand how does the …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 32: 16 So the decoder is nothing but a neural network which is trained to take inputs network which is trained to take inputs network which is trained to take inputs from the latin space and generate from th…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 33: 11 somewhere towards the right, which means that it has to be a bit bright and the that it has to be a bit bright and the that it has to be a bit bright and the pixel value is high. Bright pixels have pi…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 34: 10 now we have covered now we have covered one part of the story which explains the one part of the story which explains the one part of the story which explains the variational autoenccoder. variational…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 35: 10 part, you will understand that we have made a major assumption. made a major assumption. made a major assumption. Remember when we talked about the Remember when we talked about the Remember when we t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 36: 14 recipe and then you need to map all these handwriting styles to the secret these handwriting styles to the secret these handwriting styles to the secret recipe and make all those points and recipe and…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 37: 13 the latin space to access to generate the latin space to access to generate digit five. One option is to access all digit five. One option is to access all digit five. One option is to access all poss…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 38: 04 Wouldn't it be better if we knew which part of the Latin space to access for part of the Latin space to access for part of the Latin space to access for the type of image we want to generate? the type…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 39: 00 decoder. So the overall architecture looks like So the overall architecture looks like So the overall architecture looks like this. this. this. First I get let's say you go to any First I get let's sa…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 40: 00 this architecture is very similar to a plain autoenccoder where you have an plain autoenccoder where you have an plain autoenccoder where you have an encoder which compresses the image into encoder wh…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 41: 03 a dis or or a region which is the most a dis or or a region which is the most probable region of finding the digit probable region of finding the digit probable region of finding the digit five instea…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 42: 01 we are going to look at a nice visual we are going to look at a nice visual description to understand this much description to understand this much description to understand this much better. better. …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 43: 12 individual points. So 99% of these points, they they produce garbage. So points, they they produce garbage. So points, they they produce garbage. So your your space your latent space is not your your …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 44: 14 regions which tell you that this is the region corresponding to the dog. This is region corresponding to the dog. This is region corresponding to the dog. This is the region corresponding to the cat e…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 45: 08 space which is continuous and the space which is continuous and the garbage gaps are now filled with garbage gaps are now filled with garbage gaps are now filled with distributions. So now you can saf…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 46: 07 So the two stories put together form the variational autoenccoder. variational autoenccoder. We first looked at the decoder where we We first looked at the decoder where we We first looked at the deco…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 47: 03 the areas in the latin space? How do you know which levers to pull in the know which levers to pull in the know which levers to pull in the decoder? This is given to you by the decoder? This is given …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 48: 08 uh styles of the students when they uh styles of the students when they wrote the word hello or the handwritten wrote the word hello or the handwritten wrote the word hello or the handwritten digits a…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 49: 09 going to give you the latent space distribution given the image. So that's distribution given the image. So that's distribution given the image. So that's why you see the notation kind of flipped why …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 50: 09 the encoder like this and the decoder the encoder like this and the decoder going from a smaller line here to a going from a smaller line here to a going from a smaller line here to a bigger line. In …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 51: 16 these variational autoenccoders trained. How do you train these to generate How do you train these to generate How do you train these to generate images? Let's say you give a sample of images? Let's s…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 52: 05 and then sample from it. So we are going to take a real example in the next class to take a real example in the next class to take a real example in the next class where you will see exactly how deep …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 52: 56 is not deterministic but rather it is is not deterministic but rather it is probabilistic. probabilistic. probabilistic. And finally in your mind I want you to And finally in your mind I want you to A…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 53: 48 image which is generated at the end. This is all that is happening in the This is all that is happening in the This is all that is happening in the variation autoenccoders. variation autoenccoders. va…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 55: 04 tells you the hidden factors of variation behind that image. variation behind that image. variation behind that image. [snorts] The machine tells you the hidden factors The machine tells you the hidde…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 56: 10 can lie. So it appears to be a very can lie. So it appears to be a very smart and a clever machine. smart and a clever machine. smart and a clever machine. Okay. Now you have these hidden factors Okay…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 57: 11 the final image should be as close as possible to the original image. So you possible to the original image. So you possible to the original image. So you can see that there has to be some kind can se…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 58: 10 encode and the second neural network is the machine which learns to decode. the machine which learns to decode. the machine which learns to decode. So somehow we have to train the So somehow we have t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 59: 20 a dot like this and you pass it to the a dot like this and you pass it to the decoder and you get a value of zero decoder and you get a value of zero decoder and you get a value of zero which which me…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 60: 07 our predicted distribution which should match as close as possible to the true match as close as possible to the true match as close as possible to the true distribution which we do not really distrib…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 61: 19 points in the latin space which is also given in this formula. given in this formula. given in this formula. I I don't want to stress too much on how I I don't want to stress too much on how I I don't…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 62: 21 the visualization before where we saw that it's like finding or that it's like finding or that it's like finding or sampling for from a very giant ocean and sampling for from a very giant ocean and sa…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 63: 17 that point being real? And this actually does not even make use And this actually does not even make use And this actually does not even make use of the encoder. This is all decoder of the encoder. Th…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 64: 26 Now the question is that what is the Now the question is that what is the elbow objective? [snorts] elbow objective? [snorts] elbow objective? [snorts] Well to understand this let's understand Well to…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 65: 25 I will not get very mathematical here and I will not focus too much on the and I will not focus too much on the and I will not focus too much on the derivation but I will provide a link in derivation …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 66: 24 So this essentially says that whatever image you're looking at the end whatever image you're looking at the end whatever image you're looking at the end is that image is that image is that image uh as…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 67: 33 in any fashion but in more realistic settings the in more realistic settings the in more realistic settings the distribution is more likely to be distribution is more likely to be distribution is more…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 68: 30 will be average in terms of the will be average in terms of the slantness, in terms of the neatness etc. slantness, in terms of the neatness etc. slantness, in terms of the neatness etc. So we are So …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 69: 32 how likely is the reconstructed um output. um output. um output. What is the probability of that? And we What is the probability of that? And we What is the probability of that? And we want to maximiz…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 70: 35 our encoder distribution to stay as close as possible to a gshian which close as possible to a gshian which close as possible to a gshian which comes from our human intuition that comes from our human…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 71: 38 want to compress the earth cat into something which captures the essence of something which captures the essence of something which captures the essence of the cat. We want to send a secret recipe the…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 72: 30 objective and we try to maximize the elbow. elbow. elbow. So since the true objective is always So since the true objective is always So since the true objective is always greater than the elbow, maxi…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 73: 27 want to move our code to the yellow distribution. We want to move it as distribution. We want to move it as distribution. We want to move it as close as possible to the normal close as possible to the…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 74: 23 what remains and it makes sense from an what remains and it makes sense from an intuitive standpoint because later intuitive standpoint because later intuitive standpoint because later people forget t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 75: 20 going to come together very nicely in this last practical example. Our task is this last practical example. Our task is this last practical example. Our task is to train a variation autoenccoder to to…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 76: 21 hidden factors which we have absolutely hidden factors which we have absolutely no clue what they are but the encoder is no clue what they are but the encoder is no clue what they are but the encoder …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 77: 13 So remember our decoder setup looks like this. It's it it needs to take the this. It's it it needs to take the this. It's it it needs to take the sample from the latent space and then it sample from t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 78: 10 So it should take the latent vector and So it should take the latent vector and it should generate the reconstructed it should generate the reconstructed it should generate the reconstructed image. Ri…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 79: 07 first machine do? The first machine actually does the reverse. It takes the actually does the reverse. It takes the actually does the reverse. It takes the image as the input, the real image, and imag…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 80: 13 region which has the mu of.5 comma.5 and region which has the mu of.5 comma.5 and the sigma of 2a 2. Right? So you need the sigma of 2a 2. Right? So you need the sigma of 2a 2. Right? So you need two …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 81: 01 the GitHub repo in the description the GitHub repo in the description section. section. section. [snorts] [snorts] Okay. So overall the encoder decoder Okay. So overall the encoder decoder Okay. So ov…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 81: 56 So this is our entire encoder decoder So this is our entire encoder decoder architecture whatever we discussed as architecture whatever we discussed as architecture whatever we discussed as this machi…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 82: 51 This compares every pixel of the input with the output. If the original pixel with the output. If the original pixel with the output. If the original pixel is black and the VA predicts white, the is b…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 83: 46 zero and a standard deviation of one. To ensure that the mean is zero, we add a ensure that the mean is zero, we add a ensure that the mean is zero, we add a penalty if the mean deviates from zero. pe…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 84: 40 because log of a small value is a big uh number and there is a negative sign number and there is a negative sign number and there is a negative sign associated with it. So this will always associated …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 85: 34 the uh description. But first I just want to want to want to describe everything that we have describe everything that we have describe everything that we have conceptually described in a mathematical…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 86: 26 for longer duration maybe we'll get an for longer duration maybe we'll get an even better distribution which is even better distribution which is even better distribution which is centered at zero and…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 87: 21 this training proceeds and this is the this training proceeds and this is the quality of the reconstructions because quality of the reconstructions because quality of the reconstructions because we ha…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 88: 15 latin space uh you see that the it's it's kind of blur in some cases which it's kind of blur in some cases which it's kind of blur in some cases which we'll come to in in in some time but you we'll co…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 89: 05 is that it it produces blurry outputs and there is a very nice reason for it. and there is a very nice reason for it. and there is a very nice reason for it. I will share the link in the description I…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 89: 57 applications I have seen and uh there are encoders and distribute and and are encoders and distribute and and are encoders and distribute and and decoders right so you also have decoders right so you …