概念

Tweedie's Formula

— 给我 noisy $y$ 和 score,我直接告诉你最优去噪结果。

很多人第一次见 Tweedie's formula 会以为它是关于贝叶斯统计的某个抽象推论——什么"后验均值的封闭形式"、"指数族的共轭技巧"——听起来像统计系研究生的练习题。这种印象对一半,错一半。对的是它确实出自 1956 年的统计学传统(Tweedie 在伦敦皇家学会的会议上提到,Robbins 同年独立给出经验贝叶斯版本);错的是它在 diffusion model 里根本不是抽象统计——它是一条可执行的去噪指令:拿到一张含噪图 $y$、再问一下"score 指向哪里",就能直接算出"最佳的干净图猜测" $\mathbb{E}[x\mid y]$。Tweedie 不在告诉你"后验分布存在",它在告诉你如何用一行公式把后验均值算出来。

Vizuara《Principles of Diffusion Models》Lecture 6 的 Dr. Raj 用一个极朴素的比喻把这件事串起来——他先把 score 比作指南针:

「A score is like a compass which points towards the data. ... if you set out with a compass in the data field you will reach the true data samples as much as possible.」—— Vizuara L6(@01:15)

这个比喻在 score matching 那一讲已经反复出现:在数据流形外面的任何一点,score $\nabla \log p$ 都是一只指向密集数据区的箭头。Tweedie's formula 把这只箭头定量化——它告诉你箭头到底要走多远:从 $y$ 出发,沿 score 方向走 $\sigma^2$ 这么长,就刚好落到"后验均值"上。也就是说,score 不只是方向,它是带尺度的去噪向量。这一点很容易被滑过去:在 score matching 一章里你学到的可能只是"score 指向数据",没人告诉你它的幅值有什么物理意义;Tweedie 把这个空缺补上了——幅值就是"每单位方差需要的去噪位移"。当你把 score 网络的输出乘上 $\sigma^2$,你不是在做某种工程上的缩放,你就是在算后验均值。Vizuara 的另一处比喻把这个尺度感讲得更生动——讲师把 score 训练里的目标说成"创建黑洞 / sink":

「This means that we are creating sinks in the data. ... if you go near a black hole, you are pulled inwards.」—— Vizuara L6(@05:30)

Tweedie 给的就是黑洞的引力公式:你离数据流形多远(由 score 的大小衡量)、噪声方差多大($\sigma^2$ 决定你能被"拉回去"的力度),合起来就是一步落地的去噪位移 $y + \sigma^2 \nabla_y \log p(y)$。在训练里这件事的对应物是:score 网络的输出,本质上就是去噪残差除以方差——也就是说,扩散里你训练的 $s_\theta$ 在功能上完全等价于一个"预测干净图减去含噪图"的去噪器。这是 diffusion model 里最关键、最容易被忽略的统一关系,也是为什么"score matching"和"denoising"这两条看似独立的研究路线,最终被证明是同一件事。

🟡 教材外补充:讲师在 Lecture 6 里把 Tweedie 串到 denoising score matching 的动机里——他先讲了 Hyvärinen 2005 的 implicit score matching 在 $n^2$ 复杂度上的死结("to calculate the trace we need to calculate all these elements in the matrix"),再引入 Vincent 2011 的 denoising score matching 才"突破封锁"。Tweedie 是这条工程线路背后的理论钥匙:之所以 denoising 等价于 score matching,正是因为 Tweedie 已经证明二者只差一个 $\sigma^2$ 的缩放。你可以把 Tweedie 理解为把"为什么训练去噪器能学到 score"这件事从"经验观察"提升为"代数恒等式"的那把锁。从历史脉络看,Tweedie 1956 和 Robbins 1956 的工作几乎是同时独立提出的(两人都在伦敦皇家学会的统计讨论里给出后验均值与边缘 score 的关系),但这条公式在 50 年里几乎只在经验贝叶斯的统计学界流传,直到 Vincent 2011 把它和 score matching 串起来、再到 Song & Ermon 2019 把它推向 diffusion 模型的中心,才被深度生成模型社区重新发现。这是一个典型的"统计学早已经有答案,工程界用了半个世纪才接上"的故事——Tweedie 自己当年提出这条公式时根本不可能想到,60 多年后它会成为生成模型最热的一行公式。

主公式与几何含义

设观测模型为高斯噪声通道:

$$y = x + \sigma \epsilon, \qquad x \sim p(x), \quad \epsilon \sim \mathcal{N}(0, I).$$

Tweedie's formula 给出后验均值的封闭式:

$$\boxed{\;\mathbb{E}[x \mid y] \;=\; y \;+\; \sigma^2\, \nabla_y \log p(y).\;}$$

Why:右边只用到两样东西——含噪观测 $y$ 本身、以及噪声边缘分布 $p(y)$ 的 score。你不需要知道先验 $p(x)$ 的样子,不需要知道似然 $p(y\mid x)$ 的细节(除了它是高斯);只要能估出 $\nabla_y \log p(y)$,就能算后验均值。这一点在 diffusion 里至关重要:因为 score 网络恰好就是在估 $\nabla_y \log p(y)$。

几何解读:把它读作"从 $y$ 出发的最佳一步去噪向量"。score $\nabla_y \log p(y)$ 是单位方向(朝数据密集区),$\sigma^2$ 是步长。$\sigma$ 越大(噪声越重)→ 步长越长(要走得越远才能回到流形);$\sigma \to 0$ → 步长收缩为 0,因为 $y$ 已经接近干净数据,根本不需要修正。这条"步长与噪声水平挂钩"的几何特性正是 Tweedie 的精髓。

数据流形 supp(p(x)) y(含噪观测) y = x + σε σ² · ∇_y log p(y) score 方向 × 方差 = 去噪位移 E[x | y] 后验均值(已落到流形) 真实 x(不可见,未必等于 E[x|y]) Tweedie:y + σ²·score = E[x | y] 从含噪点出发,沿 score 走 σ² 步,落在后验均值上 y(输入) σ²·score(位移) E[x|y](输出) 数据流形

推导:把 score 写成"对含噪边缘的对数梯度"

含噪边缘分布 $p(y)$ 是干净先验 $p(x)$ 经过高斯卷积得到的:

$$p(y) \;=\; \int p(y \mid x)\, p(x)\, dx \;=\; \int \mathcal{N}(y; x, \sigma^2 I)\, p(x)\, dx.$$

Why:这是边缘化 — 把所有可能的"干净 $x$"按其先验加权求和。它对应工程里的"forward process":把数据集 $\{x_i\}$ 各加一个 $\sigma$ 高斯,得到含噪样本集。

对 $y$ 求对数梯度:

$$\nabla_y \log p(y) \;=\; \frac{\nabla_y p(y)}{p(y)} \;=\; \frac{\int \nabla_y \mathcal{N}(y; x, \sigma^2 I)\, p(x)\, dx}{\int \mathcal{N}(y; x, \sigma^2 I)\, p(x)\, dx}.$$

Why:第一步是 $\nabla \log f = \nabla f / f$;第二步可以把 $\nabla_y$ 移进积分(被积分函数对 $y$ 光滑),$p(x)$ 不含 $y$ 所以保留不变。

关键代数:高斯密度对 $y$ 的梯度有一个简洁恒等式

$$\nabla_y \mathcal{N}(y; x, \sigma^2 I) \;=\; -\frac{y - x}{\sigma^2}\, \mathcal{N}(y; x, \sigma^2 I).$$

Why:直接对 $\exp(-\|y-x\|^2 / 2\sigma^2)$ 求导,链式法则把 $-(y-x)/\sigma^2$ 拽出来,剩下的密度本身没变。这是 Tweedie 全部魔法的唯一非平凡步骤。

代回去:

$$\nabla_y \log p(y) \;=\; \frac{\int -\frac{y - x}{\sigma^2}\, \mathcal{N}(y; x, \sigma^2 I)\, p(x)\, dx}{\int \mathcal{N}(y; x, \sigma^2 I)\, p(x)\, dx} \;=\; -\frac{1}{\sigma^2}\, \mathbb{E}_{x \sim p(x \mid y)} \big[ y - x \big].$$

Why:分母正好是 $p(y)$;分子里的积分(除掉 $-1/\sigma^2$)正好是 $\int (y-x)\, p(y\mid x)\, p(x)\, dx / p(y) = \mathbb{E}[y - x \mid y]$(用 Bayes:$p(x\mid y) = p(y\mid x)p(x)/p(y)$)。这就把"score"重写成"后验期望残差"。

因为 $y$ 在期望里是常数,$\mathbb{E}[y - x \mid y] = y - \mathbb{E}[x \mid y]$。整理:

$$\sigma^2\, \nabla_y \log p(y) \;=\; \mathbb{E}[x \mid y] \;-\; y \quad\Longleftrightarrow\quad \mathbb{E}[x \mid y] \;=\; y + \sigma^2 \nabla_y \log p(y).$$

Why:这一行就是Tweedie's formula。整个证明只用到了:(1) 高斯密度对 $y$ 求导有一个干净的代数形式;(2) Bayes 规则在期望里的一次置换。没有任何工具更复杂——这也是为什么 Tweedie 1956 一篇短信函里就把它讲完了。

反向恒等:score = 缩放的去噪残差

把上式两边除以 $\sigma^2$,整理出反向关系:

$$\boxed{\;\nabla_y \log p(y) \;=\; \frac{\mathbb{E}[x \mid y] - y}{\sigma^2}.\;}$$

Why:这一行的意义被严重低估——它说"score" 不是一个抽象的对数密度梯度,它就是"最优去噪向量"除以方差。如果你训了一个 $D_\theta(y) \approx \mathbb{E}[x\mid y]$ 的去噪器,那 $(D_\theta(y) - y)/\sigma^2$ 立刻等价于一个 score 估计器;反之亦然。这就是 Vincent 2011 "Denoising Score Matching" 的全部数学根基,也是 diffusion 训练里"预测噪声 / 预测干净图 / 预测 score"三种 parameterization 等价的根本原因。换句话说,diffusion 社区里争论了很多年的"$\epsilon$-prediction 还是 $x_0$-prediction 还是 score-prediction 更好?"——从 Tweedie 的角度看,它们是同一件事的三种坐标系,三个目标网络在做训练时学到的内容是完全等价的,只是输出在不同 $\sigma$ 下的数值尺度不同。EDM (Karras 2022) 论文里那张著名的"统一参数化"表格,每一行就是把同一份信息搬到不同坐标系下;选哪一种,纯粹由数值稳定性、损失加权策略决定,不由理论决定。

在 diffusion 里的具体用法

在 DDPM 的 forward process 里,第 $t$ 步的 marginal 是高斯:$x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\, \epsilon$,可以等价改写成"$y = x + \sigma_t \epsilon$"形式(其中 $y = x_t / \sqrt{\bar\alpha_t}$,$\sigma_t^2 = (1-\bar\alpha_t)/\bar\alpha_t$)。Tweedie 立刻给出每一步的最优 $x_0$ 估计:

$$\hat x_0 \;=\; \mathbb{E}[x_0 \mid x_t] \;=\; \frac{1}{\sqrt{\bar\alpha_t}}\Big(x_t + (1-\bar\alpha_t)\, \nabla_{x_t}\log p_t(x_t)\Big).$$

Why:把 $y$ 替换成 $x_t/\sqrt{\bar\alpha_t}$,把 $\sigma^2$ 替换成 $(1-\bar\alpha_t)/\bar\alpha_t$,整理即得(一些版本里因为缩放约定不同会差一个 $\sqrt{\bar\alpha_t}$ 因子,但实质相同)。注意:每一步反向,扩散网络其实都在算 Tweedie 修正——你以为它在"预测噪声",但等价于"用 score 把含噪图拉向后验均值"。

⚙️ 工程意义:$\epsilon$-parameterization 的网络 $\epsilon_\theta(x_t,t)$ 与 score 的关系是 $\nabla_{x_t} \log p_t(x_t) = -\epsilon_\theta(x_t,t)/\sqrt{1-\bar\alpha_t}$。代回 Tweedie,立即得到 DDPM 里那个著名的 "predicted $x_0$" 公式:$\hat x_0 = (x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta) / \sqrt{\bar\alpha_t}$。这不是巧合,是 Tweedie 在 $\epsilon$-参数化下的直接翻译。DDIM 反向更新公式 $x_{t-1} = \sqrt{\bar\alpha_{t-1}}\hat x_0 + \sqrt{1-\bar\alpha_{t-1}}\epsilon$ 里那个 $\hat x_0$,就是 Tweedie 给的后验均值。

🟡 教材外补充:Tweedie's formula 在更广的指数族里也有一个推广(Robbins–Tweedie),形式是 $\mathbb{E}[\theta \mid y] = y + \frac{d}{dy}\log m(y)$,其中 $m(y)$ 是观测的边缘 CDF。但 diffusion model 几乎只用高斯版本——这是因为 forward process 被设计成必须是高斯加噪(这样 marginal 才是干净的高斯卷积,Tweedie 才能直接套用)。一旦你换成 Poisson 通道(如低剂量 CT 重建)或 binomial 通道(如离散 token),就要换对应版本的 Tweedie 推广;这也是为什么"diffusion + 离散 token"(如 SEDD、masked diffusion)需要从头重新推导损失函数与采样规则——它们不能"复用 DDPM 的公式换个分布",因为整条 Tweedie 链条在非高斯通道上根本不长那样。

三种身份的统一:score / denoiser / noise predictor

读到这里你应该已经能在脑子里把三种网络互相翻译。给定含噪 $y$(在 diffusion 里是 $x_t$):

(1) Score 网络 $s_\theta(y) \approx \nabla_y \log p(y)$ — 直接学对数密度梯度,用 denoising score matching 损失。
(2) 去噪器 $D_\theta(y) \approx \mathbb{E}[x \mid y]$ — 直接学后验均值,损失是 $\|D_\theta(y) - x\|^2$ 在 $\{y, x\}$ 配对上求期望。
(3) 噪声预测器 $\epsilon_\theta(y) \approx (y - x)/\sigma$(或等价地 $\approx -\sigma\, \nabla_y \log p(y)$)— 学被加进去的高斯噪声本身,DDPM 默认参数化。

三者间的代数换算(设 $y = x + \sigma\epsilon$):

$$D_\theta(y) \;=\; y + \sigma^2\, s_\theta(y) \;=\; y - \sigma\, \epsilon_\theta(y).$$ $$s_\theta(y) \;=\; \frac{D_\theta(y) - y}{\sigma^2} \;=\; -\frac{\epsilon_\theta(y)}{\sigma}.$$

Why:上面两行是 Tweedie 主公式的直接代数重排,再加上"噪声 = 含噪图 − 干净图除以 $\sigma$"的定义。它们告诉你:你训练的是哪一个网络,本质上不重要——三者只差线性变换。但工程上选 $\epsilon_\theta$ 还是 $D_\theta$ 还是 $s_\theta$,会改变损失函数在不同 $\sigma$ 下的权重,进而影响哪些噪声尺度被学得更准。Karras 2022 EDM 论文里花了大量篇幅证明:直接学 $D_\theta$ 在数值上最稳,因为它的输出尺度 $\sim O(1)$ 不随 $\sigma$ 变化;而 $\epsilon_\theta$ 在 $\sigma \to 0$ 时输出尺度仍是 $O(1)$ 但实际"信号"很弱,$s_\theta$ 在 $\sigma \to 0$ 时输出尺度爆炸到 $O(1/\sigma)$。这就是为什么 EDM 选择 $D_\theta$、Stable Diffusion 选择 $\epsilon_\theta$(兼容 DDPM 老训练栈)、early SDE 论文选择 $s_\theta$——选择背后是数值稳定性的工程权衡,而不是建模能力的差异。

按钮依次展开 Tweedie 的推导链。注意看什么不变量:每一步的右边都只用到含噪观测 $y$ 和 score $\nabla_y \log p(y)$,干净先验 $p(x)$ 始终没出现在最终公式里——这就是为什么 score 网络足够把整个去噪流程封装起来。

$p(y) = \int \mathcal{N}(y; x, \sigma^2 I)\, p(x)\, dx$ — 含噪边缘 = 干净先验经过高斯卷积。
$\nabla_y \log p(y) = \dfrac{\int \nabla_y \mathcal{N}(y;x,\sigma^2 I)\, p(x)\, dx}{\int \mathcal{N}(y;x,\sigma^2 I)\, p(x)\, dx}$ — 把 $\nabla_y$ 移进积分($p(x)$ 与 $y$ 无关)。
$\nabla_y \mathcal{N}(y;x,\sigma^2 I) = -\dfrac{y-x}{\sigma^2}\, \mathcal{N}(y;x,\sigma^2 I)$ — 高斯指数对 $y$ 求导的唯一非平凡一步。
$\nabla_y \log p(y) = -\dfrac{1}{\sigma^2}\, \mathbb{E}_{x \sim p(x\mid y)}[y - x]$ — 分子分母按 Bayes 重写成后验期望。
$\mathbb{E}[x\mid y] = y + \sigma^2 \nabla_y \log p(y)$ — Tweedie 主公式:含噪观测 + 方差 × score = 后验均值。
$\nabla_y \log p(y) = \dfrac{\mathbb{E}[x\mid y] - y}{\sigma^2}$ — 反过来读:score 就是"去噪残差除以方差"。

手算一例:1D 高斯先验

为了让代数闭合可验,取 $p(x) = \mathcal{N}(x; 0, 1)$,噪声水平 $\sigma = 1$。先做解析计算作为对照:含噪边缘 $p(y) = \mathcal{N}(y; 0, 2)$(两个独立高斯之和的方差是 $1+1=2$)。它的 score 是

$$\nabla_y \log p(y) \;=\; -\frac{y}{2}.$$

Tweedie 给出后验均值:

$$\mathbb{E}[x \mid y] \;=\; y + \sigma^2 \nabla_y \log p(y) \;=\; y + 1 \cdot \Big(-\frac{y}{2}\Big) \;=\; \frac{y}{2}.$$

这正好等于经典高斯-高斯共轭后验的均值(先验方差 1、似然方差 1,后验均值 = $y \cdot 1/(1+1) = y/2$)。两条独立路径给出一样的结果——Tweedie 没有作弊。

带数字:取 $y = 3$,则 $\mathbb{E}[x\mid 3] = 3/2 = 1.5$。几何上:你看到一个含噪观测落在 $y=3$,先验告诉你 $x$ 大概率在 0 附近;Tweedie 把 $y=3$ 沿 score 方向(朝 0)拉,每单位方差走 $-y/2$ 这么远,最终落在 $1.5$——介于"先验中心 0"和"观测 3"之间,符合 shrinkage 直觉。把 $\sigma$ 换成 0.1(噪声很轻):$p(y) = \mathcal{N}(0, 1.01)$,score $\approx -y/1.01$,Tweedie 给 $\mathbb{E}[x\mid 3] \approx 3 - 0.01 \cdot 3/1.01 \approx 2.97$——几乎不修正,因为观测本身已经很可信。再换 $\sigma=10$:$p(y) = \mathcal{N}(0, 101)$,score $\approx -y/101$,Tweedie 给 $\mathbb{E}[x\mid 3] \approx 3 - 100 \cdot 3/101 \approx 0.03$——几乎全靠先验,因为观测被噪声淹没。三个数字(2.97, 1.5, 0.03)把"$\sigma$ 决定 shrinkage 强度"刻画得很清楚。

这个 toy 同时演示了 Tweedie 的另一个性质:当 $\sigma$ 很小(噪声轻),Tweedie 修正几乎是恒等映射 $\mathbb{E}[x\mid y] \approx y$——也就是 diffusion 反向链的末端步骤(接近 $t=0$、$\sigma_t$ 很小)几乎不需要修正。这也解释了为什么 DDIM 在 $T$ 步压缩到 50 步、20 步、甚至 4 步仍然能产出像样图:靠后的反向步本来就接近"恒等映射 + 一点点 Tweedie 修正",省掉中间几十步对最终样本质量影响有限。反过来,靠近 $t=T$($\sigma_t$ 很大)的反向步必须老老实实跑——那里 Tweedie 给的位移又大又关键。这是采样调度策略(比如 EDM 的 $\sigma$ schedule、DDIM 的 $\tau$ subset)背后的几何直觉。

常见误区

  1. "Tweedie 给的是后验众数 / MAP 估计"。反例:取 $p(x)$ 为双峰分布(两个高斯混合),$y$ 落在两峰中间。MAP 会偏向较高峰的中心;Tweedie 给的均值则可能落在两峰之间的低密度区——它是后验期望,不是后验模式。在 diffusion 里这一点解释了一个常见现象:在 $x_t$ 含噪很重的早期步骤里,$\hat x_0 = \mathbb{E}[x_0 \mid x_t]$ 经常看起来是"模糊平均图"——比如对 MNIST 是各位数字的均值脸——因为 Tweedie 取的就是期望,不是"最像的那张图"。要拿到清晰图,必须迭代多步反向采样,把方差逐渐降下来。
  2. "Tweedie 只在高斯先验下成立"。反例:上面推导没用到 $p(x)$ 的任何形式——它对任意先验 $p(x)$(混合高斯、Laplace、自然图像分布……)都成立。真正的条件是噪声通道是高斯(即似然 $p(y\mid x) = \mathcal{N}(y;x,\sigma^2 I)$)。如果噪声是 Poisson(如低剂量 CT 探测器读数)或 Bernoulli(如离散 token mask),主公式形式变化,需要 Robbins–Tweedie 的指数族推广版本——这就是 SEDD(score entropy discrete diffusion)等离散扩散模型需要重新推导损失函数的根本原因。
  3. "score 是『梯度估计器』,Tweedie 是『另一个工具』"。反例:这条公式 $\nabla_y \log p(y) = (\mathbb{E}[x\mid y] - y)/\sigma^2$ 说明二者是同一个对象的两种参数化。如果你训一个网络去预测 score $s_\theta(y) \approx \nabla_y \log p(y)$ 和你训一个网络去预测干净图 $D_\theta(y) \approx \mathbb{E}[x\mid y]$,两个网络的输出只差一个常数缩放 $D_\theta(y) = y + \sigma^2 s_\theta(y)$。这就是为什么 EDM (Karras 2022) 的 "$D_\theta$-parameterization" 在数值上比 $\epsilon$-参数化更稳:直接预测 $\mathbb{E}[x\mid y]$ 的网络输出尺度不随 $\sigma$ 爆炸,而预测 score 或 $\epsilon$ 的输出在 $\sigma \to 0$ 时尺度会奇异。本质上是同一份信息,但工程数值表现差别很大。一个直接的检验方法:拿一个训好的 DDPM 模型,把它的 $\epsilon_\theta$ 输出按 $-\epsilon_\theta/\sqrt{1-\bar\alpha_t}$ 换算成 score,再用 Tweedie 主公式算后验均值,应该和原始 DDPM 的 "predicted $x_0$" 公式给出完全一样的数字(误差只在浮点精度量级)。如果差距很大,说明你某处掉了一个 $\sqrt{\bar\alpha_t}$ 因子——这正是 diffusion 代码 review 时最常踩的坑。

相关概念

  • Score Matching — Tweedie 是它的理论钥匙:denoising score matching 之所以成立,正因为 Tweedie 证明 score = 去噪残差除以方差。
  • DDPM — DDPM 反向步里的 "predicted $x_0$" 公式 $\hat x_0 = (x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta)/\sqrt{\bar\alpha_t}$ 就是 Tweedie 在 $\epsilon$-参数化下的直接翻译。

来源

  1. diffusion-principles-vizuara · L6 · "compass which points towards the data" · (1:15)
  2. diffusion-principles-vizuara · L6 · "creating sinks in the data, like a black hole" · (5:07)
  3. diffusion-principles-vizuara · L6 · 引出 denoising score matching 与 Tweedie 路径 · (12:37)
查看 46 条课程原文出处
CourseLectureTimeExcerpt
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 00: 03 Hello everyone, welcome to the next Hello everyone, welcome to the next lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion models. di…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 01: 15 in space and you go in that space with a compass which tells you which direction compass which tells you which direction compass which tells you which direction should I move so that I move closer to …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 02: 45 probability distribution. Right? So we do not know this term. do not know this term. do not know this term. In supervised machine learning, we often In supervised machine learning, we often In supervi…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 03: 54 requirement of this entire probability requirement of this entire probability distribution which we have no idea distribution which we have no idea distribution which we have no idea about. about. abo…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 05: 07 data which is available to us which is what makes it incredibly powerful. So we are grateful to this author to So we are grateful to this author to have made the stream of score matching have made the…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 06: 08 is close to the sink is pulled inwards. is close to the sink is pulled inwards. It's like a black hole. If you go near a It's like a black hole. If you go near a It's like a black hole. If you go near…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 06: 51 Now assume that you you are close to the data. Assume that you are the data data. Assume that you are the data data. Assume that you are the data itself. Then you don't need to move itself. Then you d…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 07: 48 for this training data at every single for this training data at every single point in the space. You can see the point in the space. You can see the point in the space. You can see the score function…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 08: 50 So just if you look at this visually you can see that wherever I start in the can see that wherever I start in the can see that wherever I start in the field and I follow the score function I field an…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 10: 02 function? So and even we have looked at one example where uh the loss function one example where uh the loss function one example where uh the loss function actually works. actually works. actually wo…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 11: 19 the diagonal elements. You can't the diagonal elements. You can't individually calculate the diagonal individually calculate the diagonal individually calculate the diagonal elements. elements. elemen…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 12: 37 in in real life use cases. And that is why for a lot of for many years actually why for a lot of for many years actually why for a lot of for many years actually uh uh uh the score matching technique …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 14: 00 the same word dn noising right and in diffusion that made sense because we diffusion that made sense because we diffusion that made sense because we started from a noisy image and then we started from…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 15: 10 These three papers are going to form the foundation of the theory of denoising foundation of the theory of denoising foundation of the theory of denoising score matching. Okay. So this technique was i…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 16: 22 formulation which was trackable. But that did not make sense because the that did not make sense because the that did not make sense because the complexity of computations squared complexity of comput…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 17: 42 and you do have access to some of them. So let's say you have access to 15 or 20 So let's say you have access to 15 or 20 So let's say you have access to 15 or 20 of these magnets but many others are …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 18: 48 nearest magnet is pulling me. If I take a point here I could say that this is a point here I could say that this is a point here I could say that this is the direction in which the nearest the directi…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 19: 45 uh I will I will superimpose the uh I will I will superimpose the magnetic fields for all these different magnetic fields for all these different magnetic fields for all these different magnets. magne…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 20: 51 We take any magnet on the table and we We take any magnet on the table and we flick it with a hand just like this. I'm flick it with a hand just like this. I'm flick it with a hand just like this. I'm…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 21: 49 the student, look, this is the ball's new location. new location. new location. and we hide the original uh magnet and we hide the original uh magnet and we hide the original uh magnet location. So we…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 23: 02 student has no idea where this new ball position came from. But we know new ball position came from. But we know new ball position came from. But we know where it came from. where it came from. where …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 24: 02 balls, all these magnets that you know balls, all these magnets that you know that the magnets are placed there. You that the magnets are placed there. You that the magnets are placed there. You place…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 25: 02 this point should be should move so that it goes back to the original position it goes back to the original position it goes back to the original position and that is exactly the direction of the and …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 26: 19 to a new location. to a new location. So the noisy new spot is represented as So the noisy new spot is represented as So the noisy new spot is represented as xhat. xhat. xhat. Uh this is I have actual…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 27: 22 score does exactly the same thing. The score tells you the direction score tells you the direction score tells you the direction to the actual data in the space. And our to the actual data in the spac…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 28: 42 of mistakes but the feedback is given of mistakes but the feedback is given from this. This is the true direction from this. This is the true direction from this. This is the true direction where whic…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 29: 59 It is not like your true It is not like your true probability distribution of data which probability distribution of data which probability distribution of data which is not tractable but it is tracta…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 31: 00 given any point in space how can I pull this point so that it how can I pull this point so that it how can I pull this point so that it becomes as close as possible to your becomes as close as possibl…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 32: 02 really making sense. I have an excellent really making sense. I have an excellent explanation to that which will come soon explanation to that which will come soon explanation to that which will come …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 33: 05 perturvation to the data. We can simplify the score function which we simplify the score function which we simplify the score function which we want to learn as follows. want to learn as follows. want…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 34: 14 uh the mean of this pixel is centered at uh the mean of this pixel is centered at 0.5 because the value is 0.5 but then you add a small noise with a but then you add a small noise with a standard devi…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 35: 36 And uh in fact this can be represented this can be represented this can be represented using a standard normal distribution using a standard normal distribution using a standard normal distribution li…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 36: 55 expression for this probability distribution. distribution. distribution. And uh because it is tractable once we And uh because it is tractable once we And uh because it is tractable once we know the …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 37: 56 anything is whatever is there in the exponential itself. So you get this at exponential itself. So you get this at exponential itself. So you get this at the bottom and then when you take the bottom a…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 39: 07 you you you might have thought why didn't I guess it before because it it didn't I guess it before because it it didn't I guess it before because it it should be the vector which points from should be…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 40: 18 predict we are trying to match the noise the noise vector which has been added in the noise vector which has been added in the noise vector which has been added in the to the clean data. the to the cl…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 41: 22 which means that the density is maximum near somewhere around this maybe minus4 near somewhere around this maybe minus4 near somewhere around this maybe minus4 and the density is maximum somewhere and…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 42: 55 So this is the neural network that we So this is the neural network that we define and the loss function is the main define and the loss function is the main define and the loss function is the main s…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 43: 51 Okay. And now what is the target score? The target score is something which The target score is something which The target score is something which you're trying to estimate which is minus you're tryi…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 44: 59 And uh finally you see the learned score And uh finally you see the learned score function which looks something like function which looks something like function which looks something like this. Now …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 45: 56 magnet. magnet. Similarly, if you are somewhere on the Similarly, if you are somewhere on the Similarly, if you are somewhere on the right, uh the score is negative because right, uh the score is nega…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 46: 42 left you reach magnet number one. If you are on the far right, you reach magnet are on the far right, you reach magnet are on the far right, you reach magnet number two. And if you are anywhere in num…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 47: 32 We first add a noise and then we try to We first add a noise and then we try to predict that noise. predict that noise. predict that noise. And in that process of predicting the And in that process of…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 48: 34 us away from the local minima so that we explore the global minimas as well. explore the global minimas as well. explore the global minimas as well. So uh this is what we get So uh this is what we get…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 49: 35 much of a difference. Something might be much of a difference. Something might be happening in the 3D space right that happening in the 3D space right that happening in the 3D space right that adding …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 50: 26 Okay. So this is it for dnoising score matching and uh in the next part matching and uh in the next part matching and uh in the next part uh I will explain about noise condition uh I will explain abou…