概念 · Module 3: Score 视角

Score Matching

— 把"训练 score 网络"变成可计算的回归问题。

很多人第一眼看到 score matching 会想:「不就是个 MSE 回归吗?把 $s_\theta(x)$ 拟合到 $\nabla_x \log p_\text{data}(x)$ 上就完事。」这个直觉的 致命错误 是:回归的目标本身是未知的——$p_\text{data}$ 我们只能从数据集里看到样本,写不出解析式;它的归一化常数 $Z$ 在高维下是天文级积分;$\nabla_x \log p_\text{data}(x)$ 自然也拿不到。所以 score matching 的精髓不在"回归"两个字上,而在于用一个数学技巧把这个无法测量的目标从损失函数里消掉——这个技巧就是分部积分。当你看完整套推导会发现:训练时根本不需要碰 $p_\text{data}$ 的真实 score,只需要数据样本本身、一点高斯噪声、以及网络自己的反向传播。这是 Hyvärinen 在 2005 年的关键洞察,也是 Vincent 在 2011 年把它推到 deep learning 工程可用形态的起点。

「A score is like a compass which points towards the data. So let's say you have data lying around in space and you go in that space with a [compass]...」—— Vizuara L6 @00:03

Vizuara 的 Dr. Raj 在 Lecture 6 用了一个非常具体的 「磁铁桌子」 比喻——想象一张大桌子上散落着许多磁铁(每个磁铁是一个数据点),你只能看见其中几个,绝大多数被布盖住了。任务是:站在桌面上任意位置,感受最近的磁铁把你往哪个方向拉。这个"拉力方向场"就是 score $\nabla_x \log p_\text{data}(x)$;它在数据密度高的地方指得最强、指得最准,在数据稀疏区域则模糊、不稳定。问题的难点是:磁铁的真实位置(即 $p_\text{data}$ 的形状)你不知道,可你能做一件事——把某个真实磁铁稍微推一下,让它带上一点高斯位移,然后训练一个模型「猜出原始磁铁在哪个方向」。这就是 denoising score matching 的入口,也是 diffusion 训练的核心套路。

「Imagine there are a bunch of magnets kept on a table and you have access to only a few of those magnets. Many of the other magnets are hidden from you.」—— Vizuara L7 @00:05

CMU 10799 的讲师在 Lecture 4 给了另一个互补视角:扩散模型的前向过程就是不断给数据加高斯噪声,直到变成纯噪声;而反向过程需要在每个噪声水平 $t$ 上知道 $\nabla_{x_t} \log p_t(x_t)$——这正是 score。能把这个量学下来,sampling 就是"顺着 score 走 + 加点随机扰动"(Langevin dynamics 或 reverse SDE)。换句话说:diffusion = forward 加噪 + 反向沿 score 去噪,而 score matching 就是那把让 score 变成可学习量的钥匙。在整个 ML 训练流程里,score matching 占据着"如何把一个无归一化的密度模型变成可优化目标"的核心位置——VAE 用 ELBO 绕开归一化常数,flow 用可逆映射回避它,而 score-based / diffusion 走的是第三条路:根本不去碰密度,只学密度对数的梯度。

三层目标递进:ESM → ISM → DSM

Score matching 不是某一个孤立公式,而是一条逐步消去未知量、逐步变成可工程实现的目标函数链。理解它最好的方式是按时间顺序看 Hyvärinen 2005 和 Vincent 2011 两篇论文的"刀法":每一刀都在解决前一刀留下的死结。下面三个版本分别叫 ESM(Explicit Score Matching,定义级别)、ISM(Implicit Score Matching,理论可计算)、DSM(Denoising Score Matching,实际可训练)。它们在最优解处给出相同的网络参数 $\theta^\star$,但在"训练时是否需要未知量"这件事上差距巨大。

① ESMExplicit · Hyvärinen 2005

最自然的写法:直接 MSE 比对网络输出和真实 score。

$$J_{ESM}(\theta) = \frac{1}{2}\,\mathbb{E}_{x \sim p_\text{data}}\big[\|s_\theta(x) - \nabla_x \log p_\text{data}(x)\|^2\big]$$

Why 这样写:score 是一个向量场(每个点都对应一个向量),衡量两个向量场差异最自然的方式就是逐点取欧氏距离的平方再求期望。这本质上就是把 MSE 从"标量预测"推广到"向量场预测"。

致命缺陷:右边那个 $\nabla_x \log p_\text{data}(x)$ 我们根本不知道。$p_\text{data}$ 只有样本而没有解析式;如果硬要用核密度估计(KDE)近似一下,那在低维玩具问题上还能跑通,但一进入图像维度($d \sim 10^5$),KDE 的方差就会因为维度灾难直接爆炸——具体来说,在 $d$ 维空间里你需要的样本数大约是 $O(k^d)$($k$ 是每维分辨率),而你只有几万张图。结果就是估计出的 score 比噪声还噪声,训练根本走不动。所以 ESM 只是一个数学上的定义、一个"如果我能算就能用"的理想目标,它本身不是可以拿去训练神经网络的损失。我们需要的是一个等价目标,但目标里不能出现任何未知量。Hyvärinen 在 2005 的工作就是在解决"如何把 ESM 等价改写成只含已知量的形式"这个问题。

② ISMImplicit · 分部积分

Hyvärinen 的"魔术":展开 ESM 的平方项,用分部积分把未知目标消掉。展开 $\|s_\theta - \nabla \log p_\text{data}\|^2$:

$$\|s_\theta\|^2 - 2\,s_\theta^\top \nabla \log p_\text{data} + \|\nabla \log p_\text{data}\|^2$$

最后一项与 $\theta$ 无关(常数 $C$),扔掉。关键是中间项的期望:

$$\mathbb{E}_{p_\text{data}}\big[s_\theta(x)^\top \nabla_x \log p_\text{data}(x)\big] = \int p_\text{data}(x)\, s_\theta(x)^\top \frac{\nabla p_\text{data}(x)}{p_\text{data}(x)} dx = \int s_\theta(x)^\top \nabla p_\text{data}(x)\, dx$$

对每个分量做分部积分(假设 $p_\text{data}$ 在无穷远处衰减),边界项归零,得:

$$\int s_\theta(x)^\top \nabla p_\text{data}(x)\, dx = -\int p_\text{data}(x)\, \nabla_x \cdot s_\theta(x)\, dx = -\mathbb{E}_{p_\text{data}}[\mathrm{tr}(\nabla_x s_\theta(x))]$$

合起来:

$$J_{ISM}(\theta) = \mathbb{E}_{x \sim p_\text{data}}\Big[\mathrm{tr}(\nabla_x s_\theta(x)) + \tfrac{1}{2}\|s_\theta(x)\|^2\Big] + C$$

Why 这一步是突破:原本未知的 $\nabla_x \log p_\text{data}(x)$ 完全消失了,取而代之的是网络自己的 Jacobian 矩阵的迹 $\mathrm{tr}(\nabla_x s_\theta(x)) = \sum_i \partial s_{\theta,i} / \partial x_i$——这是一个完全确定、可以用 autograd 反向传播算出来的量。换句话说:ISM 把"需要测量真实分布的 score"问题转化为"只需要看网络在数据点上的局部几何性质"。这是一个非常聪明的偷换:未知量不见了,代价是出现了一个网络自身的二阶量。整套推导的关键步骤就是分部积分时的边界项归零,这要求 $p_\text{data}$ 在无穷远处快速衰减——对于真实图像数据集这通常自动满足。这就是 Hyvärinen 2005 的核心贡献:他证明了 score matching 这件事在没有归一化常数 $Z$的条件下依然可以训练,这正是 EBM 这条研究线多年没能解决的死结。

新瓶颈:$\mathrm{tr}(\nabla_x s_\theta)$ 在 $d$ 维空间需要 $d$ 次反向传播——因为你要分别取每一维的偏导再加起来。在 MNIST($d = 784$)勉强能跑,在 CIFAR($d = 3072$)开始痛苦,在 ImageNet 256×256($d \approx 2 \times 10^5$)则完全不可行。所以 ISM 在理论上完美、在低维玩具上能用,但对深度生成模型这条路线来说仍然是死路一条。Yang Song 在 2019 年提出的 Sliced Score Matching 通过随机投影把这个 $O(d)$ 开销降到 $O(1)$,是一条改良路线;但真正打通工程的,是 Vincent 2011 的 DSM。

③ DSMDenoising · Vincent 2011

换思路:别去匹配干净数据的 score,去匹配加噪后的 score。定义噪声扰动分布

$$\tilde x = x + \sigma\epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$

则条件概率 $p_\sigma(\tilde x | x) = \mathcal{N}(\tilde x; x, \sigma^2 I)$,它的 score 是闭式解:

$$\nabla_{\tilde x} \log p_\sigma(\tilde x | x) = -\frac{\tilde x - x}{\sigma^2} = -\frac{\epsilon}{\sigma}$$

注意:上式分母是 $\sigma$(不是 $\sigma^2$),因为 $\tilde x - x = \sigma\epsilon$。Vincent 证明:

$$J_{DSM}(\theta) = \frac{1}{2}\,\mathbb{E}_{x, \tilde x}\Big[\big\|s_\theta(\tilde x) - \nabla_{\tilde x} \log p_\sigma(\tilde x | x)\big\|^2\Big]$$

与 ESM-on-marginal 等价(梯度相同,差一个常数)。代入闭式 score:

$$J_{DSM} = \frac{1}{2}\,\mathbb{E}_{x \sim p_\text{data}, \epsilon \sim \mathcal{N}(0,I)}\Big[\big\|s_\theta(\tilde x) + \tfrac{\epsilon}{\sigma}\big\|^2\Big]$$

Why 这是终极形式:训练时只需要四步——① 从数据集采一个 $x$,② 从标准高斯采一个 $\epsilon$,③ 做加法 $\tilde x = x + \sigma\epsilon$,④ 让网络回归到 $-\epsilon/\sigma$。每个样本只需要一次前向 + 一次反向传播,完全没有 Jacobian 迹、没有 $d$ 次循环、没有任何对网络结构的限制(不像 normalizing flow 必须可逆)。这是 deep learning 工程师梦寐以求的形态。

代价 / 妥协:你学到的并不是干净数据 $p_\text{data}$ 的 score,而是被 $\sigma$ 模糊后的分布 $p_\sigma(\tilde x) = \int p_\text{data}(x)\, \mathcal{N}(\tilde x; x, \sigma^2 I)\, dx$ 的 score $\nabla_{\tilde x} \log p_\sigma(\tilde x)$。这里有一个无法两全的 trade-off:当 $\sigma \to 0$ 时 $p_\sigma \to p_\text{data}$,但同时回归目标 $-\epsilon/\sigma$ 的方差爆炸为 $1/\sigma^2$,训练发散;当 $\sigma$ 太大时方差稳定,但学到的几乎是高斯本身的 score,对刻画 $p_\text{data}$ 没用。这就是为什么 NCSN/DDPM 不用单一 $\sigma$,而要用一整条 $\sigma_1 < \sigma_2 < \cdots < \sigma_T$ 的噪声梯度,让网络同时处理所有尺度——大尺度负责"从纯噪声中找到大致流形",小尺度负责"精修细节"。

统一时刻:DSM = DDPM 的 $\epsilon$-prediction

把 DSM 在每个噪声尺度 $\sigma_t$ 上都做一遍,并让网络显式接收 $t$ 作为输入:

$$\mathcal{L} = \mathbb{E}_{t, x_0, \epsilon}\Big[\big\|s_\theta(x_t, t) + \tfrac{\epsilon}{\sigma_t}\big\|^2\Big]$$

令 $\epsilon_\theta(x_t, t) := -\sigma_t \cdot s_\theta(x_t, t)$,整理得:

$$\mathcal{L} = \mathbb{E}_{t, x_0, \epsilon}\Big[\tfrac{1}{\sigma_t^2}\big\|\epsilon_\theta(x_t, t) - \epsilon\big\|^2\Big]$$

核心身份(贯穿整个扩散模型理论的"罗塞塔石碑"):DDPM 的简化损失 $L_\text{simple} = \|\epsilon_\theta(x_t, t) - \epsilon\|^2$(去掉前面的 $1/\sigma_t^2$ 权重后)与 DSM 在所有噪声尺度上求和在最优解处完全等价。这不是"看起来像"或"近似相等",而是同一个目标在不同 parameterization 下的两种写法。所以 CMU 10799 的讲师才会在 Lecture 4 用一句非常朴素的话总结整套 DDPM 训练:「at training time you literally just do regression on the noise that you're adding to your forward process.」——他不是在简化,而是在精确地陈述事实。
「at training time what you do is just literally do regression on the noise that you're adding to your in your forward process.」—— CMU 10799 L4 @03:24

这条身份是整个扩散模型理论的关键节点:原本看上去毫不相干的三套语言——变分推断(VAE / DDPM 视角下的 ELBO 下界)、能量模型(EBM 视角下的 score matching)、随机微分方程(Yang Song 的 SDE 反向过程)——在这里收敛到同一个训练目标上。这也是为什么 2020 年之后 score-based generative model 和 denoising diffusion probabilistic model 这两条最初彼此独立的研究线被认为是"同一件事的两种讲法"。CMU L4 在 chunk 915 解释为什么 Ho 2020 选择 $\epsilon$-prediction 而不是直接学反向 Gaussian 的均值 $\mu_\theta(x_t, t)$:「because doing this regression is much much easier to train than if you just learn this very complicated mean which like you know has like very complicated structures at different time steps.」学 $\epsilon$ 的方差和量级在所有时间步上几乎是一致的(因为 $\epsilon \sim \mathcal{N}(0, I)$ 本身就是标准化的),学 $\mu$ 则要面对每个 $t$ 截然不同的均值尺度(早期 $t$ 时 $\mu_t \approx x_0$,后期 $t$ 时 $\mu_t \approx 0$)——所以前者训练稳定、调参容易、最终性能更好。这也是为什么后来的工程文献几乎清一色用 $\epsilon$-prediction 而不是 $\mu$-prediction,即便它们在数学上等价。

另一个值得强调的点是 score matching 与 likelihood-based training 的关系。CMU L4 在 chunk 918 提到:autoregressive 模型用链式法则算似然「really really painful」,VAE 用 ELBO 是「surrogated loss」「kind of suboptimal」,normalizing flow 受可逆性约束。Score-based 模型走了完全不同的第三条路:不再追求估计概率密度本身,只学密度的对数梯度。表面看损失了概率信息,实际上对于生成任务(采样)这恰恰是充分信息——因为 Langevin 和 reverse SDE 都只需要 score。把"建模密度"换成"建模 score"是 score-based generative model 的根本范式转移,也是它能突破 likelihood-based 方法多年瓶颈的本质原因。

🟡 教材外补充(Sliced Score Matching):DSM 不是当年唯一一条试图替代 ISM 的路。Yang Song 在 2019 年提出 Sliced Score Matching(SSM):用一个随机方向 $v \sim \mathcal{N}(0, I)$ 把整个 Jacobian 迹问题降维成"一次方向上的 Hessian-vector product"——具体来说,把目标改为 $\mathbb{E}_{x, v}[v^\top \nabla_x s_\theta(x)\, v + \frac{1}{2}\|s_\theta(x)\|^2]$,每个样本只需要一次反向传播,开销变成 $O(1)$ 而不是 $O(d)$。SSM 对干净数据的 score 估计在统计意义上更准确,但在 diffusion 实际工程里被 DSM 完胜,原因有二:一是 DSM "顺手"学到的就是带噪 score,正好是反向 SDE/Langevin sampling 需要的那个量;二是 DSM 的训练循环里没有任何二阶量,对硬件友好、对网络结构无约束。这条历史脉络解释了为什么今天 score-based generative model 全部走的是 DSM 而不是 SSM。
🟡 教材外补充(EBM 与 score matching 的历史渊源):score matching 最早根本不是为生成模型设计的,而是为 EBM(Energy-Based Model)服务的。EBM 把概率写成 $p_\theta(x) = e^{-E_\theta(x)} / Z(\theta)$,其中 $Z(\theta) = \int e^{-E_\theta(x)} dx$ 是归一化常数(配分函数)。计算对数似然 $\log p_\theta = -E_\theta - \log Z$ 需要 $\log Z$,但 $Z$ 在高维下是不可解的积分。多年来人们用 MCMC、contrastive divergence 等方法估计 $Z$,都不够好。Hyvärinen 在 2005 年的关键观察是:score 关于 $\theta$ 的梯度与 $Z$ 无关——因为 $\nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x) - \nabla_x \log Z(\theta) = -\nabla_x E_\theta(x)$($\log Z$ 对 $x$ 是常数)。所以只要学 score,就完全绕过了配分函数。CMU L4 chunk 917 提到这一点:「EBM is basically just saying that okay, we're just going to learn an unnormalized version of the distribution and then we're going to normalize it later on using this thing called partition function.」从这个角度看,diffusion 模型本质是 EBM 之梦的现代落地——只不过通过 DSM + 多噪声尺度,让原本不可训练的 EBM 变成了 SOTA 生成模型。
🟡 教材外补充(Tweedie's Formula 的几何解释):DSM 的回归目标 $-\epsilon/\sigma$ 之所以指向"干净数据方向",背后是 Tweedie's Formula(1956)。它说:对于高斯加噪 $\tilde x = x + \sigma\epsilon$,最优 MMSE 去噪器满足 $\mathbb{E}[x | \tilde x] = \tilde x + \sigma^2 \nabla_{\tilde x} \log p_\sigma(\tilde x)$。变形得 $\nabla_{\tilde x} \log p_\sigma(\tilde x) = (\mathbb{E}[x|\tilde x] - \tilde x) / \sigma^2$——score 就是"从带噪点指向条件均值的向量"除以 $\sigma^2$。这就是为什么 score-based sampling 的 Langevin 步骤"看起来"像是把样本往数据流形拉:因为 score 字面上就是那个拉力方向。

下图模拟 DSM 的训练目标:橙色数据流形上分布着干净数据点 $x$,红色噪声点 $\tilde x$ 周期性地被"拉回"附近数据点 $x$ 方向。不变量:拉力向量 $-\epsilon/\sigma$ 永远指向干净数据点(即 Tweedie 几何告诉你的:score 字面上等于"指向条件均值 $\mathbb{E}[x|\tilde x]$ 的向量除以 $\sigma^2$")。这就是为什么训练 $s_\theta$ 回归到 $-\epsilon/\sigma$ 等价于"学一个把任意点拉回流形的方向场"。点击下面四个按钮可以看 DSM 推导链如何在四个步骤里把不可训练的 ESM 一步步变成 DDPM 的 simple loss。

① $\tilde x = x + \sigma\epsilon$,其中 $\epsilon \sim \mathcal{N}(0, I)$
② $\nabla_{\tilde x} \log p_\sigma(\tilde x | x) = -(\tilde x - x)/\sigma^2 = -\epsilon/\sigma$ (闭式!)
③ $J_{DSM} = \mathbb{E}\big[\|s_\theta(\tilde x) + \epsilon/\sigma\|^2\big]$,回归到 $-\epsilon/\sigma$
④ 令 $\epsilon_\theta := -\sigma \cdot s_\theta$,得 $\|\epsilon_\theta - \epsilon\|^2$ —— 这就是 DDPM 的 simple loss
data manifold x x̃ = x + σε −ε/σ = score 方向 DSM 训练:让 $s_\theta(\tilde x)$ 回归到 −ε/σ(指向干净 x 的方向)

手算:σ=0.5 的一步 DSM 训练

设数据点 $x = 2.0$(1D,方便手算),噪声尺度 $\sigma = 0.5$,从标准高斯采样 $\epsilon = 0.4$。先做加噪:$\tilde x = x + \sigma\epsilon = 2.0 + 0.5 \cdot 0.4 = 2.2$。算回归目标 $-\epsilon/\sigma = -0.4 / 0.5 = -0.8$(注意分母是 $\sigma$ 不是 $\sigma^2$,因为 $\tilde x - x = \sigma\epsilon$,所以 $(\tilde x - x)/\sigma^2 = \epsilon / \sigma$)。设此刻网络输出 $s_\theta(\tilde x) = -0.3$,则单样本 DSM 损失:$\big\|s_\theta(\tilde x) - (-\epsilon/\sigma)\big\|^2 = \big(-0.3 - (-0.8)\big)^2 = (0.5)^2 = 0.25$。这一步的梯度会把 $s_\theta(\tilde x)$ 从 $-0.3$ 往 $-0.8$ 方向推——训练结束后,网络会"知道"在 $\tilde x = 2.2$ 这个位置时应该输出大约 $-0.8$,几何含义就是"从 $\tilde x = 2.2$ 指回干净数据 $x = 2.0$ 的方向,强度大约是 $0.8$"。整个训练循环在 PyTorch 里只有一行核心代码:loss = F.mse_loss(s_theta(x_tilde, sigma), -eps/sigma)。这一行就是 Hyvärinen 2005 + Vincent 2011 + Ho 2020 三篇论文核心贡献的最终落地形式。

常见误区

  1. 「Score matching 就是把 $s_\theta$ 做 MSE 回归到真实 score」。反例:ESM 形式 $\|s_\theta(x) - \nabla_x \log p_\text{data}(x)\|^2$ 看上去确实是 MSE,但回归的目标 $\nabla_x \log p_\text{data}(x)$ 没人写得出来——把 $p_\text{data}$ 换成 1000 张 CIFAR 图像的 KDE 估计,第一个 batch 的梯度就是 NaN(因为 KDE 在样本之间的"无人区"概率几乎为零,log 之后变成 $-\infty$)。Score matching 的精髓不在"回归",而在用分部积分(ISM)或噪声扰动(DSM)把这个未知目标从损失里彻底消掉。
  2. 「DSM 学到的就是干净数据 $p_\text{data}$ 的 score」。反例:DSM 学到的严格意义上是 $\nabla_{\tilde x} \log p_\sigma(\tilde x)$——即带 $\sigma$ 模糊后的边际分布的 score。如果取 $\sigma = 0$,回归目标 $-\epsilon/\sigma$ 方差爆炸为 $\infty$,根本没法训;如果取 $\sigma = 10$,学到的几乎是标准高斯的 score($-\tilde x$),对刻画 $p_\text{data}$ 毫无用处。这就是为什么 NCSN(Song & Ermon 2019)和 DDPM(Ho 2020)都必须用一整套噪声尺度 $\sigma_1 < \cdots < \sigma_T$ 并把 $t$ 作为网络输入条件——单一 $\sigma$ 永远训不出生成模型。
  3. 「DSM 和 DDPM 是两个独立的训练目标,需要分别理解」。反例:DDPM 的简化损失 $L_\text{simple} = \|\epsilon_\theta(x_t, t) - \epsilon\|^2$ 通过参数化变换 $\epsilon_\theta(x_t, t) := -\sigma_t \cdot s_\theta(x_t, t)$ 与多尺度 DSM 完全等价(差一个 $1/\sigma_t^2$ 的权重,Ho 2020 经验性地把它去掉因为这样训练效果更好)。所以 Ho 2020 的 "$\epsilon$-prediction" 不是什么独立发明的新算法——它就是 score matching 在选择特定 parameterization 下的简化写法。把这两件事看成"两个目标"会让你在读 paper 时不断重复同一个推导。

相关概念

来源

  1. diffusion-principles-vizuara · L6 · Denoising Score Matching · 「score is a compass」 · (00:03)
  2. diffusion-principles-vizuara · L7 · Noise Conditional Score Networks · 「magnets on a table」 · (00:04)
  3. cmu-10799-diffusion-flow · L4 · Score-based Models · 「regression on the noise」 · (03:24)
  4. cmu-10799-diffusion-flow · L4 · EBM = 学 unnormalized 分布 · (05:45)
查看 154 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 00: 04 So let's begin. Previously last class uh So let's begin. Previously last class uh we have talked about what is diffusion we have talked about what is diffusion we have talked about what is diffusion m…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 00: 57 reserve sorry um all right so just a reserve sorry um all right so just a reminder what is the forward process reminder what is the forward process reminder what is the forward process basically ally …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 01: 44 sums up together is another gausian and you can just sum up a bunch of gausians. you can just sum up a bunch of gausians. you can just sum up a bunch of gausians. So that's why in the forward process …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 02: 35 is the diffusion for process. All right. Now, uh how now that we uh we have the Now, uh how now that we uh we have the Now, uh how now that we uh we have the forward process to train to construct forw…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 03: 24 mean of the gausian uh and in addition we can actually rewrite this mean this we can actually rewrite this mean this we can actually rewrite this mean this prediction of mean into something that predi…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 04: 08 last time there's some questions about why should we do this reparameterization why should we do this reparameterization why should we do this reparameterization well because doing this regression is …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 04: 54 this uh before I do that does anyone know in the audience what is normalizing know in the audience what is normalizing know in the audience what is normalizing flow by any No way. Nobody know. No way.…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 05: 45 to uh latent is the inverse of the mapping from latent to data. So this is mapping from latent to data. So this is mapping from latent to data. So this is sort of like normalizing flows. Um and sort o…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 06: 40 if you have a auto reggressive models if you have a auto reggressive models then you need to break things up by the then you need to break things up by the then you need to break things up by the chai…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 07: 25 And uh for normalizing flow just by looking at this can anyone tell me what looking at this can anyone tell me what looking at this can anyone tell me what potential problem it can it can have. Yes. Y…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 08: 23 least. Um but then it as we have learned last time it also has a problem of like last time it also has a problem of like last time it also has a problem of like very unstable training and also it very…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 09: 24 of images rather than actually having to of images rather than actually having to comput the comput the comput the >> yes that's right so you just need to >> yes that's right so you just need to >> ye…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 10: 13 model so what you do is you sort of just like follow the gra so say suppose you like follow the gra so say suppose you like follow the gra so say suppose you have a gradient that gives you like u have…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 11: 00 space right so basically what you can imagine is that you can start from imagine is that you can start from imagine is that you can start from anywhere on the data space so in this anywhere on the dat…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 11: 45 respect to the data point of the log likelihood so what is happening is that likelihood so what is happening is that likelihood so what is happening is that like so and then again like what you can li…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 12: 31 just like in diffusion right your source distribution and your target distribution and your target distribution and your target distribution should always be in the distribution should always be in th…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 13: 28 gradient of uh of of P X uh with respect to X. That's fine. But X uh with respect to X. That's fine. But X uh with respect to X. That's fine. But later on we're going to see why because later on we're…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 14: 15 is uh machine learning 101 right. So very easy very easy uh uh uh very easy very easy uh uh uh very easy very easy uh uh uh formulation. And this L2 here uh it has formulation. And this L2 here uh it …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 15: 08 like auto reggressive models. So it can like auto reggressive models. So it can sample or generate everything all at sample or generate everything all at sample or generate everything all at once once…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 15: 47 this L2 without this ground truth score. Turns out you don't even need to use the Turns out you don't even need to use the Turns out you don't even need to use the ground truth score in your loss grou…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 16: 35 simplify uh the L2 the the original L2 simplify uh the L2 the the original L2 into two parts here. And then the first into two parts here. And then the first into two parts here. And then the first pa…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 17: 27 can have the gradient of uh log of p with respect to x into you can break it with respect to x into you can break it with respect to x into you can break it into this thing right uh and then these int…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 18: 13 and then you can basically just get something like this. So um yeah this is something like this. So um yeah this is something like this. So um yeah this is a this is like we we probably wouldn't a thi…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 18: 58 need to remember is that the second part of the two things left. It's going to of the two things left. It's going to of the two things left. It's going to come boils down to the negative of the come b…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 19: 41 loss that we get, we don't really need a ground true score. It's it's like ground true score. It's it's like ground true score. It's it's like actually pretty pretty sick. So all you actually pretty p…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 20: 38 function has the same dimensionality of function has the same dimensionality of the of the data, right? Yeah. So, you the of the data, right? Yeah. So, you the of the data, right? Yeah. So, you can ju…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 21: 37 no one else. Do you have a Do you have a Do you have a Do you have a Do you have a >> Well, yes, but it's kind of related to >> Well, yes, but it's kind of related to >> Well, yes, but it's kind of re…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 22: 42 Literally Gausian. Okay. So basically what you do is uh and this is what we what you do is uh and this is what we what you do is uh and this is what we call dnoising score matching. Why so get call dn…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 23: 25 is you just need a sample data and then perturb it a little bit and now it perturb it a little bit and now it perturb it a little bit and now it becomes this like it's easier to becomes this like it's…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 24: 15 literally this. Uh so yeah, for for literally this. Uh so yeah, for for those of you who wonder why log density those of you who wonder why log density those of you who wonder why log density easier b…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 25: 25 truth it's a perturbed ground truth then truth it's a perturbed ground truth then you may have a question but like let's you may have a question but like let's you may have a question but like let's m…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 26: 04 anyway it's fine. Um uh so what you do is you first draw draw a sample from is you first draw draw a sample from is you first draw draw a sample from your data space using some easy to your data space…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 26: 50 So this is what could have happened if you do laundry dynamic essentially. So you do laundry dynamic essentially. So you do laundry dynamic essentially. So at the beginning, right, we just like at the…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 27: 44 anyway you just train your model to predict the score and then you use predict the score and then you use predict the score and then you use dynamic to sample. Uh however does it dynamic to sample. Uh…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 29: 00 survey. Let's survey. Uh we are starting from this column again. from this column again. from this column again. Hello friends. Uh Hello friends. Uh Hello friends. Uh does anyone want to share their t…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 30: 06 haven't been that I haven't been talked haven't been that I haven't been talked to today. There's so many people in the to today. There's so many people in the to today. There's so many people in the …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 31: 12 is what we call the manifold hypothesis. What does it mean is that hypothesis. What does it mean is that hypothesis. What does it mean is that basically usually the data that we want basically usually…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 31: 59 what here says like some area of the what here says like some area of the data space will not have support. What data space will not have support. What data space will not have support. What it means …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 32: 43 manifold from some point from on the manifold to some point that's not on the manifold to some point that's not on the manifold to some point that's not on the manifold the score is going to blow up m…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 33: 28 then the second thing. Yeah. Yeah. Yeah. >> What is this the left plot even showing >> What is this the left plot even showing >> What is this the left plot even showing either? Oh sorry the the the s…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 34: 10 space or entire data space now has support uh and then the score matching support uh and then the score matching support uh and then the score matching actually behave really really nicely actually be…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 34: 57 image is eventually boiled down to say 10 features or something in that case 10 features or something in that case 10 features or something in that case the manifold is load eventually how is the mani…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 35: 45 there's no probability and that's why there's no probability and that's why the score is going to blow up and adding the score is going to blow up and adding the score is going to blow up and adding s…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 36: 42 is like they they they train on the is like they they they train on the clean data. So it's like not I mean yes clean data. So it's like not I mean yes clean data. So it's like not I mean yes obviousl…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 37: 28 without any data right? So what's going to happen here is that um so say this is to happen here is that um so say this is to happen here is that um so say this is like your actual data score that your…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 38: 14 really know where to move in the low density region. So you're just going to density region. So you're just going to density region. So you're just going to move sort of randomly and then you you move…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 39: 02 high density from higher density right high density from higher density right so like say this is your like intended so like say this is your like intended so like say this is your like intended data …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 39: 48 >> it it depends on like what exactly is >> it it depends on like what exactly is this distribution say if this uh this distribution say if this uh this distribution say if this uh distribution is a m…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 40: 34 Okay. All right. Any other question? Cool. Cool. All right. So, we have all these All right. So, we have all these All right. So, we have all these problems. problems. problems. Is it problem? Is it p…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 41: 48 thing doesn't exist anymore, right? And thing doesn't exist anymore, right? And then now the perturbed like say you have then now the perturbed like say you have then now the perturbed like say you ha…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 42: 37 large of a of a noise, then it you're just learning a very noisy distribution. just learning a very noisy distribution. just learning a very noisy distribution. That's not what we want. So how much Th…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 43: 33 about which region that we are uh that about which region that we are uh that we're s we should be sampling at, then we're s we should be sampling at, then we're s we should be sampling at, then we're…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 44: 19 sample from, you are going to just gradually decrease the noise or the gradually decrease the noise or the gradually decrease the noise or the exploration term that you're adding at exploration term t…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 45: 05 have that next? uh we have the pseudo code and you know the long dynam dynamic code and you know the long dynam dynamic code and you know the long dynam dynamic left over. Yeah. But basically literall…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 45: 50 And then at training time you do this multi-level den noising score matching multi-level den noising score matching multi-level den noising score matching because at each time you basically because at…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 47: 23 Yeah, just just think about it. Uh, basically the difference between DDPM, basically the difference between DDPM, basically the difference between DDPM, which is what we learned last last time, which …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 48: 48 that the the the score is sort of it has that the the the score is sort of it has like another factor or it has another like another factor or it has another like another factor or it has another coef…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 49: 49 we have infinite numbers of noise level? we have infinite numbers of noise level? What do you what do we think is going to What do you what do we think is going to What do you what do we think is goin…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 50: 47 tailor estimation of of of the real thing and you can actually write both of thing and you can actually write both of thing and you can actually write both of them into this this format and uh if you …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 51: 42 because you see how like basically if you do xt plus delta t minus xt is going you do xt plus delta t minus xt is going you do xt plus delta t minus xt is going to be this thing right. So it's going t…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 52: 28 you just add some stoasticity. So this you just add some stoasticity. So this is why it's called stochastic is why it's called stochastic is why it's called stochastic differential equation. different…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 53: 28 and we choose the for process right so we know f and g uh and then it's we know f and g uh and then it's we know f and g uh and then it's literally just gausian right so who literally just gausian rig…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 54: 21 to use a oiler solver which is like to use a oiler solver which is like probably simple simplest solver uh that probably simple simplest solver uh that probably simple simplest solver uh that then you…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 55: 09 apply the change in x to x and then we apply the change in x to x and then we apply delta t to t apply delta t to t apply delta t to t and that's it. and that's it. and that's it. And then we iterate …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 56: 13 written as this like this function this written as this like this function this this uh so there are four process can be this uh so there are four process can be this uh so there are four process can …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 57: 20 the G as variance and in the VE formulation we do not have any formulation we do not have any formulation we do not have any adjustment of the X like whatever X you adjustment of the X like whatever X…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 58: 09 did diffusion. This class we did did diffusion. This class we did scorebased model and we also learned scorebased model and we also learned scorebased model and we also learned that they're basically …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 47: 17 otherwise it doesn't make sense I guess. Yeah. >> Yeah. Oh sorry. Yeah. >> Yeah. Oh sorry. Yeah. >> How do you weigh the different loss >> How do you weigh the different loss >> How do you weigh the d…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 00: 04 Hello everyone, welcome to the next Hello everyone, welcome to the next lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion models. di…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 01: 07 calculate the magnetic field for all the magnets and superimpose them together so magnets and superimpose them together so magnets and superimpose them together so that we have a cumulative universal …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 02: 13 idea where this ball came from. idea where this ball came from. But we have an idea where it came from. But we have an idea where it came from. But we have an idea where it came from. And essentially …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 03: 31 Okay. And the force required to pull the ball back the force required to pull the ball back the force required to pull the ball back to the start is the score field for the to the start is the score f…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 04: 34 Now this can also be represented as a noise vector. Right? noise vector. Right? noise vector. Right? [snorts] This is our flick. So [snorts] This is our flick. So [snorts] This is our flick. So essent…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 05: 37 And if this perturbed noise is very And if this perturbed noise is very small, you could assume that whatever small, you could assume that whatever small, you could assume that whatever score my I am …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 06: 33 exactly what we saw here. Minus noise divided by sigma. divided by sigma. divided by sigma. And the predicted score is what our And the predicted score is what our And the predicted score is what our …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 07: 41 as I said before uh the field of score matching has evolved matching has evolved matching has evolved through years. So in 2005 was the proof through years. So in 2005 was the proof through years. So …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 08: 49 that. It's a very beautifully written paper It's a very beautifully written paper It's a very beautifully written paper and in fact the textbook which I'm and in fact the textbook which I'm and in fac…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 09: 43 image or data sample into noise and then we predicted the reverse distribution we predicted the reverse distribution we predicted the reverse distribution from the noise back to the data. from the noi…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 10: 59 Now look at what is happening here. Your Now look at what is happening here. Your space is huge. This is the your entire space is huge. This is the your entire space is huge. This is the your entire s…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 12: 08 2005 which we explained and this is the loss that we try to minimize. Okay. Now this score is given by gradient of this score is given by gradient of this score is given by gradient of log of p of x. …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 13: 29 completely fine. Everything is defined. This the slope is defined right? You This the slope is defined right? You This the slope is defined right? You know exactly how your data changes as know exactl…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 14: 27 score vector will not be defined. score vector will not be defined. And imagine that you're going with a And imagine that you're going with a And imagine that you're going with a compass in your hand …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 15: 28 transformed from a low dimensional manifold to a higher dimensional space. manifold to a higher dimensional space. manifold to a higher dimensional space. It's taking up the space It's taking up the s…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 17: 00 Remember in the previous formulation by Remember in the previous formulation by Vincent, we just had one noise. Vincent, we just had one noise. Vincent, we just had one noise. We we added one noise wi…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 18: 24 you are flicking the magnet not just once with one level, but you're not just once with one level, but you're not just once with one level, but you're flicking it with multiple levels. flicking it wit…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 19: 42 multiple levels of noise? multiple levels of noise? We don't do this in diffusion also. In We don't do this in diffusion also. In We don't do this in diffusion also. In diffusion there is one constant…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 20: 48 is it a single neural network which is trained on trained on trained on uh for all these losses and if it's a uh for all these losses and if it's a uh for all these losses and if it's a single neural …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 22: 13 Remember now the score does not just Remember now the score does not just depend on X depend on X depend on X but it depends on X comma sigma where but it depends on X comma sigma where but it depends…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 23: 27 and land as close as possible to the and land as close as possible to the data point. Now the interesting thing is that I have Now the interesting thing is that I have a score function for a score fun…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 24: 58 The only difference is that it is performed sequentially performed sequentially performed sequentially multiple times. Let us understand how we multiple times. Let us understand how we multiple times.…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 26: 21 Okay. So you are purposely deliberately inputting a very high noise level so inputting a very high noise level so inputting a very high noise level so that that uh you move to areas where you probably…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 27: 17 here [snorts] the destination is not our data but it's far away from our data data but it's far away from our data data but it's far away from our data because we are adding so much noise. So because …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 28: 10 we are moving according to the lang dynamics update rule. This time the dynamics update rule. This time the dynamics update rule. This time the variance is a bit lower and um variance is a bit lower a…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 29: 07 sample the entire data, right? Even if, sample the entire data, right? Even if, let's say, you have handwritten digits let's say, you have handwritten digits let's say, you have handwritten digits and…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 30: 00 matching formula that it it it finally tries to predict the noise itself and tries to predict the noise itself and tries to predict the noise itself and that that is exactly what the diffusion that th…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 31: 01 please go through this for people who please go through this for people who like to see how both are connected and like to see how both are connected and like to see how both are connected and it's it…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 32: 14 uh and then which is exactly what which is exactly what which is exactly what we want to predict right we want to we want to predict right we want to we want to predict right we want to understand tha…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 33: 12 the reverse transition kernel is we are the reverse transition kernel is we are trying to predict how much noise was trying to predict how much noise was trying to predict how much noise was added in …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 34: 09 when we are inferring we do it n uh we when we are inferring we do it n uh we do it l times as as we saw here there do it l times as as we saw here there do it l times as as we saw here there are l di…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 34: 56 this and you do it for some specific time steps t so if the time step is time steps t so if the time step is time steps t so if the time step is greater than t you just stop this greater than t you ju…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 35: 59 So this is exactly how the anal So this is exactly how the anal land dynamics loop looks like. land dynamics loop looks like. land dynamics loop looks like. Let's take a look at a practical example Le…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 37: 01 you uh define your score network which is uh define your score network which is uh define your score network which is typically a conditional unit. typically a conditional unit. typically a conditiona…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 38: 08 And that is exactly what we see in the practical U notebook as well. practical U notebook as well. practical U notebook as well. [snorts] But this is done for different [snorts] But this is done for d…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 39: 15 if I run this inference for this epox where the loss is very low, you can see where the loss is very low, you can see where the loss is very low, you can see it's almost 554, it's almost 554, it's alm…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 40: 06 manifold and by adding noise we are spreading it and by adding noise we are spreading it and by adding noise we are spreading it so that it occupies the entire space. so that it occupies the entire sp…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 40: 48 which have very low density of of the data. And adding noise helps us to do data. And adding noise helps us to do data. And adding noise helps us to do that. It helps us to have the gradient that. It …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 41: 41 strength of low strength also. And once strength of low strength also. And once we train the network to understand all we train the network to understand all we train the network to understand all the…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 42: 36 images, videos or audio. As an homework, images, videos or audio. As an homework, I want you to play around with this I want you to play around with this I want you to play around with this notebook. …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 00: 03 Hello everyone, welcome to the next Hello everyone, welcome to the next lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion models. di…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 01: 15 in space and you go in that space with a compass which tells you which direction compass which tells you which direction compass which tells you which direction should I move so that I move closer to …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 02: 45 probability distribution. Right? So we do not know this term. do not know this term. do not know this term. In supervised machine learning, we often In supervised machine learning, we often In supervi…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 03: 54 requirement of this entire probability requirement of this entire probability distribution which we have no idea distribution which we have no idea distribution which we have no idea about. about. abo…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 05: 07 data which is available to us which is what makes it incredibly powerful. So we are grateful to this author to So we are grateful to this author to have made the stream of score matching have made the…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 06: 08 is close to the sink is pulled inwards. is close to the sink is pulled inwards. It's like a black hole. If you go near a It's like a black hole. If you go near a It's like a black hole. If you go near…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 06: 51 Now assume that you you are close to the data. Assume that you are the data data. Assume that you are the data data. Assume that you are the data itself. Then you don't need to move itself. Then you d…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 07: 48 for this training data at every single for this training data at every single point in the space. You can see the point in the space. You can see the point in the space. You can see the score function…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 08: 50 So just if you look at this visually you can see that wherever I start in the can see that wherever I start in the can see that wherever I start in the field and I follow the score function I field an…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 10: 02 function? So and even we have looked at one example where uh the loss function one example where uh the loss function one example where uh the loss function actually works. actually works. actually wo…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 11: 19 the diagonal elements. You can't the diagonal elements. You can't individually calculate the diagonal individually calculate the diagonal individually calculate the diagonal elements. elements. elemen…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 12: 37 in in real life use cases. And that is why for a lot of for many years actually why for a lot of for many years actually why for a lot of for many years actually uh uh uh the score matching technique …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 14: 00 the same word dn noising right and in diffusion that made sense because we diffusion that made sense because we diffusion that made sense because we started from a noisy image and then we started from…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 15: 10 These three papers are going to form the foundation of the theory of denoising foundation of the theory of denoising foundation of the theory of denoising score matching. Okay. So this technique was i…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 16: 22 formulation which was trackable. But that did not make sense because the that did not make sense because the that did not make sense because the complexity of computations squared complexity of comput…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 17: 42 and you do have access to some of them. So let's say you have access to 15 or 20 So let's say you have access to 15 or 20 So let's say you have access to 15 or 20 of these magnets but many others are …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 18: 48 nearest magnet is pulling me. If I take a point here I could say that this is a point here I could say that this is a point here I could say that this is the direction in which the nearest the directi…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 19: 45 uh I will I will superimpose the uh I will I will superimpose the magnetic fields for all these different magnetic fields for all these different magnetic fields for all these different magnets. magne…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 20: 51 We take any magnet on the table and we We take any magnet on the table and we flick it with a hand just like this. I'm flick it with a hand just like this. I'm flick it with a hand just like this. I'm…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 21: 49 the student, look, this is the ball's new location. new location. new location. and we hide the original uh magnet and we hide the original uh magnet and we hide the original uh magnet location. So we…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 23: 02 student has no idea where this new ball position came from. But we know new ball position came from. But we know new ball position came from. But we know where it came from. where it came from. where …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 24: 02 balls, all these magnets that you know balls, all these magnets that you know that the magnets are placed there. You that the magnets are placed there. You that the magnets are placed there. You place…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 25: 02 this point should be should move so that it goes back to the original position it goes back to the original position it goes back to the original position and that is exactly the direction of the and …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 26: 19 to a new location. to a new location. So the noisy new spot is represented as So the noisy new spot is represented as So the noisy new spot is represented as xhat. xhat. xhat. Uh this is I have actual…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 27: 22 score does exactly the same thing. The score tells you the direction score tells you the direction score tells you the direction to the actual data in the space. And our to the actual data in the spac…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 28: 42 of mistakes but the feedback is given of mistakes but the feedback is given from this. This is the true direction from this. This is the true direction from this. This is the true direction where whic…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 29: 59 It is not like your true It is not like your true probability distribution of data which probability distribution of data which probability distribution of data which is not tractable but it is tracta…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 31: 00 given any point in space how can I pull this point so that it how can I pull this point so that it how can I pull this point so that it becomes as close as possible to your becomes as close as possibl…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 32: 02 really making sense. I have an excellent really making sense. I have an excellent explanation to that which will come soon explanation to that which will come soon explanation to that which will come …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 33: 05 perturvation to the data. We can simplify the score function which we simplify the score function which we simplify the score function which we want to learn as follows. want to learn as follows. want…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 34: 14 uh the mean of this pixel is centered at uh the mean of this pixel is centered at 0.5 because the value is 0.5 but then you add a small noise with a but then you add a small noise with a standard devi…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 35: 36 And uh in fact this can be represented this can be represented this can be represented using a standard normal distribution using a standard normal distribution using a standard normal distribution li…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 36: 55 expression for this probability distribution. distribution. distribution. And uh because it is tractable once we And uh because it is tractable once we And uh because it is tractable once we know the …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 37: 56 anything is whatever is there in the exponential itself. So you get this at exponential itself. So you get this at exponential itself. So you get this at the bottom and then when you take the bottom a…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 39: 07 you you you might have thought why didn't I guess it before because it it didn't I guess it before because it it didn't I guess it before because it it should be the vector which points from should be…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 40: 18 predict we are trying to match the noise the noise vector which has been added in the noise vector which has been added in the noise vector which has been added in the to the clean data. the to the cl…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 41: 22 which means that the density is maximum near somewhere around this maybe minus4 near somewhere around this maybe minus4 near somewhere around this maybe minus4 and the density is maximum somewhere and…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 42: 55 So this is the neural network that we So this is the neural network that we define and the loss function is the main define and the loss function is the main define and the loss function is the main s…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 43: 51 Okay. And now what is the target score? The target score is something which The target score is something which The target score is something which you're trying to estimate which is minus you're tryi…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 44: 59 And uh finally you see the learned score And uh finally you see the learned score function which looks something like function which looks something like function which looks something like this. Now …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 45: 56 magnet. magnet. Similarly, if you are somewhere on the Similarly, if you are somewhere on the Similarly, if you are somewhere on the right, uh the score is negative because right, uh the score is nega…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 46: 42 left you reach magnet number one. If you are on the far right, you reach magnet are on the far right, you reach magnet are on the far right, you reach magnet number two. And if you are anywhere in num…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 47: 32 We first add a noise and then we try to We first add a noise and then we try to predict that noise. predict that noise. predict that noise. And in that process of predicting the And in that process of…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 48: 34 us away from the local minima so that we explore the global minimas as well. explore the global minimas as well. explore the global minimas as well. So uh this is what we get So uh this is what we get…
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 49: 35 much of a difference. Something might be much of a difference. Something might be happening in the 3D space right that happening in the 3D space right that happening in the 3D space right that adding …
diffusion-principles-vizuara L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models 50: 26 Okay. So this is it for dnoising score matching and uh in the next part matching and uh in the next part matching and uh in the next part uh I will explain about noise condition uh I will explain abou…