概念

Score Function

— $\nabla_x \log p(x)$,对输入求导而非对参数。绕开归一化常数的指南针。

第一道拦路虎是名字。很多人第一次见 "score function" 会把它脑补成「模型给样本打的分数」——像 GAN discriminator 输出的 logit、或者分类器的 softmax 概率。或者更糟——把它和反向传播里「梯度对参数」混为一谈,以为「score」就是某种 loss 的梯度。都不对。这里的 "score" 是统计学里 Fisher 的历史术语(1922 年),指的是对数密度关于输入变量的梯度:

$$s(x) \;=\; \nabla_x \log p(x), \qquad s : \mathbb{R}^d \to \mathbb{R}^d.$$

注意下标:$\nabla_x$,对输入 $x$ 求导,不是对网络参数 $\theta$ 求导。它和 loss、和反向传播、和「打分」没有任何关系。它是一个纯几何对象——给你一个密度 $p(x)$,它就是标量场 $\log p$ 的梯度向量场,与是否存在神经网络无关。即便没人训练任何东西,一维高斯 $\mathcal{N}(0, 1)$ 在 $x = 2$ 处的 score 也是确定的(等于 $-2$),由公式给出。神经网络的工作只是在我们不知道真实 $p$ 时,去逼近这个本已存在的向量场。

Vizuara 的 Dr. Raj 在 Lecture 7 用一个特别朴实的比喻把它讲清楚——「想象一张桌子上散着一堆磁铁,你只能看到其中几颗,另外许多颗藏在桌面下方看不见。问:站在桌子任意一点,最近的那颗磁铁正在把你往哪里拽?这个方向,就是 score。问题难,是因为你不知道隐藏磁铁的位置,但如果知道了,你就能算出任何一点上的合力。」—— Vizuara L7 @00:04

磁铁就是数据点,「被拽的方向」就是 $\nabla_x \log p_{\text{data}}(x)$。它告诉你:要让样本更像数据分布,应该往哪里挪。这个比喻还说明了一件容易忽略的事——score 是个向量场,每一点上都有一个箭头,不是单一标量。在训练里,这件事对应——网络 $s_\theta(x, t)$ 学的不是「这是几号样本」也不是「噪声多大」,而是「在这个位置、这个噪声水平下,往哪挪一小步会让对数密度更高」。后面我们会看到,这恰好是 Anderson 1982 反向 SDE 唯一需要的量;diffusion 的整个数学骨架就是建立在 score 这一个对象之上的。

CMU 10-799 的讲师在 Lecture 4 用另一个角度切入:「我们之前讲的是 diffusion 把噪声变成数据。今天换个视角——任何把数据建模成 score field 的方法,都和 diffusion 等价。score function is the central object.」—— CMU 10-799 L4 @00:04

两位讲师的入口不同但结论相同:Vizuara 从「绕开能量模型的归一化常数」推过来,CMU 从「diffusion 反向过程就是在跟一个 score 场走」切入。两条线在「$\nabla \log p$ 是统一对象」这一点合流。本词条会把两条线都讲完,并显式标出哪段属于哪门课。读完之后,你应该能在三个层面同时持有这个概念:统计学的(Fisher score)、物理的(势能场的负梯度)、几何的(指向数据流形的箭头场)。这三个视角在 diffusion 训练 / 采样的每一步里都同时在起作用。

关键性质:归一化常数消失

这是 score 在生成模型里地位的根源。考虑一个未归一化的密度 $\tilde p(x)$——也就是任何「形状对、但积分不等于 1」的函数。要把它变成合法概率密度,要除以配分函数 $Z = \int \tilde p(x)\,dx$:

$$p(x) \;=\; \frac{\tilde p(x)}{Z}.$$

对高维数据(比如 $256 \times 256 \times 3$ 的图像,$d \approx 2\times 10^5$),$Z$ 是一个 $d$ 维积分——没法算,没法采样估计,没法反向传梯度。这是能量模型(EBM)半个世纪卡在「只能做小图」的根本原因。

但是看 score 怎么处理:

$$\nabla_x \log p(x) \;=\; \nabla_x \big[\log \tilde p(x) - \log Z\big] \;=\; \nabla_x \log \tilde p(x) - 0 \;=\; \nabla_x \log \tilde p(x).$$

$\log Z$ 是不依赖 $x$ 的常数,对 $x$ 求导直接归零。归一化常数完全消失。

Why:这一条性质是 score-based generative model 整套范式的起点。它把「学一个密度函数」(要 $Z$)转化成「学一个向量场」(不要 $Z$)。EBM 那堵墙被绕过去了。

与能量模型的桥

能量模型(Energy-Based Model, EBM)把密度写成 Boltzmann 形式:$p(x) = e^{-E(x)}/Z$,$E_\theta : \mathbb{R}^d \to \mathbb{R}$ 是网络输出的标量「能量」——能量越低、概率越高。Vizuara L5 整节课就在讲这个范式。取对数:

$$\log p(x) \;=\; -E(x) - \log Z.$$

再对 $x$ 求导:

$$\nabla_x \log p(x) \;=\; -\nabla_x E(x).$$

所以 score 就是能量函数的负梯度——在物理直觉上正是「力场」(势能场的梯度反向给出受力方向)。这把 score-based 模型和 1980s 的 Hopfield / Boltzmann machine 直接接上了:那些早期模型也是「定义能量、低能量 = 高概率」,但训练它们要算 $Z$(partition function),高维下完全做不了。Vizuara L5 把这段历史展开讲:「EBM 在小图、低维问题上可以用 contrastive divergence、persistent CD 等近似方法训练,但在自然图像上崩。原因就是 $Z$。」

score matching 的原始动机(Hyvärinen 2005)就是给 EBM 提供一个不依赖 $Z$ 的训练目标:训练时让 $-\nabla_x E_\theta(x)$ 匹配数据 $\nabla_x \log p_{\text{data}}(x)$,整个过程 $Z$ 从未出现。Yang Song 等人 2019 之后的工作(NCSN、Score-SDE)干脆把 $E_\theta$ 也省了,直接让网络输出 $s_\theta(x)$ 是个向量,不再要求它是某个标量势的梯度——更宽松、更好训。但代价是失去了「显式能量」可用于异常检测等场景;这是一个有意识的工程取舍。

🟡 教材外补充:如果坚持让 $s_\theta = -\nabla E_\theta$(即先输出标量再求梯度),等价于强制 score 场无旋(curl-free);不强制时 $s_\theta$ 可能有非零旋度,这是和真实 score 的微小偏离。Salimans & Ho (2021) 讨论过;实践上 diffusion 不在意,因为反向 SDE 的不变量是边缘分布而非路径。

几何直觉:流向数据流形

把 $p_{\text{data}}$ 想成地形——数据多的地方是山顶,没数据的地方是低谷。score $\nabla_x \log p$ 是每一点上指向最近山顶的箭头。注意是对数密度的梯度而不是密度本身:取对数让远场的小密度差异被放大,使方向信息在概率近 0 处仍然清晰;这是技术上选 $\log p$ 而不是 $p$ 的一个隐藏好处。

  • 在数据流形上(高密度区):箭头较短,指向密度局部更大的方向;mode 处恰好是零向量(local maximum)。
  • 在数据流形附近但不在上面:箭头长,指向流形——Vizuara L7 称之为「back to data」。这正是去噪一步的方向。
  • 远离数据流形(纯噪声区):理论上 $p$ 趋近 0,$\nabla \log p$ 不稳定甚至发散;估出来的方向几乎是噪声。这是为什么必须用一族被噪声卷积过的 $p_t$、而不能只用 $p_{\text{data}}$——Song & Ermon 2019 的核心发现。
  • 多 mode 之间的鞍点:score 散度为正、方向歧义。Langevin 在这种区域容易左右摇摆,单一噪声水平往往跨不过去。
mode A mode B 鞍点:方向歧义 两团高斯混合上的 $\nabla_x \log p(x)$ 向量场——箭头指向最近的 mode

为什么用 score 而不是 $p$ 本身

这是初学者最容易问的一个问题——「为什么要绕一圈学梯度,不直接学密度?」三条理由叠加:

  1. 避开 $Z$。$p$ 需要归一化;$\nabla \log p$ 不需要。这是结构上的胜利,也是 score-based 这条线能在高维图像上突破的第一根支柱。
  2. 可以从样本估计。score matching(Hyvärinen 2005)+ denoising score matching(Vincent 2011)给出了不需要密度真值、只需要数据样本的训练目标——具体说,把「学 score」化成「学去噪」,可以用普通 MSE 训。具体见 score-matching。
  3. 直接驱动采样。有了 $s(x) \approx \nabla \log p$,Langevin dynamics $x_{k+1} = x_k + \tau s(x_k) + \sqrt{2\tau}\,z_k$ 就能从任意起点收敛到 $p$。不需要拒绝采样、不需要 MCMC 平衡、不需要 ancestral sampling 的链式分解——直接 SDE 数值积分一条龙跑通。
Why:所有「为什么 score-based 这条线在 2019–2020 突然能 work」都能追到这三件事的合力——score matching 学得到、Langevin 采样得动、噪声多尺度(NCSN)解决远场不准。三个都缺一不可。

和扩散模型的接口(最重要的一段)

这里是 score 真正进入 diffusion 的地方。前向 SDE 把数据 $p_{\text{data}}$ 慢慢扩散成噪声 $p_T \approx \mathcal{N}(0, I)$:

$$d x \;=\; f(x, t)\,dt + g(t)\,dw.$$

设 $t$ 时刻边缘分布 $p_t(x)$——也就是把所有 $x_0 \sim p_{\text{data}}$ 都做完 $t$ 步前向后的整体分布。Anderson (1982) 的反向时间公式给出:

$$d x \;=\; \big[f(x, t) - g(t)^2 \nabla_x \log p_t(x)\big]\,dt + g(t)\,d\bar w.$$

关键观察:唯一的「学习项」就是 $\nabla_x \log p_t(x)$——每一个噪声水平 $t$ 上、被噪声扰动过的边缘分布的 score。$f, g$ 是用户挑的(VP / VE / sub-VP 等不同 SDE 设计),$d\bar w$ 是反向 Brownian motion,这两者都不用学。整个反向过程的全部「难处」浓缩在一个对象上:$s_\theta(x, t) \approx \nabla_x \log p_t(x)$。

这就是为什么训练时网络的输入是 $(x, t)$ 两个量:你不是学一个 score,是学一族 score——$t$ 从 0 到 $T$ 上每一个噪声水平各一个 score 场。一个 U-Net / DiT 用 time embedding 把 $t$ 输进去,权重共享地拟合整族。

$t$ 越大,$p_t$ 越平滑(被高斯卷积过),score 越好估也越能从远场指回数据流形;$t$ 越小,$p_t$ 越接近原数据,score 在数据点之外几乎没意义。这正是 NCSN / Song-Ermon 2019 多噪声尺度的设计动机:用大 $t$ 解决「远场指向不准」,用小 $t$ 解决「贴近数据的细节」,反向过程从大到小 anneal 走一遍。这件事在 Vizuara L7 用同一个磁铁比喻继续讲——「单一噪声水平就像只看一颗磁铁,远处感受不到吸引;多噪声水平就是同时摆多颗强弱不同的磁铁,远处由强磁铁牵引,近处由弱磁铁微调。」

与 $\epsilon$-prediction 的换算

DDPM 类工作输出 $\epsilon_\theta$(直接预测「加进去的那团高斯噪声」)。两种参数化其实是一回事,只差一个尺度。在前向 $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sigma_t\, \epsilon$ 下,给定 $x_0$ 时 $x_t$ 是高斯,所以条件 score 就能写出:

$$\nabla_{x_t} \log p_t(x_t \mid x_0) \;=\; -\frac{x_t - \sqrt{\bar\alpha_t}\, x_0}{\sigma_t^2} \;=\; -\frac{\epsilon}{\sigma_t}.$$

训练目标对应(边缘 score 通过 Vincent 2011 denoising score matching 化为条件 score 的期望):

$$\boxed{\;s_\theta(x_t, t) \;\approx\; -\frac{\epsilon_\theta(x_t, t)}{\sigma_t}\;}$$

所以 DDPM 的「噪声预测」和 score-based 的「score 预测」是同一个网络的两种读法。Score-based 视角强调几何(向量场、Langevin、SDE),$\epsilon$-视角强调统计(去噪 MSE、ELBO 化简)。两边写出来的训练 loss 在加权系数上等价——这一点在 Vizuara L6 用 MSE 推导,CMU 10-799 L4 用 score matching 推导,殊途同归(详见 epsilon-prediction 和 score-matching)。

🟡 教材外补充:Yang Song et al. 2021 ICLR (Score-SDE 论文) 正式把 DDPM 和 NCSN 统一成同一个 SDE 框架的两种离散化——DDPM 是 VP-SDE 的离散化,NCSN 是 VE-SDE 的离散化。这是「diffusion = score-based」这条等价的标准出处,也是为什么今天讲 diffusion 一定要讲 score function。

退化情形:单一噪声水平为什么不够

一个自然问题:既然 score 这么强,干嘛非要在所有 $t$ 上学?直接学 $p_{\text{data}}$ 的 score 然后跑 Langevin 不行吗?答案是不行,Song & Ermon 2019 的 motivation 部分专门讨论:

  • 流形假设的诅咒。自然图像在 $\mathbb{R}^d$ 里占一个低维流形($d_{\text{intrinsic}} \ll d$)。流形外的点 $p_{\text{data}}(x) \approx 0$,$\log p_{\text{data}}(x) \to -\infty$,$\nabla \log p$ 估计极度不稳。Langevin 从高斯起点出发,几乎肯定在流形外,没有信号指回去。
  • 多 mode 之间无法跨越。真实数据是多 mode 混合(猫/狗/车…)。两个 mode 之间是低密度走廊,score 在那里近似为零或方向乱,Langevin 一旦走进某个 mode 就被困住,混合性极差。
  • 加噪声解决两件事。把数据卷积一个 $\sigma$ 大的高斯:$p_\sigma(x) = \int p_{\text{data}}(y)\,\mathcal{N}(x;y,\sigma^2 I)\,dy$。$\sigma$ 大 → $p_\sigma$ 在整个空间都非零 → score 处处有定义且指向数据;$\sigma$ 小 → $p_\sigma \approx p_{\text{data}}$,保留细节。同时学一组 $\sigma$,反向时从大到小 anneal——这就是 NCSN,也是 diffusion 反向过程的本质。

上面 SVG:橙色箭头是 score 向量场,指向两团高斯的中心;红色箭头标记鞍点附近的方向歧义(mode 之间的「分水岭」)。不变量:每个 mode 是一个 score 的零点 + 周围向内指的稳定吸引子;mode 之间出现方向反转(散度为正的鞍点)。这就是为什么 Langevin 从随机起点出发,最终落到某个 mode——score field 长得就像「向数据流形塌缩」的引力场。

手算:一维高斯的 score

设 $p(x) = \mathcal{N}(x; \mu, \sigma^2)$。先写对数密度:

$$\log p(x) \;=\; -\tfrac{1}{2}\log(2\pi\sigma^2) - \frac{(x-\mu)^2}{2\sigma^2}.$$

对 $x$ 求导,常数项 $-\tfrac{1}{2}\log(2\pi\sigma^2)$ 归零(它不依赖 $x$):

$$s(x) \;=\; \nabla_x \log p(x) \;=\; -\frac{x - \mu}{\sigma^2}.$$

验算几何直觉:$x > \mu$ 时 $s(x) < 0$,指向 $\mu$(往左拉);$x < \mu$ 时 $s(x) > 0$,往右拉;$x = \mu$ 时 $s = 0$(mode 是 score 的零点)。代具体数:$\mu = 0, \sigma = 1, x = 2 \Rightarrow s = -(2-0)/1 = -2$。要让样本更像 $\mathcal{N}(0,1)$,Langevin 一步是:

$$x \leftarrow x + \tau \cdot s(x) + \sqrt{2\tau}\,z, \quad z \sim \mathcal{N}(0,1).$$

取 $\tau = 0.1$、无噪声项时 $x \leftarrow 2 + 0.1\cdot(-2) = 1.8$——往原点挪。重复迭代,$x$ 会逐渐塌缩到 $\mu = 0$ 附近,再被 $\sqrt{2\tau}\, z$ 散开成 $\mathcal{N}(0, 1)$ 的样本。这就是 Langevin 用 score 采样的最小可工作例。

第一项 $-\tfrac{1}{2}\log(2\pi\sigma^2)$ 在这里就是归一化常数 $\log Z$ 的贡献,对 $x$ 求导后蒸发。这是「$Z$ 消失」的最小验证:即便是高斯这样简单的分布,归一化常数也是非平凡的(含 $\pi, \sigma$),但 score 完全不需要它。

推广到多维各向同性高斯 $\mathcal{N}(\mu, \sigma^2 I)$:$s(x) = -(x - \mu)/\sigma^2$,结果一样,只是变成向量。混合高斯 $p = \sum_k \pi_k \mathcal{N}(x; \mu_k, \sigma_k^2 I)$ 的 score 不再那么干净——它是带 $\pi_k$ 加权的、对每个 mode 的 score 的混合,并且在 mode 之间会有方向反转(鞍点),这正是 SVG 里红色箭头标出的位置。

常见误区

  1. 「score 是网络给样本打的分」。反例:score 是 $\nabla_x \log p$,跟有没有神经网络无关——一维高斯 $\mathcal{N}(0,1)$ 在 $x=2$ 处的 score 是 $-2$,是从公式推出来的,不是任何模型「打分」的结果。「Fisher score」的「score」是 R. A. Fisher 1922 年的统计学历史术语,意思是「对数似然对参数的梯度」(统计推断里)或「对数密度对输入的梯度」(生成模型里),都和 GAN discriminator / 分类器 logit 那种「打分」无关。混淆名词是初学最常见的坑。
  2. 「score 是对参数 $\theta$ 求导」。反例:score $= \nabla_x \log p(x)$,下标是输入 $x$;维度等于 $x$ 的维度($256 \times 256 \times 3$ 图像就是 196608 维向量,不是标量)。对参数求导得到的是 loss gradient,是反向传播用的,完全不是同一个东西。一个判别诀窍:score 在没有神经网络的场景下也有定义(比如直接对解析高斯求 $\nabla_x \log p$),loss gradient 没有网络就不存在。
  3. 「学会一个固定 $p_{\text{data}}$ 的 score 就能采样了」。反例:原始 score matching + Langevin 在高维实测不工作——纯噪声区 $p_{\text{data}}(x) \approx 0$,score 估不准(没有训练样本支持),Langevin 从随机起点走不出来;同时多 mode 之间是低密度走廊,Langevin 一旦被某个 mode 抓住就出不来。Song & Ermon 2019 的解决方案是「多噪声尺度 NCSN」:学 $p_{\sigma_1}, p_{\sigma_2}, \dots$ 一组被高斯卷积过的边缘分布的 score,从大噪声向小噪声 anneal——大噪声 score 处处有意义、能跨 mode;小噪声还原细节。这恰好就是 diffusion 反向过程的本质,也是为什么扩散模型不能只学 $\nabla \log p_{\text{data}}$,必须学一整族 $\{\nabla \log p_t\}_{t \in [0, T]}$。所有训练数据为什么要在前向里加噪声后再喂给网络?因为只有这样,模型才接触得到流形之外的样本、才知道在那里 score 应该怎么指。

相关概念

  • Score Matching——从样本估计 $\nabla \log p$ 的训练目标,跳过 $Z$;本词条提到的「能算出来」的具体方案。
  • Langevin Dynamics——有了 score 之后的「物理采样器」,$x \leftarrow x + \tau s + \sqrt{2\tau}\,z$ 的来源。
  • Noise Schedule——$t \mapsto \sigma_t$ 决定 $p_t$ 的退火曲线,即学哪些 $p_t$ 的 score。
  • DDPM——$\epsilon$-视角下的同一个东西,$s_\theta = -\epsilon_\theta/\sigma_t$ 给出两条线的桥。

来源

  1. diffusion-principles-vizuara · L5 · Energy Based Models / Score Function · (0:04)
  2. diffusion-principles-vizuara · L7 · 磁铁比喻 · NCSN 引入 · (0:04)
  3. cmu-10799-diffusion-flow · L4 · Score-based Models 视角 · (0:04)
查看 190 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 00: 04 So let's begin. Previously last class uh So let's begin. Previously last class uh we have talked about what is diffusion we have talked about what is diffusion we have talked about what is diffusion m…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 00: 57 reserve sorry um all right so just a reserve sorry um all right so just a reminder what is the forward process reminder what is the forward process reminder what is the forward process basically ally …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 01: 44 sums up together is another gausian and you can just sum up a bunch of gausians. you can just sum up a bunch of gausians. you can just sum up a bunch of gausians. So that's why in the forward process …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 02: 35 is the diffusion for process. All right. Now, uh how now that we uh we have the Now, uh how now that we uh we have the Now, uh how now that we uh we have the forward process to train to construct forw…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 03: 24 mean of the gausian uh and in addition we can actually rewrite this mean this we can actually rewrite this mean this we can actually rewrite this mean this prediction of mean into something that predi…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 04: 08 last time there's some questions about why should we do this reparameterization why should we do this reparameterization why should we do this reparameterization well because doing this regression is …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 04: 54 this uh before I do that does anyone know in the audience what is normalizing know in the audience what is normalizing know in the audience what is normalizing flow by any No way. Nobody know. No way.…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 05: 45 to uh latent is the inverse of the mapping from latent to data. So this is mapping from latent to data. So this is mapping from latent to data. So this is sort of like normalizing flows. Um and sort o…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 06: 40 if you have a auto reggressive models if you have a auto reggressive models then you need to break things up by the then you need to break things up by the then you need to break things up by the chai…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 07: 25 And uh for normalizing flow just by looking at this can anyone tell me what looking at this can anyone tell me what looking at this can anyone tell me what potential problem it can it can have. Yes. Y…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 08: 23 least. Um but then it as we have learned last time it also has a problem of like last time it also has a problem of like last time it also has a problem of like very unstable training and also it very…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 09: 24 of images rather than actually having to of images rather than actually having to comput the comput the comput the >> yes that's right so you just need to >> yes that's right so you just need to >> ye…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 10: 13 model so what you do is you sort of just like follow the gra so say suppose you like follow the gra so say suppose you like follow the gra so say suppose you have a gradient that gives you like u have…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 11: 00 space right so basically what you can imagine is that you can start from imagine is that you can start from imagine is that you can start from anywhere on the data space so in this anywhere on the dat…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 11: 45 respect to the data point of the log likelihood so what is happening is that likelihood so what is happening is that likelihood so what is happening is that like so and then again like what you can li…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 12: 31 just like in diffusion right your source distribution and your target distribution and your target distribution and your target distribution should always be in the distribution should always be in th…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 13: 28 gradient of uh of of P X uh with respect to X. That's fine. But X uh with respect to X. That's fine. But X uh with respect to X. That's fine. But later on we're going to see why because later on we're…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 14: 15 is uh machine learning 101 right. So very easy very easy uh uh uh very easy very easy uh uh uh very easy very easy uh uh uh formulation. And this L2 here uh it has formulation. And this L2 here uh it …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 15: 08 like auto reggressive models. So it can like auto reggressive models. So it can sample or generate everything all at sample or generate everything all at sample or generate everything all at once once…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 15: 47 this L2 without this ground truth score. Turns out you don't even need to use the Turns out you don't even need to use the Turns out you don't even need to use the ground truth score in your loss grou…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 16: 35 simplify uh the L2 the the original L2 simplify uh the L2 the the original L2 into two parts here. And then the first into two parts here. And then the first into two parts here. And then the first pa…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 17: 27 can have the gradient of uh log of p with respect to x into you can break it with respect to x into you can break it with respect to x into you can break it into this thing right uh and then these int…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 18: 13 and then you can basically just get something like this. So um yeah this is something like this. So um yeah this is something like this. So um yeah this is a this is like we we probably wouldn't a thi…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 18: 58 need to remember is that the second part of the two things left. It's going to of the two things left. It's going to of the two things left. It's going to come boils down to the negative of the come b…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 19: 41 loss that we get, we don't really need a ground true score. It's it's like ground true score. It's it's like ground true score. It's it's like actually pretty pretty sick. So all you actually pretty p…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 20: 38 function has the same dimensionality of function has the same dimensionality of the of the data, right? Yeah. So, you the of the data, right? Yeah. So, you the of the data, right? Yeah. So, you can ju…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 21: 37 no one else. Do you have a Do you have a Do you have a Do you have a Do you have a >> Well, yes, but it's kind of related to >> Well, yes, but it's kind of related to >> Well, yes, but it's kind of re…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 22: 42 Literally Gausian. Okay. So basically what you do is uh and this is what we what you do is uh and this is what we what you do is uh and this is what we call dnoising score matching. Why so get call dn…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 23: 25 is you just need a sample data and then perturb it a little bit and now it perturb it a little bit and now it perturb it a little bit and now it becomes this like it's easier to becomes this like it's…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 24: 15 literally this. Uh so yeah, for for literally this. Uh so yeah, for for those of you who wonder why log density those of you who wonder why log density those of you who wonder why log density easier b…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 25: 25 truth it's a perturbed ground truth then truth it's a perturbed ground truth then you may have a question but like let's you may have a question but like let's you may have a question but like let's m…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 26: 04 anyway it's fine. Um uh so what you do is you first draw draw a sample from is you first draw draw a sample from is you first draw draw a sample from your data space using some easy to your data space…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 26: 50 So this is what could have happened if you do laundry dynamic essentially. So you do laundry dynamic essentially. So you do laundry dynamic essentially. So at the beginning, right, we just like at the…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 27: 44 anyway you just train your model to predict the score and then you use predict the score and then you use predict the score and then you use dynamic to sample. Uh however does it dynamic to sample. Uh…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 29: 00 survey. Let's survey. Uh we are starting from this column again. from this column again. from this column again. Hello friends. Uh Hello friends. Uh Hello friends. Uh does anyone want to share their t…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 30: 06 haven't been that I haven't been talked haven't been that I haven't been talked to today. There's so many people in the to today. There's so many people in the to today. There's so many people in the …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 31: 12 is what we call the manifold hypothesis. What does it mean is that hypothesis. What does it mean is that hypothesis. What does it mean is that basically usually the data that we want basically usually…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 31: 59 what here says like some area of the what here says like some area of the data space will not have support. What data space will not have support. What data space will not have support. What it means …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 32: 43 manifold from some point from on the manifold to some point that's not on the manifold to some point that's not on the manifold to some point that's not on the manifold the score is going to blow up m…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 33: 28 then the second thing. Yeah. Yeah. Yeah. >> What is this the left plot even showing >> What is this the left plot even showing >> What is this the left plot even showing either? Oh sorry the the the s…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 34: 10 space or entire data space now has support uh and then the score matching support uh and then the score matching support uh and then the score matching actually behave really really nicely actually be…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 34: 57 image is eventually boiled down to say 10 features or something in that case 10 features or something in that case 10 features or something in that case the manifold is load eventually how is the mani…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 35: 45 there's no probability and that's why there's no probability and that's why the score is going to blow up and adding the score is going to blow up and adding the score is going to blow up and adding s…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 36: 42 is like they they they train on the is like they they they train on the clean data. So it's like not I mean yes clean data. So it's like not I mean yes clean data. So it's like not I mean yes obviousl…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 37: 28 without any data right? So what's going to happen here is that um so say this is to happen here is that um so say this is to happen here is that um so say this is like your actual data score that your…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 38: 14 really know where to move in the low density region. So you're just going to density region. So you're just going to density region. So you're just going to move sort of randomly and then you you move…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 39: 02 high density from higher density right high density from higher density right so like say this is your like intended so like say this is your like intended so like say this is your like intended data …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 39: 48 >> it it depends on like what exactly is >> it it depends on like what exactly is this distribution say if this uh this distribution say if this uh this distribution say if this uh distribution is a m…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 40: 34 Okay. All right. Any other question? Cool. Cool. All right. So, we have all these All right. So, we have all these All right. So, we have all these problems. problems. problems. Is it problem? Is it p…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 41: 48 thing doesn't exist anymore, right? And thing doesn't exist anymore, right? And then now the perturbed like say you have then now the perturbed like say you have then now the perturbed like say you ha…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 42: 37 large of a of a noise, then it you're just learning a very noisy distribution. just learning a very noisy distribution. just learning a very noisy distribution. That's not what we want. So how much Th…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 43: 33 about which region that we are uh that about which region that we are uh that we're s we should be sampling at, then we're s we should be sampling at, then we're s we should be sampling at, then we're…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 44: 19 sample from, you are going to just gradually decrease the noise or the gradually decrease the noise or the gradually decrease the noise or the exploration term that you're adding at exploration term t…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 45: 05 have that next? uh we have the pseudo code and you know the long dynam dynamic code and you know the long dynam dynamic code and you know the long dynam dynamic left over. Yeah. But basically literall…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 45: 50 And then at training time you do this multi-level den noising score matching multi-level den noising score matching multi-level den noising score matching because at each time you basically because at…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 47: 23 Yeah, just just think about it. Uh, basically the difference between DDPM, basically the difference between DDPM, basically the difference between DDPM, which is what we learned last last time, which …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 48: 48 that the the the score is sort of it has that the the the score is sort of it has like another factor or it has another like another factor or it has another like another factor or it has another coef…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 49: 49 we have infinite numbers of noise level? we have infinite numbers of noise level? What do you what do we think is going to What do you what do we think is going to What do you what do we think is goin…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 50: 47 tailor estimation of of of the real thing and you can actually write both of thing and you can actually write both of thing and you can actually write both of them into this this format and uh if you …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 51: 42 because you see how like basically if you do xt plus delta t minus xt is going you do xt plus delta t minus xt is going you do xt plus delta t minus xt is going to be this thing right. So it's going t…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 52: 28 you just add some stoasticity. So this you just add some stoasticity. So this is why it's called stochastic is why it's called stochastic is why it's called stochastic differential equation. different…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 53: 28 and we choose the for process right so we know f and g uh and then it's we know f and g uh and then it's we know f and g uh and then it's literally just gausian right so who literally just gausian rig…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 54: 21 to use a oiler solver which is like to use a oiler solver which is like probably simple simplest solver uh that probably simple simplest solver uh that probably simple simplest solver uh that then you…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 55: 09 apply the change in x to x and then we apply the change in x to x and then we apply delta t to t apply delta t to t apply delta t to t and that's it. and that's it. and that's it. And then we iterate …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 56: 13 written as this like this function this written as this like this function this this uh so there are four process can be this uh so there are four process can be this uh so there are four process can …
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 57: 20 the G as variance and in the VE formulation we do not have any formulation we do not have any formulation we do not have any adjustment of the X like whatever X you adjustment of the X like whatever X…
cmu-10799-diffusion-flow L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching 58: 09 did diffusion. This class we did did diffusion. This class we did scorebased model and we also learned scorebased model and we also learned scorebased model and we also learned that they're basically …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 00: 04 Hello everyone, welcome to the next Hello everyone, welcome to the next lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion models. di…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 01: 07 calculate the magnetic field for all the magnets and superimpose them together so magnets and superimpose them together so magnets and superimpose them together so that we have a cumulative universal …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 02: 13 idea where this ball came from. idea where this ball came from. But we have an idea where it came from. But we have an idea where it came from. But we have an idea where it came from. And essentially …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 03: 31 Okay. And the force required to pull the ball back the force required to pull the ball back the force required to pull the ball back to the start is the score field for the to the start is the score f…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 04: 34 Now this can also be represented as a noise vector. Right? noise vector. Right? noise vector. Right? [snorts] This is our flick. So [snorts] This is our flick. So [snorts] This is our flick. So essent…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 05: 37 And if this perturbed noise is very And if this perturbed noise is very small, you could assume that whatever small, you could assume that whatever small, you could assume that whatever score my I am …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 06: 33 exactly what we saw here. Minus noise divided by sigma. divided by sigma. divided by sigma. And the predicted score is what our And the predicted score is what our And the predicted score is what our …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 07: 41 as I said before uh the field of score matching has evolved matching has evolved matching has evolved through years. So in 2005 was the proof through years. So in 2005 was the proof through years. So …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 08: 49 that. It's a very beautifully written paper It's a very beautifully written paper It's a very beautifully written paper and in fact the textbook which I'm and in fact the textbook which I'm and in fac…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 09: 43 image or data sample into noise and then we predicted the reverse distribution we predicted the reverse distribution we predicted the reverse distribution from the noise back to the data. from the noi…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 10: 59 Now look at what is happening here. Your Now look at what is happening here. Your space is huge. This is the your entire space is huge. This is the your entire space is huge. This is the your entire s…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 12: 08 2005 which we explained and this is the loss that we try to minimize. Okay. Now this score is given by gradient of this score is given by gradient of this score is given by gradient of log of p of x. …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 13: 29 completely fine. Everything is defined. This the slope is defined right? You This the slope is defined right? You This the slope is defined right? You know exactly how your data changes as know exactl…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 14: 27 score vector will not be defined. score vector will not be defined. And imagine that you're going with a And imagine that you're going with a And imagine that you're going with a compass in your hand …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 15: 28 transformed from a low dimensional manifold to a higher dimensional space. manifold to a higher dimensional space. manifold to a higher dimensional space. It's taking up the space It's taking up the s…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 17: 00 Remember in the previous formulation by Remember in the previous formulation by Vincent, we just had one noise. Vincent, we just had one noise. Vincent, we just had one noise. We we added one noise wi…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 18: 24 you are flicking the magnet not just once with one level, but you're not just once with one level, but you're not just once with one level, but you're flicking it with multiple levels. flicking it wit…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 19: 42 multiple levels of noise? multiple levels of noise? We don't do this in diffusion also. In We don't do this in diffusion also. In We don't do this in diffusion also. In diffusion there is one constant…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 20: 48 is it a single neural network which is trained on trained on trained on uh for all these losses and if it's a uh for all these losses and if it's a uh for all these losses and if it's a single neural …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 22: 13 Remember now the score does not just Remember now the score does not just depend on X depend on X depend on X but it depends on X comma sigma where but it depends on X comma sigma where but it depends…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 23: 27 and land as close as possible to the and land as close as possible to the data point. Now the interesting thing is that I have Now the interesting thing is that I have a score function for a score fun…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 24: 58 The only difference is that it is performed sequentially performed sequentially performed sequentially multiple times. Let us understand how we multiple times. Let us understand how we multiple times.…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 26: 21 Okay. So you are purposely deliberately inputting a very high noise level so inputting a very high noise level so inputting a very high noise level so that that uh you move to areas where you probably…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 27: 17 here [snorts] the destination is not our data but it's far away from our data data but it's far away from our data data but it's far away from our data because we are adding so much noise. So because …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 28: 10 we are moving according to the lang dynamics update rule. This time the dynamics update rule. This time the dynamics update rule. This time the variance is a bit lower and um variance is a bit lower a…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 29: 07 sample the entire data, right? Even if, sample the entire data, right? Even if, let's say, you have handwritten digits let's say, you have handwritten digits let's say, you have handwritten digits and…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 30: 00 matching formula that it it it finally tries to predict the noise itself and tries to predict the noise itself and tries to predict the noise itself and that that is exactly what the diffusion that th…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 31: 01 please go through this for people who please go through this for people who like to see how both are connected and like to see how both are connected and like to see how both are connected and it's it…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 32: 14 uh and then which is exactly what which is exactly what which is exactly what we want to predict right we want to we want to predict right we want to we want to predict right we want to understand tha…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 33: 12 the reverse transition kernel is we are the reverse transition kernel is we are trying to predict how much noise was trying to predict how much noise was trying to predict how much noise was added in …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 34: 09 when we are inferring we do it n uh we when we are inferring we do it n uh we do it l times as as we saw here there do it l times as as we saw here there do it l times as as we saw here there are l di…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 34: 56 this and you do it for some specific time steps t so if the time step is time steps t so if the time step is time steps t so if the time step is greater than t you just stop this greater than t you ju…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 35: 59 So this is exactly how the anal So this is exactly how the anal land dynamics loop looks like. land dynamics loop looks like. land dynamics loop looks like. Let's take a look at a practical example Le…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 37: 01 you uh define your score network which is uh define your score network which is uh define your score network which is typically a conditional unit. typically a conditional unit. typically a conditiona…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 38: 08 And that is exactly what we see in the practical U notebook as well. practical U notebook as well. practical U notebook as well. [snorts] But this is done for different [snorts] But this is done for d…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 39: 15 if I run this inference for this epox where the loss is very low, you can see where the loss is very low, you can see where the loss is very low, you can see it's almost 554, it's almost 554, it's alm…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 40: 06 manifold and by adding noise we are spreading it and by adding noise we are spreading it and by adding noise we are spreading it so that it occupies the entire space. so that it occupies the entire sp…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 40: 48 which have very low density of of the data. And adding noise helps us to do data. And adding noise helps us to do data. And adding noise helps us to do that. It helps us to have the gradient that. It …
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 41: 41 strength of low strength also. And once strength of low strength also. And once we train the network to understand all we train the network to understand all we train the network to understand all the…
diffusion-principles-vizuara L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models 42: 36 images, videos or audio. As an homework, images, videos or audio. As an homework, I want you to play around with this I want you to play around with this I want you to play around with this notebook. …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 00: 04 Hello everyone and welcome to this next Hello everyone and welcome to this next lecture in of the course principles of lecture in of the course principles of lecture in of the course principles of dif…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 01: 15 of X. And our objective then becomes to And our objective then becomes to And our objective then becomes to determine determine determine the probability distribution P5 of X the probability distribut…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 02: 30 The objective is to match the output image with the input image as close as image with the input image as close as image with the input image as close as possible. possible. possible. And here we look…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 03: 46 intelligence can be used to produce intelligence can be used to produce images which look like they belong to images which look like they belong to images which look like they belong to the underlying…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 04: 42 it becomes uniform. We took these key ideas and we We took these key ideas and we We took these key ideas and we constructed a diffusion model which is constructed a diffusion model which is construct…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 05: 49 So you inject noise at every transition. So this is the first transition, second, So this is the first transition, second, So this is the first transition, second, third etc. third etc. third etc. Now…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 06: 45 So our objective is to minimize the mean So our objective is to minimize the mean square error between epsilon and epsilon square error between epsilon and epsilon square error between epsilon and eps…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 07: 41 something which is common in every single image on earth. And at a first single image on earth. And at a first single image on earth. And at a first glance you're not able to answer it glance you're n…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 08: 36 in the field of image generation through generative AI. Models were extremely generative AI. Models were extremely generative AI. Models were extremely important in creating a new wave in the importan…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 09: 45 models in a single cohesive framework. models in a single cohesive framework. So this is something which is very So this is something which is very So this is something which is very unique and and ve…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 10: 48 So in the energy based models our So in the energy based models our objective is to predict P5 of X using objective is to predict P5 of X using objective is to predict P5 of X using energy functions. …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 11: 57 drop this ball in this curve. What will happen is that the ball will What will happen is that the ball will What will happen is that the ball will slowly settle down, slowly fall down and slowly settl…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 12: 57 down and settle on the ground? The reason is that at on on on the ground reason is that at on on on the ground reason is that at on on on the ground the potential energy of the apple is is the potenti…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 13: 55 energy function can be a proxy for how energy function can be a proxy for how probable that particular data point or probable that particular data point or probable that particular data point or sampl…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 14: 52 course draw this schematically and we can say that okay we want this energy or can say that okay we want this energy or can say that okay we want this energy or or this probability curve to follow or …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 15: 51 my function should satisfy my function should satisfy and people use an exponential function and people use an exponential function and people use an exponential function to relate the energy to the p…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 17: 03 density as p5 of x. density as p5 of x. Then from the above graph we can find Then from the above graph we can find Then from the above graph we can find out a function which represents the out a func…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 17: 56 system u every system finally tries to go in a position with the minimum go in a position with the minimum go in a position with the minimum possible energy and this is something possible energy and t…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 18: 46 the whole idea behind energy based models models okay now let us discuss after this point okay now let us discuss after this point okay now let us discuss after this point let's take an example suppos…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 20: 00 add up to one? We need the probability to add up to one because if we are to add up to one because if we are to add up to one because if we are creating a lot of samples and we are creating a lot of s…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 20: 52 so let's let's take that example again. so let's let's take that example again. Uh let's say you have a dice. Okay. Uh Uh let's say you have a dice. Okay. Uh Uh let's say you have a dice. Okay. Uh and…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 21: 42 and the solution is that we can simply and the solution is that we can simply normalize the probabilities by dividing normalize the probabilities by dividing normalize the probabilities by dividing it…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 22: 48 So for the discrete states it looked So for the discrete states it looked like this summation of e ra to e5 of x like this summation of e ra to e5 of x like this summation of e ra to e5 of x x goes fr…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 23: 46 energy based models. In other words, how do we find these energy functions which do we find these energy functions which do we find these energy functions which correspond to each of these states? cor…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 24: 58 push the good data up. push the good data up. So I want to move this curve like this. So I want to move this curve like this. So I want to move this curve like this. The left side I want to push down …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 25: 49 higher probability. All of us understand higher probability. All of us understand this. this. this. Now we will again use the same Now we will again use the same Now we will again use the same formula…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 26: 47 Uh here E5 of X denotes the energy function. Okay. So now if you break this down, Okay. So now if you break this down, this is the first term that you get. uh this is the first term that you get. uh t…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 28: 00 so p5 of x can be written as integral of p5 of x given z into p of z dz. p5 of x given z into p of z dz. p5 of x given z into p of z dz. And there we said that this integral is And there we said that …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 29: 00 bypasses the partition function. So you might be thinking that what is So you might be thinking that what is So you might be thinking that what is this new terminology which I've this new terminology …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 29: 56 And because of that we come up with a alternative objective function which alternative objective function which alternative objective function which uses something called as the score uses something c…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 31: 05 appreciation for people who actually sit appreciation for people who actually sit down, they do the mathematics and they down, they do the mathematics and they down, they do the mathematics and they c…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 32: 17 to occur. to occur. Let's let let's take a sample example. Let's let let's take a sample example. Let's let let's take a sample example. Let us say these black dots these are Let us say these black do…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 33: 18 score function does. It tells you the score function does. It tells you the direction to where the data is located. direction to where the data is located. direction to where the data is located. So i…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 34: 09 know where the data is where the score is the maximum is the maximum is the maximum we can start from any point and finally we can start from any point and finally we can start from any point and fina…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 34: 58 trajectory? Is is is it can I define it? Can I write Is is is it can I define it? Can I write Is is is it can I define it? Can I write an equation which can tell me how do I an equation which can tell…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 35: 58 Let's say I calculate the score for this Let's say I calculate the score for this point. Where will this score point point. Where will this score point point. Where will this score point towards to th…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 36: 53 right or the left uh the value will decline at x= to infinity or minus decline at x= to infinity or minus decline at x= to infinity or minus infinity the value becomes zero. infinity the value becomes…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 37: 55 Now this is just the first term. We have to take a gradient of this also. to take a gradient of this also. to take a gradient of this also. Gradient of a constant is zero and the Gradient of a constan…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 38: 44 So uh this is what we get if we actually plot the score function. You can see plot the score function. You can see plot the score function. You can see that the arrows are pointed towards the that the…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 39: 41 score-based framework but score-based framework but score-based methods score-based methods score-based methods is something which has come before is something which has come before is something which…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 40: 40 challenges which we started out with. We wanted something which can uh help us wanted something which can uh help us wanted something which can uh help us offer a tractable alternative to offer a trac…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 42: 09 Now what what happens is that when you take this gradient uh the the first term take this gradient uh the the first term take this gradient uh the the first term is minus grad of e5 of x which is simp…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 43: 06 that's all. So the partition function becomes zero So the partition function becomes zero So the partition function becomes zero and is not included in the calculation and is not included in the calcu…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 44: 02 makes sure that the score of the predicted model matches the score of the predicted model matches the score of the predicted model matches the score of the true probability distribution. true probabil…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 45: 33 So ideally we want a loss which looks like this. Now the a loss which looks like this. Now the a loss which looks like this. Now the question again becomes how do we question again becomes how do we q…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 46: 22 Let us assume that we have done the training. We have matched this score training. We have matched this score training. We have matched this score with the uh score for the true model for with the uh …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 47: 09 data point and I know where to go I can navigate my way and I can find samples navigate my way and I can find samples navigate my way and I can find samples from the distribution Okay. And uh this fro…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 48: 18 Here I can visually see that this is the deepest valley and I can see the deepest valley and I can see the deepest valley and I can see the treasure is hidden there. treasure is hidden there. treasure…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 49: 10 valley and I start here. and I want to reach the bottom of the valley. So I'll reach the bottom of the valley. So I'll reach the bottom of the valley. So I'll make a step like this. I'll again make a …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 50: 12 would get your next location. But what is the slope here? How do we But what is the slope here? How do we But what is the slope here? How do we find the slope? This is my energy find the slope? This i…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 51: 07 exactly what we are doing here. We are exactly what we are doing here. We are calculating the gradient of the energy calculating the gradient of the energy calculating the gradient of the energy and u…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 51: 59 this which is gradient of the energy function. So the negative of this function. So the negative of this function. So the negative of this actually is the score function itself. actually is the score …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 52: 52 gradient is zero. So you are just sitting there but there is another pit sitting there but there is another pit sitting there but there is another pit which is even below which is even below which is …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 53: 40 need to provide a shake which gives you enough random energy to kick you out of enough random energy to kick you out of enough random energy to kick you out of these small potholes so that you can the…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 54: 29 we are sampling from a gshian we are sampling from a gshian distribution with a mean of xi and a distribution with a mean of xi and a distribution with a mean of xi and a standard deviation of beta. s…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 55: 30 the total variance of this xt + 1 is maybe a constant or it's just one that maybe a constant or it's just one that maybe a constant or it's just one that is my hypothesis is my hypothesis is my hypoth…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 56: 32 Now we what we do is we replace this Now we what we do is we replace this grad of energy by s of x which we have grad of energy by s of x which we have grad of energy by s of x which we have already s…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 57: 16 trajectory let's say I start from here with this update I'll go like this with this update I'll go like this with this update I'll go like this and then I'll finally I I I need to come and then I'll f…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 57: 59 the score function which takes you from one location to the another and finally one location to the another and finally one location to the another and finally you end up at a space or a region where …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 58: 55 x is known to us. It looks something like this. The probability of finding like this. The probability of finding like this. The probability of finding the samples is maximum at these two the samples i…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 59: 55 comments. So first we define the setup which are So first we define the setup which are So first we define the setup which are the two mountain setup as as we saw the two mountain setup as as we saw t…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 60: 40 noise. We use the exact same equation which we discuss in theory which we discuss in theory which we discuss in theory and then we visualize the path. and then we visualize the path. and then we visua…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 61: 28 It's it's called a drunk hiker. It's like someone is drunk and walking It's like someone is drunk and walking It's like someone is drunk and walking here and there and finally reaching the here and th…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 62: 18 So I want all of you to run this tweak a little things here and there have fun little things here and there have fun little things here and there have fun with the notebook and really develop with the…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 63: 15 since sampling with langu dynamics only requires the score like we saw in this requires the score like we saw in this requires the score like we saw in this formula it only requires the score we formu…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 64: 27 and score matching fits this vector and score matching fits this vector field by minimizing the mean square field by minimizing the mean square field by minimizing the mean square error between the tr…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 65: 40 know how to pronounce this name. uh however they they came up with a uh however they they came up with a uh however they they came up with a technique to technique to technique to distill that objecti…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 66: 47 a summation of this and a constant. So basically minimizing this meant that So basically minimizing this meant that So basically minimizing this meant that we want to minimize this and you see here we…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 68: 01 like this. A1 A2 A3 A4 A5 A6 A4 A5 A6 A4 A5 A6 A7 A8 and A9. So the trace of this A7 A8 and A9. So the trace of this A7 A8 and A9. So the trace of this matrix is simply the addition of a1 + a5 matrix …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 69: 11 uh it is it is completely fine for our uh it is it is completely fine for our purposes what we need to understand is purposes what we need to understand is purposes what we need to understand is the i…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 70: 05 Which makes a lot of sense, right? In Which makes a lot of sense, right? In the previous example that we looked at the previous example that we looked at the previous example that we looked at these t…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 71: 14 where we are trying to make the data samples appear as syncs. samples appear as syncs. samples appear as syncs. which is what you see over here also. If which is what you see over here also. If which …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 72: 07 more to this term. So the this term drives So the this term drives So the this term drives scores to zero in the higher probability scores to zero in the higher probability scores to zero in the highe…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 73: 01 Now uh once we have understood this, Now uh once we have understood this, this is where the actual training of the this is where the actual training of the this is where the actual training of the uh …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 73: 57 distribution. So in a way we will take a look at the whole uh whole pipeline. look at the whole uh whole pipeline. look at the whole uh whole pipeline. Okay, let's try to understand this. Okay, let's …
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 75: 18 loss is made up of two terms. The first loss is made up of two terms. The first is the trace of the Jacobian is the trace of the Jacobian is the trace of the Jacobian which is what we discussed before…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 76: 10 predictions for the x score and the y score to create a vector. score to create a vector. score to create a vector. Next we have a training loop which is a Next we have a training loop which is a Next…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 77: 00 completely appear like a sync but there completely appear like a sync but there are arrows coming in from the left and are arrows coming in from the left and are arrows coming in from the left and the…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 77: 49 we go to any other point. So we are we go to any other point. So we are going to see trajectories like these. going to see trajectories like these. going to see trajectories like these. And in this ex…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 78: 33 perturve ourselves from the valleys so perturve ourselves from the valleys so that we don't miss any other treasures that we don't miss any other treasures that we don't miss any other treasures which…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 79: 19 them yet. We will make the connection in the next lecture. the next lecture. the next lecture. So initially used to enable efficient So initially used to enable efficient So initially used to enable e…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 80: 15 Scores are given by gradient of logarithm of uh the probability. It logarithm of uh the probability. It logarithm of uh the probability. It turns out that the scores appear as turns out that the score…
diffusion-principles-vizuara L6: Lecture 5 - Energy Based Models | Score Function | Principles of Diffusion Models 81: 02 neural network to match the scores and neural network to match the scores and to do that we use a score matching loss to do that we use a score matching loss to do that we use a score matching loss wh…