概念

Classifier Guidance

— 用独立训练的分类器梯度把无条件扩散"拉"向某个类别。

很多人第一次看到 "classifier guidance" 这个词,会以为它是"先训一个扩散模型,再训一个普通分类器,把两者拼起来"。这句话对一半,错一半。对的是"两个模型分开训"的拓扑结构;错的是"普通分类器"那部分。一个在 ImageNet 上训练好的 ResNet—— 它见过的输入永远是干净的自然图像;可是在扩散采样的中段(比如 $t=500$,对应 $\bar\alpha_t$ 大约 0.5),$x_t$ 已经被高斯噪声糊到肉眼几乎看不出原图了。把这种噪声图喂进 ResNet,它给出的预测分布几乎是均匀的 0.001 量级,梯度方向几乎完全由数值噪声决定,而不是由"图像里到底像不像猫"决定。这条信息一旦进入反向 SDE,就会把采样轨迹推到一个跟目标类别毫无关系的地方。Dhariwal & Nichol 在 2021 年那篇被称作 "Diffusion Models Beat GANs" 的论文里给出的 trick,正是额外训一个能吃下"任意噪声级别 $x_t$"的分类器 $p_\phi(c \mid x_t)$——它的训练集是 $\{(x_t, t, c)\}$ 三元组,把整个 forward process 的所有 $t$ 都覆盖一遍。

那为什么要费这么大劲?CMU 10799 Lecture 7 的讲师用了一个非常自然的苏格拉底式引入:他先让学生回答"条件生成最朴素的做法是什么"。台下马上有人答"把条件 $c$(文本、属性向量、深度图……)和噪声一起扔进网络,从头训一个 $\epsilon_\theta(x_t, t, c)$"。也有学生补充"我会再加一个 feature extractor,从 condition 里提取一个 embedding,沿着像素输入一起送进去"。讲师肯定了这两个都是正确答案,但马上反问:"这个写法的瓶颈在哪儿?"教室里安静了几秒,他自己把答案点破:

「Where does the label come from? You need different training for different conditions. For example, if this model is text-conditioned, then we cannot use it for class-label condition; if it's class-label conditioned, we cannot use it for depth map. Even if you could formulate your model to condition on everything you can think of, you also need to collect data for everything. That's really really expensive.」—— CMU 10799 L7(@07:53–08:45)

这一段把 classifier guidance 出现的真正动机讲得最透:扩散模型训练贵——一个 ImageNet 256×256 的 ADM 跑下来要数千 GPU-hours——可是无条件模型已经"知道怎么画一张真实的人脸"了。为什么每来一种新条件(文本、深度图、骨架、CLIP embedding……)就要把这几千 GPU-hours 重烧一遍?讲师在课程末尾把这个问题正式抛出来:「Is it possible to reuse the unconditional model to do conditional generation?」这句话就是 classifier guidance 的灵魂——把条件信息从生成器里剥离出来,放到一个独立的、轻量的、可以更换的"控制模块"里。

🟡 教材外补充:讲师在课上没有给出形象比喻,但理解上你可以把它想象成无条件扩散是一辆在数据流形上自动驾驶的车,分类器梯度是副驾驶不停地往方向盘上"轻推"——车本身仍然只懂"画真实图",副驾驶专门负责"往猫的方向偏一点"。$w$ 就是副驾驶的力气:$w=0$ 时副驾驶完全不动,采样退化为无条件;$w$ 适中(论文里通常 1.0–3.0)时车被温柔地引导到目标类别附近;$w$ 过大(10 以上)时副驾驶猛打方向,车被拽离数据流形,画出过饱和、形态怪异的"猫的卡通漫画"。这种诚意过度的现象在 Dhariwal & Nichol 论文 Table 3 里有详细量化,被称为 "guidance scale trade-off"——这一发现后来在 CFG、CLIP-guided diffusion 乃至 Stable Diffusion 的 CFG scale 参数里都能看到同样的 U 形曲线。

读到这里你应该已经能在脑子里画出整个故事:扩散模型 $\epsilon_\theta(x_t, t)$ 是不变的知识基座,存着"如何画一张真实图"的全部先验;分类器 $p_\phi(c \mid x_t)$ 是可插拔的控制头,存着"什么样的 $x$ 看起来更像类别 $c$"的判别知识;采样阶段把两者输出按 Bayes 拆解的方式相加,就得到一个朝向条件 $c$ 的采样轨迹。整个 trick 的工程优雅之处在于:换条件只需要换分类器(而且分类器比扩散模型小得多,参数量、显存、训练时间都低一个数量级);扩散模型本身一次训好、永远复用,无论后续要做猫狗分类、年龄回归还是属性控制,底座都不动。在 2021 年那个 GPU 极其稀缺、训练一个 ImageNet 扩散模型动辄几千 GPU-hours 的时代,这种"复用性"的工程价值再怎么强调都不为过。

从 Bayes 到 score 的两行推导

classifier guidance 的全部数学只需要两步。出发点是条件分布的 Bayes 拆解:

$$p(x_t \mid c) = \frac{p(x_t)\, p(c \mid x_t)}{p(c)}.$$

两边取对数、再对 $x_t$ 求梯度($\log p(c)$ 不含 $x_t$,导数为 0):

$$\nabla_{x_t} \log p(x_t \mid c) = \nabla_{x_t} \log p(x_t) + \nabla_{x_t} \log p(c \mid x_t).$$

Why:第一项是无条件 score——你训练好的扩散网络(在 $\epsilon$-参数化下等价于 $-\epsilon_\theta(x_t,t)/\sqrt{1-\bar\alpha_t}$,见 Tweedie 与 score 的换算)就在算它。第二项是分类器 log-似然对输入的梯度——只要你有一个能吃 $x_t$ 的分类器 $p_\phi(c \mid x_t)$,反向传播一下就能拿到。

把第二项乘上一个 guidance scale $w$,就得到 Dhariwal & Nichol 真正用的"加权"版本:

$$\tilde\nabla_{x_t} \log p(x_t \mid c) \;=\; \nabla_{x_t} \log p(x_t) \;+\; w \cdot \nabla_{x_t} \log p_\phi(c \mid x_t).$$

Why:$w=1$ 对应"诚实的 Bayes 拆解";但实证上 $w>1$(论文里常用 $w=10$ 量级)会让样本更像 目标类别,代价是多样性下降。这是 Dhariwal & Nichol 论文里第一次以可量化的 FID/IS 给出 guidance scale 这个超参——后来 CFG、CLIP guidance 全部沿用同一个 $w$ 概念。

反向采样里的具体写法

在 $\epsilon$-参数化的反向步里,把上式翻译成对 $\epsilon$ 的修正(用关系 $\epsilon = -\sqrt{1-\bar\alpha_t}\,\nabla \log p$):

$$\hat\epsilon(x_t, t, c) \;=\; \epsilon_\theta(x_t, t) \;-\; w \sqrt{1-\bar\alpha_t}\; \nabla_{x_t} \log p_\phi(c \mid x_t).$$

Why:负号源于 Tweedie 关系里的负号;$\sqrt{1-\bar\alpha_t}$ 是把 score 换算成 $\epsilon$ 单位的尺度因子。然后照常喂给 DDPM 或 DDIM 的反向更新——只多了一次分类器的 forward + backward,无条件模型完全不动。

⚠️ 工程地雷:分类器 $p_\phi(c \mid x_t)$ 必须在带噪 数据上训。具体做法是先按 forward process 公式 $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$ 给训练集每张图加上对应 $t$ 的噪声,把 $(x_t, t, c)$ 当作分类器输入。Dhariwal & Nichol 用了一个 U-Net 编码器结构、参数量约 8× 大于普通 ImageNet 分类器,因为它要同时嵌入时间步 $t$。
x_t 含噪图 ε_θ(x_t, t) 无条件扩散模型 (冻结) ∇log p_φ(c | x_t) 噪声鲁棒分类器 (在所有 t 上训过) + ε_θ w · √(1−ᾱ_t) · ∇log p_φ ε̂(x_t, t, c) 送入 DDPM/DDIM

读图要点:蓝色框是训练阶段就固定的扩散网络,整个 guidance 流程中它的权重一字都不会改;橙色框是额外训的噪声鲁棒分类器;两条数据流在采样的每一步都各跑一次,叠加后才进 DDPM 或 DDIM 的反向更新。注意 $x_t$ 同时进两个网络——也就是说,每个反向步要付出"1 次扩散前向 + 1 次分类器前向 + 1 次分类器反向"的算力,比无条件采样贵大约 1.3× 到 2×(取决于分类器规模)。这点开销换来的好处是巨大的:你可以事后给任何已经训好的扩散模型加上类别控制,不必重训生成器。这在 2021 年是一项极有冲击力的工程胜利——它意味着 OpenAI 的 unconditional ADM checkpoint 可以被一个外部分类器"接管方向盘",而不需要把那几千 GPU-hours 重烧一次。

为什么"开放条件"会让 classifier guidance 崩溃

CMU 讲师在课上把这个限制讲得很直接:「You can't just condition on everything you can think of」。在 classifier guidance 框架下,"条件 $c$"必须能被一个 softmax 输出表示——也就是说,$c$ 得是一个有限、离散、可枚举的类别集合。 ImageNet 的 1000 类、CIFAR 的 10 类,没问题;可一旦 $c$ 是"一句任意的英文文本"——比如 "an astronaut riding a horse on Mars"——你不可能训一个分类器去枚举所有自然语言句子。即使你硬要做,CLIP 风格的"text-image 对齐"也只是把文本嵌入到一个连续 embedding 空间,不是 classifier guidance 严格要求的 softmax 输出。这是 classifier guidance 在 2021 年之后迅速被 CFG 取代的根本原因,不是因为 CFG 数学更优雅,而是因为 CFG 能处理 classifier guidance 在原理层面无法处理的开放条件。

另外两个工程痛点也值得记一下。第一,分类器训练成本:噪声鲁棒分类器虽然比扩散网络小,但仍然要在整个扩散数据集上训练,而且要把 $t$ 当作输入(通常用类似 sinusoidal embedding 的方式)。第二,guidance 信号的尺度不稳:分类器 logits 的范数随 $t$ 变化巨大——靠近 $t=T$ 时图像几乎全噪声,分类器输出接近均匀,梯度小;靠近 $t=0$ 时图像清晰,梯度大。这意味着固定 $w$ 在不同 $t$ 段的"实际推力"非常不均匀,工程上经常需要再加一个 $t$-相关的归一化项。

🟡 教材外补充:CFG 的精妙之处恰好在于它完全不用分类器—— 它让同一个扩散网络在训练时偶尔丢掉条件 $c$(典型 10% 概率),采样时把"有条件预测 $\epsilon_\theta(x_t,t,c)$"与"无条件预测 $\epsilon_\theta(x_t,t,\varnothing)$"的差当作隐式的 $\nabla \log p(c \mid x_t)$ 估计,于是开放条件(文本、CLIP embedding、姿势骨架……)也能用了。可以这样理解:classifier guidance 把"控制信号"放在外部、需要单独训练;CFG 把"控制信号"内化进生成器、共享同一组权重——这正是 Stable Diffusion / DALL-E 2 / Imagen 全部选 CFG 的原因。

历史定位与现代回响

从时间线看,classifier guidance 是 2020–2021 年扩散模型从"能跑"走向"能打 GAN"的关键节点。在它之前,无条件扩散的样本质量虽然稳定但缺乏可控性;GAN 派阵营则一直握着 ImageNet 上最好的条件生成 FID。Dhariwal & Nichol 用 classifier guidance 在 ImageNet 128×128 上把 FID 压到 2.97,第一次正面击败 BigGAN-deep——这条新闻直接导致 OpenAI 后续把研究方向全面押注扩散,催生了 GLIDE、DALL-E 2 等模型。换句话说,classifier guidance 是扩散模型"统治视觉生成"叙事里的起爆点。

不过 classifier guidance 本身的寿命相当短:CFG 在同年(2021)由 Ho & Salimans 提出,几乎立刻把它替代——所有需要开放条件的应用都只能选 CFG。但 classifier guidance 的思想仍然活着,并且以"外挂式 guidance"的形态广泛存在。CLIP-guided diffusion 把分类器换成 CLIP 图文相似度;DPS(diffusion posterior sampling)把分类器换成测量模型 $\nabla \log p(y \mid x_t)$,用于图像修复 / 反卷积 / 压缩感知;DragDiffusion、ControlNet 的早期版本里都能看到 classifier guidance 的影子——所有"用外部信号在采样时校正扩散轨迹"的方法,都是它的现代变体。所以理解 classifier guidance 不是在学一个过时的算法,而是在学所有 guidance 方法的母版。

下面按钮按顺序展示反向一步的等式链。注意看什么不变量:无条件项 $\epsilon_\theta(x_t,t)$ 始终不变,变化的只是分类器梯度项的权重 $w$——这就是"复用无条件模型"的具体含义。

$p(x_t \mid c) = \dfrac{p(x_t)\, p(c \mid x_t)}{p(c)}$ — 仅是 Bayes 公式,没有任何近似。
$\nabla_{x_t} \log p(x_t \mid c) = \underbrace{\nabla_{x_t} \log p(x_t)}_{\text{无条件 score}} + \underbrace{\nabla_{x_t} \log p(c \mid x_t)}_{\text{分类器梯度}}$
$\tilde\nabla = \nabla \log p(x_t) + \boldsymbol{w} \cdot \nabla \log p_\phi(c \mid x_t)$ — $w>1$ 牺牲多样性换"更像目标类别"。
$\hat\epsilon(x_t,t,c) = \epsilon_\theta(x_t,t) - w\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\log p_\phi(c\mid x_t)$ — 用 Tweedie 把 score 换成 $\epsilon$。
把 $\hat\epsilon$ 代进 DDIM 的反向更新 $x_{t-1} = \sqrt{\bar\alpha_{t-1}}\,\hat x_0 + \sqrt{1-\bar\alpha_{t-1}}\,\hat\epsilon$,迭代到 $t=0$。

手算一步:1D toy

设 $x_t = 0.8$,$t$ 对应 $\bar\alpha_t = 0.64$(即 $\sqrt{1-\bar\alpha_t} = 0.6$)。扩散网络输出 $\epsilon_\theta = 0.5$。再设你有一个二分类器("猫 / 非猫"),它在 $x_t=0.8$ 处对类别"猫"的 log-likelihood 梯度为 $\nabla_{x_t} \log p_\phi(\text{cat}\mid x_t) = 1.5$。先取 $w=2$。

代入修正公式:

$$\hat\epsilon \;=\; \epsilon_\theta \;-\; w \sqrt{1-\bar\alpha_t}\, \nabla \log p_\phi \;=\; 0.5 \;-\; 2 \cdot 0.6 \cdot 1.5 \;=\; 0.5 \;-\; 1.8 \;=\; -1.3.$$

解读:无条件预测说"图里大约有 0.5 单位噪声需要扣除";分类器说"往 +x 方向走 1.5 单位能更像猫"。两者合成后,等效噪声变成 $-1.3$——也就是反向步里会朝 +x 方向多走 $1.8$ 个单位。$w$ 越大,这"多走的部分"越激进。把同一道题里的 $w$ 换成 $w=0$,立即得到 $\hat\epsilon = 0.5$——退化回无条件采样;换成 $w=5$,得 $\hat\epsilon = 0.5 - 5 \cdot 0.6 \cdot 1.5 = -4$,朝 +x 方向猛冲 $4.5$ 单位,几乎肯定冲出数据流形。这个一维玩具把 $w$ 的作用刻画得很直白:它是"分类器投票权重"的旋钮。

再用同一个例子算一下 $w=1$(诚实 Bayes 拆解):$\hat\epsilon = 0.5 - 1 \cdot 0.6 \cdot 1.5 = -0.4$。可以看到在 $w=1$ 与 $w=2$ 之间数值变化是 $-0.4 \to -1.3$,差距已经接近 1 个 $\epsilon$ 单位——而 DDIM 一步的反向更新里,$\epsilon$ 的尺度直接决定 $x_{t-1}$ 偏离无条件预测的程度。这就是为什么 Dhariwal & Nichol 论文里 $w$ 的甜点要靠 grid search 找:每个数据集、每个 $t$ schedule 下,$w$ 的"最佳值"都不一样。把这一节读到这里,你就能从单步公式直接想象多步采样的累积效应:一个反向轨迹由几十到几百步组成,每一步都有这样一次无条件预测 + 分类器修正的合成,累积起来就是图像被一步步"推"到条件 $c$ 所代表的区域。

常见误区

  1. "任何预训练分类器都能拿来当 $p_\phi$"。反例:把 ImageNet 上训好的 ResNet-50 拿来,对一张 $t=500$ 的 noisy $x_t$(信噪比约 0 dB)做 forward,top-1 输出几乎是均匀分布的 0.001 量级,梯度的数值噪声远大于真实信号——采样会朝完全错误的方向漂移,最终图像跟目标类别毫无关系。必须在 $\{(x_t, t, c) : t \in [0, T]\}$ 的全噪声范围上重训分类器,并且把 $t$ 作为额外输入(通常是 sinusoidal time embedding),让它"知道"当前噪声水平。Dhariwal & Nichol 用的就是一个 U-Net 编码器结构的分类器,专门为这个目的设计。
  2. "classifier guidance 就是 CFG,只是名字不同"。反例:CFG 在同一个扩散网络里训 $\epsilon_\theta(x_t, t, c)$ 和 $\epsilon_\theta(x_t, t, \varnothing)$(通过条件 dropout 实现),采样时算 $(1+w)\epsilon_\theta(x_t,t,c) - w\epsilon_\theta(x_t,t,\varnothing)$——整个流程里根本没有独立分类器。两者数学起点都是 Bayes 拆解 $\nabla \log p(x|c) = \nabla \log p(x) + \nabla \log p(c|x)$,但实现路径完全不同:classifier guidance 需要 2 个独立网络(扩散+分类器),CFG 只需 1 个;classifier guidance 限离散类别集合,CFG 支持任意 embedding(文本、CLIP、姿势骨架……);classifier guidance 每步多 1 次分类器反向传播,CFG 每步多 1 次扩散模型前向。理解这两条不同后,就能解释为什么 Stable Diffusion 等所有现代文本到图像模型选 CFG 而不是 classifier guidance。
  3. "$w$ 越大越好,因为更像目标类别"。反例:Dhariwal & Nichol 论文 Table 3 在 ImageNet 128×128 上的实测数据,$w=1.0$ 时 FID = 2.97;当 $w$ 增大到 10 左右,FID 飙到 11+,因为样本被推离数据流形、出现过饱和颜色、重复纹理、肢体扭曲等典型 "guidance artifacts"。$w$ 的甜点通常在 $[1, 3]$ 区间,并且对类别敏感——画"豹"的最佳 $w$ 和画"花瓶"的最佳 $w$ 可能差一倍。这一现象后来在 CFG 里被复现为"CFG scale 旋钮",几乎每个 Stable Diffusion 用户都遇到过 "$w=15$ 出来的图像质感很塑料" 的经验——根源就是 classifier guidance 的同一条 trade-off 曲线。

相关概念

  • Classifier-Free Guidance — 直接继承者:去掉独立分类器,把 guidance 内化进扩散网络本身。
  • DDIM — classifier guidance 通常与 DDIM 反向步合用,因为后者步数少、guidance 信号每步占比更显著。
  • DDPM — guidance 也兼容原始 DDPM 随机反向;论文同时报告两种采样器结果。
  • Image Generation — 2021 年 "Diffusion Models Beat GANs" 的核心实证就是 classifier guidance 让 ImageNet 类条件生成首次超过 BigGAN。

来源

  1. cmu-10799-diffusion-flow · L7 · "haven't talked about conditional generation yet" 引入 · (3:35)
  2. cmu-10799-diffusion-flow · L7 · "different training for different conditions" 瓶颈讨论 · (7:53)
  3. cmu-10799-diffusion-flow · L7 · "reuse the unconditional model" 引出 guidance · (9:27)
查看 83 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 00: 03 All right. So let's start get started All right. So let's start get started for the lecture. for the lecture. for the lecture. Last time we learned DDIM which is like Last time we learned DDIM which i…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 00: 52 these things can also speed things up or these things can also speed things up or get better results. Now we go like uh time z Now we go like uh time z like t minus or time t to tus 10 like t minus or…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 01: 47 and uh one thing about this design space things is that I misspoke um last time things is that I misspoke um last time things is that I misspoke um last time about about about essentially basically wh…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 02: 32 medium time steps rather than at the beginning or at the end. Any questions beginning or at the end. Any questions beginning or at the end. Any questions here? Okay, cool. All right, so now that we Ok…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 03: 35 now we haven't talked about anything about conditional generation yet. So about conditional generation yet. So about conditional generation yet. So far, everything is unconditional. We far, everything…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 04: 34 usually what people would do like the first thing you would try uh is to first thing you would try uh is to first thing you would try uh is to basically you still have your noise that basically you st…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 05: 17 conditional um sample based on whatever the condition that you specified right the condition that you specified right the condition that you specified right okay so what what is the what is the okay s…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 06: 16 essentially have another feature from for example maybe another image or like for example maybe another image or like for example maybe another image or like some uh maybe another like 3D mesh or some…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 07: 00 norm like we want dark hair or some norm like we want dark hair or some different hair? different hair? different hair? Um, so you mean like basically how would Um, so you mean like basically how woul…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 07: 53 label condition, right? If if this model is class label condition then we cannot is class label condition then we cannot is class label condition then we cannot use it for like say condition on like u…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 08: 45 like the data collection as we uh may like the data collection as we uh may know now is like very very expensive know now is like very very expensive know now is like very very expensive thing. And th…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 09: 27 um which is that basically if you think about it right the unconditional model about it right the unconditional model about it right the unconditional model that we learned already has a lot of that w…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 10: 17 unconditional model to do conditional unconditional model to do conditional generation at all? sort of. But there's like a very very sort of. But there's like a very very simple answer to this. Okay, …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 11: 17 probability of the label given the model or the the image. Right? So if you write or the the image. Right? So if you write or the the image. Right? So if you write it in terms of if you write it in te…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 12: 08 plus a uh the gradient from a discriminative model. Yeah, discriminative model. Yeah, discriminative model. Yeah, >> here influence like both the model >> here influence like both the model >> here in…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 13: 00 That's it. And you just run your That's it. And you just run your sampling as normal and you get a sampling as normal and you get a sampling as normal and you get a conditional sample. conditional sam…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 14: 02 >> it's a classifier. So it's either like >> it's a classifier. So it's either like it can be anything. So depending on your it can be anything. So depending on your it can be anything. So depending o…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 14: 46 training data. You cannot do some interpolation to generate some other interpolation to generate some other interpolation to generate some other thing. thing. thing. >> Um yes uh that is a very great …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 15: 29 like in painting and this is like colorization. Yeah. So you can just like colorization. Yeah. So you can just like colorization. Yeah. So you can just like use the same unconditional model to to use …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 16: 22 need to actually specifically train this classifier uh to to be able to recognize classifier uh to to be able to recognize classifier uh to to be able to recognize a noisy image and and and be able to…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 17: 13 nodding their head. nodding their head. Oh, someone someone on online. Is that Oh, someone someone on online. Is that Oh, someone someone on online. Is that >> Oh, no. >> Oh, no. >> Oh, no. >> Oh, oka…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 18: 10 basically just input your um your the the the less noisy the clean data the the less noisy the clean data the the less noisy the clean data estimation and the condition that you estimation and the con…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 19: 02 gradient that you got from your clean data estimation um and your data estimation um and your data estimation um and your off-the-shelf classifier off-the-shelf classifier off-the-shelf classifier uh …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 20: 27 so, uh, we we took the grade in regards so, uh, we we took the grade in regards to XT. Uh, but like that's not where the to XT. Uh, but like that's not where the to XT. Uh, but like that's not where t…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 21: 19 model and one large size model right so this thing is just going to takes you this thing is just going to takes you this thing is just going to takes you like a lot of compute that's one thing like a …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 22: 06 like a ball and then the the the noise level the manifold for each noise level level the manifold for each noise level level the manifold for each noise level is just just like layered shells that is …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 22: 56 distance from the origin. Uh and then and then like basically we're going to and then like basically we're going to and then like basically we're going to see that around like 90% of the density see t…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 23: 44 the manifold at all. So the guided sample can completely go off the sample can completely go off the sample can completely go off the manifold and just like wander off and manifold and just like wande…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 24: 30 these things basically just contribute to the fact that DPS is really really to the fact that DPS is really really to the fact that DPS is really really really slow and if you try to reduce the really…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 25: 49 little bit like how do we is there any little bit like how do we is there any other ways that we can apply guidance. >> Yeah. Yeah. So basically the first thing >> Yeah. Yeah. So basically the first t…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 26: 36 expensive thing to do. So because the because the classifier is usually really because the classifier is usually really because the classifier is usually really small. So yeah, so the first thing that…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 27: 29 manifold right so what should we do next manifold right so what should we do next the other very easy thing to do here is the other very easy thing to do here is the other very easy thing to do here i…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 28: 11 like a property of the like the low low dimensional manifold. So basically you dimensional manifold. So basically you dimensional manifold. So basically you can you can basically just like have a can …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 28: 58 class or something. Uh but basically you can actually get access to the data can actually get access to the data can actually get access to the data manifold via the decoder of an manifold via the dec…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 29: 44 the like the the yeah the low dimensional like representation space. dimensional like representation space. dimensional like representation space. Um so essentially this is like kind of Um so essentia…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 30: 32 samples and label them with noise levels, right? Nobody. Nobody would do levels, right? Nobody. Nobody would do levels, right? Nobody. Nobody would do that. Um but if you that. Um but if you that. Um …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 31: 24 we have one for clean IM. Oh yes yes uh we have one for clean IM. Oh yes yes uh you can speak up by the way the the you can speak up by the way the the you can speak up by the way the the people onlin…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 32: 02 to assume that we have access to autoenccoders? Uh we're going to talk autoenccoders? Uh we're going to talk autoenccoders? Uh we're going to talk about the next class actually but about the next clas…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 32: 52 uh and you can just uh and and this this algorithm is applicable to all algorithm is applicable to all algorithm is applicable to all differentiable guidance or classifiers differentiable guidance or …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 33: 37 into a text condition model without training anything by just applying clip training anything by just applying clip training anything by just applying clip clip guidance to your to to your clip guidan…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 34: 29 go to your classifier, right? So all you need to do is to take gradient all the need to do is to take gradient all the need to do is to take gradient all the way back and that's that gradient will way…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 35: 15 your uh like your your your image. Say your uh like your your your image. Say for example you have like a gausian blur for example you have like a gausian blur for example you have like a gausian blur…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 36: 12 Now that we're in the realm of training Now that we're in the realm of training free uh guidance, is there any other way free uh guidance, is there any other way free uh guidance, is there any other w…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 37: 15 nonisotropic noise um uh yeah sort of basically it's the same thing right I basically it's the same thing right I basically it's the same thing right I guess Adrian so like uh like basically guess Adr…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 37: 57 so like B and if your model is really so like B and if your model is really large the it's like a lot of memory large the it's like a lot of memory large the it's like a lot of memory requirement righ…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 39: 22 >> Okay. The classifier cannot be trained, >> Okay. The classifier cannot be trained, right? They cannot sample, right? You right? They cannot sample, right? You right? They cannot sample, right? You …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 40: 14 called like LGD loss guided diffusion or something um uh that paper that paper something um uh that paper that paper something um uh that paper that paper was actually done by uh like the same was act…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 40: 56 actually a great trick that you should do. create a proxy distribution. This is do. create a proxy distribution. This is do. create a proxy distribution. This is um that's kind of something that we're…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 41: 37 generated. Um but the problem is we don't have what we want to generate yet, don't have what we want to generate yet, don't have what we want to generate yet, right? Because if we do, why why are we r…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 42: 25 right? So, this needs to have a condition in it. condition in it. condition in it. Okay. Hold on. The people on in person Okay. Hold on. The people on in person Okay. Hold on. The people on in person …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 43: 07 a really rough uh or preliminary version a really rough uh or preliminary version of what we want to generate. But the of what we want to generate. But the of what we want to generate. But the problem…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 44: 18 that as x0ero. How about that? It turns out you can definitely do that. It turns out you can definitely do that. It turns out you can definitely do that. So say like this is like your input and So say…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 45: 06 um swapping in as one of the x0 in the um swapping in as one of the x0 in the middle and just just continue on middle and just just continue on middle and just just continue on diffusion as if nothing…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 45: 59 like rough ideas of like what do you want to generate and then just like want to generate and then just like want to generate and then just like hijack the the diffusion process by hijack the the diff…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 46: 49 to your input and and this way it's to your input and and this way it's going to create something that like kind going to create something that like kind going to create something that like kind of lo…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 47: 35 looking image from it based on your pre-trained diffusion model and say like pre-trained diffusion model and say like pre-trained diffusion model and say like you dislike the gray sky of um you dislik…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 48: 14 then the diffusion model can uh the SD is going to generate something that's is going to generate something that's is going to generate something that's realistic looking but with the correct realisti…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 48: 59 this is a very terribly masked uh image this is a very terribly masked uh image but basically you see how like if you do but basically you see how like if you do but basically you see how like if you …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 49: 34 better masking skills than I do then then you would have get a better um then you would have get a better um then you would have get a better um image. Yeah. like I think similar to image. Yeah. like …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 50: 29 majority of the the image is going to be noise right so you just going to noise right so you just going to noise right so you just going to generate unconditionally like normal but generate unconditio…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 51: 16 like how abstract I guess your image is, like how abstract I guess your image is, um essentially you you will need to um essentially you you will need to um essentially you you will need to adjust lik…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 51: 54 not never mind um but basically the reason why this is happening is that um reason why this is happening is that um reason why this is happening is that um the diffusion process the diffusion process …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 52: 44 reconstruct the uh the high frequency details more and more. And this is why details more and more. And this is why details more and more. And this is why basically based on like how much detail basic…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 53: 31 like determine like which time step should I go and stuff like that which is should I go and stuff like that which is should I go and stuff like that which is what you said I think this is what we wha…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 54: 17 generation uh if you start at time zero generation uh if you start at time zero which means that there's no diffusion at which means that there's no diffusion at which means that there's no diffusion …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 55: 11 people were saying is that like at the people were saying is that like at the beginning of the diffusion it's like beginning of the diffusion it's like beginning of the diffusion it's like really chao…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 55: 50 basically what you can do is say you're basically what you can do is say you're like unhappy with your uh w with this like unhappy with your uh w with this like unhappy with your uh w with this partic…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 56: 26 steps uh in your longriven dynamic for you to reach the the good part of the of you to reach the the good part of the of you to reach the the good part of the of the distribution. But yeah, but the di…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 57: 18 Yeah. Could you speak to like theoretical like Could you speak to like theoretical like Could you speak to like theoretical like I understand the concept of a GAN being I understand the concept of a G…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 58: 02 problem. Yeah, like if you do not train for this particular task, you just try for this particular task, you just try for this particular task, you just try to use your pre-trained unconditional to us…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 58: 45 search based thing like Monte Carlo research type of type of style type of research type of type of style type of research type of type of style type of thing or you can use like a like like thing or …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 59: 32 part or the other the part. >> Oh, so like masking here meaning that >> Oh, so like masking here meaning that >> Oh, so like masking here meaning that only the mask only the part that is only the mask…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 60: 26 learn good information. Uh usually unconditional models can give you very unconditional models can give you very unconditional models can give you very good uh image qualities and conditional good uh …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 61: 23 what you can do is if you do a weighted average of the two the two distributions average of the two the two distributions average of the two the two distributions what you can do is you can actually g…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 62: 16 conditions that is like that everyone conditions that is like that everyone has, right? So, for example, like say has, right? So, for example, like say has, right? So, for example, like say you're um …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 63: 00 score solo you kind of just like do a weighted mixture of of both. Uh so like weighted mixture of of both. Uh so like weighted mixture of of both. Uh so like your so basically it's gamma is usually yo…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 64: 01 have your uh input image and then you have your uh input image and then you can literally just like specify your can literally just like specify your can literally just like specify your condition to …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 64: 42 you'll have to have both the te the image and then the text and then extract image and then the text and then extract image and then the text and then extract the feature. But like so this is just to …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 65: 26 to like map your text into some feature to like map your text into some feature space and this feature extractor could space and this feature extractor could space and this feature extractor could be …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 66: 09 it's uh it it just gets you like clearer features and this is why it it actually features and this is why it it actually features and this is why it it actually just like usually gives you like more u…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 66: 55 but all right now you know how to turn but all right now you know how to turn an unconditional model into a an unconditional model into a an unconditional model into a conditional one. Congratulations…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 67: 39 Max. Please come to come in person. If Max. Please come to come in person. If you want to ask questions, it's going to you want to ask questions, it's going to you want to ask questions, it's going to…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 68: 18 is not final class but like the the fourth class in the next two weeks we're fourth class in the next two weeks we're fourth class in the next two weeks we're going to be hearing from uh Alex who's go…