概念 · Module 5: 采样、引导与应用

Classifier-Free Guidance

— 同一个网络兼任「有 prompt」与「没 prompt」两种角色,靠外推放大条件信号。

很多人第一次看 CFG 公式会以为:"$w$ 是一个 0 到 1 的混合系数,$w=1$ 完全用条件,$w=0$ 完全无条件"——正好反了。CFG 不是在两个预测之间插值,而是在它们连线的延长线上外推。$w=0$ 才是普通条件采样;$w=5$、$w=10$ 是把条件信号放大了几倍——这就是为什么 Stable Diffusion 常用 $w \in [5, 12]$ 仍然能跑、Imagen 论文给出 $w$ 最大可以推到 30,而这些数值在"插值"的解读下完全说不通。CFG 在向量空间里画的是"比条件还要更条件"的方向。

另一个常见误解:以为 CFG 需要一个外接分类器去打分。这是把它和 classifier guidance 混了。CFG 的全名是 Classifier-Free Guidance——Ho & Salimans 2022 这篇论文的核心贡献就是:把分类器扔掉,靠一个网络兼任两种角色就够了。它本质上是一种训练技巧加一行推理公式,不需要任何辅助模型。

CMU 10799 Lecture 7 的 Allen 是这样进入 CFG 的:上节课讲完 DDIM、ODE 求解器、加速到 10×,他抛出一个"反着来"的问题。

「我们之前从 $t$ 跳到 $t-10$ 来加速。那有没有人想过反过来跑?」——「Yeah, we're going to talk about it today.」—— CMU L7 @00:52

这句"反过来跑"就是 CFG 的几何直觉:先用条件得到一个去噪方向,再用无条件得到第二个方向;两者相减得到「条件独有的那部分梯度」,然后沿这部分继续往前走——这一步就是"反着退回",也是 CFG 在 score 向量上的本质。把它念出来:「用无条件方向把条件方向往前再外推一些」。

这节课正式引出条件生成时,Allen 让学生从最朴素的做法开始,一步步逼到 CFG:

「现在我们做的还都是无条件生成。如果要做条件生成——最直接的写法是什么?」「把 label 当 feature 塞进网络。」「对,那这种写法的核心瓶颈是什么?」「每加一种条件都要重训一次。」「Perfect answer。」—— CMU L7 @04:35–08:00

这就是 CFG 要解决的痛点:训练时不希望把模型按条件类型切碎。如果文本条件、类别条件、深度图条件各训一个模型,数据收集与算力成本爆炸;即便都塞进一个模型,朴素 conditioning 在条件信号弱时(文本与图像配对噪声大、CLIP embedding 维度高)会让网络偷懒——退化成"忽略条件、按训练集平均生成"。

Allen 在课上接着说:

「能不能用一个模型做所有条件?把它们全部嵌进去?——这就是大家在做的事。但其实还有另一种选择,这是我最喜欢的 diffusion 的一点。」—— CMU L7 @08:45

这个"另一种选择"正是 CFG:一个网络、一种训练流程,靠 condition dropout 同时学到 $p(x|c)$ 与 $p(x)$,采样时把两者的预测做外推,得到比"直接条件预测"更锐的样本。

在机器学习的上下文里,CFG 对应的是采样阶段的修改,而不是训练阶段——它和 epsilon-prediction、score matching、ELBO 都没有冲突。换句话说:你可以拿任何一个训好的条件扩散模型(只要训练时做了 condition dropout),在推理时零成本把无条件预测多算一次,就能用上 CFG。这也是它成为工业级标配的原因。

从 score 视角看:CFG 在做什么

扩散模型预测的 $\epsilon_\theta(x_t, t, c)$ 与条件 score 是同一物体(差一个常数 $\sigma_t$):

$$\epsilon_\theta(x_t, t, c) = -\sigma_t \nabla_{x_t} \log p_t(x_t \mid c)$$

这条对应关系在 DDPM 训练里隐式成立——它来自 Tweedie 公式:给定带噪样本 $x_t$,最小均方误差去噪等价于沿 score 方向走一步。换句话说,训扩散模型就是在拟合 score。明白了这一点,CFG 就可以完全在 score 空间里推导。

朴素条件采样沿着 $\nabla \log p(x|c)$ 走。CFG 想往「比条件更条件」的方向多迈一步——把条件相对无条件多出来的那部分梯度放大 $w$ 倍:

$$\hat s(x_t) = \underbrace{\nabla \log p(x_t \mid c)}_{\text{条件 score}} + w \cdot \underbrace{\big(\nabla \log p(x_t \mid c) - \nabla \log p(x_t)\big)}_{\text{条件相对于无条件的"差量"}}$$

把这一项化简,等价于:

$$\hat s(x_t) = (1+w)\,\nabla \log p(x_t \mid c) - w\,\nabla \log p(x_t)$$

用 $\epsilon = -\sigma_t \nabla \log p$ 换元,立刻得到工程实现里那条公式:

$$\boxed{\hat\epsilon = (1+w)\,\epsilon_\theta(x_t, t, c) - w\,\epsilon_\theta(x_t, t, \emptyset)}$$

Why:第一行是几何想法(往"条件 - 无条件"方向多走 $w$ 步);第二行是把 $p(x|c) - p(x)$ 的差合并;第三行只是把 score 翻成 $\epsilon$、符号反一下——三步一一对应。

等价的分布视角

到这里只是几何上的"外推一步"——但更深的问题是:外推后的 score 对应的是哪个分布?把 $\hat s$ 反推回它对应的密度,CFG 实际上是在采样这个非归一化的分布:

$$p_w(x \mid c) \propto \frac{p(x \mid c)^{1+w}}{p(x)^{w}} = p(x \mid c) \cdot \Big(\frac{p(x \mid c)}{p(x)}\Big)^{w}$$

Why:$\log p_w(x|c) = (1+w)\log p(x|c) - w \log p(x) + \text{const}$,求 $\nabla_x$ 之后常数消去,正好等于上面那条 $\hat s$。括号里的比值 $p(x|c)/p(x)$ 是条件相对无条件的似然比——CFG 把它 $w$ 次幂放大,等价于"对那些在条件下比无条件更可能的 $x$,再额外加权 $w$ 次"。这一步揭示了 CFG 真正的本质:它在改采样目标的分布形状,而不是在改去噪算法。

这条分布幂的形式还有一个直觉解读:温度退火。把 $p(x|c)^{1+w}/p(x)^w$ 看作"用先验 $p(x)^w$ 做归一化的条件后验幂"——$w$ 越大,分布越集中在条件似然比的众数上。这正是 CFG 提升样本锐度同时损失多样性的数学根源:高 $w$ 让分布变得尖锐 (peaked),所有采样轨迹都被吸向相同的几个高密度模态。

🟡 教材外补充:似然比 $p(x|c)/p(x)$ 在 Bayes 下等价于 $p(c|x)/p(c)$——这就是 classifier guidance 用一个外接分类器 $p(c|x)$ 显式给出的东西。Ho & Salimans 2022 的核心观察:把这个似然比内嵌到 ε 网络里,分类器就不用单独训了。一张网,两顶帽子,省掉了一整个判别器流水线。

训练侧:一行 dropout,两顶帽子

到目前为止的推导都建立在同一个网络能同时给出 $\epsilon_\theta(x_t, t, c)$ 与 $\epsilon_\theta(x_t, t, \emptyset)$ 这两个预测之上。怎么让一个网络学会两件事?答案出奇地朴素:训练时以一定概率把条件丢掉。

Ho & Salimans 2022 的训练伪代码只比无条件 DDPM 多一行:

# 训练循环
for x, c in loader:
    if random() < p_drop:        #  ← 只多这一行
        c = NULL_TOKEN              # 把条件替换成"占位符 ∅"
    t = uniform(1, T)
    noise = randn_like(x)
    x_t = sqrt(alpha_bar_t) * x + sqrt(1 - alpha_bar_t) * noise
    loss = ||noise - eps_theta(x_t, t, c)||^2

Why:以 $p_{\text{drop}}$ 概率把条件替换成 $\emptyset$(实际实现中是一个可学习的 null embedding 或全零向量),等价于让同一个网络以 $(1-p_{\text{drop}})$ 的样本学 $p(x|c)$ 的 score、以 $p_{\text{drop}}$ 的样本学 $p(x)$ 的 score。两个分布共享所有权重——网络在前向时看到 $c$ 还是 $\emptyset$,决定它进入条件分支还是无条件分支。这就是 CFG 比 classifier guidance 工程友好的关键:不用维护第二个网络,也不用为每个时间步重训分类器。

实现细节有三个容易忽略的地方:第一,null token 必须是可学习的(论文里写作 $\emptyset$ 实际是一个 embedding),而不是字面意义的全零——全零会让 attention layer 的 softmax 退化到均匀分布、削弱无条件分支的表达。第二,dropout 是按 batch 元素独立采样的,不是整个 batch 一起 drop;这样每个 batch 都同时贡献条件和无条件梯度。第三,条件如果是多模态拼接(文本 + image embedding),通常整体一起 drop而不是各自独立 drop——否则会出现"只有文本没图"的奇怪混合状态,CFG 时无对应的 $\emptyset$ 可用。

典型 $p_{\text{drop}}$:图像扩散 10–20%(Stable Diffusion 用 10%、Imagen 用 10%);视频扩散更高(25–30%)以保证时序无条件先验也学得到;condition-rich 任务(ControlNet 这类已经有强 spatial 条件的)有时 5% 就够。$p_{\text{drop}}$ 太大→无条件样本占据训练预算太多、条件分支欠拟合、整体 prompt-following 变差;太小→无条件分支训练数据不足、$\epsilon_\theta(x_t, t, \emptyset)$ 在采样时输出近似噪声,CFG 公式的减法操作放大噪声、样本质量崩盘。10% 是经验上的工程甜蜜点。

为什么 CFG 击败了 classifier guidance

Dhariwal & Nichol 2021 的 classifier guidance 需要一个对有噪声的 $x_t$ 都能正确分类的判别器——这件事在工程上非常贵:

  • 必须在每一个时间步 $t$ 的噪声分布上重训分类器:ImageNet 上的标准 ResNet 在 $t=500$ 的 $x_t$ 上准确率会跌到接近随机猜测,因为输入与训练分布完全脱节。
  • 分类器的梯度 $\nabla_x \log p_\phi(c|x_t)$ 噪声大,又容易把样本推到 adversarial 方向——视觉上看着像"金毛",实际是高频对抗噪点。Dhariwal & Nichol 不得不用大量 trick(gradient clipping、scale)才能稳定下来。
  • 类别空间一旦变化就要换分类器;文本条件下还得专门训一个对噪声鲁棒的 CLIP-style 模型,再算梯度。
  • 推理时每一步要做两次反向传播(一次去噪网络、一次分类器),延迟翻倍。

CFG 把似然比"内化"到 ε 网络里——训练只多一行 dropout,推理只多一次前向,零额外网络。代价仅仅是:每一步采样要对同一个网络调用两次(条件分支 + 无条件分支),可以做 batched forward 几乎无开销。这也是为什么 Stable Diffusion / Imagen / DiT / SD3 全线倒向 CFG。

CFG 的两个变体(值得一记)

原始 CFG 是静态的:整个采样过程用同一个 $w$。但在工程实践中人们发现两个改良:

  • Dynamic / interval CFG:只在中间时间步开 CFG,靠近 $t=0$ 和 $t=T$ 关掉。原因:早期高噪声步骤 score 估计不准,外推放大误差;末期低噪声步骤已经接近清晰图,再外推容易过饱和。Common-DiT 和 SD 3 都报告这种 interval guidance 能显著降低 FID。
  • Self-attention guidance / Perturbed-Attention Guidance:把"无条件分支"替换成"attention 被扰动的分支",相当于让 CFG 沿着注意力清晰度的方向外推,而不是沿着条件信号外推。这对 unconditional 模型也能用。

这些变体都没有改变 CFG 的核心结构(仍然是"主分支 + 弱分支的外推"),只是把"弱分支"换了一种构造方式——理解了原始 CFG 的几何,这些变体一望即知。

Linqi Zhou 的现场实战提醒:random CFG 的梯度噪声

CMU L11 的 guest lecturer 是 Luma AI 的 Linqi (Alex) Zhou。他讲到一个工程细节:

「我们实验里发现,training-time random CFG(也就是 condition dropout)会让 mean-flow 的梯度范数剧烈波动,但我们提出的 TVM 目标对此非常稳定。」—— CMU L11 @55:39

这条来自工业实战的提醒很有价值:CFG 的 condition dropout 是噪声源。每个 batch 里有些样本带条件、有些没条件,对应不同分布;如果你的训练目标对分布切换不鲁棒(比如某些 flow matching 变体),梯度会反复震荡。这是 condition dropout rate 选得不好(过大或过小)容易踩的坑——这也间接解释了为什么 SD 把 $p_{\text{drop}}$ 锁在 10% 而不是看起来"更均衡"的 50%。

score 空间 $x_t$ ∇log p(x)(无条件) ∇log p(x|c)(条件) 差量(条件 − 无条件) CFG(w=3)= 条件 + 3·差量

下面六步分别对应 CFG 推导链的一行。点按钮看 不变量:等式两边、每一步代数化简之后采样的目标分布都没变,变的只是表达形式(从"差量外推"换到"分布幂"再换到"ε 网络外推"再退回"似然比放大")。这个不变量很重要——它告诉你 CFG 的核心思想可以用四种等价语言描述,工程实现只用第三种,但理解必须靠第一、第四种。

想法:条件已经给了一个方向,但我们想"比条件更条件"——往(条件 − 无条件)的差量方向多走 $w$ 步。$$\hat s = s(x|c) + w \cdot \big(s(x|c) - s(x)\big)$$
把括号展开:$$\hat s = s(x|c) + w \cdot s(x|c) - w \cdot s(x)$$
合并 $s(x|c)$ 的系数:$$\hat s = (1+w)\,s(x|c) - w\,s(x)$$
用 $\epsilon = -\sigma_t \nabla \log p$,符号反向后系数不变:$$\hat\epsilon = (1+w)\,\epsilon_\theta(x_t, t, c) - w\,\epsilon_\theta(x_t, t, \emptyset)$$ 这就是代码里那一行。
$\hat s$ 是哪个分布的 score?积分回去:$\log p_w \propto (1+w)\log p(x|c) - w\log p(x)$,即 $$p_w(x|c) \propto \frac{p(x|c)^{1+w}}{p(x)^{w}}$$ CFG 在改采样目标。
写成似然比形式:$p_w(x|c) \propto p(x|c)\cdot\big(p(x|c)/p(x)\big)^{w}$。由 Bayes,$p(x|c)/p(x) = p(c|x)/p(c)$ ——后者正是 classifier guidance 用分类器显式提供的东西。CFG 把它隐式内嵌,省去分类器。

手算:$w$ 怎么放大条件信号

设某一步:条件预测 $\epsilon_\theta(x_t, t, c) = 0.8$(标量化简),无条件预测 $\epsilon_\theta(x_t, t, \emptyset) = 0.5$。差量 = $0.8 - 0.5 = 0.3$(条件多出来的部分)。

$w$$\hat\epsilon = (1+w)\cdot 0.8 - w\cdot 0.5$展开解读
0$1\cdot 0.8 - 0\cdot 0.5$= 0.80纯条件采样
1$2\cdot 0.8 - 1\cdot 0.5$= 1.10外推 1 步(差量 ×2)
3$4\cdot 0.8 - 3\cdot 0.5$= 1.70外推 3 步(差量 ×4)
7$8\cdot 0.8 - 7\cdot 0.5$= 2.90SD 默认附近
−1$0\cdot 0.8 - (-1)\cdot 0.5$= 0.50退化成无条件

验证差量倍数:$\hat\epsilon - \epsilon(x|c) = w\cdot 0.8 - w\cdot 0.5 = w\cdot(0.8 - 0.5) = w\cdot 0.3$。Why:CFG 相对纯条件采样多出来的修正量正好是 $w$ 倍的差量——这就是为什么人们把 $w$ 叫"guidance strength"或"guidance scale",并且只有相对差量被放大、绝对 $\epsilon$ 大小本身并不重要。

对 Stable Diffusion 的 $w=7.5$,差量被放大 7.5 倍。这也解释了高 $w$ 下样本变锐、颜色饱和、但多样性骤降——所有样本都被推到"最像条件"的那个尖锐众数;当差量被放大到 $\hat\epsilon$ 远超 $\epsilon$ 网络见过的训练分布时,去噪过程会跑出训练流形,产生饱和过曝、毛刺、伪影。这就是为什么 Imagen 引入了 dynamic thresholding(把超出范围的像素值动态截断)来配合高 $w$ 使用。

把这条手算结果上升到一般规律:

  • $w \in [0, 1]$:温和外推,FID 改善、多样性几乎不损失。多数 video diffusion 用 $w \approx 1$。
  • $w \in [3, 8]$:典型文生图区间,质量与多样性的甜蜜点。
  • $w \in [8, 15]$:高锐度但开始出现饱和和重复模式。
  • $w > 15$:失真显著,几乎只能用在 class-conditional ImageNet 这种条件信号极强的场景。

常见误区

  1. "$w$ 是 0–1 的混合权重"。反例:SD 推荐 $w = 7.5$,DiT 论文报告 ImageNet 上 $w=4$ 附近 FID 最优,CFG 实测最佳点常在 5–12。$w=0$ 不是"无条件",而是"纯条件采样";要做纯无条件,要么取 $w=-1$、要么直接调用 $\epsilon_\theta(x_t, t, \emptyset)$。把 $w$ 当混合权重写 UI 滑块的人,永远调不到 7.5。
  2. "$w$ 越大越好"。反例:$w=20$ 在 ImageNet class-conditional 上 FID 反而比 $w=4$ 差——样本过饱和、纹理失真、模式坍塌(所有"金毛"长得几乎一样、所有"鹦鹉"色彩同样过曝)。CFG 是 quality–diversity trade-off,不是单调改善。Imagen 的解决方法是 dynamic thresholding,让高 $w$ 仍然可用;不这么做的话 SD 在 $w > 12$ 已经明显退化。
  3. "训练时不丢条件,推理直接用 CFG 公式就行"。反例:若 $p_{\text{drop}} = 0$,$\epsilon_\theta(x_t, t, \emptyset)$ 是网络从没见过的输入——输出几乎是噪声。CFG 公式照样能算(不会报错),但 $\hat\epsilon$ 会包含一个完全错误的 $\epsilon_\theta(x_t, t, \emptyset)$ 项,结果是有规律的彩噪而不是图像。Ho & Salimans 反复强调:训练必须显式 condition dropout;这一行代码缺了,CFG 整套方法就垮了。

相关概念

  • Classifier Guidance — CFG 的前驱;用外接分类器显式提供 $\nabla \log p(c|x)$,工程上更贵。
  • DDIM — CFG 通常与 DDIM/DPM-Solver 等确定性采样器配合,扰动方向更清晰。
  • Latent Diffusion — Stable Diffusion 即 LDM + CFG + cross-attention,是 CFG 工业级实战的标杆。
  • DDPM — CFG 的去噪基底;$\epsilon_\theta$ 的训练目标与 DDPM 完全相同,只是多了 condition dropout。

来源

  1. cmu-10799-diffusion-flow · L7 · 「能不能反过来跑?我们今天就讲这个」 · (00:52)
  2. cmu-10799-diffusion-flow · L7 · 「现在全是无条件生成——怎么做条件生成?」 · (03:35)
  3. cmu-10799-diffusion-flow · L7 · 「每加一种条件就要重训一次——核心瓶颈」 · (07:53)
  4. cmu-10799-diffusion-flow · L7 · 「能不能一个网络做所有条件?这才是 diffusion 的妙处」 · (08:45)
  5. cmu-10799-diffusion-flow · L11 · Luma AI Linqi Zhou:random CFG 训练梯度噪声分析 · (55:39)
查看 85 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 55: 39 So this is actually exactly opposite of meanflow which takes partial on both xt meanflow which takes partial on both xt meanflow which takes partial on both xt and t. and t. and t. Yeah. Uh so I will …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 56: 38 very stable. And if we track also the norm of the And if we track also the norm of the And if we track also the norm of the marginal velocity uh for both mean flow marginal velocity uh for both mean f…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 00: 03 All right. So let's start get started All right. So let's start get started for the lecture. for the lecture. for the lecture. Last time we learned DDIM which is like Last time we learned DDIM which i…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 00: 52 these things can also speed things up or these things can also speed things up or get better results. Now we go like uh time z Now we go like uh time z like t minus or time t to tus 10 like t minus or…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 01: 47 and uh one thing about this design space things is that I misspoke um last time things is that I misspoke um last time things is that I misspoke um last time about about about essentially basically wh…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 02: 32 medium time steps rather than at the beginning or at the end. Any questions beginning or at the end. Any questions beginning or at the end. Any questions here? Okay, cool. All right, so now that we Ok…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 03: 35 now we haven't talked about anything about conditional generation yet. So about conditional generation yet. So about conditional generation yet. So far, everything is unconditional. We far, everything…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 04: 34 usually what people would do like the first thing you would try uh is to first thing you would try uh is to first thing you would try uh is to basically you still have your noise that basically you st…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 05: 17 conditional um sample based on whatever the condition that you specified right the condition that you specified right the condition that you specified right okay so what what is the what is the okay s…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 06: 16 essentially have another feature from for example maybe another image or like for example maybe another image or like for example maybe another image or like some uh maybe another like 3D mesh or some…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 07: 00 norm like we want dark hair or some norm like we want dark hair or some different hair? different hair? different hair? Um, so you mean like basically how would Um, so you mean like basically how woul…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 07: 53 label condition, right? If if this model is class label condition then we cannot is class label condition then we cannot is class label condition then we cannot use it for like say condition on like u…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 08: 45 like the data collection as we uh may like the data collection as we uh may know now is like very very expensive know now is like very very expensive know now is like very very expensive thing. And th…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 09: 27 um which is that basically if you think about it right the unconditional model about it right the unconditional model about it right the unconditional model that we learned already has a lot of that w…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 10: 17 unconditional model to do conditional unconditional model to do conditional generation at all? sort of. But there's like a very very sort of. But there's like a very very simple answer to this. Okay, …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 11: 17 probability of the label given the model or the the image. Right? So if you write or the the image. Right? So if you write or the the image. Right? So if you write it in terms of if you write it in te…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 12: 08 plus a uh the gradient from a discriminative model. Yeah, discriminative model. Yeah, discriminative model. Yeah, >> here influence like both the model >> here influence like both the model >> here in…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 13: 00 That's it. And you just run your That's it. And you just run your sampling as normal and you get a sampling as normal and you get a sampling as normal and you get a conditional sample. conditional sam…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 14: 02 >> it's a classifier. So it's either like >> it's a classifier. So it's either like it can be anything. So depending on your it can be anything. So depending on your it can be anything. So depending o…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 14: 46 training data. You cannot do some interpolation to generate some other interpolation to generate some other interpolation to generate some other thing. thing. thing. >> Um yes uh that is a very great …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 15: 29 like in painting and this is like colorization. Yeah. So you can just like colorization. Yeah. So you can just like colorization. Yeah. So you can just like use the same unconditional model to to use …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 16: 22 need to actually specifically train this classifier uh to to be able to recognize classifier uh to to be able to recognize classifier uh to to be able to recognize a noisy image and and and be able to…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 17: 13 nodding their head. nodding their head. Oh, someone someone on online. Is that Oh, someone someone on online. Is that Oh, someone someone on online. Is that >> Oh, no. >> Oh, no. >> Oh, no. >> Oh, oka…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 18: 10 basically just input your um your the the the less noisy the clean data the the less noisy the clean data the the less noisy the clean data estimation and the condition that you estimation and the con…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 19: 02 gradient that you got from your clean data estimation um and your data estimation um and your data estimation um and your off-the-shelf classifier off-the-shelf classifier off-the-shelf classifier uh …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 20: 27 so, uh, we we took the grade in regards so, uh, we we took the grade in regards to XT. Uh, but like that's not where the to XT. Uh, but like that's not where the to XT. Uh, but like that's not where t…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 21: 19 model and one large size model right so this thing is just going to takes you this thing is just going to takes you this thing is just going to takes you like a lot of compute that's one thing like a …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 22: 06 like a ball and then the the the noise level the manifold for each noise level level the manifold for each noise level level the manifold for each noise level is just just like layered shells that is …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 22: 56 distance from the origin. Uh and then and then like basically we're going to and then like basically we're going to and then like basically we're going to see that around like 90% of the density see t…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 23: 44 the manifold at all. So the guided sample can completely go off the sample can completely go off the sample can completely go off the manifold and just like wander off and manifold and just like wande…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 24: 30 these things basically just contribute to the fact that DPS is really really to the fact that DPS is really really to the fact that DPS is really really really slow and if you try to reduce the really…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 25: 49 little bit like how do we is there any little bit like how do we is there any other ways that we can apply guidance. >> Yeah. Yeah. So basically the first thing >> Yeah. Yeah. So basically the first t…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 26: 36 expensive thing to do. So because the because the classifier is usually really because the classifier is usually really because the classifier is usually really small. So yeah, so the first thing that…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 27: 29 manifold right so what should we do next manifold right so what should we do next the other very easy thing to do here is the other very easy thing to do here is the other very easy thing to do here i…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 28: 11 like a property of the like the low low dimensional manifold. So basically you dimensional manifold. So basically you dimensional manifold. So basically you can you can basically just like have a can …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 28: 58 class or something. Uh but basically you can actually get access to the data can actually get access to the data can actually get access to the data manifold via the decoder of an manifold via the dec…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 29: 44 the like the the yeah the low dimensional like representation space. dimensional like representation space. dimensional like representation space. Um so essentially this is like kind of Um so essentia…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 30: 32 samples and label them with noise levels, right? Nobody. Nobody would do levels, right? Nobody. Nobody would do levels, right? Nobody. Nobody would do that. Um but if you that. Um but if you that. Um …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 31: 24 we have one for clean IM. Oh yes yes uh we have one for clean IM. Oh yes yes uh you can speak up by the way the the you can speak up by the way the the you can speak up by the way the the people onlin…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 32: 02 to assume that we have access to autoenccoders? Uh we're going to talk autoenccoders? Uh we're going to talk autoenccoders? Uh we're going to talk about the next class actually but about the next clas…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 32: 52 uh and you can just uh and and this this algorithm is applicable to all algorithm is applicable to all algorithm is applicable to all differentiable guidance or classifiers differentiable guidance or …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 33: 37 into a text condition model without training anything by just applying clip training anything by just applying clip training anything by just applying clip clip guidance to your to to your clip guidan…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 34: 29 go to your classifier, right? So all you need to do is to take gradient all the need to do is to take gradient all the need to do is to take gradient all the way back and that's that gradient will way…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 35: 15 your uh like your your your image. Say your uh like your your your image. Say for example you have like a gausian blur for example you have like a gausian blur for example you have like a gausian blur…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 36: 12 Now that we're in the realm of training Now that we're in the realm of training free uh guidance, is there any other way free uh guidance, is there any other way free uh guidance, is there any other w…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 37: 15 nonisotropic noise um uh yeah sort of basically it's the same thing right I basically it's the same thing right I basically it's the same thing right I guess Adrian so like uh like basically guess Adr…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 37: 57 so like B and if your model is really so like B and if your model is really large the it's like a lot of memory large the it's like a lot of memory large the it's like a lot of memory requirement righ…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 39: 22 >> Okay. The classifier cannot be trained, >> Okay. The classifier cannot be trained, right? They cannot sample, right? You right? They cannot sample, right? You right? They cannot sample, right? You …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 40: 14 called like LGD loss guided diffusion or something um uh that paper that paper something um uh that paper that paper something um uh that paper that paper was actually done by uh like the same was act…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 40: 56 actually a great trick that you should do. create a proxy distribution. This is do. create a proxy distribution. This is do. create a proxy distribution. This is um that's kind of something that we're…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 41: 37 generated. Um but the problem is we don't have what we want to generate yet, don't have what we want to generate yet, don't have what we want to generate yet, right? Because if we do, why why are we r…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 42: 25 right? So, this needs to have a condition in it. condition in it. condition in it. Okay. Hold on. The people on in person Okay. Hold on. The people on in person Okay. Hold on. The people on in person …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 43: 07 a really rough uh or preliminary version a really rough uh or preliminary version of what we want to generate. But the of what we want to generate. But the of what we want to generate. But the problem…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 44: 18 that as x0ero. How about that? It turns out you can definitely do that. It turns out you can definitely do that. It turns out you can definitely do that. So say like this is like your input and So say…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 45: 06 um swapping in as one of the x0 in the um swapping in as one of the x0 in the middle and just just continue on middle and just just continue on middle and just just continue on diffusion as if nothing…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 45: 59 like rough ideas of like what do you want to generate and then just like want to generate and then just like want to generate and then just like hijack the the diffusion process by hijack the the diff…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 46: 49 to your input and and this way it's to your input and and this way it's going to create something that like kind going to create something that like kind going to create something that like kind of lo…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 47: 35 looking image from it based on your pre-trained diffusion model and say like pre-trained diffusion model and say like pre-trained diffusion model and say like you dislike the gray sky of um you dislik…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 48: 14 then the diffusion model can uh the SD is going to generate something that's is going to generate something that's is going to generate something that's realistic looking but with the correct realisti…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 48: 59 this is a very terribly masked uh image this is a very terribly masked uh image but basically you see how like if you do but basically you see how like if you do but basically you see how like if you …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 49: 34 better masking skills than I do then then you would have get a better um then you would have get a better um then you would have get a better um image. Yeah. like I think similar to image. Yeah. like …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 50: 29 majority of the the image is going to be noise right so you just going to noise right so you just going to noise right so you just going to generate unconditionally like normal but generate unconditio…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 51: 16 like how abstract I guess your image is, like how abstract I guess your image is, um essentially you you will need to um essentially you you will need to um essentially you you will need to adjust lik…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 51: 54 not never mind um but basically the reason why this is happening is that um reason why this is happening is that um reason why this is happening is that um the diffusion process the diffusion process …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 52: 44 reconstruct the uh the high frequency details more and more. And this is why details more and more. And this is why details more and more. And this is why basically based on like how much detail basic…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 53: 31 like determine like which time step should I go and stuff like that which is should I go and stuff like that which is should I go and stuff like that which is what you said I think this is what we wha…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 54: 17 generation uh if you start at time zero generation uh if you start at time zero which means that there's no diffusion at which means that there's no diffusion at which means that there's no diffusion …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 55: 11 people were saying is that like at the people were saying is that like at the beginning of the diffusion it's like beginning of the diffusion it's like beginning of the diffusion it's like really chao…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 55: 50 basically what you can do is say you're basically what you can do is say you're like unhappy with your uh w with this like unhappy with your uh w with this like unhappy with your uh w with this partic…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 56: 26 steps uh in your longriven dynamic for you to reach the the good part of the of you to reach the the good part of the of you to reach the the good part of the of the distribution. But yeah, but the di…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 57: 18 Yeah. Could you speak to like theoretical like Could you speak to like theoretical like Could you speak to like theoretical like I understand the concept of a GAN being I understand the concept of a G…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 58: 02 problem. Yeah, like if you do not train for this particular task, you just try for this particular task, you just try for this particular task, you just try to use your pre-trained unconditional to us…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 58: 45 search based thing like Monte Carlo research type of type of style type of research type of type of style type of research type of type of style type of thing or you can use like a like like thing or …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 59: 32 part or the other the part. >> Oh, so like masking here meaning that >> Oh, so like masking here meaning that >> Oh, so like masking here meaning that only the mask only the part that is only the mask…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 60: 26 learn good information. Uh usually unconditional models can give you very unconditional models can give you very unconditional models can give you very good uh image qualities and conditional good uh …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 61: 23 what you can do is if you do a weighted average of the two the two distributions average of the two the two distributions average of the two the two distributions what you can do is you can actually g…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 62: 16 conditions that is like that everyone conditions that is like that everyone has, right? So, for example, like say has, right? So, for example, like say has, right? So, for example, like say you're um …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 63: 00 score solo you kind of just like do a weighted mixture of of both. Uh so like weighted mixture of of both. Uh so like weighted mixture of of both. Uh so like your so basically it's gamma is usually yo…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 64: 01 have your uh input image and then you have your uh input image and then you can literally just like specify your can literally just like specify your can literally just like specify your condition to …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 64: 42 you'll have to have both the te the image and then the text and then extract image and then the text and then extract image and then the text and then extract the feature. But like so this is just to …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 65: 26 to like map your text into some feature to like map your text into some feature space and this feature extractor could space and this feature extractor could space and this feature extractor could be …
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 66: 09 it's uh it it just gets you like clearer features and this is why it it actually features and this is why it it actually features and this is why it it actually just like usually gives you like more u…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 66: 55 but all right now you know how to turn but all right now you know how to turn an unconditional model into a an unconditional model into a an unconditional model into a conditional one. Congratulations…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 67: 39 Max. Please come to come in person. If Max. Please come to come in person. If you want to ask questions, it's going to you want to ask questions, it's going to you want to ask questions, it's going to…
cmu-10799-diffusion-flow L6: CMU 10799 S26: Lecture 7 - Guidance & Controllable Generation - Diffusion & Flow Matching 68: 18 is not final class but like the the fourth class in the next two weeks we're fourth class in the next two weeks we're fourth class in the next two weeks we're going to be hearing from uh Alex who's go…