概念

Rectified Flow

— 不是新模型,是把任何 flow 的轨迹「拉直」的训练 trick。

很多人第一次听到 Rectified Flow 会把它理解成"另一种生成模型",跟 DDPM、Score Matching、Flow Matching 并列起来记。这是误读。Rectified Flow 本身不是一个新模型族,也没有新的网络架构,更没有改动损失函数的形式——它是一个针对已有 flow 的训练技巧,目的只有一个:把生成时的 ODE 轨迹从弯曲变直。轨迹一旦变直,1 步 Euler 就足以从噪声跳到数据,这才有了 Stable Diffusion 3、FLUX 等少步数生成器,也才有了 SDXL Turbo、SDXL Lightning 等"实时出图"产品的工程基础。

它出现的动机直接来自 CMU 10799 Lecture 10 的开场提问。讲师先回顾完 DDIM、各种 ODE solver 之后抛出一个问题:

「我们已经讲了快速采样的方法,比如 DDIM,还可以换不同的 ODE solver。但除此之外,还有别的办法加速推理吗?」 — CMU 10799 L10 @00:04

讲师接着引导出两条路径:第一条是 distillation——把一个慢的、多步的预训练模型蒸馏成一个快的、少步的学生模型,例如把"一步学生 = 两步老师"层层套娃,得到 progressive distillation;第二条则是 Rectified Flow,思路完全不同——不是事后蒸馏,而是从源头改训练目标,让模型一开始就学一个轨迹接近直线的 flow,少步数采样自然就稳。本页讲后者,但理解时不妨把它放在加速采样的整张技术地图上看:Rectified Flow 与 distillation 是互补关系而非竞争关系。

在训练里,这件事对应 Liu et al. 2022 提出的直线条件路径 + reflow 迭代。它不引入新损失、不改网络结构,只换一个采样路径和一个数据重耦合循环。代码层面的改动可能只有十几行,但概念层面的跨度很大——它揭示了"训练成本"和"推理成本"之间可以做时间换空间式的转移。

1. 直线插值路径:所有玄机的起点

标准 Flow Matching 允许任意条件路径 $x_t = \alpha_t x_0 + \sigma_t x_1$(高斯调度、cosine 调度、variance-preserving 调度都行)。Rectified Flow 把路径锁死成最简单的形式:

$$x_t = (1-t)\,x_0 + t\,x_1, \quad t \in [0, 1]$$

其中 $(x_0, x_1) \sim \pi$ 是一对耦合样本:$x_0$ 来自噪声分布(典型为标准高斯 $\mathcal{N}(0, I)$),$x_1$ 来自数据分布。注意 $x_t$ 是 $x_0$ 与 $x_1$ 的线性插值——画在二维平面上就是一条连接两端的直线段。两端时间分别取 $x_0 = x_t|_{t=0}$、$x_1 = x_t|_{t=1}$,速度 $\dot x_t = x_1 - x_0$ 是一个与 $t$ 无关的常向量。这一点跟 DDPM 的方差爆炸调度、Karras EDM 的对数 sigma 调度完全不同——后两者都有时间相关的非线性项。

Why 直线:如果训练得到的速度场 $v_\theta$ 真的能完美匹配这个线性路径的速度 $x_1 - x_0$,那么积分这条 ODE 时,路径本身也是直线,1 步 Euler 就够:$\hat x_1 \approx x_0 + v_\theta(x_0, 0)$。直线是 ODE 数值积分的"圣杯"——任意步长都零截断误差,因为 Euler 法的误差项正比于二阶导数 $\ddot x_t$,而直线的二阶导数恒为零。这就是为什么 Rectified Flow 一直瞄准"轨迹直度"这个不变量打。

2. 条件速度的取期望:$u_t(x)$ 的来源

问题是:给定一个具体的中间状态 $x_t = x$,可能对应很多不同的 $(x_0, x_1)$ 对——多条直线穿过同一点 $x$。比如平面上点 $(1, 0.5)$ 可以是直线 $(0, 0) \to (4, 2)$ 上 $t = 0.25$ 处的点,也可以是直线 $(2, 1) \to (-2, -1)$ 上 $t = 0.25$ 处的点。所以"在 $x$ 这点的速度"不能直接读取,而是要对所有穿过该点的轨迹取条件期望:

$$u_t(x) = \mathbb{E}_{(x_0, x_1) \sim \pi}\bigl[\,x_1 - x_0 \,\big|\, x_t = x\bigr]$$

Why 期望:这是 marginal velocity field(边缘速度场)——把所有穿过 $x$ 的"小直线"在该点的方向做加权平均,得到的才是边缘分布意义下的瞬时速度。神经网络 $v_\theta(x, t)$ 学的就是这个 $u_t(x)$。这里要警觉一个微妙的事实:尽管每一条条件路径都是直线,但边缘速度场 $u_t(x)$ 的积分曲线却可以是弯的——因为不同轨迹在同一点 $x$ 的方向不一致,期望出来的方向是它们的某种"中间值",这条平均方向线本身就不再是直线了。这正是单次 Rectified Flow 训练之后轨迹依然会弯的根本原因,也是后面 reflow 要解决的问题。

3. Conditional Flow Matching 损失

训练 $v_\theta$ 不需要显式算 $u_t(x)$ 的条件期望(那是一个内嵌的积分,没法直接评估)。利用 Conditional Flow Matching 的简化技巧——本质上跟 Tweedie 公式同一精神——目标可以等价改写成在条件路径上做回归:

$$\mathcal{L}_{\text{RF}}(\theta) = \mathbb{E}_{t \sim U[0,1],\,(x_0, x_1) \sim \pi}\Bigl\|\,v_\theta\bigl((1-t)x_0 + t x_1,\,t\bigr) - (x_1 - x_0)\,\Bigr\|^2$$

Why 等价:对一个 MSE 损失,最优解就是条件期望——$v_\theta^*(x_t, t) = \mathbb{E}[x_1 - x_0 \mid x_t]$。所以训练这个看似简单的"目标速度 = $x_1 - x_0$",最终学到的就是 $u_t(x)$。这一招让训练完全 simulation-free:每个 step 只需要采 $(x_0, x_1, t)$,算插值点 $x_t$,前向一次,反传一次,结束。完全不需要在训练时跑 ODE 求解器,比 distillation 类方法(每一训练步都得跑 4~8 步 ODE 老师推理)便宜一个数量级。

实现上还有几个细节值得知道。第一,$t$ 通常从 $U[0, 1]$ 均匀采,但 SD3 论文用 logit-normal 偏向中段时间步——因为中间 $t$ 的样本更接近真实"扩散途中"的状态,对最终生成质量影响最大。第二,损失对所有像素求和或求均值都行,但要跟下游 reflow 阶段保持一致;通常用 mean。第三,由于条件路径是直线,在 $t = 0$ 和 $t = 1$ 附近网络的输入分布会退化——前者全是噪声、后者全是干净数据——所以实务里会把 $t$ 限制在 $[\epsilon, 1 - \epsilon]$。

4. Reflow:本概念的真正灵魂

第一次训练用的耦合是 $\pi_0 = p_\text{noise} \times p_\text{data}$(独立采样,完全随机配对)。此时虽然每条条件路径都是直线,但 $u_t(x)$ 这条边缘速度场不是常向量场——多条直线在 $x$ 处方向不同,期望出来的轨迹依然弯曲。直观上:随机配对让"猫的图像"可能跟某团噪声配在一起,但跟"另一团相似噪声"配的可能是狗,两条路径在某点几乎相交时速度方向却南辕北辙,平均出来就是弯的。

Liu et al. 2022 的关键观察:训完 $v^{(1)}$ 后,可以用它定义新的耦合:

  1. 采一个 $x_0 \sim p_\text{noise}$。
  2. 求解 ODE $\dot z_t = v^{(1)}(z_t, t)$ 从 $t = 0$ 积到 $t = 1$,得到终点 $x_1 = z_1$。
  3. 记录这对 $(x_0, x_1)$,把它们当作新的耦合 $\pi_1$ 的一个样本。
  4. 重复采足够多次,得到完整的 $\pi_1$ 分布。

然后用同样的 $\mathcal{L}_{\text{RF}}$ 在 $\pi_1$ 上重训一遍,得到 $v^{(2)}$。这一步叫 reflow。

Why reflow 让轨迹变直:在 $\pi_1$ 下,每个 $x_0$ 都只对应一个它经 ODE 走到的 $x_1$——耦合已经被前一轮 ODE "整理"过。同一点 $x$ 上多条直线交叉的情况大幅减少,期望出来的 $u_t^{(2)}$ 更接近常向量场,对应的 ODE 轨迹也就更直。可以继续 $\pi_2, \pi_3, \dots$,每次都更直。从最优运输的角度看,reflow 在逼近 Monge 最优运输方案——即"每个 $x_0$ 走最短路径到达对应的 $x_1$",而最短路径就是直线。

论文里证明了两条关键性质:第一,reflow 操作单调降低 transport cost,即每轮迭代后 $\mathbb{E}\|x_1 - x_0\|^2$ 不增;第二,边缘分布严格不变——即 $v^{(k)}$ 仍把噪声映成数据,只是路径更"经济"。这两条性质保证了 reflow 是"白吃的午餐"——质量不退、速度提升。

(a) 标准 diffusion:弯曲轨迹 x₀ x₁ 需要 50~1000 步 ODE solver (b) Rectified Flow:拉直轨迹 x₀ x₁ 1~4 步 Euler 即可 Reflow 迭代效果: k=1(弯) k=2(更直) k=3(近乎直线)
🟡 教材外补充:CMU L10 把 Rectified Flow 放在"distillation 之后"讲是有道理的——progressive distillation 也是为了少步数采样,但它把一个慢模型蒸馏成一个快模型;Rectified Flow 则是把一个慢路径重新参数化成一个快路径,再单次训练就够。两条路径互相补——SD3 用 Rectified Flow 做基础,FLUX 在其上再叠 distillation/LCM 风格 trick。课堂里还讨论了一个相关问题:为什么不直接训练"一步到位"的模型?讲师的回答是训练时跑多步采样太贵,而且最终任务从高斯映到复杂数据分布这件事本身就是病态的。Rectified Flow 巧妙绕开了这个困境——它依然是多步训练目标(每个 $t$ 都有监督信号),只是让最终学到的轨迹结构使得推理时一步就够。

5. 工程上的几个真实数字

原论文在 CIFAR-10 上的数字很有说服力:随机耦合训出来的 $v^{(1)}$ 用 1 步 Euler 采样 FID 高达 378(基本是随机噪声);用 100 步 RK45 solver 才能降到 2.58。一轮 reflow 之后的 $v^{(2)}$,1 步 FID 直接降到 12.21——下降了 30 倍。再做一轮 reflow 加上 distillation,1 步 FID 可以达到 4.85,已经接近多步采样的基线。这组数字说明两件事:(1)reflow 的收益巨大但有边际递减;(2)reflow 后再做一层 distillation 是现行最佳实践,SD3、FLUX Schnell 都采用这种"reflow + distill"双阶段流水线。

6. 与其他加速路线的对比

把 Rectified Flow 放在整张加速地图上,能更清楚看到它的定位。DDIM 是采样器层面的改进——预训练模型不变,只换 ODE 离散化策略;它能从 1000 步压到约 50 步,但更少就会掉质量。Progressive distillation 是训练层面的层级压缩——每次砍一半步数、训一个新学生,能压到 4~8 步。Consistency Models 直接学"从任何 $t$ 跳到 $t = 0$ 的函数"——理论一步到位,但训练目标更复杂、需要 EMA teacher 稳定性技巧。Rectified Flow 则是路径层面的重参数化——它改的是 $x_t$ 怎么生成、$\pi$ 怎么定义,目标函数本身没换。这种"低侵入性"使得 Rectified Flow 能很容易跟其它任何加速技术叠加。

实务里,SD3 选择 Rectified Flow 作为底座,主要是看中两点:训练稳定性(loss 是简单的 MSE,没有 EMA、teacher-student 等技巧)、以及无须 simulation(不像 distillation 类方法每 step 都要前向多次)。而它的"代价"在于:reflow 阶段需要先用 $v^{(1)}$ 跑大量 ODE 离线生成 $\pi_1$——这是一次性的数据准备成本,但相比每步训练都跑 ODE 还是便宜得多。

7. 从最优运输的角度再看一遍

Rectified Flow 的几何画面在最优运输里更清晰。给定两个分布 $p_\text{noise}$ 与 $p_\text{data}$,理论上存在无穷多种"运输方案"——每种方案就是一个耦合 $\pi$。其中代价最低的那个叫 Monge 最优运输,它的特点是:每个 $x_0$ 都沿着直线走到对应的 $x_1$,且不同轨迹互不交叉。reflow 的每一步迭代都让当前耦合更接近这个最优运输——单次 reflow 严格降低 $\mathbb{E}\|x_1 - x_0\|^2$,但通常不会一步到达最优。这也解释了为什么实务里两到三次 reflow 就够了:每次都把"代价"砍掉一大块,但完全收敛要无限多次,性价比断崖式下降。从这个角度看,Rectified Flow 与最优运输文献里的 Schrödinger Bridge、IDFM 等方法属于同一家族,只是切入点更工程化、可以直接套在已有的 diffusion 训练框架上。

8. 与 score / 噪声预测的等价转换

最后值得记住的一点:尽管 Rectified Flow 学的是速度场 $v_\theta$,但它在数学上跟 DDPM 类模型的 score $\nabla_x \log p_t$ 和噪声预测 $\epsilon_\theta$ 是同一族量的不同参数化。在线性路径 $x_t = (1-t)x_0 + t x_1$ 下三者可以互相换算:知道 $v_\theta$ 与当前的 $x_t$,就能反推 $x_0$ 和 $x_1$ 的预测,进而得到等价的 noise/score 估计。这意味着已有 DDPM 预训练权重在概念上能"翻译"到 Rectified Flow 框架——SD3 实际上就利用了这一点做权重初始化,而不是从零训练。理解了这条等价性,你就能把扩散模型领域看似分裂的多个学派(score-based、noise-prediction、flow-matching、rectified flow)放在同一个底座上比较,而不是把它们当成相互替代的方案。换句话说,模型架构和优化器都不变,只是预测目标和采样路径换了一套坐标系;这种"语言切换"在扩散文献里反复发生,Rectified Flow 不过是最近的一次。

点按钮依次推进 reflow 轮次。不变量:每一轮 $v^{(k)}$ 仍把 $p_\text{noise}$ 映到 $p_\text{data}$(边缘分布不变),变化的只是轨迹的直度。

$\pi_0 = p_\text{noise} \times p_\text{data}$:每个噪声 $x_0$ 跟数据点 $x_1$ 随机配对。条件路径直,但边缘速度场弯。

手算:2D 玩具例子

设 $x_0 = (0, 0)$,$x_1 = (4, 2)$,$t = 0.25$。则线性插值点:

$$x_t = (1 - 0.25)(0,0) + 0.25 \cdot (4, 2) = (0.75 \cdot 0 + 0.25 \cdot 4,\ 0.75 \cdot 0 + 0.25 \cdot 2) = (1, 0.5)$$

目标速度 $x_1 - x_0 = (4, 2) - (0, 0) = (4, 2)$。损失(假设此时 $v_\theta(x_t, t) = (3.5, 2.1)$):

$$\|v_\theta - (x_1 - x_0)\|^2 = (3.5 - 4)^2 + (2.1 - 2)^2 = 0.25 + 0.01 = 0.26$$

训练完后 1 步 Euler 复原:$\hat x_1 = x_0 + 1 \cdot v_\theta(x_0, 0) \approx (3.5, 2.1)$。距离真实 $x_1 = (4, 2)$ 的误差约为 $\sqrt{0.25 + 0.01} \approx 0.51$;reflow 之后这个误差会持续下降到接近 0。

再看一个交叉的例子说明"为什么 reflow 重要"。考虑两对配对:$(x_0^A, x_1^A) = ((0,0), (4,0))$ 和 $(x_0^B, x_1^B) = ((0,2), (4,2))$。在 $t = 0.5$ 时两条路径分别经过 $(2, 0)$ 和 $(2, 1)$,互不相交,速度均为 $(4, 0)$。但若换成"对角配对" $(x_0^A, x_1^B) = ((0,0), (4,2))$ 和 $(x_0^B, x_1^A) = ((0,2), (4,0))$,两条路径在 $(2, 1)$ 处相交,但速度分别是 $(4, 2)$ 和 $(4, -2)$,取平均得到 $(4, 0)$——一个不指向任何真实终点的方向。这正是随机耦合的病态。reflow 后的耦合自动倾向于第一种"平行配对",避免交叉。

常见误区

  1. "Rectified Flow 是新模型,跟 Flow Matching 并列"。反例:训练损失就是 Conditional Flow Matching 的形式,只是把条件路径固定成线性、再叠加 reflow 迭代。SD3 技术报告里直接写的是 "Conditional Flow Matching with linear interpolation path"——Rectified Flow 是它的一个特例加上后处理流程,不是平行的新分支。
  2. "训一次就能 1 步采样"。反例:Liu et al. 报告 CIFAR-10 上 $v^{(1)}$ 单步 FID 高达 378,几乎是噪声水平;$v^{(2)}$ 降到 12.21,$v^{(3)}$ 才接近多步 baseline 的水平。"直" 是 reflow 多轮迭代的产物,不是初次训练的产物。如果你只跑一次训练就期待 1 步采样能用,你会得到一堆灰雾。
  3. "reflow 改变了生成分布"。反例:reflow 的数学保证就是 $\pi_k$ 的两个边缘 $\int \pi_k(x_0, x_1) dx_0 = p_\text{data}$ 与 $\int \pi_k(x_0, x_1) dx_1 = p_\text{noise}$ 始终成立——它只重新分配 $x_0$ 与 $x_1$ 的配对方式,不动两端各自的分布。实测里生成质量不退化、有时还会略升,这是 SD3 选择它作为底座的核心理由。

相关概念

来源

  1. cmu-10799-diffusion-flow · L10 · Distillation, Consistency Models & Flow Maps · (0:04) — 加速推理动机
  2. cmu-10799-diffusion-flow · L10 · 直接学 t-step 的可行性讨论 · (5:59)
  3. cmu-10799-diffusion-flow · L10 · t 步训练的计算可行性瓶颈 · (8:17)
查看 162 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 00: 03 Okay, so let's get started. uh so far uh Okay, so let's get started. uh so far uh our last class we have talked about you our last class we have talked about you our last class we have talked about yo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 00: 48 Okay. Uh so this is just like a quick like reminder of like like reminder of like like reminder of like what happened next last class. So what happened next last class. So what happened next last clas…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 01: 39 lastly we talked about like the easiest lastly we talked about like the easiest way to do discrete diffusion which is way to do discrete diffusion which is way to do discrete diffusion which is mass d…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 02: 46 probability path, right? to go from like some noise or like some the full mass to some noise or like some the full mass to some noise or like some the full mass to clean data, right? Yeah, basically t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 03: 32 let let's look at them in a in the discrete settings. So last time we kind discrete settings. So last time we kind discrete settings. So last time we kind of skipped through this part uh or of skipped…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 04: 23 distribution, right? So the probability the sum of the probability over all the sum of the probability over all the sum of the probability over all possible states or all possible tokens possible stat…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 05: 13 transitions should add up to zero. And this is like and this is literally just this is like and this is literally just this is like and this is literally just a con conversation uh conservation of a c…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 06: 09 likely that you're going to go basically it's like how how how likely that you're it's like how how how likely that you're it's like how how how likely that you're going to go jump from this state to …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 07: 03 It's a velocity, right? Essentially. So It's a velocity, right? Essentially. So it's basically just saying that like um it's basically just saying that like um it's basically just saying that like um …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 07: 53 this is also why we're trying to do reparameterization trick yeah but that's reparameterization trick yeah but that's reparameterization trick yeah but that's a great question. All right. Anything a g…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 08: 58 this state that's it right this is like super uh very intuitive I hope is super uh very intuitive I hope is super uh very intuitive I hope is All good. What can we observe here? Can we write What can …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 10: 17 not equal anymore. So just sum of all the z all the possible z's and then qzx the z all the possible z's and then qzx the z all the possible z's and then qzx ptz right and z could be equal to x as ptz…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 11: 21 oiler, right? So you just get you you just literally plug everything in here just literally plug everything in here just literally plug everything in here and then you'll get this thing and the and th…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 12: 15 like the the core of the the story in all the discrete diffusion papers is all the discrete diffusion papers is all the discrete diffusion papers is that like no matter what you do try to that like no…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 13: 03 just use cross entropy to learn this, right? Because we have the ground truth right? Because we have the ground truth right? Because we have the ground truth clean data. So we just need to use cross c…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 14: 01 flips right. So you start from t t flips right. So you start from t t equals z which and then you sample from equals z which and then you sample from equals z which and then you sample from your prior…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 14: 45 you just increment the time and then you do it over over again until you reach do it over over again until you reach do it over over again until you reach the end point. Okay, make sense? This this is…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 15: 42 things that we developed somehow but basically there are two discreful basically there are two discreful basically there are two discreful matching papers uh this time they are matching papers uh this…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 16: 22 know but anyway it's like really like I really hope this guy get rich and famous really hope this guy get rich and famous really hope this guy get rich and famous soon. Uh anyway and then the second o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 17: 05 of learn the rate matrix but we don't of learn the rate matrix but we don't actually learn the rate matrix directly actually learn the rate matrix directly actually learn the rate matrix directly we w…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 17: 50 your par paragraph is in order to build this model and this is just like super this model and this is just like super this model and this is just like super like not good right this is a very very lik…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 18: 36 like it's the same problem as the fixed length thing, right? Yeah. length thing, right? Yeah. length thing, right? Yeah. >> I heard this from like who's doing >> I heard this from like who's doing >> …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 19: 20 right. So let's take a look at it. Uh right. So let's take a look at it. Uh yeah. So essentially variable length you yeah. So essentially variable length you yeah. So essentially variable length you s…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 19: 57 like something that we can think about, right? And it's like now we have like right? And it's like now we have like right? And it's like now we have like two um like frameworks that like sort of two u…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 20: 56 >> Yeah. What else? >> Yeah. What else? >> Inserting at but I saw that you can do >> Inserting at but I saw that you can do >> Inserting at but I saw that you can do decoding in blocks like you can pr…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 21: 29 >> Okay, so tatting at the end. So you can >> Okay, so tatting at the end. So you can discard some of the tokens. Yeah, that discard some of the tokens. Yeah, that discard some of the tokens. Yeah, th…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 22: 15 paragraph and for the next diffusion process that paragraph is that one process that paragraph is that one process that paragraph is that one element for you. You noise the paragraph element for you. …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 23: 09 the same thing and that this is like this is corresponding to one paper that this is corresponding to one paper that this is corresponding to one paper that we're going to talk about but not now we're…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 23: 49 different tokens in between tokens, right? I don't need to insert at the right? I don't need to insert at the right? I don't need to insert at the end. Why do I have to insert at the end? end. Why do …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 24: 31 >> Does this work in the diffusion case? >> Good question. I I'm gonna talk about >> Good question. I I'm gonna talk about >> Good question. I I'm gonna talk about it. Okay. So, how do we even model t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 25: 15 you use it to to to essentially construct construct the the ground truth construct construct the the ground truth construct construct the the ground truth of like what this like what the full of like …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 25: 57 right? What is this other thing? >> The position or the whether to generate >> The position or the whether to generate >> The position or the whether to generate it between these it between these it b…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 26: 52 basically if we're missing tokens and basically if we're missing tokens and like essentially like essentially like essentially note that like it's more likely or like note that like it's more likely o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 27: 45 what you're predicting and then the second objective from your model is second objective from your model is second objective from your model is predict what tokens are missing so this predict what tok…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 28: 26 like I don't know. >> Yeah. >> Yeah. >> Yeah. But basically what Okay. So >> Yeah. But basically what Okay. So >> Yeah. But basically what Okay. So basically what happens in the model or basically wha…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 29: 10 >> Yeah. Yeah. So basically what's happening in what's hap what's happening happening in what's hap what's happening happening in what's hap what's happening in like implementation is that you are in …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 29: 55 is that is missing out of all the tokens is that is missing out of all the tokens that are missing? Right. Okay. So these that are missing? Right. Okay. So these that are missing? Right. Okay. So thes…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 30: 48 missing versus you know how many tokens you predict to be missing and then the you predict to be missing and then the you predict to be missing and then the second thing is uh like literally just seco…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 31: 48 flow And why is it a flow model? Well, it's And why is it a flow model? Well, it's And why is it a flow model? Well, it's actually because this thing like the actually because this thing like the actu…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 32: 35 another thing then you should be doing another thing then you should be doing this operation where it's a ratio or this operation where it's a ratio or this operation where it's a ratio or like a rate…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 33: 23 divergence that is basically just a generalized version of KL. Uh but this generalized version of KL. Uh but this generalized version of KL. Uh but this is like too complicated. We're not going is lik…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 34: 11 actually like already good enough for whatever you want to do. So you can whatever you want to do. So you can whatever you want to do. So you can literally just train a insertion based literally just …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 34: 56 we kind of fixed a veryable length uh problem with insertion based length uh problem with insertion based length uh problem with insertion based generation. Uh we have other things to generation. Uh w…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 35: 40 So like basically this thing is very nice because you can um like essentially nice because you can um like essentially nice because you can um like essentially you can do paralyze generation like you …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 36: 20 that you have generated like in your previous blocks. The only thing you need previous blocks. The only thing you need previous blocks. The only thing you need to like do birectional attention to is t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 37: 00 people think about this. Um but yeah, so people think about this. Um but yeah, so this is like a nice way to do you know this is like a nice way to do you know this is like a nice way to do you know t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 37: 50 >> not in the current block but but we can K catch all the previous blocks. >> No because we don't we haven't generated >> No because we don't we haven't generated that yet, right? So like the block a…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 38: 29 of like also like but but but but if you have too small of a blog then it just have too small of a blog then it just have too small of a blog then it just Yeah. It doesn't really it just is just Yeah.…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 39: 16 in I think. Yeah. in I think. Yeah. >> Do you think the second one is >> Do you think the second one is >> Do you think the second one is um like it's it's the it's a problem of um like it's it's the …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 40: 11 That's correct. Yeah. >> For the for the second case, the mass >> For the for the second case, the mass >> For the for the second case, the mass diffusion, why can't we do something diffusion, why can…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 40: 57 Anything else? Yep. >> Can you overhand why mass diffusion is >> Can you overhand why mass diffusion is >> Can you overhand why mass diffusion is lower? lower? lower? >> Uh okay. Who wants to tell who…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 41: 49 but it doesn't mean that you you cannot train a good enough diffusion model to train a good enough diffusion model to train a good enough diffusion model to blm that's not the case it's just that blm …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 42: 45 the the optimal that you're going to get the the optimal that you're going to get from whatever algorithm that they from whatever algorithm that they from whatever algorithm that they develop will loo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 43: 32 this is not image and text but like we're going to look at image and text we're going to look at image and text we're going to look at image and text later. But for those of you who are in later. But …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 44: 31 >> What are we cogenerating? >> So, we're co-generating something uh >> So, we're co-generating something uh >> So, we're co-generating something uh discreet and something uh continuous at discreet an…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 45: 15 like alternating between the two that's like alternating between the two that's pretty much that right and like pretty much that right and like pretty much that right and like basically And then if yo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 46: 03 well right so what you can do is you can well right so what you can do is you can uh have like just like I think some uh have like just like I think some uh have like just like I think some someone al…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 46: 50 >> I mean the problem is like this is a >> I mean the problem is like this is a research project right so it's like very research project right so it's like very research project right so it's like ve…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 47: 37 and then the the the image for example like flower or like cat and then and like flower or like cat and then and like flower or like cat and then and then the the the the image are going to then the t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 48: 13 text and like image at the same time text and like image at the same time because like you start from nothing and because like you start from nothing and because like you start from nothing and then y…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 48: 49 the way to become omni but this is like just like two two modalities I guess. just like two two modalities I guess. just like two two modalities I guess. Yeah. Yeah. Omni model just means that Yeah. Y…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 49: 50 you cannot access the ponapto anymore, it's still going to be on YouTube. Uh I it's still going to be on YouTube. Uh I it's still going to be on YouTube. Uh I kind of do some editing uh for the kind o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 50: 25 your turn to take what you've learned into the real world so I will highly into the real world so I will highly into the real world so I will highly recommend if you are proud of what recommend if you…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 51: 05 something like that. I think it could be something like that. I think it could be a good experience if you guys are a good experience if you guys are a good experience if you guys are interested in ge…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 51: 39 research slash you know when you go to the AGI company you know make the AGI a the AGI company you know make the AGI a the AGI company you know make the AGI a diffusion model please we don't want to d…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 52: 25 >> padding tokens and tokens after the OS, right? right? >> Yeah. >> Yeah. >> Uh so one thing I implemented for >> Uh so one thing I implemented for >> Uh so one thing I implemented for homework 3 and…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 53: 06 I thought that's what they did in I thought that's what they did in refflow but maybe not. If it's not then refflow but maybe not. If it's not then refflow but maybe not. If it's not then this is a go…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 53: 42 than a 2 RF model because the only than a 2 RF model because the only errors accumulate. errors accumulate. errors accumulate. >> Yeah. No, that's your homework. Four. >> Yeah. No, that's your homewor…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 54: 19 in general I think that's a better idea than the normal the the the way that than the normal the the the way that than the normal the the the way that they they did. Okay. Any other question? they the…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 55: 03 >> Yeah. Yeah. So actually there multiple >> Yeah. Yeah. So actually there multiple uh so the most famous one in the uh so the most famous one in the uh so the most famous one in the robotics world is…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 55: 49 gradient you should apply from those buffer data to your current time to one buffer data to your current time to one buffer data to your current time to one of the time that you sampled. Um so this of…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 56: 25 in that framework but you can turn you in that framework but you can turn you can essentially turn an S OD like a flow can essentially turn an S OD like a flow can essentially turn an S OD like a flow…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 57: 06 basically is there any other ways that we can do it besides turning into a SDE we can do it besides turning into a SDE we can do it besides turning into a SDE um yeah so like let's hope we can get um …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 57: 58 essentially you just sort of like uh they sort of extend the flow map uh uh they sort of extend the flow map uh uh they sort of extend the flow map uh uh what you call framework into the what you call…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 58: 37 Uh, so let's see if he actually post it online this week. Uh, but yeah, I don't online this week. Uh, but yeah, I don't online this week. Uh, but yeah, I don't actually know how how he did it, but act…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 59: 19 of similar to uh one of the other previous uh framework and equilibrium previous uh framework and equilibrium previous uh framework and equilibrium model. So equilibrium model is something model. So e…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 60: 02 or the underlying uh, you know, uh, math is a little bit related, but like this is a little bit related, but like this is a little bit related, but like this is definitely something new. I think is de…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 00: 04 Okay. So, previously uh we have learned Okay. So, previously uh we have learned about fast sampling techniques, right? about fast sampling techniques, right? about fast sampling techniques, right? And…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 00: 59 pre-trained model, right? So suppose we already have a pre-trained diffusion already have a pre-trained diffusion already have a pre-trained diffusion model like before. Um then what is the model like…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 02: 00 second time step we do a loss comparison with the 500 time step. Hey yes very with the 500 time step. Hey yes very with the 500 time step. Hey yes very very good very good very very good very good ver…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 02: 45 t and uh how about okay let's just do a few more steps right how about we'll few more steps right how about we'll few more steps right how about we'll train a new model such that um like one train a n…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 03: 44 A So this is like yep >> exact as >> exact as >> exact as many as you want essentially. So like many as you want essentially. So like many as you want essentially. So like log to t I guess. log to t I…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 04: 30 Right. Right. >> So if in the first example we took the >> So if in the first example we took the >> So if in the first example we took the two two two steps to V1 of my new models step and in steps t…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 05: 11 way this is the paper that tells you that you should do V prediction like V that you should do V prediction like V that you should do V prediction like V prediction is the best parameterization predic…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 05: 59 >> Oh, why wouldn't it be a good model? I >> Oh, why wouldn't it be a good model? I guess guess guess Marco, Marco, Marco, >> you still have to learn the same amount >> you still have to learn the sam…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 06: 44 network's job to just take this teeny tiny Gaussian to that complex version tiny Gaussian to that complex version tiny Gaussian to that complex version without getting enough. So it's very without get…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 07: 23 what about like the feasibility of training this kind of model like how training this kind of model like how training this kind of model like how about let's just look at uh training a about let's jus…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 08: 17 to take a very long time and it's like if you try to scale it up to t time if you try to scale it up to t time if you try to scale it up to t time steps, right? Uh let me finish maybe. Um steps, right…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 08: 54 sounds kind of not great. Yeah hold on. Yeah, Yeah, Yeah, >> I think the four power could only be for >> I think the four power could only be for >> I think the four power could only be for the teache…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 09: 31 just going to become very slow. So, so do you think that like five total So, so do you think that like five total So, so do you think that like five total four of the teacher student four of the teach…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 10: 18 here >> um and what was the question >> um and what was the question >> um and what was the question step. step. step. >> Yeah. >> Yeah. >> You said you require like four forward >> You said you requi…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 11: 02 That's your forward pass. That's your forward pass. >> Yeah but just create a new data set with >> Yeah but just create a new data set with >> Yeah but just create a new data set with this feature mod…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 11: 37 yeah. Um, so what do we think? Is it possible to train a model that goes from possible to train a model that goes from possible to train a model that goes from t directly xt directly to x0? What do we…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 12: 30 right even even if you stop red over there it's not good right okay so anyway there it's not good right okay so anyway there it's not good right okay so anyway let's take a look at this uh problem let…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 13: 15 yeah, what does it mean is that basically for any two points like any basically for any two points like any basically for any two points like any arbitrary two points on the same OD arbitrary two poin…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 14: 26 is this the only criteria that we need for this new model to generate something for this new model to generate something for this new model to generate something to to generate the correct thing? to t…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 15: 13 like there it says nothing about the output, right? Like we have no guarantee output, right? Like we have no guarantee output, right? Like we have no guarantee about what the output should look like a…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 16: 03 condition essentially so this is the condition essentially so this is the thing that's going to like basically thing that's going to like basically thing that's going to like basically just like make …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 16: 48 B or something. It's like the when when B or something. It's like the when when your gradient of the the loss spikes up. your gradient of the the loss spikes up. your gradient of the the loss spikes u…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 17: 42 consistency model he was at open eye and then now he's apparently some big name then now he's apparently some big name then now he's apparently some big name in meta super intelligence lab or in meta …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 18: 26 DDPM. Yeah. Yeah. So, that guy is also a DDPM. Yeah. Yeah. So, that guy is also a genius and he's now has his own startup. genius and he's now has his own startup. genius and he's now has his own star…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 19: 15 green one which is like only one step away is the one that we're going to use away is the one that we're going to use away is the one that we're going to use here. So essentially basically what's here…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 20: 01 next time step that is getting on the same trajectory solved by our solver. same trajectory solved by our solver. same trajectory solved by our solver. and the original teacher model. So this and the …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 20: 47 just going to become a little bit messier if you if you do not use L2 but messier if you if you do not use L2 but messier if you if you do not use L2 but in practice in practice in practice uh percept…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 21: 41 plus delta t? plus delta t? >> Ah, that's also another great question. >> Ah, that's also another great question. >> Ah, that's also another great question. So, uh here everything so this f theta So, …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 22: 24 that you need to take from the t-shirt doesn't scale with t. Yeah. Uh, okay. Do doesn't scale with t. Yeah. Uh, okay. Do doesn't scale with t. Yeah. Uh, okay. Do you want your first you want your firs…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 23: 13 with the larger jump. We do not update with the larger jump. We do not update our model with the with the shorter our model with the with the shorter our model with the with the shorter jump. jump. ju…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 24: 31 parameterization that we're using sort of and uh so we are trying to predict of and uh so we are trying to predict of and uh so we are trying to predict the clean data and um usually it the clean data…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 25: 39 when t is zero the skip connection is when t is zero the skip connection is one and then the output the weight is one and then the output the weight is one and then the output the weight is zero right…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 26: 26 can be producing garbage. Right now all this saying is that if my model sees a this saying is that if my model sees a this saying is that if my model sees a very good image already and also is the ver…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 27: 09 think I don't I I agree with the thing that like the loss function on the last that like the loss function on the last that like the loss function on the last line can if you just do training with lin…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 27: 53 should not because if we just ask it to do the same thing at S and T that same do the same thing at S and T that same do the same thing at S and T that same thing could just be end up predicting thing…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 28: 37 this. Sorry. The the skip connection weight is kind of like a schedule. It weight is kind of like a schedule. It weight is kind of like a schedule. It has a scheduling. It's only one when has a schedu…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 29: 33 trajectory that's like kind of similar trajectory that's like kind of similar to what we what we would have got from a to what we what we would have got from a to what we what we would have got from a…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 30: 25 scale differently essentially. So that scale differently essentially. So that so that it's like slightly less noisy. so that it's like slightly less noisy. so that it's like slightly less noisy. Uh th…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 31: 20 about the previous uh slide, but >> uh so if we if we already know that the >> uh so if we if we already know that the >> uh so if we if we already know that the image is close enough to x0 where it's…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 31: 59 boundary condition. Yeah. Okay. All right. Any any questions about training right. Any any questions about training right. Any any questions about training consistency model from scratch? consistency …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 32: 50 Um and we're not trying to solve it here. Well we kind of are trying to here. Well we kind of are trying to here. Well we kind of are trying to solve it here actually. Um solve it here actually. Um so…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 33: 47 So one step but how do you do multi-step right? Can you even do multi-step? right? Can you even do multi-step? right? Can you even do multi-step? >> Yeah like we're turning on a lot of like >> Yeah li…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 34: 20 >> Why do you want to do that's a great question. The next slide is going to question. The next slide is going to question. The next slide is going to answer that. Yeah. answer that. Yeah. answer that…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 35: 04 or compute that you can afford to do four step generation say but like this four step generation say but like this four step generation say but like this model is not trained for that it doesn't model…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 35: 44 do whatever I guess. Yeah, doesn't do whatever I guess. Yeah, doesn't really matter anymore. Um yeah so really matter anymore. Um yeah so really matter anymore. Um yeah so essentially essentially esse…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 36: 50 >> instead of doing like this one step to >> instead of doing like this one step to the final noise you can do it like two the final noise you can do it like two the final noise you can do it like two…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 37: 27 be able to jump to any point from any point to any point. Um so yeah so this point to any point. Um so yeah so this point to any point. Um so yeah so this xt is also arbitrary so you should be xt is a…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 38: 20 >> Like two points on the trajectory then you saw the you saw the OD between them you saw the you saw the OD between them you saw the you saw the OD between them and then you essentially just take the…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 39: 04 meaningful when you're dealing with clean data. So think about it right the clean data. So think about it right the clean data. So think about it right the perceptual loss is like something like a per…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 39: 52 perceptual distance between the final X0 perceptual distance between the final X0 output. Um the only thing that you need output. Um the only thing that you need output. Um the only thing that you nee…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 40: 30 >> Yeah. Yeah. So like basically so the this the first part uses solvers so that this the first part uses solvers so that this the first part uses solvers so that this thing this part is actually on t…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 41: 18 im style at at noise bag thing right and and that's still valid actually in the and that's still valid actually in the and that's still valid actually in the in the paper like we have this thing in th…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 42: 02 kicked off that >> uh what do you mean? Oh you mean like >> uh what do you mean? Oh you mean like >> uh what do you mean? Oh you mean like the the Oh yeah. So so like you mean the the Oh yeah. So so l…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 42: 59 Yeah. But basically because we do not Yeah. But basically because we do not always match the end point now we we always match the end point now we we always match the end point now we we instead of a …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 43: 52 score right from jumping from T. Now um you you get your exact log likelihood you you get your exact log likelihood you you get your exact log likelihood back by because you can take the back by becau…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 44: 43 change that you're going from a that you're going from any point should you're going from any point should you're going from any point should follow the instant like the tangent line follow the instan…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 45: 30 when S is not equal to T and then score when S is not equal to T and then score matching loss which is just go from T to matching loss which is just go from T to matching loss which is just go from T …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 46: 13 >> Uh which one? Uh you're saying that we >> Uh which one? Uh you're saying that we add like a discriminator to improve FID add like a discriminator to improve FID add like a discriminator to improve …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 47: 17 otherwise it doesn't make sense I guess. Yeah. >> Yeah. Oh sorry. Yeah. >> Yeah. Oh sorry. Yeah. >> How do you weigh the different loss >> How do you weigh the different loss >> How do you weigh the d…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 48: 24 so I guess someone asked right do Do you so I guess someone asked right do Do you even need a solver here? Uh the answer even need a solver here? Uh the answer even need a solver here? Uh the answer i…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 49: 20 Is it just like all the questions are like, "Oh, the thing that we're going to like, "Oh, the thing that we're going to like, "Oh, the thing that we're going to talk about next." Very nice. Okay. talk…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 50: 04 OD or PD or whatever um and then the flow maps because we also look at ODEs flow maps because we also look at ODEs flow maps because we also look at ODEs right especially if If you're doing flow right…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 51: 03 It's very sad. Anyway, um but how do you uh build a valid flow map? Actually the uh build a valid flow map? Actually the uh build a valid flow map? Actually the answer to this question uh can be found…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 51: 49 prediction of XS. Okay. So flow map is prediction of XS. Okay. So flow map is the predict prediction of XS the the predict prediction of XS the the predict prediction of XS the endpoint of the jump. O…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 52: 30 guess. Uh let's just like take this analogy. Okay. Yes. analogy. Okay. Yes. analogy. Okay. Yes. >> Call it progressive. Is that the same >> Call it progressive. Is that the same >> Call it progressive…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 53: 21 that is flowing down this one river. I that is flowing down this one river. I can teleport the duck so that the duck's can teleport the duck so that the duck's can teleport the duck so that the duck's…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 54: 00 get like basically okay. So I guess this is a better uh uh visualization. So this is a better uh uh visualization. So this is a better uh uh visualization. So this is generated by nano banana the thin…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 54: 39 check for you. And then and then the the cop be like, huh, let's see if there's cop be like, huh, let's see if there's cop be like, huh, let's see if there's any duck that is like suspicious. Like, an…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 55: 24 instantaneous jump so that you can fold the cop and then and then be like, "Oh, the cop and then and then be like, "Oh, the cop and then and then be like, "Oh, okay. This is a normal duck." Basically,…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 56: 25 >> Yes. >> So if you take a langian from S to S >> So if you take a langian from S to S >> So if you take a langian from S to S plus DT, it should be the same as the is plus DT, it should be the same …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 57: 37 well for consistency model we don't really match the the the derivative. I really match the the the derivative. I really match the the the derivative. I guess that's the problem. guess that's the prob…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 58: 33 I can do I can be a little bit like I can essentially be a a little bit um can essentially be a a little bit um can essentially be a a little bit um like late. I can I I can start the jump like late. …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 59: 57 jumping, they should end up at the same jumping, they should end up at the same place. So, as long as they're following place. So, as long as they're following place. So, as long as they're following …
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 60: 59 really mention the flow map thing in really mention the flow map thing in their in their paper. We by we I mean their in their paper. We by we I mean their in their paper. We by we I mean one of my pa…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 61: 43 displacement that you're predicting right so this is like so the right so this is like so the right so this is like so the displacement that you're predicting is displacement that you're predicting is…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 62: 36 they call a mean flow identity. they call a mean flow identity. Okay, that does it make sense for people? that does it make sense for people? Good. Okay. So why is mean flow a oiler Good. Okay. So why…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 63: 31 that's it. Um and one nice thing about the mean flow identity is that uh so in the mean flow identity is that uh so in the mean flow identity is that uh so in the middle here if you take s equals t th…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 64: 47 like but an interesting thing is that CTN was developed in 2023 CTN was developed in 2023 CTN was developed in 2023 and um the the how to train your like and um the the how to train your like and um t…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 65: 43 think this is the most intuitive one. So this is literally just saying that uh this is literally just saying that uh this is literally just saying that uh basically if your flow map jump from S basica…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 66: 44 How many any of you guys have seen this model? Shortcut model. No one. No. I model? Shortcut model. No one. No. I model? Shortcut model. No one. No. I think this is a Peter Ail or something. think thi…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 67: 33 making sure that your flow oh my god I'm so sorry we're running out of time but so sorry we're running out of time but so sorry we're running out of time but let's get let's finish everything uh the l…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 68: 31 likelihood. Uh so why why is that? Well, it's basically because yeah because it's it's basically because yeah because it's it's basically because yeah because it's only jump to in the sampling space a…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 69: 13 can use the same algorithms to solve another OD right so this is what we're another OD right so this is what we're another OD right so this is what we're doing so this is like my new paper doing so th…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 69: 53 use the same full map method to distill any OD. So if you have like you know an any OD. So if you have like you know an any OD. So if you have like you know an interesting OD that is associated with i…
cmu-10799-diffusion-flow L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching 70: 40 He was a PhD student at Stanford. Uh he He was a PhD student at Stanford. Uh he co-founded something and then got co-founded something and then got co-founded something and then got acquired by Luma. …