概念

Flow Matching

— 直接回归一个速度场把噪声推向数据,绕开"每个噪声层级都学一遍"。

很多人初看 Flow Matching 会以为它"只是 DDPM 换了个 loss"——既然结果差不多,何必再起一个名字?这是被论文里的等价定理误导了。Flow Matching 的真正立场是:不再把"加噪—去噪"当作建模出发点,而是先在先验 $p_0$(一般是 $\mathcal{N}(0, I)$)和数据 $p_1$ 之间钦定一条概率路径 $p_t$,然后训一个速度场把质点沿这条路径搬运。它解决的核心痛点是:score matching 必须"在每个噪声层级都学一遍" $\nabla \log p_t(x)$,路径的形状、$\beta$ 调度、参数化方式($\epsilon$ 还是 $x_0$ 还是 $v$)全都耦合到模型上;Flow Matching 把这层耦合切开:你先挑路径,再回归速度,路径设计与模型训练彻底解耦。这就是 Lipman、Chen、Le 等人 2023 年那篇 ICLR 论文("Flow Matching for Generative Modeling")真正贡献的东西。

"我们上一节课讲了扩散如何把噪声变成数据——通过逐步加噪、再逐步去噪。我们又学了 score-based models,它做的事情非常非常相似。如果你把时间步推到无穷,两者其实就是同一种连续时间随机过程,可以写成随机微分方程。"
—— CMU 10-799 S26 · L5 @00:02

Vizuara 课程的 Dr. Raj 在 Lecture 9 / 10 选了一个更朴素的物理图像:盒子里很多粒子 $P_1, P_2, \dots$ 在按各自轨迹运动;每个粒子有它自己的轨迹,但只要你能写出"每个位置在每个时刻的速度"——也就是速度场 $u_t(x)$——你就等价于知道了所有粒子的去向。Flow Matching 把数据生成翻译成这件物理事:噪声样本是粒子初始位置,目标是设计一个让所有粒子"恰好流到数据分布"的速度场。这套粒子语言是 Vizuara 选择把 Flow Matching 拆成四节课讲的原因——它要从连续介质力学的最朴素直觉一步步爬上来,最后落到和 CMU 同一套公式。这两条路径——CMU 的"SDE → ODE → 速度场"和 Vizuara 的"粒子 → 轨迹 → 速度场"——殊途同归,是不同教学风格在同一概念上的两种入口。

"关于流模型理论有一件有意思的事:同一个主题可以用两种方式讲。你可以用一节课讲完,也可以用四节课讲。看似矛盾,但走难走的四节课路线,你最后到达的地方非常简单——简单到你会问,为什么一开始不直接告诉我这个公式?"
—— Vizuara · L9(Introduction to Flow Models)@00:04

CMU 的 Tianqi 在 Lecture 13 把同一件事推到离散域时说得更直白——他指着白板上的转移率矩阵 $Q_t$ 说:

"这个 $Q_t$ 就是流——本质上就是速度。它告诉你:从这个状态跳到那个状态的可能性有多大。所有流加起来必须为零——这就是质量守恒。"
—— CMU 10-799 S26 · L13 @07:03

在训练里,这件事对应一个朴素的 L2 回归损失:采样 $(t, x_1, x)$、查表得到条件向量场 $u_t(x|x_1)$ 的闭式值、用 L2 拟合 $v_\theta$。代码改动量小到一行——把 DDPM 训练循环里"预测噪声 $\epsilon$" 换成"预测速度 $x_1 - x_0$" 就完事。Stable Diffusion 3、Rectified Flow、Meta Movie Gen、Black Forest Labs 的 Flux 系列,全部在用这条骨架。在机器学习的上下文里,这件事对应的训练目标可以这样总结:不再学"哪里该减噪",而是学"质点此刻应该往哪走、走多快",把生成问题从概率密度估计退化成最优传输式的回归问题。

为什么这层语言切换重要?因为它把"diffusion 必须从 Gaussian 噪声出发"这条隐含假设给打破了。score matching 需要 $\sigma \to 0$ 的极限来定义噪声层级,必然以 Gaussian 为起点;速度场则不挑先验——只要你能在 $p_0$ 和 $p_1$ 之间画一条概率路径,就有对应的速度场可学。这就是为什么 image-to-image translation、protein backbone 生成、3D shape generation 等任务在 2024 年后大量切到 Flow Matching:它们的"先验"本来就不是 Gaussian 噪声,是另一份数据。

第一步:概率路径与连续性方程

钦定一条概率路径 $\{p_t\}_{t \in [0,1]}$,满足 $p_0 = \mathcal{N}(0, I)$(或任何易采样先验)、$p_1 \approx p_{\text{data}}$。再找一个时变向量场 $u_t(x)$,使 $p_t$ 是 ODE $\dot x = u_t(x)$ 的边际分布。这两件事由连续性方程串起来:

$$\frac{\partial p_t(x)}{\partial t} + \nabla \cdot \bigl(p_t(x)\, u_t(x)\bigr) = 0.$$

Why:连续性方程就是物理里的"质量守恒"——单位时间内流出某区域的概率等于该区域内概率的下降速率。CMU 在 L13 用一整张幻灯片强调流模型必须满足两件事——概率加总恒等于 1、变化必须连续;连续性方程正是这两件事的微分形式。如果你训出来的 $v_\theta$ 近似满足它,把先验质点沿 ODE 推到 $t=1$,得到的样本边际就是 $p_{\text{data}}$。这里有个微妙之处值得点出:连续性方程并不唯一确定 $u_t$——同一个概率路径 $p_t$ 可以由无数个速度场驱动(差一个"无散度"的旋转项),Flow Matching 隐含地通过 L2 回归挑出其中"L2 模长最小"的那一个,对应物理上的最小动能传输。

第二步:直接学 marginal $u_t$ 行不行?

最朴素的想法是写一个 marginal loss:$\mathbb{E}_{t, x \sim p_t}\|v_\theta(x,t) - u_t(x)\|^2$。问题在于 $u_t(x)$ 是 marginal——它涉及对整个数据分布积分

$$u_t(x) = \mathbb{E}_{x_1 \sim p(x_1 | x_t = x)}\bigl[u_t(x|x_1)\bigr],$$

这个期望写不出闭式、也没法用 Monte Carlo 估计(你不知道后验 $p(x_1|x_t=x)$)。这正是 score matching 当年踩到的同一个坑:marginal score $\nabla \log p_t(x)$ 也不可知,必须靠 denoising score matching 的"加噪—预测原 $x$"trick 绕开。Flow Matching 也需要一个类似的绕开技巧——这就是条件化。

第三步:条件 Flow Matching(CFM)

Lipman et al. 2023 的关键想法:不直接学 marginal,而学每一对端点 $(x_0, x_1)$ 的条件路径。固定一个数据点 $x_1$,沿"从噪声到 $x_1$ 的直线插值"定义条件概率路径 $p_t(x|x_1)$,对应的条件向量场 $u_t(x|x_1)$ 是这条直线的导数——闭式、好算。最常用的独立耦合 + 线性插值路径写作

$$x_t = (1-t)\, x_0 + t\, x_1, \qquad x_0 \sim \mathcal{N}(0, I),\; x_1 \sim p_{\text{data}}.$$

对 $t$ 求导得到条件速度 $u_t(x_t|x_1) = \frac{d x_t}{dt} = x_1 - x_0$——和 $t$ 无关、和 $x_t$ 无关,仅是端点之差。代入 CFM 损失:

$$\mathcal{L}_{\mathrm{CFM}}(\theta) = \mathbb{E}_{t \sim \mathcal{U}[0,1],\, x_0 \sim p_0,\, x_1 \sim p_1}\bigl\|v_\theta\!\bigl((1-t)x_0 + tx_1,\,t\bigr) - (x_1 - x_0)\bigr\|^2.$$

Why:每次梯度更新只需要采一对 $(x_0, x_1)$、一个 $t$,做一次 forward——和 DDPM 同样的复杂度,但 target 是端点差而不是噪声。代码改动只有 loss 一行。这是 Flow Matching"简单到不可思议"的根源——CMU L5 的 Tianqi 在白板上一句话就写完了整个训练循环。值得注意的是 $u_t(x_t|x_1) = x_1 - x_0$ 这个表达式里没有出现 $x_t$——这意味着同一对 $(x_0, x_1)$ 在任何 $t$ 的速度都相同。你在训练时给 $v_\theta$ 喂的输入是 $x_t$ 和 $t$,但回归 target 与 $t$ 无关;正是这种"target 与 $t$ 解耦"的简洁结构,让 Flow Matching 的优化曲面比 DDPM(噪声 target 随 $\sigma_t$ 缩放)更平坦、训练更稳。

第四步:核心定理——CFM ≡ FM(梯度相同)

看似 CFM 学的是条件速度 $u_t(x|x_1)$,跟 marginal $u_t(x)$ 不是一回事。Lipman 等人证明:

$$\nabla_\theta \mathcal{L}_{\mathrm{CFM}}(\theta) = \nabla_\theta \mathcal{L}_{\mathrm{FM}}(\theta),$$

两者梯度完全相同,差一个与 $\theta$ 无关的常数。直觉是:marginal 向量场恰好是条件向量场对后验 $p(x_1|x_t = x)$ 的条件期望,而 L2 回归 $v_\theta \to u_t(\cdot|x_1)$ 在最优解处自然收敛到这个条件期望。这和 denoising score matching 通过 Tweedie 公式吃掉 marginal score 是同一个数学魔法的两面。

Why:你训练的明明是"每对 $(x_0, x_1)$ 的直线速度",最后却得到了 marginal 速度场。这件事的代价是:网络输出在数据空间的每个交叉点上做"平均化",得到的 marginal 轨迹不再是直线,而是被"平均"过的弯曲路径。这就是为什么 CFM 训完直接采样仍需要 5~10 步——marginal 轨迹弯了,大步长 Euler 误差就出来了。Rectified Flow 解决的恰好是这个问题:用模型 sample 出来的成对 $(x_0, x_1^{\text{pred}})$ 作为新的训练对再训一次,新模型对应的 marginal 轨迹被拉得更直,2-step 甚至 1-step 推理变得可行。所以"条件期望吃掉 marginal 不可知"这件事不是免费午餐——它把不可知问题转化成了"轨迹弯曲"问题,留给 reflow 收尾。
$t=0$ · 噪声 $p_0$ $p_t$ 概率路径 $t=1$ · 数据 $p_1$ 速度场 $v_\theta(x,t)$(箭头方向 = 局部速度) 5 个粒子沿三条概率路径流动,端点不变,仅中间位置随 $t$ 变化

第五步:训练与采样

训练(一行伪代码可写完):每个 minibatch 采 $(t \sim \mathcal{U}[0,1],\, x_0 \sim \mathcal{N}(0,I),\, x_1 \sim \mathcal{D})$,把插值点 $x_t = (1-t)x_0 + tx_1$ 和 $t$ 送进 $v_\theta$,L2 回归 target $x_1 - x_0$。反向传播、更新。完成。没有 KL,没有 ELBO,没有噪声预测——只有一个最小二乘回归。

采样:固定 $x_0 \sim \mathcal{N}(0, I)$,按 ODE $\dot x = v_\theta(x, t)$ 从 $t=0$ 积到 $t=1$。Euler 步长 $\Delta t = 0.1$ 通常已经够好——5~10 步 Euler 就能出可看的样本,是 DDPM 1000 步、DDIM 50 步之后的又一次提速。如果用 Heun、RK4 等高阶 ODE 积分器,4 步就够。注意采样过程是完全确定性的——只有初始 $x_0$ 带随机性,ODE 积分本身没有噪声项;这和 DDPM 反向 SDE 每步都加噪声形成对比,也是 Flow Matching 推理可复现性优于 DDPM 的根因。

Why:线性插值路径下,每个粒子的"真值轨迹"是直线,速度恒定。训出来的 $v_\theta$ 是 marginal 速度场——它在轨迹交叉点会被平均化、有点弯曲,但整体仍接近分段直线。直线 ODE 用大步长 Euler 误差很小,这就是 5 步采样可行的几何原因。Rectified Flow(Liu et al. 2023)顺着这条直觉做"reflow"——用模型生成的 $(x_0, x_1)$ 对作为新的训练数据再训一遍,把轨迹进一步拉直,把推理步数推到 1~4。值得一提的工程细节:训练时 $t$ 的采样分布不一定取均匀分布——SD3 用 logit-normal 把更多权重放在 $t \approx 0.5$,因为中间段的速度场更难拟合(端点附近被 $x_0, x_1$ 牢牢锚定,中段不确定性最大);这种"采样重加权"是 Flow Matching 时代的小型调参艺术。

第六步:和 score matching 的等价

当概率路径选成高斯扩散路径 $p_t(x|x_1) = \mathcal{N}(\alpha_t x_1, \sigma_t^2 I)$(DDPM 前向过程的反向取向),CFM 的条件速度场可以写成

$$u_t(x|x_1) = \frac{\dot \alpha_t}{\alpha_t} x - \Bigl(\dot \sigma_t - \frac{\dot \alpha_t}{\alpha_t}\sigma_t\Bigr) \frac{x - \alpha_t x_1}{\sigma_t},$$

这是 score $\nabla \log p_t(x|x_1) = -(x - \alpha_t x_1)/\sigma_t^2$ 的仿射函数。换句话说:DDPM 的 $\epsilon$-prediction、score matching 的 score-prediction、Flow Matching 的 velocity-prediction,在高斯扩散路径下的最优解处互相等价,只是参数化和路径设计自由度不同。这就是 Tianqi 在 L5 开篇说"两者其实就是同一种连续时间随机过程"的精确含义。线性插值路径并不在 DDPM 的"variance-preserving"或"variance-exploding"两个标准家族里——它对应 $\alpha_t = t,\, \sigma_t = 1-t$ 这种"variance-blowing-up-then-zero"的非标准 schedule,传统 score matching 不会自然选到,但 Flow Matching 视角下它反而最自然,因为速度场是常数。这种"路径选择的自由度"是 score matching 框架下被遮蔽、Flow Matching 框架下被暴露的设计空间。

🟡 教材外补充:

实践层面,SD3、Movie Gen、Flux 之所以全部转向 Flow Matching + 线性路径,不是因为它"理论更优"——理论上 DDPM、EDM、Flow Matching 在无限步极限下是同一个对象。真正的工程理由有三个:(1) 路径与模型解耦:团队可以独立调路径形状(线性、cosine、optimal-transport-aware),不必牵扯模型架构;(2) 大步长 Euler 几乎无损:线性路径下推理 5 步即可,把 H100 上的视频生成成本压到原来的 1/10;(3) 训练数值更稳:DDPM 在 $t \to 0$ 处 score $\propto 1/\sigma$ 爆炸,必须 clip;Flow Matching 速度 $x_1 - x_0$ 是有界的,不需要 clip。这些是 2023 年之后的工程沉淀,Lipman 原论文里并未强调。

另一条 my-addition:Flow Matching 不局限于 Gaussian 先验。Pooladian et al. 2023 的 "Multisample Flow Matching"、Tong et al. 2023 的 "OT-CFM" 都把 $p_0$ 换成任意可采分布(包括另一份图像数据集),训练出来的模型直接做 image-to-image translation。这点把 Flow Matching 推到了比 score matching 更通用的位置——score 的 $\sigma \to 0$ 极限只能从 $\mathcal{N}$ 出发,速度没有这个限制。在蛋白质结构生成、3D 点云、分子构象、视频帧预测这些"先验分布天然非 Gaussian"的任务上,Flow Matching 几乎是默认选择。

再一条 my-addition:CMU L13 关心的"离散 Flow Matching"是把速度场换成连续时间 Markov 链的转移率矩阵 $Q_t$。每个状态向其他状态跳的速率构成一个矩阵,对角元为负、非对角元为正、行和为零("all the flows will add up to zero")。它和连续 Flow Matching 共享同一套语言:先选概率路径(mask diffusion 选"逐渐把 token 换成 [MASK]"),再回归条件转移率。这套统一视角让 SEDD(Score Entropy Discrete Diffusion)、Discrete Flow Matching、Edit Flow 都能写在同一个白板上——这是 Tianqi 在 L13 用整堂课打通的事情。

拖动滑块改变时间 $t$。不变量:每条轨迹的起点(左)和终点(右)不动,只有蓝色粒子的位置沿直线按 $t$ 滑动,速度箭头方向恒定——这正是线性插值路径下"条件速度场 $u_t = x_1 - x_0$ 与 $t$ 无关"的几何含义。

t = 0.30 x₀(噪声,t=0) x₁(数据,t=1) xₜ = (1−t)x₀ + t·x₁;速度 uₜ = x₁ − x₀(方向恒定)

手算一步 Flow Matching

设数据点 $x_1 = 4$、噪声 $x_0 = -1$、采样时间 $t = 0.3$。插值点 $x_t = (1-0.3)\cdot(-1) + 0.3 \cdot 4 = -0.7 + 1.2 = 0.5$。条件速度 $u_t(x_t|x_1) = x_1 - x_0 = 4 - (-1) = 5$。若网络当前预测 $v_\theta(0.5, 0.3) = 3.5$,则单点 CFM 损失:

$$\ell = \bigl(v_\theta - u_t\bigr)^2 = (3.5 - 5)^2 = (-1.5)^2 = 2.25.$$

训练梯度推 $v_\theta(0.5, 0.3)$ 朝 $5$ 走。采样反过来:从 $x_0 = -1$ 开始,Euler 步长 $\Delta t = 0.2$ 共 5 步,假设理想情况下每步 $v_\theta \approx 5$,则 $x_{0.2} = -1 + 0.2 \cdot 5 = 0$、$x_{0.4} = 0 + 0.2 \cdot 5 = 1$、$x_{0.6} = 2$、$x_{0.8} = 3$、$x_{1.0} = 4$——正好命中目标 $x_1 = 4$。这就是"5 步 Euler 出图"的最小工作例子。

常见误区

  1. "Flow Matching 是更简单的 score matching"。反例:CFM 学的是 $u_t = x_1 - x_0$,单位是速度(量纲 $[x]/[t]$);score 学的是 $\nabla \log p_t$,单位是对数密度梯度(量纲 $1/[x]$)。两者只有在高斯扩散路径下才能仿射互换;路径换成 optimal transport coupling 或非高斯先验,关系就不成立。把 Flow Matching 仅仅当作 score matching 的代数变形,就错过了"路径可设计"这个核心自由度。
  2. "训练目标里出现的 $u_t(x|x_1)$ 是 marginal 速度"。反例:固定 $x_1 = 4$、$x_0 = -1$,条件速度恒为 $5$;但同一个插值点 $x_t = 0.5$ 在 marginal 视角下既可能来自 $(x_0, x_1) = (-1, 4)$,也可能来自 $(x_0, x_1) = (0.36, 0.6)$,对应的条件速度分别是 $5$ 和 $0.24$。marginal $u_t(0.5)$ 是对所有这些条件速度按后验 $p(x_1|x_t=0.5)$ 加权平均,不是任何单条直线的速度。CFM 之所以工作,正是因为 L2 回归在最优解处恰好逼出这个条件期望——这一点和 denoising score matching 用同一种数学技巧,但 Flow Matching 框架下表达更直白。
  3. "线性路径就是 Flow Matching 的定义、换路径就不是 Flow Matching 了"。反例:线性插值只是最常用的路径,并非定义本身。Flow Matching 的定义是"在任意概率路径 $p_t$ 上回归对应的条件向量场"——线性路径下 Euler 5 步、CIFAR-10 FID ≈ 6;换成 $x_t = \alpha_t x_1 + \sigma_t x_0$ 配合 cosine schedule 得到的是和 DDPM 等价的 variance-preserving 路径;换成 spherical 插值得到的是 OT-CFM 的球面路径。线性路径只是因为速度恒定、最容易推导,才被原论文选作入门版本;把"步数多 = 质量好"也是从 DDPM 1000 步训练目标遗留下来的偏见,Flow Matching 的速度场近似分段直线,结构性允许大步长,Rectified Flow 一次 reflow 后 1 步采样 FID 仍能 < 10。

相关概念

来源

  1. cmu-10799-diffusion-flow · L5 · Flow Matching 引入:从 SDE 到 ODE · (0:02)
  2. cmu-10799-diffusion-flow · L13 · Discrete Flow Matching 与 Edit Flow 开场 · (0:03)
  3. cmu-10799-diffusion-flow · L13 · 概率路径与 velocity 在离散域的类比 · (2:46)
  4. cmu-10799-diffusion-flow · L13 · "Q_t 就是流,本质上就是速度" · (6:09)
  5. diffusion-principles-vizuara · L9 · 介绍 Flow Models:粒子轨迹直觉 · (0:03)
  6. diffusion-principles-vizuara · L10 · Flow Models 训练目标构造 · (0:04)
查看 248 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 00: 03 Okay, so let's get started. uh so far uh Okay, so let's get started. uh so far uh our last class we have talked about you our last class we have talked about you our last class we have talked about yo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 00: 48 Okay. Uh so this is just like a quick like reminder of like like reminder of like like reminder of like what happened next last class. So what happened next last class. So what happened next last clas…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 01: 39 lastly we talked about like the easiest lastly we talked about like the easiest way to do discrete diffusion which is way to do discrete diffusion which is way to do discrete diffusion which is mass d…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 02: 46 probability path, right? to go from like some noise or like some the full mass to some noise or like some the full mass to some noise or like some the full mass to clean data, right? Yeah, basically t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 03: 32 let let's look at them in a in the discrete settings. So last time we kind discrete settings. So last time we kind discrete settings. So last time we kind of skipped through this part uh or of skipped…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 04: 23 distribution, right? So the probability the sum of the probability over all the sum of the probability over all the sum of the probability over all possible states or all possible tokens possible stat…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 05: 13 transitions should add up to zero. And this is like and this is literally just this is like and this is literally just this is like and this is literally just a con conversation uh conservation of a c…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 06: 09 likely that you're going to go basically it's like how how how likely that you're it's like how how how likely that you're it's like how how how likely that you're going to go jump from this state to …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 07: 03 It's a velocity, right? Essentially. So It's a velocity, right? Essentially. So it's basically just saying that like um it's basically just saying that like um it's basically just saying that like um …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 07: 53 this is also why we're trying to do reparameterization trick yeah but that's reparameterization trick yeah but that's reparameterization trick yeah but that's a great question. All right. Anything a g…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 08: 58 this state that's it right this is like super uh very intuitive I hope is super uh very intuitive I hope is super uh very intuitive I hope is All good. What can we observe here? Can we write What can …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 10: 17 not equal anymore. So just sum of all the z all the possible z's and then qzx the z all the possible z's and then qzx the z all the possible z's and then qzx ptz right and z could be equal to x as ptz…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 11: 21 oiler, right? So you just get you you just literally plug everything in here just literally plug everything in here just literally plug everything in here and then you'll get this thing and the and th…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 12: 15 like the the core of the the story in all the discrete diffusion papers is all the discrete diffusion papers is all the discrete diffusion papers is that like no matter what you do try to that like no…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 13: 03 just use cross entropy to learn this, right? Because we have the ground truth right? Because we have the ground truth right? Because we have the ground truth clean data. So we just need to use cross c…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 14: 01 flips right. So you start from t t flips right. So you start from t t equals z which and then you sample from equals z which and then you sample from equals z which and then you sample from your prior…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 14: 45 you just increment the time and then you do it over over again until you reach do it over over again until you reach do it over over again until you reach the end point. Okay, make sense? This this is…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 15: 42 things that we developed somehow but basically there are two discreful basically there are two discreful basically there are two discreful matching papers uh this time they are matching papers uh this…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 16: 22 know but anyway it's like really like I really hope this guy get rich and famous really hope this guy get rich and famous really hope this guy get rich and famous soon. Uh anyway and then the second o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 17: 05 of learn the rate matrix but we don't of learn the rate matrix but we don't actually learn the rate matrix directly actually learn the rate matrix directly actually learn the rate matrix directly we w…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 17: 50 your par paragraph is in order to build this model and this is just like super this model and this is just like super this model and this is just like super like not good right this is a very very lik…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 18: 36 like it's the same problem as the fixed length thing, right? Yeah. length thing, right? Yeah. length thing, right? Yeah. >> I heard this from like who's doing >> I heard this from like who's doing >> …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 19: 20 right. So let's take a look at it. Uh right. So let's take a look at it. Uh yeah. So essentially variable length you yeah. So essentially variable length you yeah. So essentially variable length you s…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 19: 57 like something that we can think about, right? And it's like now we have like right? And it's like now we have like right? And it's like now we have like two um like frameworks that like sort of two u…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 20: 56 >> Yeah. What else? >> Yeah. What else? >> Inserting at but I saw that you can do >> Inserting at but I saw that you can do >> Inserting at but I saw that you can do decoding in blocks like you can pr…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 21: 29 >> Okay, so tatting at the end. So you can >> Okay, so tatting at the end. So you can discard some of the tokens. Yeah, that discard some of the tokens. Yeah, that discard some of the tokens. Yeah, th…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 22: 15 paragraph and for the next diffusion process that paragraph is that one process that paragraph is that one process that paragraph is that one element for you. You noise the paragraph element for you. …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 23: 09 the same thing and that this is like this is corresponding to one paper that this is corresponding to one paper that this is corresponding to one paper that we're going to talk about but not now we're…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 23: 49 different tokens in between tokens, right? I don't need to insert at the right? I don't need to insert at the right? I don't need to insert at the end. Why do I have to insert at the end? end. Why do …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 24: 31 >> Does this work in the diffusion case? >> Good question. I I'm gonna talk about >> Good question. I I'm gonna talk about >> Good question. I I'm gonna talk about it. Okay. So, how do we even model t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 25: 15 you use it to to to essentially construct construct the the ground truth construct construct the the ground truth construct construct the the ground truth of like what this like what the full of like …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 25: 57 right? What is this other thing? >> The position or the whether to generate >> The position or the whether to generate >> The position or the whether to generate it between these it between these it b…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 26: 52 basically if we're missing tokens and basically if we're missing tokens and like essentially like essentially like essentially note that like it's more likely or like note that like it's more likely o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 27: 45 what you're predicting and then the second objective from your model is second objective from your model is second objective from your model is predict what tokens are missing so this predict what tok…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 28: 26 like I don't know. >> Yeah. >> Yeah. >> Yeah. But basically what Okay. So >> Yeah. But basically what Okay. So >> Yeah. But basically what Okay. So basically what happens in the model or basically wha…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 29: 10 >> Yeah. Yeah. So basically what's happening in what's hap what's happening happening in what's hap what's happening happening in what's hap what's happening in like implementation is that you are in …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 29: 55 is that is missing out of all the tokens is that is missing out of all the tokens that are missing? Right. Okay. So these that are missing? Right. Okay. So these that are missing? Right. Okay. So thes…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 30: 48 missing versus you know how many tokens you predict to be missing and then the you predict to be missing and then the you predict to be missing and then the second thing is uh like literally just seco…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 31: 48 flow And why is it a flow model? Well, it's And why is it a flow model? Well, it's And why is it a flow model? Well, it's actually because this thing like the actually because this thing like the actu…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 32: 35 another thing then you should be doing another thing then you should be doing this operation where it's a ratio or this operation where it's a ratio or this operation where it's a ratio or like a rate…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 33: 23 divergence that is basically just a generalized version of KL. Uh but this generalized version of KL. Uh but this generalized version of KL. Uh but this is like too complicated. We're not going is lik…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 34: 11 actually like already good enough for whatever you want to do. So you can whatever you want to do. So you can whatever you want to do. So you can literally just train a insertion based literally just …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 34: 56 we kind of fixed a veryable length uh problem with insertion based length uh problem with insertion based length uh problem with insertion based generation. Uh we have other things to generation. Uh w…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 35: 40 So like basically this thing is very nice because you can um like essentially nice because you can um like essentially nice because you can um like essentially you can do paralyze generation like you …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 36: 20 that you have generated like in your previous blocks. The only thing you need previous blocks. The only thing you need previous blocks. The only thing you need to like do birectional attention to is t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 37: 00 people think about this. Um but yeah, so people think about this. Um but yeah, so this is like a nice way to do you know this is like a nice way to do you know this is like a nice way to do you know t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 37: 50 >> not in the current block but but we can K catch all the previous blocks. >> No because we don't we haven't generated >> No because we don't we haven't generated that yet, right? So like the block a…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 38: 29 of like also like but but but but if you have too small of a blog then it just have too small of a blog then it just have too small of a blog then it just Yeah. It doesn't really it just is just Yeah.…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 39: 16 in I think. Yeah. in I think. Yeah. >> Do you think the second one is >> Do you think the second one is >> Do you think the second one is um like it's it's the it's a problem of um like it's it's the …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 40: 11 That's correct. Yeah. >> For the for the second case, the mass >> For the for the second case, the mass >> For the for the second case, the mass diffusion, why can't we do something diffusion, why can…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 40: 57 Anything else? Yep. >> Can you overhand why mass diffusion is >> Can you overhand why mass diffusion is >> Can you overhand why mass diffusion is lower? lower? lower? >> Uh okay. Who wants to tell who…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 41: 49 but it doesn't mean that you you cannot train a good enough diffusion model to train a good enough diffusion model to train a good enough diffusion model to blm that's not the case it's just that blm …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 42: 45 the the optimal that you're going to get the the optimal that you're going to get from whatever algorithm that they from whatever algorithm that they from whatever algorithm that they develop will loo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 43: 32 this is not image and text but like we're going to look at image and text we're going to look at image and text we're going to look at image and text later. But for those of you who are in later. But …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 44: 31 >> What are we cogenerating? >> So, we're co-generating something uh >> So, we're co-generating something uh >> So, we're co-generating something uh discreet and something uh continuous at discreet an…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 45: 15 like alternating between the two that's like alternating between the two that's pretty much that right and like pretty much that right and like pretty much that right and like basically And then if yo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 46: 03 well right so what you can do is you can well right so what you can do is you can uh have like just like I think some uh have like just like I think some uh have like just like I think some someone al…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 46: 50 >> I mean the problem is like this is a >> I mean the problem is like this is a research project right so it's like very research project right so it's like very research project right so it's like ve…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 47: 37 and then the the the image for example like flower or like cat and then and like flower or like cat and then and like flower or like cat and then and then the the the the image are going to then the t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 48: 13 text and like image at the same time text and like image at the same time because like you start from nothing and because like you start from nothing and because like you start from nothing and then y…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 48: 49 the way to become omni but this is like just like two two modalities I guess. just like two two modalities I guess. just like two two modalities I guess. Yeah. Yeah. Omni model just means that Yeah. Y…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 49: 50 you cannot access the ponapto anymore, it's still going to be on YouTube. Uh I it's still going to be on YouTube. Uh I it's still going to be on YouTube. Uh I kind of do some editing uh for the kind o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 50: 25 your turn to take what you've learned into the real world so I will highly into the real world so I will highly into the real world so I will highly recommend if you are proud of what recommend if you…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 51: 05 something like that. I think it could be something like that. I think it could be a good experience if you guys are a good experience if you guys are a good experience if you guys are interested in ge…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 51: 39 research slash you know when you go to the AGI company you know make the AGI a the AGI company you know make the AGI a the AGI company you know make the AGI a diffusion model please we don't want to d…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 52: 25 >> padding tokens and tokens after the OS, right? right? >> Yeah. >> Yeah. >> Uh so one thing I implemented for >> Uh so one thing I implemented for >> Uh so one thing I implemented for homework 3 and…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 53: 06 I thought that's what they did in I thought that's what they did in refflow but maybe not. If it's not then refflow but maybe not. If it's not then refflow but maybe not. If it's not then this is a go…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 53: 42 than a 2 RF model because the only than a 2 RF model because the only errors accumulate. errors accumulate. errors accumulate. >> Yeah. No, that's your homework. Four. >> Yeah. No, that's your homewor…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 54: 19 in general I think that's a better idea than the normal the the the way that than the normal the the the way that than the normal the the the way that they they did. Okay. Any other question? they the…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 55: 03 >> Yeah. Yeah. So actually there multiple >> Yeah. Yeah. So actually there multiple uh so the most famous one in the uh so the most famous one in the uh so the most famous one in the robotics world is…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 55: 49 gradient you should apply from those buffer data to your current time to one buffer data to your current time to one buffer data to your current time to one of the time that you sampled. Um so this of…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 56: 25 in that framework but you can turn you in that framework but you can turn you can essentially turn an S OD like a flow can essentially turn an S OD like a flow can essentially turn an S OD like a flow…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 57: 06 basically is there any other ways that we can do it besides turning into a SDE we can do it besides turning into a SDE we can do it besides turning into a SDE um yeah so like let's hope we can get um …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 57: 58 essentially you just sort of like uh they sort of extend the flow map uh uh they sort of extend the flow map uh uh they sort of extend the flow map uh uh what you call framework into the what you call…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 58: 37 Uh, so let's see if he actually post it online this week. Uh, but yeah, I don't online this week. Uh, but yeah, I don't online this week. Uh, but yeah, I don't actually know how how he did it, but act…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 59: 19 of similar to uh one of the other previous uh framework and equilibrium previous uh framework and equilibrium previous uh framework and equilibrium model. So equilibrium model is something model. So e…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 60: 02 or the underlying uh, you know, uh, math is a little bit related, but like this is a little bit related, but like this is a little bit related, but like this is definitely something new. I think is de…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 00: 02 So last time we learned about how So last time we learned about how diffusion turn noise into data by diffusion turn noise into data by diffusion turn noise into data by gradually adding noise and gra…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 00: 52 by an SDE with this score function and then we can just do score matching to then we can just do score matching to then we can just do score matching to learn uh how to solve the uh the learn uh how t…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 01: 41 That's exactly correct. Okay. Why don't we just do linear interpolation of the we just do linear interpolation of the we just do linear interpolation of the two things, right? Um so exact how two thin…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 02: 26 this trajectory in this way. Um yeah so remember the same noise same data. remember the same noise same data. remember the same noise same data. Then learning the transformation along Then learning th…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 03: 28 progress. So like 0.25 time um times uh progress. So like 0.25 time um times uh data minus noise, right? Uh so basically data minus noise, right? Uh so basically data minus noise, right? Uh so basical…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 04: 20 this is literally what we need to learn, this is literally what we need to learn, right? This is like literally the only right? This is like literally the only right? This is like literally the only t…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 05: 17 Right? How do you progress through the chain? You just compute the displacement chain? You just compute the displacement chain? You just compute the displacement for x and then directly apply the for …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 06: 26 full matching. Class is over. Hey, no, not really. Okay, fine. Uh but no, not really. Okay, fine. Uh but no, not really. Okay, fine. Uh but basically the the question that we're basically the the ques…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 07: 10 people start to de develop this thing. Okay. So basically back in 2018 there Okay. So basically back in 2018 there Okay. So basically back in 2018 there this like pretty seminal paper um uh this like …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 08: 05 can just solve this OD by integrating can just solve this OD by integrating through uh the velocity and then you through uh the velocity and then you through uh the velocity and then you you'll be abl…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 08: 55 Anyone else I haven't talked to today? Nobody remember what is normalizing Nobody remember what is normalizing Nobody remember what is normalizing flow? >> You can reconstruct the process. >> You can …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 09: 50 invertible transformation because invertible transformation because everything is deterministic. everything is deterministic. everything is deterministic. Okay, any question? Okay, any question? Okay,…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 11: 01 no new mass uh is is going to get added and no mass is going to disappear and no mass is going to disappear and no mass is going to disappear getting dropped. So like translate it getting dropped. So …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 11: 50 probability flux. So imagine a flux is like the amount of flow the amount of like the amount of flow the amount of like the amount of flow the amount of water uh per unit per time that's going water u…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 12: 55 in a container. It's in like a contained in a container. It's in like a contained tank. So, the probability mass never tank. So, the probability mass never tank. So, the probability mass never change.…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 14: 00 in math in this way. So this is like in math in this way. So this is like basically like how much probability gets basically like how much probability gets basically like how much probability gets cha…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 15: 12 two assumptions. >> Uh two assumptions. One is that >> Uh two assumptions. One is that >> Uh two assumptions. One is that add up to one. The second one is that add up to one. The second one is that ad…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 16: 04 thing that like flow out >> as physically inclined. >> as physically inclined. >> as physically inclined. >> It is okay. >> It is okay. >> It is okay. >> I'm just a civil engineer. >> I'm just a civil…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 17: 08 the divergence connection again the divergence connection again >> yes I I think I missed a P here actually >> yes I I think I missed a P here actually >> yes I I think I missed a P here actually by t…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 18: 21 that like so this is the correct that like so this is the correct divergence. Sorry that that that one I divergence. Sorry that that that one I divergence. Sorry that that that one I missed P. Um but …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 19: 13 the log probability of of the log density. Okay. And this OD is actually density. Okay. And this OD is actually density. Okay. And this OD is actually equal to so the derivative of the total equal to …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 20: 52 So exactly so so how do you train your continuous how do you train your continuous how do you train your continuous normalizing flow from here? What do we normalizing flow from here? What do we normal…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 22: 04 we actually need to do numerical integration right. But this is at integration right. But this is at integration right. But this is at training time. So that just means that training time. So that jus…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 23: 07 numerical integration, right? So it still has a lot of steps. Can we do still has a lot of steps. Can we do still has a lot of steps. Can we do something that doesn't require a lot of something that d…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 24: 11 just need to make sure that the velocity actually match with the ground truth actually match with the ground truth actually match with the ground truth velocity. velocity. velocity. So you just do L2 …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 25: 12 changing every day be like actually I think that the the storm is moving north think that the the storm is moving north think that the the storm is moving north or maybe not. We don't know, right? So …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 25: 58 Yeah. That's right. That's right. You just go straight. just go straight. just go straight. Yeah, exactly. Uh so basically this is Yeah, exactly. Uh so basically this is Yeah, exactly. Uh so basically…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 26: 57 just start from some prior which we can just start from some prior which we can choose. So it doesn't matter we can choose. So it doesn't matter we can choose. So it doesn't matter we can choose the p…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 27: 43 the the data point then you can get a marginal marginal marginal uh yeah and also uh just the boundary uh yeah and also uh just the boundary uh yeah and also uh just the boundary condition essentially…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 29: 03 probability path and also just like how probability path and also just like how like and reweed by how likely in general like and reweed by how likely in general like and reweed by how likely in gener…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 30: 38 this average is like averaging by like this average is like averaging by like how likely is the data point given your how likely is the data point given your how likely is the data point given your cu…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 31: 44 regularization condition. This can can can be done. Uh and then uh basically can be done. Uh and then uh basically can be done. Uh and then uh basically you uh input your um because we know you uh inp…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 32: 46 can construct can generate the marginal property path which is what we want. property path which is what we want. property path which is what we want. Oh, I saw so many confusion. Oh, I saw so many co…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 33: 50 Think about that. Isn't that pretty Think about that. Isn't that pretty cool? Okay. Why? Why? Why is that? Okay. cool? Okay. Why? Why? Why is that? Okay. cool? Okay. Why? Why? Why is that? Okay. So, t…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 34: 46 Like basically you if you swap uh the Like basically you if you swap uh the dx1 term inside so you swap the order of dx1 term inside so you swap the order of dx1 term inside so you swap the order of t…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 35: 50 we're compressing the bullet. >> It's literally the same. It's literally >> It's literally the same. It's literally >> It's literally the same. It's literally the same. It's equivalent. It's not even …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 36: 53 Well we first have the the the endpoint distribution and because we're distribution and because we're distribution and because we're transforming it straight to the point transforming it straight to t…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 37: 52 >> Yeah. No no this is what we chose. This is a design choice. So that so this is a design choice. So that so this is a design choice. So that so this thing is generalizable to any kind of thing is ge…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 39: 09 we can directly sample from >> therefor you >> like previous lectures like gausian >> like previous lectures like gausian assumption is used like this closure on assumption is used like this closure o…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 40: 22 we talked about stuff when we u when we actually uh decrease the the velocity at actually uh decrease the the velocity at actually uh decrease the the velocity at which we adding noise. which we addin…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 41: 26 this particular path. Only on this this particular path. Only on this particular x0 to x1 path. But it's not particular x0 to x1 path. But it's not particular x0 to x1 path. But it's not in general. i…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 42: 24 >> because if you look at the if you look >> because if you look at the if you look at the this the the left hand side right at the this the the left hand side right at the this the the left hand side…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 43: 16 literally the same conference and they all got into the same conference. all got into the same conference. all got into the same conference. So yeah, so the first group is a group So yeah, so the firs…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 44: 00 but you may have this question, right? I feel like some people are having this feel like some people are having this feel like some people are having this question but they didn't ask. So like questio…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 44: 58 do is now let's say you you after training with the linear interpolation training with the linear interpolation training with the linear interpolation you already got this coupling. So the you already…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 45: 50 right? So you can map from one point in right? So you can map from one point in P 0 to another point in P1 and then you P 0 to another point in P1 and then you P 0 to another point in P1 and then you …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 46: 34 because you don't know like basically if you if you only do one step there can be you if you only do one step there can be you if you only do one step there can be like a lot of um uh uh like like lik…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 47: 37 >> The third image is the linear >> The third image is the linear interpolation of the new coupling. interpolation of the new coupling. interpolation of the new coupling. >> That what is that doing? >…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 48: 17 connected this side to that side and connected this side to that side and that side to this side. There's no that side to this side. There's no that side to this side. There's no there's no this anymo…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 48: 56 perfectly trained into this process, >> then you do it again. Yeah. Yeah. So, >> then you do it again. Yeah. Yeah. So, >> then you do it again. Yeah. Yeah. So, this is just you you get it this is just…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 49: 44 well. you're just like kind of random random walking a little bit. Whereas for random walking a little bit. Whereas for random walking a little bit. Whereas for flow matching, you don't really do flow…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 50: 27 question is is flow matching also the question is is flow matching also the same thing because if you think about it same thing because if you think about it same thing because if you think about it y…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 51: 12 can write everything as if it is flow can write everything as if it is flow matching as well and then you can matching as well and then you can matching as well and then you can actually just recover …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 52: 03 still need to take care of that part. But it turns out that we actually do But it turns out that we actually do But it turns out that we actually do know how to take care of that part. And know how to…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 52: 44 OD, right? So this is the just um say we OD, right? So this is the just um say we have this particular OD and it turns out have this particular OD and it turns out have this particular OD and it turns…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 53: 30 change for the probability means that the marginalss are all the same. So this the marginalss are all the same. So this the marginalss are all the same. So this just means that now if you have your ju…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 54: 36 >> That's a great question. Um so for >> That's a great question. Um so for example, I don't know if you guys example, I don't know if you guys example, I don't know if you guys familiar with the RL. …
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 55: 40 Use the use the corresponding PF OD of Use the use the corresponding PF OD of the SDE. Right? So you just plug it in the SDE. Right? So you just plug it in the SDE. Right? So you just plug it in and t…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 57: 07 >> A lot of uh uh lot of PD getting converted to OD. Is it are we just doing converted to OD. Is it are we just doing converted to OD. Is it are we just doing it because it's simpler or do we you it b…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 57: 53 diversity but but yes so like essentially um if you're sampling from a essentially um if you're sampling from a essentially um if you're sampling from a SDE then it's going to be yeah because SDE then…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 59: 06 sampling chain through your by by by sampling chain through your by by by backward integrating your velocity and backward integrating your velocity and backward integrating your velocity and then duri…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 60: 12 you know just like you know you just add something divide something you it's the something divide something you it's the something divide something you it's the same thing. Okay. same thing. Okay. sam…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 61: 01 next week we're gonna be able to see like like like uh different options to further uh different options to further uh different options to further improve what we have learned. Um so for improve what…
cmu-10799-diffusion-flow L5: CMU 10799 S26: Lecture 5 - Flow Matching - Diffusion & Flow Matching 61: 41 the generation faster? And we're going to learn uh to do it with or without to learn uh to do it with or without to learn uh to do it with or without training retraining of the model. And training ret…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 00: 04 Hello everyone and welcome to the next Hello everyone and welcome to the next lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion mode…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 01: 27 Similarly, every single point in the box is following a specific trajectory is following a specific trajectory is following a specific trajectory and the particles are moving as and the particles are …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 02: 49 with not just space but it is changing with not just space but it is changing with time also. [snorts] Why it is with time also. [snorts] Why it is with time also. [snorts] Why it is changing with spa…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 03: 44 Now we approach another person. Let's Now we approach another person. Let's call that person uh detective number call that person uh detective number call that person uh detective number two. And we a…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 04: 36 from trajectories and then calculating from trajectories and then calculating the velocity field. [snorts] Now they the velocity field. [snorts] Now they the velocity field. [snorts] Now they are thin…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 06: 04 which is a specific point in space. which is a specific point in space. [snorts] Let's say this is x0. Now what is x1? X1 is equal to Now what is x1? X1 is equal to X0 + delta X X0 + delta X X0 + delt…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 07: 11 particles. We have looked at their particles. We have looked at their trajectories. We have looked at the trajectories. We have looked at the trajectories. We have looked at the mysterious force that …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 08: 08 their trajectories and you collect all these trajectories and you collect all these trajectories and you collect all these trajectories and you call them as a flow map or just and you call them as a f…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 09: 01 can get a fair idea about what the flow can get a fair idea about what the flow is representing is representing is representing by by by having this square grid and having this square grid and having …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 09: 50 flow models in the first place. So far whatever we have discussed can be whatever we have discussed can be whatever we have discussed can be understood by a physics student easily understood by a phys…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 10: 47 okay generate a new image of a cat. Now the way you solve this problem is Now the way you solve this problem is Now the way you solve this problem is you try to find a distribution you try to find a d…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 11: 50 a gshian distribution that is noise and you end up with the actual distribution. you end up with the actual distribution. you end up with the actual distribution. So the path that we are going to foll…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 12: 55 in a way which is similar to how the in a way which is similar to how the particles are moving in space and time. particles are moving in space and time. particles are moving in space and time. What d…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 14: 00 So we are casting the deep generative modeling problem modeling problem modeling problem as a problem where we are trying to as a problem where we are trying to as a problem where we are trying to pre…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 15: 05 [snorts] and it depends on time as well. So this is exactly the same equation So this is exactly the same equation So this is exactly the same equation that we looked at before. that we looked at befo…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 15: 54 to predict this. So this is approximated by a neural network. by a neural network. by a neural network. Okay. And these are the conditions we Okay. And these are the conditions we Okay. And these are …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 16: 56 distribution that we want to predict in the first place. the first place. the first place. And it turns out that this is at the end And it turns out that this is at the end And it turns out that this …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 17: 52 But what if you start with a gshian But what if you start with a gshian distribution and you end up with distribution and you end up with distribution and you end up with something that does not repre…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 18: 45 is the price of the house. In this case, is the price of the house. In this case, we want to minimize the loss function so we want to minimize the loss function so we want to minimize the loss functio…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 19: 42 this target is not very easy. In fact we have absolutely no idea what this target have absolutely no idea what this target have absolutely no idea what this target is. is. is. And that is going to be …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 20: 46 uh how the training target is predicted uh how the training target is predicted in the first place. in the first place. in the first place. And whatever I'm going to discuss next And whatever I'm goin…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 21: 51 breed. So we have different So we have different So we have different we have a probability distribution that we have a probability distribution that we have a probability distribution that you know t…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 22: 46 going to follow next and uh we are going to go through a little bit of to go through a little bit of to go through a little bit of mathematics right now so please don't be mathematics right now so ple…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 23: 38 Okay. So, first concept that we will Okay. So, first concept that we will discuss is called as So let's let's try to understand what is So let's let's try to understand what is the meaning of [snorts]…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 24: 51 to something I do know even if it is like just one simple single data point like just one simple single data point like just one simple single data point [snorts] and that is exactly what [snorts] and…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 25: 51 know the data which is given to us. We know this initial distribution also. know this initial distribution also. know this initial distribution also. So it is very rational to think that we So it is v…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 27: 05 go to the point Z. So it is conditioned on the point Z. on the point Z. So what can I do so that each individual So what can I do so that each individual So what can I do so that each individual distr…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 28: 46 such that at every time step I will have such that at every time step I will have a gshian and a gshian and a gshian and the probability distribution will change the probability distribution will chan…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 29: 39 probability path which is linear in probability path which is linear in time. time. time. So the mean increases from zero to the So the mean increases from zero to the So the mean increases from zero …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 30: 39 going to generate those trajectories. uh and this is exactly what we are going uh and this is exactly what we are going uh and this is exactly what we are going to do next. But before we move, I want …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 31: 23 anyways uh for now just think about this as being a conditional probability path as being a conditional probability path as being a conditional probability path which is conditioned on the point Z whi…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 32: 15 single point that is the probability path and we want to know at every single path and we want to know at every single path and we want to know at every single time step in this transition how does ti…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 33: 17 trajectories the reverse problem trajectories the reverse problem we simply sample the initial point which we simply sample the initial point which we simply sample the initial point which was x0 in o…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 34: 06 this transformation? There is a vector field which is responsible for this field which is responsible for this field which is responsible for this transformation transformation and [snorts] and [snort…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 35: 06 every single point the velocity vector is pointed towards them. And And for points which are far away the for points which are far away the for points which are far away the vectors are longer in magn…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 36: 05 this is because as as as time proceeds this is because as as as time proceeds we probably want to uh move the velocity we probably want to uh move the velocity we probably want to uh move the velocity…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 36: 56 we don't need anything else using this formula we can simply calculate the formula we can simply calculate the formula we can simply calculate the trajectories of each and every point so trajectories …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 37: 46 Now this u of t is is called as Now this u of t is is called as conditional So you you might look at this and think So you you might look at this and think uh well if you look at the training uh well …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 38: 40 our target and train our neural network to predict it for every single point to predict it for every single point to predict it for every single point zed. zed. zed. It turns out that what we will do …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 39: 37 velocity vector field which is pointed from our data to the data point where we from our data to the data point where we from our data to the data point where we want to go. So it's it's a it's arrow …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 40: 32 final flow that you get the the final trajectories that you get those are the trajectories that you get those are the trajectories that you get those are the marginal probability paths. marginal proba…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 41: 24 to us. So here if you see to us. So here if you see the final distribution that we get which the final distribution that we get which the final distribution that we get which is which are these um is …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 42: 50 want to reach which is a sample of eight washians washians washians and initially we start with a random and initially we start with a random and initially we start with a random noise. noise. noise. …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 43: 45 Now uh here let's play this piece of code to understand this piece of code to understand this piece of code to understand this u and you'll probably get something like and you'll probably get somethin…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 44: 39 in our target. But uh the the issue is that we do not have access to this that we do not have access to this that we do not have access to this because we only have access to our because we only have …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 45: 39 by z minus x divided by 1 minus t. by z minus x divided by 1 minus t. [snorts] So these these these field [snorts] So these these these field [snorts] So these these these field lines are almost linea…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 46: 30 we do not have access to infinite samples. we have access to finite samples. we have access to finite samples. we have access to finite samples samples samples and uh this this uh credit goes to uh an…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 47: 18 understand what we are doing here. Essentially we are treating data Essentially we are treating data Essentially we are treating data generation as a flow problem. We know generation as a flow problem…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 48: 17 field is impossibly complex. You cannot write down an equation that You cannot write down an equation that You cannot write down an equation that defines what a Giblly movie looks like. defines what a…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 49: 06 be true and it does actually work. We we be true and it does actually work. We we have seen this in other uh aspects of have seen this in other uh aspects of have seen this in other uh aspects of deep…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 49: 54 uh I I hope this lecture was interesting for all of you. I haven't yet given the for all of you. I haven't yet given the for all of you. I haven't yet given the final reveal. Uh I'm I'm delaying it fi…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 50: 45 this this came out last year this this came out last year and this is based on uh flow matching. and this is based on uh flow matching. and this is based on uh flow matching. So here essentially what …
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 51: 32 is which is the action expert here which is a it's based on flow matching and uh is a it's based on flow matching and uh is a it's based on flow matching and uh the training target that you see here i…
diffusion-principles-vizuara L1: Lecture 10 - Constructing Training Target for Flow Models | Principles of Diffusion Models 52: 09 lecture which is going to be like a magician's reveal. Thank you.
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 00: 03 Good morning everyone. Good morning everyone. Today we are starting with the next Today we are starting with the next Today we are starting with the next lecture of our course which is lecture of our …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 01: 10 simple which you could have got in the simple which you could have got in the first lecture itself. first lecture itself. first lecture itself. But I don't want to show you that route But I don't want…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 02: 06 keep the mathematics as simple as possible [snorts] and I have taken possible [snorts] and I have taken possible [snorts] and I have taken inspiration from an uh MIT course to inspiration from an uh M…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 03: 01 model is a representation of a phenomena a neural network is an example of a a neural network is an example of a a neural network is an example of a model or any mathematical function which model or a…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 04: 22 we have looked at different techniques to solve this broader problem. It to solve this broader problem. It to solve this broader problem. It started out with variational started out with variational s…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 05: 34 one of the most consequential techniques in modern [snorts] in modern [snorts] in modern [snorts] generative modeling. And the reason it generative modeling. And the reason it generative modeling. And…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 06: 33 and it works exceptionally well. You can see here they have this action expert see here they have this action expert see here they have this action expert and this action expert is built [snorts] and …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 07: 26 formula comes about which is very simple. So flow modeling at the heart of simple. So flow modeling at the heart of simple. So flow modeling at the heart of it is very simple and it has huge number it…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 08: 45 maybe you're sitting on a table right maybe you're sitting on a table right now or you have a table in front of you. now or you have a table in front of you. now or you have a table in front of you. U…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 09: 51 with time. So the magnets that we have placed on the table, what if these placed on the table, what if these placed on the table, what if these magnets are constantly moving? magnets are constantly mo…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 10: 59 take at multiple time steps and hence you will get a vector field which is you will get a vector field which is you will get a vector field which is changing with time. So now we are discussing time d…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 12: 12 depends on not just the position in space but also it depends on the time space but also it depends on the time space but also it depends on the time snapshot. So it is a timed dependent snapshot. So …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 13: 07 timed dependent vector field. So another way of defining this is So another way of defining this is So another way of defining this is people usually write this t as a people usually write this t as a…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 14: 11 So for example, if we take this example at at the top at time t equal to0, this at at the top at time t equal to0, this at at the top at time t equal to0, this is what my velocity field looks like. is…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 15: 16 So now we are asking the question how do the position of particles which are the position of particles which are the position of particles which are inside the space change according to inside the spa…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 16: 24 particle if I know the velocity vector field. This is exactly what this formula field. This is exactly what this formula field. This is exactly what this formula says. says. says. So if I have a parti…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 17: 59 of x0 okay so now this if I write as delta x okay so now this if I write as delta x okay so now this if I write as delta x by delta t by delta t by delta t is equal to mu of u of x0. is equal to mu of…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 19: 08 box end up at time t equal to 10 let's box end up at time t equal to 10 let's say. Now if you know the exact velocity say. Now if you know the exact velocity say. Now if you know the exact velocity at…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 20: 02 box follows the vector field and the box follows the vector field and the question I'm trying to address is where question I'm trying to address is where question I'm trying to address is where does t…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 21: 06 they represent the vector field which is time dependent. That is why the arrows time dependent. That is why the arrows time dependent. That is why the arrows are changing with time. And if we know are…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 22: 21 but it is not quite correct because at but it is not quite correct because at the end of it we are doing an the end of it we are doing an the end of it we are doing an approximation. approximation. ap…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 23: 38 everywhere in a lot of papers. But flow everywhere in a lot of papers. But flow means something very specific. In our means something very specific. In our means something very specific. In our mind, …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 24: 34 flow is a collection of trajectories which evolve according to the velocity which evolve according to the velocity which evolve according to the velocity field. We have just seen that if we know field…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 25: 30 now let us look at one single snapshot because we have we have restricted with because we have we have restricted with because we have we have restricted with the screen in in front of us. Now you the…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 26: 21 trajectories and we get a overall sense of the flow. You can see that this this of the flow. You can see that this this of the flow. You can see that this this is probably something which is swirling.…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 27: 09 let's let's see how the grid deforms. Now note that the grid is going to Now note that the grid is going to Now note that the grid is going to deform in a way that the vector field deform in a way tha…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 28: 09 according to a trajectory which is governed by an ordinary differential governed by an ordinary differential governed by an ordinary differential equation and if we collect all these equation and if w…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 29: 26 understand how the grid deforms. Yeah. So this this does work. So this this does work. So this this does work. Okay. So you can actually change these Okay. So you can actually change these Okay. So yo…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 30: 34 using the word vector field also. Second is the trajectories which are calculated is the trajectories which are calculated is the trajectories which are calculated from the velocity field using the si…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 31: 46 It is only one dimensional minus theta x and it is independent of time. So at and it is independent of time. So at and it is independent of time. So at every time snapshot you take a camera every time…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 32: 52 is dxt by dt is equal to minus theta of xt. xt. xt. Now let's see if this is indeed the Now let's see if this is indeed the Now let's see if this is indeed the flow. flow. If we substitute this in xt,…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 34: 06 is itself completely zero. So the position stays zero. But position stays zero. But position stays zero. But let's look at yeah if x0 is any any any let's look at yeah if x0 is any any any let's look …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 35: 12 example x0 into e to minus theta t and we get the exact same plot. and we get the exact same plot. So we can actually play this and see how So we can actually play this and see how So we can actually …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 36: 03 field and how the flow is a collection of the trajectories. of the trajectories. of the trajectories. [snorts] Okay. Now uh once we are getting Okay. Now uh once we are getting confident with the defi…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 37: 06 We take small steps in the direction of the velocity field and we see how the the velocity field and we see how the the velocity field and we see how the particle trajectory changes with time. particl…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 37: 56 long is your simulation carried on. Okay. So we initialize time as t equal Okay. So we initialize time as t equal Okay. So we initialize time as t equal to0. The step size which is delta t is 1 to0. T…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 38: 56 position. That is all is mentioned in the flowchart. So basically the flowchart. So basically the flowchart. So basically if we have a particle and if we know the if we have a particle and if we know …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 39: 45 For example, when we are using flow models for robotics or protein models for robotics or protein models for robotics or protein synthesis, synthesis, synthesis, people calculate the flow which is the…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 40: 49 Now we have defined the word flow in the Now we have defined the word flow in the initial part of the lecture. We will initial part of the lecture. We will initial part of the lecture. We will underst…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 41: 41 So this is exactly what we are going to So this is exactly what we are going to do in flow models. Also our goal is to do in flow models. Also our goal is to do in flow models. Also our goal is to con…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 43: 05 So we use the same ordinary differential So we use the same ordinary differential equation dx by dt is equal to u. But equation dx by dt is equal to u. But equation dx by dt is equal to u. But this ti…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 43: 58 we have added in flow model is that the we have added in flow model is that the collection of these particles represent collection of these particles represent collection of these particles represent …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 44: 54 evolves from t =0 to t = 1. evolves from t =0 to t = 1. So this is sort of what we want. We want So this is sort of what we want. We want So this is sort of what we want. We want to start from the ini…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 45: 57 the start distribution which is a simple gshian. It is uniformly distributed. gshian. It is uniformly distributed. gshian. It is uniformly distributed. There are maybe around 2,000 particles There are…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 46: 47 from the start distribution to the end distribution. Now here there are a lot of options. We Now here there are a lot of options. We can convert can convert can convert we can change the end distribut…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 47: 42 replaced by a neural network. replaced by a neural network. So there is another word for this in So there is another word for this in So there is another word for this in literature which is also call…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 48: 29 model is being trained and the loss is going down. What is the meaning of this going down. What is the meaning of this going down. What is the meaning of this statement? The flow model is going to be …
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 49: 25 see the animation over here once this code runs. Yeah. So let's see uh as expected we Yeah. So let's see uh as expected we start with the initial gshian start with the initial gshian start with the in…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 50: 30 models we had a number of encoders and a number of decoders. number of decoders. number of decoders. So they were discrete. Here something a So they were discrete. Here something a So they were discre…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 52: 03 the particles who are compressing inside here they are moving out and here they are moving out and here they are moving out and slowly resembling that spiral shape. So slowly resembling that spiral sh…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 52: 56 initial problem of converting the uh initial problem of converting the uh gshian distribution to our actual gshian distribution to our actual gshian distribution to our actual distribution. What we sa…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 53: 50 So the differential equation is also termed as a neural ordinary differential termed as a neural ordinary differential termed as a neural ordinary differential equation. So the terms that I have equat…
diffusion-principles-vizuara L2: Lecture 9 - Introduction to Flow Models | Principles of Diffusion Models 54: 43 ends our first lecture and in the next ends our first lecture and in the next lecture we will understand what is the lecture we will understand what is the lecture we will understand what is the targe…