概念

Video Diffusion

— 不是「图像扩散乘 T 帧」;时间一致性需要结构性改造。

很多人以为 video diffusion 就是「把图像扩散重复 T 次、每帧生成一张图、再拼起来」。这是错的。如果真的逐帧独立采样,相邻帧之间的角色五官、衣服颜色、背景树叶位置都会随机抖动——没有任何机制保证「第 t 帧的猫」和「第 t+1 帧的猫」是同一只猫。这种逐帧独立的做法在 2022 年之前确实有人尝试过,结果是视频里的物体每隔几帧就「变身」一次,连最基本的物理直觉都没有。时间一致性(temporal coherence)需要结构性改造:让网络在去噪时同时看到所有帧,并在帧之间传递信息——本质上是把生成的随机变量从「一张图」升级为「一段时空体」。

第二个常见误解是「视频扩散应该有完全不同的损失函数」。这也是错的。CMU 10799 第 11 讲请来了 Luma AI 的 Linqi (Alex) Zhou 做客座讲座。他在开场就肯定了一个事实:今天市面上几乎所有 text-to-video 模型背后的核心技术,都是你们已经学过的 diffusion + flow matching——只是数据维度升了一档。

「过去五年里,diffusion models 和 flow matching 主导了视觉生成 AI——既出现了 GPT-4o image、Nano Banana、Midjourney、Flux 这样强大的 text-to-image 模型,也出现了我们 Luma 在做的、专攻真实感(realism)的 text-to-video 模型。当下市面上几乎所有 text-to-video 模型,背后都是你们学过的这些技术。」
—— Linqi Zhou, CMU 10799 L11 @01:36

他播放 Luma 最新视频模型 R3 的样片时,还自嘲了一下「视频好像没播放好,有点 jitter」——但即便如此,从样片里依然能看到流畅的人物动作和角色动画(character animation)。这正好暗示了 video diffusion 的两个评价维度:motion 是否真实、character 是否一致。前者要求模型懂物理;后者要求模型懂「同一性」(identity)。

Alex 紧接着强调了「问题」:扩散模型在推理时要解一条概率流 ODE,需要许多步才能采样准确——他原话是 you will still need many many steps to simulate the OD accurately。视频比图像更大、更慢——一段几秒钟的 720p 视频,潜空间维度可能是图像的几十倍,单步前向就比图像贵几十倍。这就把 video diffusion 的真正难题摆了出来:不仅要建模时间,还要算得起、跑得动。这也是为什么 Luma 整个研究路线(inductive moment matching、terminal velocity matching)都在追求「one or few step sampling」——视频场景下,把 50 步压到 4 步意味着用户体验从「等一分钟」变成「等几秒」。

在训练里,video diffusion 对应这件事:把训练数据从 $(H, W, C)$ 的图像换成 $(T, H, W, C)$ 的视频张量,把去噪网络从 2D U-Net 改造成能处理时间维的网络,把损失函数原封不动地用 L2。架构的差异不在「学什么」,而在「网络如何在张量内部传播时间信息」。

三种主流架构配方

把「图像扩散」推广到「视频扩散」,工程上有三条主流路径。三者解决的本质问题相同(如何让网络看到时间维),但归纳偏置、训练成本和能力上限差别很大。下面这张表给出结构对比,下文逐一拆解。

🟡 教材外补充:CMU L11 的客座讲座聚焦于 Luma 在推理加速(inductive moment matching、terminal velocity matching)上的研究,并未系统讲解 video diffusion 的三种架构。下面三种 recipe(Ho 2022、AnimateDiff、Sora)是为了让你对 video diffusion 这个概念有完整的图景所做的补充——三个工作分别代表了 2022、2023、2024 三个阶段的主流设计思路,理解它们的演化能帮你判断「下一步会怎么走」。
配方代表工作核心做法代价 / 限制
3D U-Net + temporal attention Video Diffusion Models (Ho 2022)
Stable Video Diffusion (2023)
Gen-2 (Runway)
2D 空间卷积 + 时间注意力层;直接在 $(T, H, W, C)$ 张量上去噪 compute 随 $T$ 线性甚至更高阶上升;短片(< 4s)效果最佳
Image-as-frame + motion module AnimateDiff (2023)
EmuVideo (Meta)
冻结预训练 image diffusion,在其上新加一个 motion module 学时间相关性 训练便宜;时间一致性中等;可复用 LoRA / DreamBooth 等图像生态
Joint spacetime latent Sora (OpenAI 2024)
Veo (Google)
Luma Dream Machine / R3
3D VAE 把 $(T, H, W, C)$ 压成 $(T', H', W', C')$ 的 spacetime patches;diffusion 在 4D 潜空间 VAE 训练昂贵;但 patch 化后可生成长视频、变分辨率

配方 1:3D U-Net with temporal attention

Ho 等人 2022 年的 Video Diffusion Models 是开山之作,随后 Stable Video Diffusion (Stability AI 2023)、Gen-2 (Runway) 都沿用了这条路线。U-Net 里的 2D 空间卷积块照旧,但在每个 block 后新增一层「时间注意力(temporal attention)」——它把张量 reshape 成 $(B \cdot H \cdot W, T, C)$,让每个空间位置上的不同帧互相 attend。换句话说:网络在同一个像素坐标 $(h, w)$ 上,跨时间维度做 self-attention,从而知道「这个位置在第 5 帧应该和第 4、6 帧保持什么样的关系」。Why:空间卷积负责「这一帧长什么样」,时间注意力负责「这一帧和别的帧的对应位置怎么关联」——两种归纳偏置正交分工。损失函数依然是 L2,训练目标依然是预测噪声或速度场——Alex 在讲座里特别强调过这点:

「最终的 loss 就归结为一个简单的 L2 loss——它非常稳定,扩展到很大参数量也没问题。」
—— Linqi Zhou, CMU 10799 L11 @04:13

工程上的常见做法还包括:先用图像数据预训练空间块,再加入时间块做联合微调;时间维上插入 1D temporal conv 与时间 attention 交替;在条件化阶段把 video FPS 也作为一个额外输入,让模型能生成不同速度的视频。代价是:训练时一个 batch 必须装下整段视频,显存占用随 $T$ 线性上升;推理时单步前向也比图像贵 $T$ 倍。对短片(4 秒、24 帧)效果最好,超过 16 秒就开始捉襟见肘。

配方 2:Image-as-frame + motion module

AnimateDiff 的洞察是:既然 Stable Diffusion 这种 image diffusion 已经被训得很好了,为什么要从头训练一个视频模型?做法是冻结 image diffusion 的权重,只在每个 transformer block 之间插入一个「motion module」——本质上是一个时间维的 self-attention 层。训练只更新这个 motion module,输入是真实视频;Stable Diffusion 的图像先验完全保留不变。Why:图像生成能力来自冻结的 base model;时间一致性来自新加的 motion module——把两种知识解耦后,单独训练后者只需要少量视频数据。

这样设计的另一个好处是 image diffusion 的所有生态(LoRA、ControlNet、个性化模型)都能直接复用——给某个角色训了 LoRA,AnimateDiff 直接就能让这个角色「动起来」,不需要重新收集这个角色的视频数据。代价是:因为底层 image diffusion 从未见过视频,motion module 必须在「不破坏图像质量」的约束下学会时间一致性,最终的视频在快速运动、复杂物理交互场景下会出现飘移和闪烁。Meta 的 EmuVideo 走的也是类似思路,只是先用 image diffusion 生成一张「关键帧」,再让视频模型从这张关键帧扩展出整段视频。

配方 3:Joint spacetime latent(Sora 的路线)

Sora 在 2024 年带火了「spacetime patches」这个说法,Google 的 Veo、Luma 的 Dream Machine / R3 都属于这一路线。流程是:先训一个 3D VAE,把 $(T, H, W, C)$ 的原始视频压成 $(T', H', W', C')$ 的低维潜张量(典型压缩比:时间 4×、空间 8×8——一段 16 帧 $512 \times 512$ 的视频可能被压成 $4 \times 64 \times 64$ 的潜张量);然后把这个 4D 潜张量切成小 patch(例如 $2 \times 2 \times 2$ 的立方体),把所有 patch 展平成一个 token 序列,喂给 diffusion transformer(DiT 风格)。

Why:把时间维当成「另一个空间维」处理,让 VAE 在编码阶段就吸收掉帧间冗余(相邻帧本来就很像,压缩比可以很高),diffusion 不再需要专门的「时间注意力」——所有注意力都是统一的 spacetime self-attention,每个 token 都能跨时间和空间任意 attend。这种设计的最大好处是 patch 数量可以根据视频长度和分辨率灵活变化:想生成 10 秒视频就多几个 token,想生成 60 秒就再多几个,网络结构完全不变。这是 Sora 能生成长视频、不同长宽比的根本原因——也是为什么 Sora 报告里强调 spacetime patches 是「OpenAI 的核心贡献」。

缺点是 3D VAE 的训练昂贵且不稳定:要在重建质量和压缩比之间反复调参,时间维的压缩不当会导致快速运动的视频被「糊掉」。但一旦 VAE 训好,后续 diffusion 阶段的工作量就比配方 1 小得多。

训练目标的扩展

从图像到视频,损失函数本身几乎不变。Alex 在 L11 中讲了 flow matching 的通用形式:在 prior 分布 $\pi$ 和数据分布之间存在一个 ground-truth 速度场 $u_t$(slides 里画成红色矢量),ODE 轨迹(蓝线)从噪声出发沿速度场积分到数据;这个 ground truth velocity field 通常没有 closed form,神经网络的任务就是用 L2 回归把它学出来。视频扩散只是把这个 ODE 搬到更高维空间:

$$\mathcal{L}_{\mathrm{video}} = \mathbb{E}_{t, x_0, \epsilon} \big\| u_\theta(x_t, t) - u_t(x_t) \big\|^2, \quad x \in \mathbb{R}^{T \times H \times W \times C}$$

Why:高维不改变 L2 loss 的形式,只改变神经网络的参数量和数据吞吐量;速度场仍然是切向量,仍然可以通过 DDIM/Euler sampler 在推理时积分出来。Alex 也指出,扩散模型在推理时间扩展性上「scale very poorly with inference compute」——你给它的 number of steps 少,性能就差;多了才能接近最优。视频场景下这个问题被放大数十倍:因为单步前向就比图像贵几十倍,视频用户不可能像图像那样接受 50 步采样,这才有了 Luma 在 IMM、TVM 这些工作上的努力——他们想把视频生成压到一两步完成,同时保持质量。

三种 video diffusion 配方 ① 3D U-Net + temporal attn 输入: (T, H, W, C) 2D conv temp attn 联合去噪所有帧 代价: T× compute ② image+motion module 冻结 SD + 插入新模块 SD (冻) motion 仅训 motion 模块 代价: 时间一致性中等 ③ spacetime latent 3D VAE → 4D patches 3D VAE DiT 空时统一 patch 优势: 长视频可行 共同损失:$\mathcal{L} = \|u_\theta - u_t\|^2$(CMU L11 @04:13: 「simple L2 loss」) 条件化方式:text → cross-attn | image-to-video → 首帧 concat | camera → 额外 embedding

条件化(conditioning)

视频扩散的条件化方式继承自图像扩散,但多了几种新形式:

  • Text → video:和 Stable Diffusion 一样,文本编码器(CLIP 或 T5)输出的 embedding 通过 cross-attention 注入 U-Net / DiT。每个 token 都能 attend 到文本,所有帧共享同一份文本条件——这是为什么用 prompt 控制视频「主题」很容易,但控制「时间顺序」(先 A 再 B)依然困难。
  • Image-to-video:首帧(image)通过 concat 或 cross-attention 喂入网络,让模型「从这张静态图开始动起来」。Stable Video Diffusion 就是这条路线,用户给一张图,模型生成接下来的 25 帧。在训练时把视频第一帧拿来作为条件就好——本质上是给模型一个「初值」。
  • Camera-motion conditioning:把相机的位姿轨迹(pose sequence)编码后注入网络——Stable Video 3D、MotionCtrl 都属于此类。它让用户能指定「镜头向左横移」「环绕物体一周」这种导演意图。技术上是把每一帧对应的 camera pose(外参矩阵)展平成向量,通过 AdaIN 或 cross-attention 注入。
  • Motion conditioning:除了相机,还可以条件化「场景里物体的运动轨迹」——画一条曲线说「让这个鸟沿这条路径飞」。代表工作是 DragNUWA 和 MotionCtrl 的物体控制部分。

评价指标的难题

video diffusion 比 image diffusion 多了一个评价维度——时间维。常用指标是 FVD(Fréchet Video Distance),它把 I3D 视频分类网络的特征拿出来,计算生成视频和真实视频特征分布的 Fréchet 距离——是 FID 的视频版。但 FVD 对「快速运动 + 闪烁」非常不敏感(因为 I3D 主要看动作类别,不看物理一致性),所有大公司在做产品时都会同时做大规模 human study——让人类标注员看 A/B 视频判断哪个更好。Alex 在讲座里也提到 Luma 训练时会引入 RL 和 preference fine-tuning,本质上就是把 human preference 注入训练 loop。

与图像扩散的连接

把上面三种配方放在一起看,会发现一条共同的演化线索:时间维总要被某种方式「吸收」。在 3D U-Net 路线里,时间维由专门的 temporal attention 层显式处理;在 AnimateDiff 路线里,时间维交给后挂的 motion module;在 Sora 路线里,时间维直接被 3D VAE 折叠进潜空间——之后的 DiT 根本不再「区分」时间和空间。从工程趋势看,越往后走,越倾向于让 VAE 承担时间相关性的压缩,把 diffusion 留在一个更平等的「时空 token」域里。这也意味着,理解了 image diffusion 之后,理解 video diffusion 的额外门槛主要在于「数据张量怎么压、怎么 patch」,而不是「损失函数怎么写」。

下面四步对应「video diffusion 一个 forward pass 在做什么」。点击按钮按顺序展开等式链。不变量:每一步的 loss 形式始终是 L2(CMU L11 @04:13),只有输入张量的形状从 $(H, W, C)$ 变成了 $(T, H, W, C)$。

$x_0 \sim p_{\text{data}}$,shape $= (T, H, W, C)$,例如 $(16, 64, 64, 4)$ 的潜视频。
$x_t = \alpha_t x_0 + \sigma_t \epsilon$,$\epsilon \sim \mathcal{N}(0, I)$。整个 4D 张量统一加噪。
$\hat{u} = u_\theta(x_t, t, \text{text}, \text{cam})$。3D U-Net / DiT 跨时间联合处理。
$\mathcal{L} = \|\hat{u} - u_t\|^2$。「a simple L2 loss」,与图像扩散同构。

算一算:3D U-Net 一个 forward pass 的代价

设潜视频形状 $(T, H, W, C) = (16, 64, 64, 4)$。temporal attention 把张量 reshape 成 $(H \cdot W, T, C) = (4096, 16, 4)$,每个空间位置做一次 attention:

单次 attention FLOPs $\approx 2 \cdot T^2 \cdot C = 2 \cdot 16^2 \cdot 4 = 2048$ FLOPs/位置。总计 $4096 \cdot 2048 \approx 8.4 \times 10^6$ FLOPs 仅 attention 一项。

对比同尺寸 image diffusion($T=1$):没有这一项,单层 attention FLOPs 为零。所以仅这一层就把 compute 相对图像版本提升了约 16×;再叠加 U-Net 的多个层级、多个 block,整体「T× compute」的代价就这样累积起来。如果再考虑采样步数(diffusion 通常 25–50 步),生成一段 16 帧潜视频比生成 16 张独立图像贵 16 倍以上——这就是为什么 Luma 要把推理压到一两步、为什么 Sora 要先用 3D VAE 把 $T$ 从 16 压到 4。

常见误区

  1. 「video diffusion = image diffusion × T 次」。反例:取 Stable Diffusion 对同一个 prompt「a cat walking in a garden」固定种子跑 16 次,把 16 张图拼成视频——你会得到 16 个不同角度、不同光线、甚至不同品种的「猫」。即使把种子固定、噪声步数固定,prompt 描述里没有约束的所有自由度都会随机变动。temporal coherence 必须由网络架构在 forward pass 中显式建模,不可能靠外部采样技巧补救。
  2. 「时间一致性 = 帧间像素差小」。反例:让镜头快速横扫,每一对相邻帧像素差可能大到 50%——但视频依然是 coherent 的,因为语义一致(同一只猫,只是位置在变)。反过来,一个抖动严重的静止镜头帧间像素差很小,但人眼立刻能看出「闪烁」。FVD 等指标试图衡量这种语义级时间一致性,而不是 L1/L2 像素差——这也是为什么所有产品级评估都依赖 human study。
  3. 「Sora 是全新算法的发明」。反例:Sora 的训练目标依然是 diffusion + L2 loss——Alex 在 CMU L11 的原话「behind pretty much all of these text-to-video models on the market right now are diffusion models and flow matching」。Sora 的创新主要在 spacetime patches 的表示方式(让 patch 数量随视频长度变化)和大规模训练 scale 上,损失函数和你在第一节课学的 image diffusion 是同一个 L2。把它当成全新算法去读论文会找不到「新的损失」——因为根本没有。

相关概念

来源

  1. cmu-10799-diffusion-flow · L11 · Linqi Zhou (Luma AI): 「diffusion + flow matching 主导视觉生成 AI」 · (00:44)
  2. cmu-10799-diffusion-flow · L11 · Luma R3 视频样本与 motion / character animation · (01:36)
  3. cmu-10799-diffusion-flow · L11 · 「simple L2 loss、extremely stable、scalable」 · (04:13)
查看 83 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 00: 03 Cool. Cool. Yeah. Uh and thanks again Cool. Cool. Yeah. Uh and thanks again for the invite. Um I'm very excited to for the invite. Um I'm very excited to for the invite. Um I'm very excited to see you…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 00: 44 done without my fantastic colleagues at luma. Yeah. luma. Yeah. luma. Yeah. So um in the past five years or so uh So um in the past five years or so uh So um in the past five years or so uh diffusion …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 01: 36 Yeah, there's some jitter. But um yeah, Yeah, there's some jitter. But um yeah, we we show some samples from our latest we we show some samples from our latest we we show some samples from our latest …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 02: 26 uh so to visualize this process you know uh so to visualize this process you know on a on a simple 2D graph um we show on a on a simple 2D graph um we show on a on a simple 2D graph um we show that he…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 03: 21 And these trajectories can be seen as And these trajectories can be seen as these uh blue lines uh here. And the these uh blue lines uh here. And the these uh blue lines uh here. And the ground truth …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 04: 13 and uh ultimately the loss comes down to a simple L2 loss uh which is extremely a simple L2 loss uh which is extremely a simple L2 loss uh which is extremely stable and very scalable to large number s…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 05: 10 naturally it suffers from first uh the OD simulation error uh and slow OD simulation error uh and slow OD simulation error uh and slow inference inference inference so um so um so um so as a result uh…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 06: 03 scaling and this can be best uh represented by the green line here where represented by the green line here where represented by the green line here where where you know when you have very few where y…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 07: 00 problem uh with these types of models is that uh first it it can often suffer that uh first it it can often suffer that uh first it it can often suffer from training instability. For example, from tra…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 07: 55 these different tasks we need to do step step distillation Um and this usually step distillation Um and this usually step distillation Um and this usually comes after the RL or preference or comes aft…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 08: 43 today I'll be cover two works that we did at luma uh which is inductive moment did at luma uh which is inductive moment did at luma uh which is inductive moment matching uh and terminal velocity match…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 09: 36 yeah let's first look at uh how we can achieve efficient inference by design. So to think about efficient inference uh So to think about efficient inference uh we we first need to look at what makes w…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 10: 35 your neuronet network taking in XT&T. And the result the the result of one And the result the the result of one And the result the the result of one step DDIM uh at time S can be best step DDIM uh at …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 11: 34 relationship. Um so we we I I I like to term ter term Um so we we I I I like to term ter term Um so we we I I I like to term ter term this the capacity problem uh of DDIM. this the capacity problem uh…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 12: 32 So so that comes with the power that's uh that's from the um the neuronet uh that's from the um the neuronet uh that's from the um the neuronet network that that's being injected with network that tha…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 13: 34 >> Okay. Do you guys have questions regarding the first part? regarding the first part? regarding the first part? >> No >> No >> No question. question. question. >> Okay. Okay. Yeah. Like I I I told A…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 14: 12 model on S then we can like s like hope model on S then we can like s like hope that it learns the OD better. that it learns the OD better. that it learns the OD better. >> So what about let's say uh …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 15: 03 neural network you are still trying to fit the tangent velocity at every single fit the tangent velocity at every single fit the tangent velocity at every single point. >> I hope that answers the ques…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 16: 00 few step uh network uh directly from from scratch. Okay. So the first method uh inductive Okay. So the first method uh inductive mode matching. So for for this uh for mode matching. So for for this uh…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 17: 05 and we call the two intermediate distributions uh QS and QT where where distributions uh QS and QT where where distributions uh QS and QT where where QS is closer to our data distribution QS is closer…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 18: 00 to to learn this mapping uh that transforms QT to QS uh a naive approach transforms QT to QS uh a naive approach transforms QT to QS uh a naive approach right to learn this mapping uh is is to right t…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 19: 08 is it is kind of kind of like a GAN but is it is kind of kind of like a GAN but uh the optimal discriminator is already uh the optimal discriminator is already uh the optimal discriminator is already …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 20: 12 And um a second component uh for IM is the concept of inductive learning. Uh so the concept of inductive learning. Uh so the concept of inductive learning. Uh so consider the same learning scheme wher…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 21: 01 that is the case here when QT is very far away from from QS. Um so naively far away from from QS. Um so naively far away from from QS. Um so naively matching these two distributions with matching thes…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 21: 58 QQS. um and and and learning the model um and and and learning the model um and and and learning the model learning the model distribution uh learning the model distribution uh learning the model dist…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 22: 54 So what is the base case here? The base case is if you take QS. case is if you take QS. case is if you take QS. Oh uh yeah and here we need to yeah Oh uh yeah and here we need to yeah Oh uh yeah and h…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 23: 55 true regardless of your theta and even true regardless of your theta and even when your theta is untrained um that the when your theta is untrained um that the when your theta is untrained um that the…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 24: 53 is going to be the base case itself. So this is going to be identically equal to this is going to be identically equal to this is going to be identically equal to QS. QS. QS. So what you're matching a…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 25: 51 >> question. Yeah, >> about the choice of MMD. I mean uh we >> about the choice of MMD. I mean uh we >> about the choice of MMD. I mean uh we can formalize the the flow matching as can formalize the t…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 26: 40 some iterative scheme uh to to actually optimize uh that objective. So yeah, so optimize uh that objective. So yeah, so optimize uh that objective. So yeah, so that's part of the reason that we don't …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 27: 31 RBF and and uh Llas kernel both work in RBF and and uh Llas kernel both work in our case. our case. our case. >> Yeah. Yeah. Thank you. >> Yeah. Yeah. Thank you. >> Yeah. Yeah. Thank you. >> Yeah. Yea…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 28: 20 target to to train the new the new time target to to train the new the new time step um the the new model distribution step um the the new model distribution step um the the new model distribution and…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 29: 11 and improved consensity training uh in discrete time. Uh and this is done on discrete time. Uh and this is done on discrete time. Uh and this is done on imageet. So consensity training imageet. So con…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 30: 14 transformer size. Uh so so so the more compute you throw at it, it gets uh compute you throw at it, it gets uh compute you throw at it, it gets uh better performance in general better performance in g…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 31: 09 to consistency training as a special case if our MMD objective uses only one case if our MMD objective uses only one case if our MMD objective uses only one particle. Uh and instead of using LLAS part…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 32: 04 and G GMMN can also be seen as a special and G GMMN can also be seen as a special case of Imm. Uh case of Imm. Uh case of Imm. Uh you can easily see that when if you set you can easily see that when i…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 32: 53 essentially in this paper the drift field for their uh for their model is field for their uh for their model is field for their uh for their model is also parameterized via uh via MMD also parameteriz…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 33: 53 per GPU can can usually be as low as per GPU can can usually be as low as one. uh and in this case you know you one. uh and in this case you know you one. uh and in this case you know you need to gath…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 34: 45 distinguish between nearby R&T then it distinguish between nearby R&T then it it's generally uh harder to scale. it's generally uh harder to scale. it's generally uh harder to scale. So these are so t…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 35: 36 so kernels is is a very uh standard um concept I guess in machine learning um concept I guess in machine learning um concept I guess in machine learning um it's it it's basically a a similarity it's i…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 36: 34 to to some um vector where each entry is like a uh a power of this real number like a uh a power of this real number like a uh a power of this real number and that is kind of a augmented feature and t…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 37: 28 the and the and the keys right and this is sort of like basically we project the is sort of like basically we project the is sort of like basically we project the query and the keys to some feature sp…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 38: 15 and also maybe like if you guys have tried like empirically like how do would tried like empirically like how do would tried like empirically like how do would you compare the three types of models? y…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 39: 20 point XR say. Um so what what we found point XR say. Um so what what we found is that uh because of the correlation is that uh because of the correlation is that uh because of the correlation through …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 40: 19 particle for training uh for for for particle for training uh for for for training the model means that they are training the model means that they are training the model means that they are ignoring …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 41: 16 >> uh, yeah, but but for the next work, I guess it'll be much closer to flow maps. guess it'll be much closer to flow maps. guess it'll be much closer to flow maps. >> All right. Cool. Cool. >> All ri…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 42: 10 curved path in space which I'm sure all of you know um and on the right uh we of you know um and on the right uh we of you know um and on the right uh we have uh have uh have uh a a model train trend …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 43: 11 prior x1. So the tangent along this OD trajectory So the tangent along this OD trajectory So the tangent along this OD trajectory is the marginal velocity is the marginal velocity is the marginal velo…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 44: 15 displacement f theta uh which we can define um you know as uh uh which we can define um you know as uh uh which we can define um you know as uh s minus t * capital f where where s minus t * capital f …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 45: 11 map that directly takes xt to x0 map that directly takes xt to x0 so naively uh what we want to learn oh so naively uh what we want to learn oh so naively uh what we want to learn oh how How we can le…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 46: 00 actually want to look at is this quantity um which which characterizes quantity um which which characterizes quantity um which which characterizes the the terminal velocity uh the the the the terminal…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 47: 03 if it happens to match the marginal velocity at this point then you have velocity at this point then you have velocity at this point then you have automatically learned this this automatically learned…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 48: 07 and this is best shown uh in in this figure uh in the previous slides. Um so figure uh in the previous slides. Um so figure uh in the previous slides. Um so this is the case when when uh the bound thi…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 49: 06 these quantities exist but we don't know that during pre pre-training time so how that during pre pre-training time so how that during pre pre-training time so how do we so how do we kind of have acce…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 50: 02 your own network u approximates the the your own network u approximates the the ground truth velocity at at every single ground truth velocity at at every single ground truth velocity at at every sing…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 50: 56 that uh TVM loss upper bounds the two that uh TVM loss upper bounds the two wound distance up to some constant and wound distance up to some constant and wound distance up to some constant and this is…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 51: 54 actually not lies continuous because of the dot product self attention and layer the dot product self attention and layer the dot product self attention and layer norms and there's actually a lot of n…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 52: 51 we you know add add RMS norm as uh we you know add add RMS norm as uh without any parameters uh to the time without any parameters uh to the time without any parameters uh to the time step modulation …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 53: 53 and you can do some quick mental math uh to just take derivative with respect to to just take derivative with respect to to just take derivative with respect to s on both sides, right? And s on both s…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 54: 49 have any stop gradient um which means have any stop gradient um which means that we ideally want to back propagate that we ideally want to back propagate that we ideally want to back propagate through…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 55: 39 So this is actually exactly opposite of meanflow which takes partial on both xt meanflow which takes partial on both xt meanflow which takes partial on both xt and t. and t. and t. Yeah. Uh so I will …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 56: 38 very stable. And if we track also the norm of the And if we track also the norm of the And if we track also the norm of the marginal velocity uh for both mean flow marginal velocity uh for both mean f…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 57: 32 network predicted velocity and uh and um network predicted velocity and uh and um conditional velocity and there are conditional velocity and there are conditional velocity and there are additional va…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 58: 34 >> and yeah um experimentally we achieve still there one step result on imaget still there one step result on imaget still there one step result on imaget and it can also surpass uh dn and xit and it …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 59: 27 scale and this model is is trained with scale and this model is is trained with pure TVM loss uh closely following what pure TVM loss uh closely following what pure TVM loss uh closely following what …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 60: 16 for uh FSTP if you naively call JVP on for uh FSTP if you naively call JVP on the entire FSTP sharded model the entire FSTP sharded model the entire FSTP sharded model uh then FSTP will usually error …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 61: 07 first challenge with uh FSTP with JVP first challenge with uh FSTP with JVP and the second challenge was is uh is and the second challenge was is uh is and the second challenge was is uh is writing JV…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 61: 58 uh I will take some questions. Uh but before that I will I have also have I before that I will I have also have I before that I will I have also have I was also given a task of of recruiting. was also…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 62: 51 reach out to me or to to our our team and uh also the inventors of DDIM and and uh also the inventors of DDIM and and uh also the inventors of DDIM and Nerf both both work here. So, the mentor Nerf bo…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 63: 46 >> may I ask question so I might be wrong but it seems that when you do the but it seems that when you do the but it seems that when you do the terminal velocity matching at a specific terminal veloci…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 64: 27 time team? Oh uh do you mean the integration integration integration here? here? here? >> Yeah. Over a different time. >> Yeah. Over a different time. >> Yeah. Over a different time. >> Oh yeah. Yeah.…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 65: 17 the the integration only comes in um if the the integration only comes in um if you want to measure its relationship you want to measure its relationship you want to measure its relationship with resp…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 66: 14 any sweet spot for the NFP like with four NFP you are getting good quality four NFP you are getting good quality four NFP you are getting good quality but if you kind of increase it to 50, but if you …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 67: 03 scratch like it's just a distillation approach how is it compared to EVM approach how is it compared to EVM approach how is it compared to EVM >> uh sorry for do you mean comp compared >> uh sorry for…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 67: 56 your own network then then that's basically a dissolution technique. Yeah. basically a dissolution technique. Yeah. basically a dissolution technique. Yeah. >> Thank you. >> Thank you. >> Thank you. >…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 68: 52 step sampling and if you spend more more more compute on on more intermediate more compute on on more intermediate more compute on on more intermediate step it doesn't give you a huge gain um step it …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 69: 54 inspired me uh to write this paper was a inspired me uh to write this paper was a work called um uh physics informed work called um uh physics informed work called um uh physics informed dissolation f…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 70: 48 know. >> Is there any computational overhead >> Is there any computational overhead >> Is there any computational overhead since we have a much more complex final since we have a much more complex fin…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 71: 50 objectives at the same at the same time. objectives at the same at the same time. Um uh so we have observed that if you Um uh so we have observed that if you Um uh so we have observed that if you opti…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 72: 39 terms of performance or in terms of like terms of performance or in terms of like training stabilities like like yeah training stabilities like like yeah training stabilities like like yeah basically …
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 73: 34 models in general still has some advantage in terms of training training advantage in terms of training training advantage in terms of training training uh uh training simplicity I guess. Um uh uh tra…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 74: 32 >> no. Uh but that's a great uh I think to to study uh because for we only used the to study uh because for we only used the to study uh because for we only used the DDIM uh uh the the the DDM form of…
cmu-10799-diffusion-flow L10: CMU 10799 S26: Lecture 11 - Guest Lecture Linqi (Alex) Zhou from Luma AI - Diffusion & Flow Matching 75: 28 out. >> Yeah, please reach out. Um yeah and Alex >> Yeah, please reach out. Um yeah and Alex >> Yeah, please reach out. Um yeah and Alex can if you can share with me with the can if you can share with…