概念 · Module 6 离散与语言扩散

Masked Diffusion

— 把"加噪"换成"按概率盖 [MASK]";reverse 就是把 mask 一步步填回去。

初学者听到 "masked diffusion" 几乎都会反应:"这不就是 BERT 吗?BERT 不就是把 token 盖住、让模型预测吗?" —— Masked diffusion is NOT BERT:BERT 是 fixed-rate(固定 mask 比例 15%)的一次性预测;MD 是 sample-a-rate-then-iterate(先采一个 mask 率、再迭代解 mask)的生成模型。这是最关键、也是最常见的误解,必须先纠正。BERT 在训练时每个 batch 都按固定 15% 比例随机盖住 token,模型只学这一个噪声级别;推理时它一次性把所有 mask 位置同时预测出来;它根本不是生成模型,本质是表示学习——你拿一个全 mask 的序列输入 BERT,它会输出乱码,因为"99% mask"这种 regime 它从未在训练时见过。

Masked diffusion 做了两个关键扩展:

  1. 把"mask 比例"提升成连续时间变量 $t \in [0,1]$。每个训练样本随机采一个 $t$,按比例 $\bar\beta_t$ 盖 token;这意味着同一个模型既要学会"只盖 1%"的微小扰动,也要学会"盖 99%"的近乎从头生成。BERT 是单点训练,MD 是覆盖整条噪声曲线训练。
  2. 推理时迭代。从全 mask 序列 $x_T$ 出发,模型对所有 mask 位置同时输出预测分布,但只 commit 置信度最高的 top-$k$ 个位置;其余继续作为 mask 喂回网络,循环若干步直到全部解开。BERT 是 one-shot,MD 是 iterative refinement。

记住这个口诀就不会再混淆:"BERT 是固定比例的 fill-in-the-blank;MD 是变比例的、迭代的 fill-in-the-blank。"前者做表示学习,后者做生成。

讲师的入题方式(CMU 10799 L12 @ 00:02):Eric Xing 在系统讲完 DDPM、score-based、flow matching 三视角后,抛出转折问题——"前面所有东西都在连续空间(图像归一到 $[-1,1]$),那离散数据怎么办?分子、DNA、文本——甚至图像本身存的也是 $0\text{–}255$ 的离散像素。"这是从连续扩散通往 MD 的天然过渡。
"so far everything we have learned are about image or image that's represented in a continuous space ... is it possible for us to extend what we have learned to discrete data?" —— CMU 10799 L12 @ 01:36

讲师紧接着抛了一个半玩笑的动机:"Anthropic 的人跟我说——大概是 ICML 2024——凡是不能压成文本的信息都是无聊的信息,图像生成根本不在通往 AGI 的路上"(CMU L12 @ 01:45)。讲师补了一句:"咱们等着看 Anthropic 据传要发的图像模型是不是真的在通往 AGI 的路上吧。不管怎么说,AGI bros 真的相信文本是通往超级智能的路径,所以这话题从投资角度看也极其重要。"半开玩笑地说明为什么要做离散/文本扩散:除了文本本身的研究价值,分子、DNA 序列、甚至量化后的图像像素都是离散数据。

"anything that cannot be expressed or compressed into text is uninteresting information ... image generation is just not on the path to AGI." —— CMU 10799 L12 @ 02:00(讲师转述 Anthropic 观点)

当学生提议"把 token 嵌入到连续空间、做连续扩散、再 nearest-neighbor 回到离散 token"时,讲师评价:"这是一个很好的思路,事实上 Stanford 团队最早就这么做的(Diffusion-LM, Li et al. 2022)。不幸的是它效果不如我们今天要讲的方法"(CMU L12 @ 04:43)—— 这就是从 latent-continuous diffusion 转向纯离散吸收态扩散的历史动机。

Vizuara 的 Dr. Raj 在 Lecture 12–17 走的是另一条互补的路径:从直觉出发把整套 diffusion LM 类比成"流体旋转 + 反向预测"。他在 L13 @ 00:07 说:"就像你慢慢搅动液体——你希望预测每一步加了多少噪声,这样反向去噪时才能恢复原状态。"把图像扩散里的"加高斯噪声"换成"按概率盖 mask",整套训练管线就自然映射到 MD。

"if you take a look at this video again, what we want to do is that when we start rotating the fluid slowly, we want to predict how much noise is added at each step. So that during denoising, it will help us recover the original state." —— Vizuara L13 @ 00:07

Dr. Raj 强调 MD-LM 与 AR-LM 的目标对照(L12 @ 00:03):"我们想得到一个尽量逼近真实文本分布的预测分布,但不用自回归模型,而用 diffusion 模型。"放回 ML 上下文一句话总结:训练时你在做"任意 mask 比例下的填空";推理时你在做"从全 mask 出发的多步去噪"——loss 是熟悉的 cross-entropy,只在被 mask 的位置上算。

Forward:吸收态转移矩阵(absorbing transition matrix)

设 token 词表大小 $V$,加一个特殊符号 $\texttt{[MASK]}$,总状态数 $V+1$。每一步、每一个位置独立地按一个 $(V+1) \times (V+1)$ 的转移矩阵 $Q_t$ 做马尔可夫跳变。离散扩散一般框架(D3PM, Austin et al. 2021)允许多种 $Q_t$ 选择——比如 uniform(任意 token 可以变成任意其他 token),比如 gaussian-like 近邻(按词向量距离衰减),但 masked diffusion 选的是吸收转移(absorbing transition):所有 token 都有概率被吸收到唯一一个"黑洞状态" $\texttt{[MASK]}$,进入黑洞后再也出不去。

Why 吸收态? 吸收态指 $\texttt{[MASK]}$ 一旦进入就再也出不去——$Q_t$ 的 $\texttt{[MASK]}$ 行是 $[0,\dots,0,1]$。这带来三个工程优势:(1) $q(x_t \mid x_0)$ 有闭式,因为位置 $i$ 在 $t$ 时刻要么仍是原 token、要么已是 mask,没有"变成另一个 token"的中间态;(2) reverse 网络只需在 mask 位置预测,未 mask 的位置直接保留——目标函数大幅简化;(3) 整个状态轨迹是"单调消失"的,比 uniform 的"乱跳"更稳定,更容易训练。代价是 endpoint 不再有近似均匀分布的样本多样性——这在文本里反而是好事(你不希望终点是随机 token 串)。

单步 forward 在位置 $i$ 上的转移分布:

$$ q(x_t^{(i)} \mid x_{t-1}^{(i)}) \;=\; (1-\beta_t)\,\delta_{x_{t-1}^{(i)}} \;+\; \beta_t\,\delta_{\texttt{[MASK]}}. $$

含义:以概率 $1-\beta_t$ 保持当前 token,以概率 $\beta_t$ 跳到 mask 状态。已经是 mask 的位置永远保持 mask(吸收)。各位置之间独立采样——这是连续扩散里"每个像素独立加噪"的离散对应物,是闭式可解的根源。

闭式 $q(x_t \mid x_0)$:跳过中间时间步

因为吸收态不可逆,"截止 $t$ 步内被 mask 至少一次"的概率就是 $1$ 减去"每一步都没被 mask"的概率:

$$ \bar\beta_t \;=\; 1 - \prod_{s=1}^{t}(1-\beta_s). $$

类比 DDPM 的 $\bar\alpha_t = \prod_s \alpha_s$——这里 $\bar\beta_t$ 是"截止到 $t$ 已被吸收的累积概率"。于是每个位置独立有:

$$ q(x_t^{(i)} \mid x_0^{(i)}) \;=\; (1-\bar\beta_t)\,\delta_{x_0^{(i)}} \;+\; \bar\beta_t\,\delta_{\texttt{[MASK]}}. $$
Why 闭式重要? 训练时你不想真的跑 $T$ 步前向链——一个 batch 一千个样本、每个跑 $T=1000$ 步,根本算不动。闭式让你只采一个 $t$、按 $\bar\beta_t$ 比例独立 mask 每个位置,一次到位地得到 $x_t$。这和图像 DDPM 的"重参数到任意 $t$"($x_t = \sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\epsilon$)是同一类技巧,背后都是马尔可夫链的"半群性质"。

Endpoint:全 MASK 序列就是"纯噪声"

当 $t \to T$(或连续时间下 $t \to 1$),$\bar\beta_t \to 1$,$x_T$ 几乎确定地是全 $\texttt{[MASK]}$ 序列。这就是 MD 的"纯噪声"——一个完全没有信息的序列。注意:这和图像 DDPM 的 $x_T \sim \mathcal{N}(0,I)$ 是结构对应但工程不同的"先验"——MD 的先验是 deterministic(全 mask),DDPM 的先验是 stochastic(高斯采样)。MD 推理时你不需要采样初始噪声,直接拿全 mask 序列开始。

Reverse:神经网络预测原 token,只在 mask 位置算 loss

reverse 过程是一个神经网络 $p_\theta(x_0 \mid x_t)$(一般是双向 Transformer,因为没有 causal mask 的需要),输入带 mask 的序列,输出每个位置上的 token 分布。但 loss 只在 mask 位置上累加:

$$ p_\theta(x_0^{(i)} \mid x_t) \;=\; \mathrm{softmax}\bigl(f_\theta(x_t)_i\bigr),\qquad \mathcal{L}(t) \;=\; -\sum_{i \in M_t} \log p_\theta\!\bigl(x_0^{(i)} \mid x_t\bigr), $$

其中 $M_t = \{i : x_t^{(i)} = \texttt{[MASK]}\}$ 是当前被 mask 的位置集合。未 mask 的位置不算 loss——它们是"已知信息",让模型把容量浪费在"把 cat 预测成 cat"这种零信息任务上是低效的。完整训练目标对 $t$ 取期望并加权:

$$ \mathcal{L}_{\text{MDLM}} \;=\; \mathbb{E}_{t \sim \mathcal{U}[0,1]}\,\mathbb{E}_{x_0,\,x_t \sim q(\cdot\mid x_0)}\!\left[\,\frac{1}{\bar\beta_t}\sum_{i \in M_t} -\log p_\theta\!\bigl(x_0^{(i)} \mid x_t\bigr)\right]. $$

权重 $1/\bar\beta_t$ 源自 ELBO 推导(Sahoo et al. 2024 Eq. 7),直觉是"早期(小 $t$、少量 mask)单条 loss 信息含量低,后期(大 $t$、大量 mask)单条 loss 信息含量高,归一化后让两端权重平衡"。实践中这个权重在最简形式 $\bar\beta_t = t$ 下就是 $1/t$,是 MDLM 比 D3PM 训练更稳的关键工程改进之一。

MDLM 的三处关键升级(Sahoo et al., Cornell, NeurIPS 2024)

🟡 教材外补充 D3PM (2021) 是离散时间、固定噪声调度、loss 形式偏 ELBO 早期版本。MDLM (Simple and Effective Masked Diffusion Language Models) 在 2024 年做了三件事,让 MD-LM 真正在语言建模指标上能打:

  1. 连续时间公式化:$t \sim \mathcal{U}[0,1]$,最简调度 $\bar\beta_t = t$(线性),训练时无需离散化 $T$ 步;loss 变成简单的 weighted negative log-likelihood,没有 KL 项也没有辅助 loss。这让代码量从 D3PM 的几百行收缩到核心几十行。
  2. 变 mask 率覆盖整条 $[0,1]$:BERT 永远在 15% 上训;MDLM 训练时每个样本采一个 $t$,从 1% 到 99% 的 mask 比例都见过。因此同一个模型既能做"几乎没盖"的填空(高密度 context),也能做"几乎全盖"的从头生成(低密度 context)。这是 MD 能做生成、而 BERT 不能的根本原因——不是网络结构差异,是训练分布差异。
  3. 规模对标 GPT-2:在 LM1B / OpenWebText 上达到与 GPT-2 medium 接近的 perplexity,首次让 MD-LM 在语言建模指标上和同等规模的 AR 模型同台竞争。在那之前,扩散语言模型一直被认为"原理优雅但实际效果不行"。
后续工作链(教材外补充):LLaDA (Nie et al. 2025) 把 MDLM 思路 scale 到 8B 参数,对话能力接近 LLaMA-3 8B;DiffuLLaMA 直接从预训练 AR 模型蒸馏出 MD-LM,复用 LLaMA 的预训练权重;Google 的 Gemini Diffusion(2025)也是这条路线的工程化产品。这些工作都共享同一套吸收态数学骨架——你掌握了 MDLM,就掌握了它们的核心。

三种语言建模范式的并列对照

🟡 教材外补充 把 MD 放进语言建模的版图里,最有用的对照是 AR-LM(GPT 系列)、Masked-LM(BERT)、Masked Diffusion LM(MDLM/LLaDA)三者的训练任务和推理协议差异。三者都用 Transformer,差别全在训练目标和如何"用"模型:

维度AR-LM (GPT)Masked-LM (BERT)Masked Diffusion (MDLM)
注意力Causal(单向)BidirectionalBidirectional
训练 mask 率0(无 mask)固定 15%$t \sim \mathcal{U}[0,1]$,所有比例
训练目标下一个 token 的 NLLmask 位置 NLL(一次性)mask 位置 NLL,按 $1/\bar\beta_t$ 加权
推理协议左→右,逐 token一次性预测 mask从全 mask 起,迭代 32–256 步
能生成?能不能(OOD)能
并行解码?难(KV cache 串行)—原生支持,每步多位置
可控位置编辑?难能(infilling)能(任意 mask 模式)

从这张表能看出 MD 的位置:它继承 BERT 的双向架构和 mask 范式,但通过"变 mask 率训练 + 迭代推理"获得了 AR 才有的生成能力。这是 MD 之所以在 2024–2025 年突然引起注意的根本原因——它不是"另一种生成模型",而是第一个把 BERT-style 表示学习和 GPT-style 生成统一在同一个数学框架里的方案。

为什么不走"嵌入到连续空间再扩散"那条路?

CMU L12 @ 03:36 学生提议:"把文本 token 先 embedding 到连续向量,在连续空间做标准 diffusion,最后做 nearest-neighbor 找回最近的 token。"讲师回应:"这是 Stanford 团队(Li et al. 2022 的 Diffusion-LM)最早的尝试,思路很自然——但它效果不如吸收态扩散。"为什么?两个核心问题:

  1. 连续 ↔ 离散的语义鸿沟。token 嵌入空间不是凸的——两个合法 embedding 之间的中点未必是合法 token 的 embedding。连续扩散过程会把样本带到"语义空白区",nearest-neighbor 投影时引入大量量化误差。吸收态扩散全程在离散状态空间运动,不存在这种鸿沟。
  2. 语言的组合性 vs 像素的局部连续性。像素 RGB 是天然连续的,加少量高斯噪声后图像还能看;token 不是——把 "cat" 的 embedding 加点噪声,对应回去可能就成 "bat"、"rat"、或一个完全不相干的词。文本的局部结构对连续扰动极其敏感。

这就是为什么 2022 年之后的研究主线从"连续 latent + 离散投影"转向"全程离散 + 吸收态"——MDLM、LLaDA、SEDD 都是这条线的产物。

切换下方五个按钮,对比 BERT 的一次性 mask 和 MD 的迭代去 mask。注意看这个不变量:BERT 只在 15% 这一个 mask 比例上工作(按钮 ①);而 MD 在每一步上 mask 比例都不同——从 100% (步骤 ②) 一路降到 0% (步骤 ⑤),同一个网络要在所有这些比例下都能输出合理预测。这就是为什么"BERT 不能生成、MD 可以生成"的核心机制。

BERT: 固定 15% mask,一次性预测 The cat sat on the mat ? ? ? ? ? ?
BERT 训练时固定挑 15% 的 token 替换为 [MASK](这里挑了 cat),模型一次性预测,loss 只算被 mask 的位置。它不会生成整段文本——拿掉所有 token 它就废了。

手算:$\bar\beta_t$、加噪采样、单步 loss

设 6 个 token 的句子 $x_0 = $ "The cat sat on the mat"(位置 0–5),离散时间 $T=4$,线性调度 $\beta_t = 0.25, 0.30, 0.35, 0.40$。

第一步:算 $\bar\beta_t$

按递推式 $\bar\beta_t = 1 - \prod_{s=1}^{t}(1-\beta_s)$:

$$ \bar\beta_1 \;=\; 1 - 0.75 \;=\; 0.250, $$ $$ \bar\beta_2 \;=\; 1 - 0.75 \cdot 0.70 \;=\; 1 - 0.525 \;=\; 0.475, $$ $$ \bar\beta_3 \;=\; 1 - 0.525 \cdot 0.65 \;=\; 1 - 0.341 \;=\; 0.659, $$ $$ \bar\beta_4 \;=\; 1 - 0.341 \cdot 0.60 \;=\; 1 - 0.205 \;=\; 0.795. $$

含义:到 $t=4$ 时,每个位置独立地有 79.5% 概率已变 mask,平均 $0.795 \times 6 \approx 4.77$ 个位置被盖。注意 $\bar\beta_t$ 单调递增但不会到 1——除非 $T$ 足够大。要让 $x_T$ 接近全 mask,调度尾段必须够大、或 $T$ 够长。

第二步:采样 $x_2$

在 $t=2$,每个位置独立采 Bernoulli(0.475):假设抽样结果是 (no, mask, no, no, mask, no),则

$$ x_2 \;=\; (\text{The},\, \texttt{[MASK]},\, \text{sat},\, \text{on},\, \texttt{[MASK]},\, \text{mat}). $$

mask 位置集合 $M_2 = \{1, 4\}$。

第三步:单样本 loss

假设网络 $f_\theta$ 接收 $x_2$,对位置 1 输出 $p_\theta(\text{cat} \mid x_2) = 0.6$,对位置 4 输出 $p_\theta(\text{the} \mid x_2) = 0.8$(位置 0/2/3/5 的输出被丢弃)。未加权 loss:

$$ \mathcal{L}_{\text{raw}}(t{=}2) \;=\; -\log 0.6 - \log 0.8 \;=\; 0.511 + 0.223 \;=\; 0.734. $$

按 MDLM 的 ELBO 权重 $1/\bar\beta_t = 1/0.475 \approx 2.105$:

$$ \mathcal{L}_{\text{MDLM}}(t{=}2) \;=\; 2.105 \times 0.734 \;=\; 1.545. $$

未 mask 的 4 个位置(The/sat/on/mat)不进入 loss——这是 MD 和普通 LM cross-entropy 最容易混的差别。普通 LM 对所有位置算 loss,MD 只对 $M_t$ 中的位置算。把这一点写错,模型 80% 的梯度信号会浪费在"已知 token 自我预测"上,训练几乎不收敛——这是实现 MD 时最常见的 bug。

常见误区

  1. "Masked diffusion = BERT,就是换了个包装。" 反例:把训好的 BERT 推理时输入一个 100% mask 的 16-token 序列让它"生成文章"——它会输出语法看似合理但语义全错的乱码(实测 perplexity 比 MDLM 高 20+ 倍),因为 BERT 训练时只见过 15% mask 这一个 regime,99% mask 对它是 out-of-distribution。MDLM 在同样输入下能产出连贯文本,因为它在训练时确实见过 $t \approx 1$(接近全 mask)的样本,并且反复迭代解 mask。结构上 BERT 和 MD 的网络几乎一样(都是双向 Transformer),差别在训练分布和推理协议。
  2. "Reverse step 一步把所有 mask 都填回去就好了。" 反例:你训了一个 MDLM,推理时输入全 mask 序列,一次 forward 就 argmax 出所有 token——结果是一堆高频词的语法填充("the the and and is is"),因为模型对每个位置都独立采 argmax,相邻 token 之间没有联合一致性。实际 MDLM 推理通常分 32–256 步,每步只 commit 置信度 top-$k$ 个位置(通常 $k = \lceil L/N_\text{steps} \rceil$,$L$ 是序列长度),其余继续以 mask 形式重新送入网络,让已知 context 越来越多,后面的位置在"看到前面预测的上下文"的条件下做更靠谱的预测。一步生成等价于退化成 BERT,质量崩盘。
  3. "loss 要对所有位置算 cross-entropy,否则模型不知道完整的语言分布。" 反例:你写错代码,对全部 6 个位置都算 loss,包括位置 0/2/3/5 的 "The/sat/on/mat" 这些未被 mask的位置。结果:模型 80%+ 梯度信号花在"输入 cat 输出 cat"这种 identity mapping 任务上,真正需要学的 mask 位置反而被淹没;训练 loss 看起来在下降(因为 identity 部分太容易),但 PPL 完全不收敛。正确做法是 loss 仅在 $M_t$ 上累加,再按 $1/\bar\beta_t$ 加权得到 ELBO(MDLM 论文 Eq. 7)。这是实现 MD 最常见的、且最难一眼看出的 bug。

相关概念

来源

  1. cmu-10799-diffusion-flow · L12 · Discrete Diffusion & Masked Diffusion · (00:02) — 概览,承接前面图像扩散,引出离散数据动机
  2. cmu-10799-diffusion-flow · L12 · 三视角到离散的映射问题 · (05:26) — denoising / score / flow 三视角各自如何离散化
  3. cmu-10799-diffusion-flow · L12 · 从 naive 嵌入空间扩散到 absorbing · (06:19) — 为什么 nearest-neighbor 思路不 work,引出真正的离散扩散
  4. diffusion-lm-vizuara · L12 · Diffusion LLM Noising Schedule · (00:03) — Vizuara 的 noising schedule 设计
  5. diffusion-lm-vizuara · L13 · Diffusion LLM Training Pipeline · (00:04) — 训练管线与 noise prediction 的 LM 类比
  6. diffusion-lm-vizuara · L16 · Coded from Scratch Part 1 · (00:02) — 从零实现 MD-LM,H100 训练
  7. diffusion-lm-vizuara · L17 · Coded from Scratch Part 2 · (00:03) — 200K iter 后的采样可视化
查看 159 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 00: 02 All right. All right. Right. So, so far what we have learn we Right. So, so far what we have learn we Right. So, so far what we have learn we have learned about like pretty much have learned about lik…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 00: 56 to do distillation or uh you know self to do distillation or uh you know self dissolations or flow maps or consistency dissolations or flow maps or consistency dissolations or flow maps or consistency…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 01: 45 who, actually, I'll just name who, like the people from Anthropic have claimed the people from Anthropic have claimed the people from Anthropic have claimed to me that uh I I believe it's ICML 2024 to…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 02: 37 uh I mean obviously you know we have other important modalities right for other important modalities right for other important modalities right for example like molecules or like DNA example like mole…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 03: 35 >> Near space and do like a nearest neighbors type thing to match it. neighbors type thing to match it. neighbors type thing to match it. >> Uh so diffusion model on a continuous >> Uh so diffusion mo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 04: 44 then embedding is what goes through >> so you first take the text and go to >> so you first take the text and go to >> so you first take the text and go to embedding embedding existing embedding embed…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 05: 26 running out of the allergyfree options. Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy sensitive option. Okay. Anyway, you're sensitive option…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 06: 19 And then uh for flow matching is pretty And then uh for flow matching is pretty much the same thing, but instead of much the same thing, but instead of much the same thing, but instead of learning the…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 07: 03 adding noise process is the forward process and the gradually d noiseis process and the gradually d noiseis process and the gradually d noiseis process is the reverse process. Right. process is the re…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 08: 09 version of a tweet from Andre Kapathi uh so let's identify some of the noise in so let's identify some of the noise in so let's identify some of the noise in this tweet what do we see this tweet what …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 09: 15 then you get some weird emojis and you then you get some weird emojis and you get some like out of place like six, get some like out of place like six, get some like out of place like six, seven like …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 10: 13 >> Yeah. Okay. >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh do you have allergy? Okay. This is also do you have a…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 11: 23 from the like original uh clean data to some random tokens right in the some random tokens right in the some random tokens right in the vocabulary. So that basically say like vocabulary. So that basic…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 12: 23 vector. Uh then basically say we have a vector. Uh then basically say we have a beta chance. This is just a some chance beta chance. This is just a some chance beta chance. This is just a some chance …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 13: 09 is why this is like one minus you know is why this is like one minus you know every everything else has one beta over every everything else has one beta over every everything else has one beta over th…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 14: 08 sentences with real vocabulary, it's not sentences with real vocabulary, it's not necessary that the noise sentence is necessary that the noise sentence is necessary that the noise sentence is actuall…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 15: 10 then you know so you just so the first then you know so you just so the first row is uh I second row is love uh third row is uh I second row is love uh third row is uh I second row is love uh third ro…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 16: 21 that by induction we can not only get like basically it's the same thing as like basically it's the same thing as like basically it's the same thing as like you know two gausians added like you know t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 17: 11 this is like literally how to jump this is like literally how to jump directly from x0 to xt is you just like directly from x0 to xt is you just like directly from x0 to xt is you just like kind of li…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 18: 24 as the beta t in uh DDPM. as the beta t in uh DDPM. >> Yeah. Don't worry. Everything's gonna be Don't worry. Everything's gonna be simplified later on in this class. right simplified later on in this …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 19: 40 >> Estimate these points and then Q >> Estimate these points and then Q transpose. transpose. transpose. >> Yes, pretty much. And then specifically, >> Yes, pretty much. And then specifically, >> Yes,…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 20: 37 because you're getting you're marovian like everything is marovian here. So you like everything is marovian here. So you like everything is marovian here. So you can ignore the x0 here. So then this c…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 21: 35 categorical distribution that is defined categorical distribution that is defined by this black font term here. >> Oh dot is like elementwise >> Oh dot is like elementwise multiplication. Anyway, hone…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 22: 42 difficult to define in this case. Right? What is the other thing? What is the other thing? What is the other thing? >> Tokens. >> Tokens. >> Tokens. >> The what >> The what >> The what >> the probabil…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 23: 30 prediction right so literally you need to only you only need to predict the to only you only need to predict the to only you only need to predict the logist of the final clean output given logist of t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 24: 18 a D3PM. All right, any questions about D3PM? Yes, Yes, >> this won't have like an equivalence to >> this won't have like an equivalence to >> this won't have like an equivalence to like score matching…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 25: 17 like discrete diffusion models >> great question and the answer is Yes. >> great question and the answer is Yes. >> great question and the answer is Yes. And we're going to talk about it next And we'r…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 26: 33 fixed length. >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is fixed length. Yeah. fixed length. Yeah. fixed length. Yeah. >> …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 27: 24 Alrighty. Okay. So h just a reminder what is a uh what is a continuous score what is a uh what is a continuous score what is a uh what is a continuous score and why is it weird to think about score an…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 28: 10 no continuous flow anymore. There's no gradient like you cannot like really do gradient like you cannot like really do gradient like you cannot like really do any gradient stuff anymore here, right? a…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 28: 56 type of thing. Uh which is not a healthy thing to do but like you know just for thing to do but like you know just for thing to do but like you know just for the sake of the class anyway um you know t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 29: 41 basically what people does is that they define something called uh the concrete define something called uh the concrete define something called uh the concrete score where it's literally just the scor…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 30: 49 apparently uh if you read this particular paper uh if you read this particular paper uh if you read this particular paper they have some connections to something they have some connections to somethin…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 31: 48 people actually call it a rate matrix. people actually call it a rate matrix. So it's a instantaneous rate to jump So it's a instantaneous rate to jump So it's a instantaneous rate to jump like just j…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 32: 43 thing, then you're actually basically if thing, then you're actually basically if you do the math then then then you're you do the math then then then you're you do the math then then then you're actu…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 33: 27 everything well you can represent everything in a well you can represent everything in a well you can represent everything in a fully connected graph and then assign fully connected graph and then ass…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 34: 22 So if you want your chain to satisfy detail balance you're going to get this detail balance you're going to get this detail balance you're going to get this thing so that your chain is reversible, thi…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 35: 19 I see everyone's confused by how this whole thing but we need basically we whole thing but we need basically we whole thing but we need basically we need a neuronet network to predict this need a neur…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 36: 48 score versus your uh you know your your score versus your uh you know your your your your your real score. And you your your your real score. And you your your your real score. And you literally just …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 37: 47 non-zero number divided by a non-zero number or like or sorry non- negative number or like or sorry non- negative number or like or sorry non- negative number divided by a non- negative number number …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 38: 35 good. Exactly. That's exactly correct. good. Exactly. That's exactly correct. Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you yo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 39: 20 like a like a like a like a like a what we call uh constant so let's just ignore we call uh constant so let's just ignore we call uh constant so let's just ignore that for now so we basically we can t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 40: 06 then the person is going to be this thing, right? So what you do is you thing, right? So what you do is you thing, right? So what you do is you essentially so K is like you you jump to essentially so …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 40: 54 then this part becomes one uh s minus r uh times the sum of all the possible all uh times the sum of all the possible all uh times the sum of all the possible all the probabilities and then this one i…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 41: 37 us something that is that has that has exponential in it and also literally exponential in it and also literally exponential in it and also literally right the the the easiest right the the the easies…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 42: 20 right? So you the the S minus the right? So you the the S minus the likelihood ratio times the log of the likelihood ratio times the log of the likelihood ratio times the log of the your S. Um so beca…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 43: 12 calculate the ground truth by literally the same trick the den noising score the same trick the den noising score the same trick the den noising score matching. This is just the noising score matching…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 44: 40 the reverse rate, right? So, you just replace this concrete score with the replace this concrete score with the replace this concrete score with the concrete score that you trained. And concrete score…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 45: 39 >> earlier how are we choosing why do we just randomly sample any token choices just randomly sample any token choices just randomly sample any token choices >> yeah based on >> yeah based on >> yeah …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 46: 38 basically like how about we can only do masking and unmasking, right? So in the masking and unmasking, right? So in the masking and unmasking, right? So in the forward process all you can do is you fo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 47: 20 like basically the only two columns that you're going to have non- negative value you're going to have non- negative value you're going to have non- negative value is the columns on the diagonal and t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 48: 06 >> But if we go all the way end of the day everything will be masked. everything will be masked. everything will be masked. >> Yeah. And it's still a distribution >> Yeah. And it's still a distributio…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 48: 55 multiplication and stuff anymore. Uh because we simplified everything, right? because we simplified everything, right? because we simplified everything, right? So basically what what we're essentially…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 49: 51 stay unmasked and then with one minus alpha t uh probability you're going to alpha t uh probability you're going to alpha t uh probability you're going to stay masked and uh basically if we if stay ma…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 51: 08 when uh xt equals x0 and x t minus one also equals x zero um like basically if also equals x zero um like basically if also equals x zero um like basically if if you have not been masked uh at time t …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 52: 02 simplified into this really really simple uh ratio. And similarly simple uh ratio. And similarly simple uh ratio. And similarly because the only other option for you is because the only other option f…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 53: 11 over. You should you should have the same value because it hasn't been same value because it hasn't been same value because it hasn't been masked. And if XT is masked, then you masked. And if XT is ma…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 54: 23 times the clean data estimation. times the clean data estimation. That's it. >> We have that problem because we'll start >> We have that problem because we'll start with full mask. So if you start wit…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 55: 26 >> So you're always starting with the full mask. mask. mask. >> That's right. >> That's right. >> That's right. >> So if that one reverse gets you to I >> So if that one reverse gets you to I >> So if…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 56: 12 actually a good point. Basically, uh the the the mercury um founder, Inception the the mercury um founder, Inception the the mercury um founder, Inception Lab founder, Stephano, who is the one of Lab …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 57: 09 you can also do like continuous time training with this. You literally just training with this. You literally just training with this. You literally just take the continuous time limit of the of take …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 57: 58 >> uh you yes kind of so it's the same thing it's uh like analogy to you choose thing it's uh like analogy to you choose thing it's uh like analogy to you choose the number of sampling steps right in …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 58: 42 this is purely depending on like the the this is purely depending on like the the model that you got trained but usually model that you got trained but usually model that you got trained but usually i…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 59: 34 basically the mask ratio is depending on this alpha t thing which you set. So you this alpha t thing which you set. So you this alpha t thing which you set. So you you can you can choose uh but like y…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 60: 34 the tokens. the tokens. >> Yeah. >> Yeah. >> But then in the last you have something >> But then in the last you have something >> But then in the last you have something like XT equal to M. So like X…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 61: 16 >> uh what do you mean by latent text well like so you mean like like so you mean like like so you mean like a region space for images and then use a region space for images and then use a region spac…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 62: 02 >> Does it produce something coherent if >> Does it produce something coherent if all the things are independently getting all the things are independently getting all the things are independently get…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 62: 49 >> It is pretty much just birth literally if you think about it. Yeah. if you think about it. Yeah. if you think about it. Yeah. >> Well, between like >> Well, between like >> Well, between like >> yo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 63: 40 when they when he said like oh we can correct hallucination but like here we correct hallucination but like here we correct hallucination but like here we cannot really it's the same as cannot really …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 64: 24 like a sort of like a like a like a like a interesting moment in science. But if a interesting moment in science. But if a interesting moment in science. But if you're one of the authors, you may get,…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 65: 08 >> very low because it's 2024. Okay. Anyway um but since like you know people show um but since like you know people show um but since like you know people show saw the potential of this they have saw…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 65: 55 say and seems like super compatible uh say and seems like super compatible uh to comparable to like large language to comparable to like large language to comparable to like large language models like…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 66: 39 Please come up and uh can get rich as well. All right, see you guys next well. All right, see you guys next well. All right, see you guys next class.
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 00: 03 So what we aim to do is as follows. We So what we aim to do is as follows. We have a true probability distribution, have a true probability distribution, have a true probability distribution, right? S…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 00: 57 an image using a noising schedule maybe an image using a noising schedule maybe a goshian schedule. The predicting noise a goshian schedule. The predicting noise a goshian schedule. The predicting noi…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 01: 41 very crucial role. Right? So let's see how noising is included. how noising is included. how noising is included. What is the noising process in the text? What is the noising process in the text? What…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 02: 32 that noise and then when we start with that noise and then when we start with noise, we can predict the original text noise, we can predict the original text noise, we can predict the original text ju…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 03: 32 embedding. So this is a 384 dimensional embedding. So this is a 384 dimensional vector. vector. vector. This is a 384 dimensional vector. This is a 384 dimensional vector. This is a 384 dimensional ve…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 04: 44 this is the true sentences probability distribution. sentences probability distribution. sentences probability distribution. This and this sentence the next day is This and this sentence the next day …
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 05: 25 a portion of this, this is no longer a a portion of this, this is no longer a true sentence. It becomes a cor, it true sentence. It becomes a cor, it true sentence. It becomes a cor, it becomes a corr…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 06: 33 schedule. Right? So first I define a time step. Let's say I go from time time step. Let's say I go from time time step. Let's say I go from time equal to 1 till time equal to 6. equal to 1 till time e…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 07: 42 going haywire. So maybe initially when only one mask is added, I go to a lower only one mask is added, I go to a lower only one mask is added, I go to a lower probability. When two masks are added, I …
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 08: 56 way it happens in text is that uh at time equal to 1, you almost have an time equal to 1, you almost have an time equal to 1, you almost have an entire clean text with maybe some mask entire clean tex…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 09: 40 it's a Bernoli distribution. So the probability of masking is way higher. probability of masking is way higher. probability of masking is way higher. Probability of masking all tokens is way Probabili…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 10: 44 3,000. Right? Now these are masked. So this is masked and this is masked. So this is masked and this is masked. So this is masked and this is masked. So this will be replaced with 1 0 0 0 1 this will …
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 11: 35 a noisy input with mask tokens. Okay. a noisy input with mask tokens. Okay. Everything else remains exactly the Everything else remains exactly the Everything else remains exactly the same. Now we'll …
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 12: 23 architecture if you compare them side by architecture if you compare them side by side is the input which goes into this side is the input which goes into this side is the input which goes into this i…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 13: 03 targets then we sorry we get the predictions then we sorry we get the predictions then we sorry we get the predictions then we get the loss then we get the loss then we get the loss then we update the…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 13: 51 what is the noising process in the case of diffusion language models. The next of diffusion language models. The next of diffusion language models. The next step which we are going to see is that step…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 00: 04 Okay. So the next step in these key Okay. So the next step in these key characteristics which we mentioned and characteristics which we mentioned and characteristics which we mentioned and probably th…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 01: 01 You take a noisy image. You take a noisy image. Let's say this is a noisy image of a Let's say this is a noisy image of a Let's say this is a noisy image of a panda which is the example which we have …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 01: 44 function, right? We need to predict the function, right? We need to predict the noise and then we need to compare it noise and then we need to compare it noise and then we need to compare it with the …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 02: 44 How do I predict the true values here? So what I want to do what does it mean So what I want to do what does it mean So what I want to do what does it mean prediction of noise? I somehow want to predi…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 03: 41 at this position? And how do you get the loss function? loss function? loss function? Well, the answer is extremely simple, Well, the answer is extremely simple, Well, the answer is extremely simple, …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 04: 32 Right. So the next token which is predicted at this position I'll compare it with the true next token I'll compare it with the true next token which should be next. which should be next. which should …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 05: 25 predictions, right? For the next, these predictions, right? For the next, these are my predictions. are my predictions. are my predictions. These are my predictions. I'm just going These are my predic…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 06: 11 um second characteristics the model which predicts the noise which was a which predicts the noise which was a which predicts the noise which was a unit in the case of diffusion for unit in the case of…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 07: 14 are. Right? In the ARM architecture, we are. Right? In the ARM architecture, we remember we had the uh token embedding remember we had the uh token embedding remember we had the uh token embedding and…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 08: 06 we add the token embedding plus position embedding the one major difference in embedding the one major difference in embedding the one major difference in diffusion language models is that along diffu…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 08: 51 time embedding vector is added to all my tokens. So now the input embedding is tokens. So now the input embedding is tokens. So now the input embedding is not just the token embedding plus not just th…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 09: 35 change. So now if you see if you compare these block diagrams these block diagrams these block diagrams in the ARM architecture, we only had the in the ARM architecture, we only had the in the ARM arc…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 10: 25 exactly the same. The difference is in the attention mechanism. the attention mechanism. the attention mechanism. Now think about it right here. We no Now think about it right here. We no Now think ab…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 11: 08 means that when I'm calculating the attention so the mask the mask day the mask day mask bright right which means that if mask bright right which means that if mask bright right which means that if I'…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 12: 28 I essentially have this entire causal attention matrix which is available to attention matrix which is available to attention matrix which is available to me. I don't have to set these values to me. I…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 13: 15 major difference? The third major difference comes in the output token difference comes in the output token difference comes in the output token prediction, right? Uh let's say here we prediction, rig…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 13: 56 these losses. That's my loss. I don't have to find the loss for the I don't have to find the loss for the I don't have to find the loss for the other positions at all. other positions at all. other po…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 15: 04 So if this is let's say 0.1 this probability and this is2 my loss in this probability and this is2 my loss in this probability and this is2 my loss in this case will be - 1 by2 log of.1 log of.1 plus …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 16: 15 we have a time embedding now which was we have a time embedding now which was not there auto reggressive model in not there auto reggressive model in not there auto reggressive model in multi attentio…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 17: 02 on u the time step. More the time step, on u the time step. More the time step, more is the masking. Less the time step, more is the masking. Less the time step, more is the masking. Less the time ste…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 17: 43 transformer block. So I have my layer normalization multi head attention I normalization multi head attention I normalization multi head attention I have my dropout I have my skip have my dropout I ha…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 18: 31 batch size. We choose the time step, right? Let's say batch size is 128. We right? Let's say batch size is 128. We right? Let's say batch size is 128. We assemble an input sequence of assemble an inpu…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 19: 30 tokens. Based on this loss, we find the partial derivative of the loss with partial derivative of the loss with partial derivative of the loss with respect to all the parameters. And respect to all th…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 20: 27 case is the better we pre-train a language diffusion model the better we language diffusion model the better we language diffusion model the better we are able to discover what noise is added are able…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 21: 18 Start with all masks and then we uncover Start with all masks and then we uncover tokens correct tokens at that position. tokens correct tokens at that position. tokens correct tokens at that position…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 22: 02 predict is that the probability of actually recovering the probability of actually recovering the probability of actually recovering the true true distribution or the the true true distribution or the…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 22: 48 proved to be an upper bound on the proved to be an upper bound on the negative log likelihood making it a negative log likelihood making it a negative log likelihood making it a principled objective f…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 23: 34 the masked positions. Okay. So, until now we have seen the first two So, until now we have seen the first two So, until now we have seen the first two steps of diffusion based language steps of diffus…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 24: 22 the third difference. Noisy input here the third difference. Noisy input here that's the four difference. There are that's the four difference. There are that's the four difference. There are four maj…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 25: 09 embedding matrix and we construct a time embedding matrix also. And I show you embedding matrix also. And I show you embedding matrix also. And I show you how everything is done. So at each how everyt…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 25: 54 model can be summarized here. We have model can be summarized here. We have token ids convert them into token token ids convert them into token token ids convert them into token embedding vectors add …
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 00: 03 So here you can see the file which has So here you can see the file which has been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was getting som…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 00: 56 iterations you'll definitely see samples which are as co inference samples which which are as co inference samples which which are as co inference samples which are as coherent as what you are seeing …
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 01: 37 you can even chain different GPUs together to do distributed computing if together to do distributed computing if together to do distributed computing if possible possible possible so let's actually p…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 02: 36 before. Um yeah if you train for 200,000 iterations you'll see an amazing iterations you'll see an amazing iterations you'll see an amazing coherent language diffusion language coherent language diffu…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 03: 23 all the necessary things which you will need to learn about language diffusion need to learn about language diffusion need to learn about language diffusion models. All right. So yeah we have this All…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 04: 19 shown on the screen right now serves as shown on the screen right now serves as a reference for you to build multiple a reference for you to build multiple a reference for you to build multiple such s…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 05: 08 fraction of the cost which might be awesome for the future. But now you know awesome for the future. But now you know awesome for the future. But now you know how to build this fully yourself. Um, oka…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 00: 02 Okay. So now I'm going to take you Okay. So now I'm going to take you through the code of assembling this through the code of assembling this through the code of assembling this diffusion language mod…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 01: 00 here. The H100 GPU is much faster. If you don't have access to the Google If you don't have access to the Google If you don't have access to the Google Collab Pro, you can start with the T4 Collab Pro…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 01: 43 number of training steps is only 2,000. Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 which is if you have compute budget or w…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 02: 34 take a look at the whiteboard you'll see that whiteboard you'll see that whiteboard you'll see that this dimension which I keep talking this dimension which I keep talking this dimension which I keep …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 03: 20 essentially here in these examples we have a sequence length of four tokens, have a sequence length of four tokens, have a sequence length of four tokens, right? Whereas in actual code we have right? …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 04: 13 chained to each other. So these are the uh this variable is the number of uh this variable is the number of uh this variable is the number of layers. The number of heads is layers. The number of heads…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 05: 11 that's this uh four times the D model. that's this uh four times the D model. So DFF is the feed forward neural So DFF is the feed forward neural So DFF is the feed forward neural network hidden layer…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 06: 01 optimizer. Okay. So you run this initial configuration and then you move forward. configuration and then you move forward. configuration and then you move forward. You install a bunch of dependencies …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 06: 49 go to this uh hugging face repository and I load my data. I load my training and I load my data. I load my training and I load my data. I load my training data set. I load my validation data set data …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 07: 36 we I'm connected to an L4 GPU at the moment. See that's fine. So let's see. Um Um yeah so here you'll see that I'm yeah so here you'll see that I'm yeah so here you'll see that I'm connected to an L4 …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 08: 32 token, beginning of sequence, end of token, beginning of sequence, end of sequence and a mask token. This is very sequence and a mask token. This is very sequence and a mask token. This is very import…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 09: 20 with a tokenizer directly borrowed from with a tokenizer directly borrowed from tick token. I I I think the code should tick token. I I I think the code should tick token. I I I think the code should …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 10: 21 because in the noising process we are because in the noising process we are going to replace the tokens with masks going to replace the tokens with masks going to replace the tokens with masks and the…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 11: 13 becomes clearer to all of you. Yeah. And if you see that you first define a And if you see that you first define a And if you see that you first define a token embedding, position embedding and token …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 12: 20 a transformer um we are going to do a transformer um we are going to do a transformer um we are going to do a transformer encoder layer over here and uh let's see and attention mask equal to none. Thi…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 13: 11 embedding. We add the time embedding to it. The resultant input embedding we it. The resultant input embedding we it. The resultant input embedding we pass through the transformer block. Then pass thr…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 13: 59 at the time embedding matrix at the time embedding matrix if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need to know at what tim…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 14: 47 to based on the time step we have to add to based on the time step we have to add the mask tokens. This is what is done the mask tokens. This is what is done the mask tokens. This is what is done here…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 15: 29 done because we have already seen this on the whiteboard right to calculate the on the whiteboard right to calculate the on the whiteboard right to calculate the loss I only need to look at the mask l…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 16: 13 Well, it's included over here where we Well, it's included over here where we pass an attention mask equal to none. pass an attention mask equal to none. pass an attention mask equal to none. These ar…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 16: 56 So this is the cross entropy loss. So you can run this right now. Once we have you can run this right now. Once we have you can run this right now. Once we have this defined, the last step is just to …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 17: 35 time step right so when you go to the diffusion loss function you'll see that diffusion loss function you'll see that diffusion loss function you'll see that uh where is the diffusion loss yeah when u…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 18: 19 Now this thing which I'm telling right now the corrupted input sequence is now the corrupted input sequence is now the corrupted input sequence is passed through the model is exactly passed through th…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 19: 11 Yeah, see the training process has started. Now I'm on the L I think let's started. Now I'm on the L I think let's started. Now I'm on the L I think let's see which GPU I'm on. I'm on the L4 GPU see w…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 19: 57 in u the convergence speed so if you see the auto reggressive model in 15,000 the auto reggressive model in 15,000 the auto reggressive model in 15,000 epochs I go from a loss of 8 to a loss epochs I …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 20: 44 can see the training has started over here and we can print the loss we can here and we can print the loss we can here and we can print the loss we can print the learning rate etc. While the print the…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 21: 34 here. I am not going to go through this here. I am not going to go through this code in detail because it just code in detail because it just code in detail because it just implements the same thing, …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 22: 19 Uh so this is run right now and uh here Uh so this is run right now and uh here I'm just seeing a sample inference. Once I'm just seeing a sample inference. Once I'm just seeing a sample inference. On…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 23: 27 iterations did not finish. But if you actually run um actually run um actually run um 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iteratio…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 24: 09 Lily's mom asked her to help clean up Lily's mom asked her to help clean up her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom p…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 24: 50 we can also do this on runpod. So if you have credentials or if you have some have credentials or if you have some have credentials or if you have some budget even $6 to $7 what I'll suggest budget ev…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 25: 45 I'll set overrides. Oops. I think it was I'll set overrides. Oops. I think it was 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set overrides over here. A…