概念

Discrete Diffusion

— 让 token 在有限词表里"跳",不是在向量空间里"飘"。

很多初学者听到 "discrete diffusion" 就会想:是不是把 DDPM 的高斯噪声换成"小一点的噪声",再用 round() 离散化就行?不是。CMU 10-799 Lecture 12 讲师明确指出:离散扩散里根本没有"加噪声"这个连续操作,只有"以某个概率跳到别的 token"。词不是向量,"the cat" 和 "a cat" 之间不存在"中间一半的词",你也没办法在 token 上做加法。这一点是和连续扩散最根本的差异——连续扩散里 $x_t = x_{t-1} + \sigma_t \epsilon$ 是有意义的;离散扩散里这个公式没意义,你只能问"下一步落到哪个 token 的概率有多大"。

「Nearly add noise kind of in discrete setting. Do you have like a fixed sequence of jumping from word to word? — Yeah. So this Q_t thing is like a transition mixtures that represent a jump essentially.」—— CMU 10-799 L13 @06:09

CMU 讲师的核心比喻是 "jump 跳跃":每一步前向,token 都按照一个转移矩阵 $Q_t$ 决定"留在原位 / 跳到 MASK / 跳到随机其他 token"。把这件事写成连续时间,就是 CTMC(continuous-time Markov chain,连续时间马尔可夫链);把它截成离散 $T$ 步,就是 Austin 等 (2021) 提出的 D3PM 框架。讲师还把 $Q_t$ 类比为 "velocity 速度场"——它扮演的角色和连续扩散里 flow matching 的 $v_\theta(x, t)$ 一样,只不过"速度"现在不是一个向量,而是一张"我此刻从哪个 token 跳到哪个 token 的概率表"。这把离散扩散、SEDD、discrete flow matching 三件事缝到了同一个数学框架下。

为什么要费这个劲做离散扩散?因为语言、代码、DNA、化学式天然是离散的。把它们硬编码成连续向量(如 word embedding),再做 DDPM,会丢掉"词的身份"这一约束——你训出来的中间结果可能根本不是任何合法 token,需要后处理 nearest-neighbor 投影才能落回词表,这一步会引入难以分析的误差。离散扩散从一开始就让噪声"住在"合法 token 上:前向链每一时刻的样本都是真实词表里的某个 token,反向链每一步采样出的也是合法 token。整个训练和推理过程都在离散空间里完成,理论分析(ELBO、收敛速度)也都用 Markov 链工具。

在训练里,这件事对应一个比 DDPM 更接近 BERT 的目标:网络读入"一句被加噪的句子"——某些 token 被替换成了 MASK 或其它随机 token——然后预测"原始 token 的 categorical 分布"。和 BERT 不同的是,MASK 比例不再固定(BERT 是 15%),而是由扩散时间 $t$ 决定:$t$ 接近 0 时几乎不 mask,$t$ 接近 $T$ 时几乎全 mask。最后用 ELBO 把所有时间步的预测损失加权求和。

前向链:一行公式定义一切

设词表大小 $K$(GPT-2 大约 $K = 50{,}257$;DNA 则 $K = 4$;蛋白质 $K = 20$)。把每个 token 写成 one-hot 向量 $x \in \{0,1\}^K$。第 $t$ 步的前向跳变完全由一个 $K \times K$ 转移矩阵 $Q_t$ 决定:

$$q(x_t \mid x_{t-1}) = \operatorname{Cat}\!\bigl(x_t \,\big|\, Q_t \, x_{t-1}\bigr).$$

读法:$Q_t$ 的第 $j$ 列是"从 token $j$ 出发,下一步落到各 token 上的概率向量"。在词表是 $\{a, b, c\}$ 的玩具例子里,$(Q_t)_{2,1} = 0.3$ 意思是"如果当前 token 是 $a$,下一步以 0.3 的概率变成 $b$"。

CMU 讲师强调一个易被忽视的守恒律:每一列加起来必须 = 1(mass conservation,概率守恒)。直觉上很显然——从任意一个状态出发,下一步"落到某处"的总概率必须是 1;它不能跑出去也不能凭空多出来。在连续时间版(CTMC)里,把 $Q_t = I + \Delta t \cdot R_t$ 改写后,对应的速率矩阵 $R_t$ 每一列加起来必须 = 0——"流出 = 流入"。这两个守恒律是讲师在 L13 反复强调的"两件事"之一,另一件是 Kolmogorov 连续性方程(continuity equation)。

「The amount of flow flowing out of a certain state should equal to the amount of flow that's flowing in. This is literally just conservation of mass.」—— CMU 10-799 L13 @05:13

讲师在 L13 用"水流"打比方解释这两个守恒律:想象 $K$ 个水桶(每个 token 一个),每一时刻有水从桶之间流来流去,但所有桶里的水加起来必须恒等于 1(这就是 mass conservation);某个桶的水位变化率等于"流入的水流减去流出的水流",这就是 Kolmogorov forward equation(也叫 Chapman-Kolmogorov 方程)。讲师原话:"the infinitesimal change in probability equals the amount of probability that comes into this particular state minus the probability that goes out of this state."(L13 @08:58)这正是连续扩散里 Fokker-Planck 方程的离散版本。

三种 Q_t:D3PM 给出的设计空间

类型$Q_t$ 结构$T \to \infty$ 时的终态典型用途
Uniform(均匀替换)以 $\beta_t$ 概率均匀跳到任意 token;以 $1-\beta_t$ 留下词表上的均匀分布D3PM 早期实验
Absorbing(吸收态 / MASK)以 $\beta_t$ 概率跳到 [MASK];MASK 是 absorbing state(不会再跳出)整段全是 [MASK]BERT-style;MDLM、MaskGIT 的根
Gaussian-like(嵌入式)把 token 嵌入连续空间 → 扩散 → 投影回最近 token连续高斯处理"近义词更易混"的场景,如像素值 0–255

讲师特别点出:absorbing 是最容易实现、效果也最好的一种。它把"加噪"简化为"以概率 $\bar\beta_t$ 把每个位置独立替换成 MASK",这正是 masked language modeling 的扩散化版本。讲师原话:"the easiest way to do discrete diffusion is mask diffusion — you always try to interpolate between your clean data and the full mask."(L13 @01:39)端点很清晰:$t = 0$ 时是干净文本,$t = T$ 时整段都是 MASK——这是一个标准的"两端皆已知"的插值问题。uniform 设定下,端点是"词表上的均匀分布",反向时网络要从"完全随机的句子"还原;而 absorbing 设定下,反向时网络面对的是"已经填了一部分、剩下打了 MASK 的句子",问题更接近 BERT,更容易学。

三种 $Q_t$ 还对应了三种生成阶段。Absorbing 给了一个干净的"已知 vs 未知"二分——已经填好的 token 不动,剩下的 MASK 一步步被填上;这种"单调减"的特性让推理时可以很自然地搭配 confidence-based parallel decoding(MaskGIT 的思路)。Uniform 没有这种二分——每个 token 在每一步都可能被改写,网络要不断和自己的旧预测博弈,训练上更难。Gaussian-like 因为保留了"邻近性",特别适合像素值或分子键能这种"近的状态更像"的离散数据。

闭式:跳过中间步直接到 $x_t$

因为 Markov 链可以做矩阵连乘,定义累积转移:

$$\bar{Q}_t \;=\; Q_t \, Q_{t-1} \cdots Q_1, \qquad q(x_t \mid x_0) = \operatorname{Cat}\!\bigl(x_t \mid \bar{Q}_t \, x_0\bigr).$$

Why:这一步对应连续扩散里的"从 $x_0$ 一步到 $x_t$"闭式。没有它,训练时每个样本都要从 $t=0$ 一步步跑到 $t$,开销 $O(t)$;有了它,给定 $x_0$ 和随机抽到的 $t$,直接采样一次就行,开销 $O(1)$。这是 DDPM 在连续域 $\bar\alpha_t$ 闭式($x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$)的离散对应物。在 absorbing 情形下,闭式更简单:$\bar{Q}_t$ 等价于"以概率 $1 - \prod_{s \leq t}(1 - \beta_s)$ 把每个位置独立替换为 MASK"——你只需要知道一个累积保留概率 $\bar\alpha_t$,就能一步采到任意 $t$ 时刻。这种"逐位置独立"的 factorization 是后续 MDLM 简化训练损失的关键。

反向:神经网络预测 categorical

反向过程也是 Markov 链,但每一步的转移概率由神经网络给出:

$$p_\theta(x_{t-1} \mid x_t) = \operatorname{Cat}\!\bigl(x_{t-1} \mid f_\theta(x_t, t)\bigr).$$

注意:你不预测 $\epsilon$(离散世界没有可加的噪声向量),而是直接预测下一时刻 token 的概率向量。网络输出是一个长度 $K$ 的 logits,经过 softmax 自动满足列和 = 1。在 absorbing 情况下,更高效的做法是预测 $\hat{x}_0$——给定当前被 MASK 掉一部分的 $x_t$,网络猜原始干净 token;然后用 Bayes 解析地反推 $p_\theta(x_{t-1}|x_t) = q(x_{t-1}|x_t, \hat{x}_0)$。这个 reparameterization 把"预测 $K$ 个 token 的全概率"换成了"预测 $K$ 个 token 中哪个是原始的",更接近分类任务,训起来比直接预测 transition 稳得多。

训练目标用同一个 ELBO,逐项替换为 categorical KL(两个 categorical 之间的 KL 有闭式 $\sum_k q_k \log(q_k / p_k)$,不需要 reparameterization trick):

$$\mathcal{L} = \mathbb{E}\!\left[\sum_{t=2}^{T} D_{\mathrm{KL}}\!\bigl(q(x_{t-1}|x_t, x_0)\,\|\,p_\theta(x_{t-1}|x_t)\bigr)\right] - \log p_\theta(x_0|x_1).$$

当 $Q_t$ 是 absorbing 时,这个 ELBO 可以进一步化简为一个 "加权交叉熵"——只在"当前是 MASK"的位置算损失,权重和 $t$ 的位置有关。这是 MDLM (Sahoo et al. 2024) 的核心发现:复杂的离散扩散 ELBO 在 absorbing 设定下退化为一个像 BERT 那样直白的目标,于是工程上"训扩散模型"变得和"训 MLM"一样简单。

「We can train it using the same ELBO loss from DDPM essentially. We also talked about how to define the discrete score (concrete score), and learn it using a more stable loss called score entropy.」—— CMU 10-799 L13 @00:48

SVG:三种 $Q_t$ 的轨迹示意

下图把同一句话(or 同一个像素序列)在三种 $Q_t$ 下的前向轨迹画在一起,从左到右是时间。注意三条路径在终点的分布完全不同——这是后续选哪一种 $Q_t$ 的依据。

三条前向轨迹(同一句话,不同 $Q_t$) Uniform the cat sat → the dog sat → XYZ Absorbing the cat sat → the [M] sat → [M] 终点:每个位置都是 [M]——MASK 是吸收态,进去就出不来。 Gaussian-like 128 130 132 → 127 131 130 → 64 附近的整数比远处更易混淆——保留"顺序"信息(适合像素值)。 不变量:每一行的"跳"都按 $Q_t$ 给的概率发生,列和 = 1。

组合爆炸:$K = 50{,}000$ 怎么办?

这是讲师在 L13 @07:53 学生提问时的关键时刻——一位同学问:"This is represented by a matrix for state transitions. It already seems like a very big matrix."(这个用矩阵表示状态转移,看起来已经是个很大的矩阵了。)讲师笑着回了一句 "It is. It is."(确实是。)然后给出了离散扩散最重要的工程妥协。

问题:$Q_t \in \mathbb{R}^{K \times K}$ 对 GPT-2 词表就是 $50{,}257^2 \approx 2.5 \times 10^9$ 个数;如果是 LLaMA 词表 $K = 128{,}000$,这个数字飙到 $1.6 \times 10^{10}$。每一步都存这么大矩阵,显存爆炸;更糟糕的是,神经网络要预测的对象(rate 矩阵)大小是 logit 维度的二次方——讲师原话:"the rate matrix is quadratics of the logit dimension"。解法:永远不显式构造 $Q_t$,而是用结构化的 $Q_t$,让矩阵-向量乘 $Q_t x$ 在 $O(K)$ 甚至 $O(1)$ 而非 $O(K^2)$ 完成。

  • Uniform 情形:$Q_t = (1-\beta_t) I + \frac{\beta_t}{K} \mathbf{1}\mathbf{1}^\top$——只需要存一个标量 $\beta_t$,矩阵-向量乘只需 $O(K)$。
  • Absorbing 情形:$Q_t = (1-\beta_t) I + \beta_t \, e_{\text{MASK}} \mathbf{1}^\top$——只需要标量 $\beta_t$ 和一个特殊 token 索引。前向采样直接退化为"对每个位置抛一次硬币:留下或换 MASK",无任何矩阵运算。
  • Gaussian-like 情形:$Q_t$ 是一个 band 矩阵(只有主对角线和邻近几条副对角线非零),每行存稀疏几项即可;和"卷积"在 $O(K)$ 实现是同构的。
  • SEDD(Lou et al. 2024):直接预测"concrete score" $s_\theta(x_t, y) \approx \frac{q_t(y)}{q_t(x_t)}$ 的 logits,避开了显式 rate 矩阵。讲师原话:"This is also why we're trying to do the reparameterization trick."(L13 @08:24)这种 score-based formulation 让训练损失变成 score entropy——比直接学 rate 矩阵的 KL 数值上更稳定。
工程细节:网络通常预测 logits 而不是直接预测 rate 矩阵。原因有两个——一是参数量从 $O(K^2)$ 降到 $O(K)$;二是 logits 用 softmax 自动满足"列和 = 1"约束,rate 矩阵则要手动投影才能保证守恒律。这就是为什么所有现代离散扩散语言模型(D3PM、SEDD、MDLM)的网络输出形状都是 $[\text{batch}, \text{seq\_len}, K]$,和一个标准 Transformer 完全一致——你拿一个预训练的 BERT 改改 loss 就能开始训练。

三种范式如何统一

讲师在 L12 → L13 的衔接里完成了一个重要梳理:离散扩散不是一个单一算法,而是三种等价表述。

  1. Categorical noise + denoiser:对应 DDPM 在离散上的版本(D3PM)。前向用 $Q_t$ 加噪,反向网络预测下一时刻的 categorical 分布或者 $\hat{x}_0$;损失是 ELBO 上的 KL 求和。
  2. Score matching 形式:对应 SEDD(score-entropy discrete diffusion,Lou et al. 2024)。把"learn $p_\theta$"换成"learn concrete score"——比值 $q_t(y) / q_t(x_t)$ 的对数,类似连续扩散里 $\nabla \log p_t(x)$ 的角色。讲师说:"score entropy loss is a more stable loss function."(L13 @01:23)
  3. Flow matching 形式:CTMC + Kolmogorov 方程,预测"概率路径"(probability path)上的速度场。讲师在课堂上让学生回答这个对应物:"What is the analog to velocity in flow matching in our discrete setting? — Probability part."(L13 @02:23)"probability path" 就是从干净数据流向纯噪声的一族 $p_t$。

三者都从同一个连续时间 Markov 链出发,只是损失函数不同——和连续扩散里 "DDPM = score matching = flow matching 都等价" 的图景对应。讲师在 L13 @08:24 总结了为什么会出现三种范式:在大词表上直接学 rate 矩阵不稳定且参数量爆炸,于是大家用不同的 reparameterization——预测 categorical 概率、预测 concrete score、或者预测 probability flow——结果在数学上等价但在工程上区别巨大。

预告:MDLM 与 SEDD 是这套框架的两个高效实例

读到这里你已经掌握了离散扩散的全部数学骨架:$Q_t$、累积矩阵 $\bar{Q}_t$、ELBO、守恒律、结构化降阶。剩下两件事是把这套骨架"工程化"成可训练的语言模型。

  • MDLM(Masked Diffusion Language Model):选 $Q_t$ 为 absorbing,证明 ELBO 退化为加权 cross-entropy;训练几乎和 BERT 一样,推理时按时间步逐步揭开 MASK。它最像"BERT 的连续时间外推"。
  • SEDD(Score-Entropy Discrete Diffusion):选择 score-matching 形式,预测 concrete score $s_\theta$;用 score entropy 损失替代 KL,数值上更稳。它最像"score-based generative model 的离散版"。

这两条路在 2024–2025 年都已经能在 LM1B、WikiText 等基准上接近或追上自回归 GPT,且支持任意顺序填空、并行解码、可控生成——这是离散扩散从理论走向"被严肃考虑替代 AR LM"的关键节点。下一节我们会单独讲 MDLM 和 SEDD 的细节,重点是它们如何把这一节抽象的 ELBO 落地成几行 PyTorch。

🟡 教材外补充:当 $Q_t$ 是 absorbing 时,前向就是 "随时间增大 MASK 比例",反向就是 "并行预测被 MASK 位置的 token"——这正是 MDLM (Sahoo et al. 2024) 和 MaskGIT 在做的事。换句话说,BERT 的 MLM 训练目标可以看作 absorbing 离散扩散 ELBO 的一个特例(取 $T = 1$ 且固定 mask 比例 15%)。MDLM 的贡献是证明 simplified ELBO 在 absorbing 设定下等价于 weighted cross-entropy,让训练像 BERT 一样简单;而推理时仍然保留了多步 refinement 的能力。此外,SEDD 把同样的物理过程用 concrete score 表述,参数共享、稳定性更好;2024–2025 年它和 MDLM 并列成为开源社区里最常被复现的离散扩散语言模型骨架。GenGPT、LLaDA 等纯扩散语言模型都是在这条线上长出来的。

点击按钮,按顺序展开 ELBO 在离散扩散中的推导链。不变量:每一步 $q(x_t|x_{t-1})$ 都是合法的 categorical 分布——列和恒等于 1。

$q(x_t \mid x_{t-1}) = \operatorname{Cat}(x_t \mid Q_t \, x_{t-1})$
$Q_t \in \mathbb{R}^{K \times K}$;列和 = 1。Token 按概率"跳"。
$\bar{Q}_t = Q_t Q_{t-1} \cdots Q_1$
$q(x_t \mid x_0) = \operatorname{Cat}(x_t \mid \bar{Q}_t \, x_0)$
不再需要逐步采样——一次到位(对应 DDPM 的 $\bar\alpha_t$ 闭式)。
$p_\theta(x_{t-1} \mid x_t) = \operatorname{Cat}(x_{t-1} \mid f_\theta(x_t, t))$
网络输出 categorical 的概率向量,不是 $\epsilon$;softmax 自带列和 = 1。
$\mathcal{L} = \mathbb{E}\!\left[\sum_t D_{\mathrm{KL}}\!\bigl(q(x_{t-1}|x_t,x_0)\,\|\,p_\theta(x_{t-1}|x_t)\bigr)\right]$
两个 categorical 之间的 KL 有闭式:$\sum_k q_k \log \frac{q_k}{p_k}$。
$\sum_j (Q_t)_{ij} = 1 \;\;\forall i$(离散时间)
$\sum_j (R_t)_{ij} = 0 \;\;\forall i$(连续时间 rate 矩阵)
"流入 = 流出"——CMU 称之为 conservation of mass。

手算:$K = 3$ 的 absorbing 链两步

设词表 $\{a, b, \text{MASK}\}$,$K = 3$;$\beta_1 = \beta_2 = 0.4$。Absorbing 矩阵(列代表"从某 token 出发",行代表"落到哪个 token"):

$$Q_1 = \begin{pmatrix} 0.6 & 0 & 0 \\ 0 & 0.6 & 0 \\ 0.4 & 0.4 & 1 \end{pmatrix}.$$

从 $x_0 = a = (1, 0, 0)^\top$ 出发:$Q_1 \, x_0 = (0.6,\, 0,\, 0.4)^\top$——以 0.6 概率仍是 $a$,以 0.4 概率变 MASK,不会变 $b$(因为 absorbing 只允许"留下"或"跳到 MASK")。守恒律检查:第 1 列之和 $= 0.6 + 0 + 0.4 = 1$ ✓;第 2 列 $= 0 + 0.6 + 0.4 = 1$ ✓;第 3 列 $= 0 + 0 + 1 = 1$ ✓(MASK 是吸收态,不会跳出)。

再走一步:$Q_2 = Q_1$,累积矩阵 $\bar{Q}_2 = Q_2 Q_1$ 的第 1 列 = $Q_1 \cdot (0.6, 0, 0.4)^\top = (0.36,\, 0,\, 0.64)^\top$——两步后仍是 $a$ 的概率 $0.6 \times 0.6 = 0.36$,已 MASK 概率 $1 - 0.36 = 0.64$。对比从 $\bar{Q}_2$ 直接读出来的"累积保留概率" $\bar\alpha_2 = 0.36 = (1 - 0.4)^2$,与公式 $\bar\alpha_t = \prod_{s \leq t}(1 - \beta_s)$ 完全吻合。这就是 absorbing 离散扩散最大的工程优势:你从不需要真的把 $Q_t$ 算出来,一个标量 $\bar\alpha_t$ 就够了。

常见误区

  1. "在 token embedding 上加 Gaussian 噪声 = 离散扩散"。反例:把 "cat" 的 embedding 加噪到 $(0.3, -0.2, \ldots)$,再 nearest-token 投影回去可能落到 "dog"——但路径上 $(0.3, -0.2, \ldots)$ 不是任何合法 token,违背"住在合法状态"的原则;这是 latent diffusion 不是 discrete diffusion。真正的离散扩散每一步采样出的都必须是词表里的某个合法 token。
  2. "$Q_t$ 要显式存 $50{,}000 \times 50{,}000$ 大矩阵"。反例:uniform 情形只需 1 个标量 $\beta_t$,矩阵-向量乘退化为线性组合;absorbing 只需标量 $\beta_t$ + MASK 的 token index,前向甚至不用乘法。讲师 L13 @08:24 明确说:"we're trying to do reparameterization trick" 来避开 $O(K^2)$。SEDD 进一步用神经网络直接输出 logits,从根本上不和 $Q_t$ 打照面。
  3. "absorbing discrete diffusion 和 BERT 的 MLM 是两回事"。反例:absorbing $Q_t$ 把每个位置独立 mask 到吸收态,反向预测 $\hat{x}_0$——这正是 BERT cloze 任务的"随时间版本"。BERT 固定 mask 比例 15%,而 absorbing 扩散让 mask 比例随 $t$ 从 0 滑到 1。MDLM (2024) 严格证明二者 ELBO 等价(仅相差一个时间相关的 reweight 系数)。所以"扩散语言模型"和"BERT" 不是相反方向,而是同一个数学物体的不同时间切片。

相关概念

来源

  1. cmu-10799-diffusion-flow · L12 · Discrete Diffusion & Masked Diffusion · (00:02)
  2. cmu-10799-diffusion-flow · L13 · "the easiest way to do discrete diffusion is mask diffusion" · (01:39)
  3. cmu-10799-diffusion-flow · L13 · "Q_t is a jump / velocity essentially" · (06:09)
  4. cmu-10799-diffusion-flow · L13 · "represented by a matrix... it is a very big matrix" + reparameterization trick · (07:03)
查看 150 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 00: 02 All right. All right. Right. So, so far what we have learn we Right. So, so far what we have learn we Right. So, so far what we have learn we have learned about like pretty much have learned about lik…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 00: 56 to do distillation or uh you know self to do distillation or uh you know self dissolations or flow maps or consistency dissolations or flow maps or consistency dissolations or flow maps or consistency…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 01: 45 who, actually, I'll just name who, like the people from Anthropic have claimed the people from Anthropic have claimed the people from Anthropic have claimed to me that uh I I believe it's ICML 2024 to…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 02: 37 uh I mean obviously you know we have other important modalities right for other important modalities right for other important modalities right for example like molecules or like DNA example like mole…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 03: 35 >> Near space and do like a nearest neighbors type thing to match it. neighbors type thing to match it. neighbors type thing to match it. >> Uh so diffusion model on a continuous >> Uh so diffusion mo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 04: 44 then embedding is what goes through >> so you first take the text and go to >> so you first take the text and go to >> so you first take the text and go to embedding embedding existing embedding embed…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 05: 26 running out of the allergyfree options. Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy sensitive option. Okay. Anyway, you're sensitive option…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 06: 19 And then uh for flow matching is pretty And then uh for flow matching is pretty much the same thing, but instead of much the same thing, but instead of much the same thing, but instead of learning the…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 07: 03 adding noise process is the forward process and the gradually d noiseis process and the gradually d noiseis process and the gradually d noiseis process is the reverse process. Right. process is the re…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 08: 09 version of a tweet from Andre Kapathi uh so let's identify some of the noise in so let's identify some of the noise in so let's identify some of the noise in this tweet what do we see this tweet what …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 09: 15 then you get some weird emojis and you then you get some weird emojis and you get some like out of place like six, get some like out of place like six, get some like out of place like six, seven like …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 10: 13 >> Yeah. Okay. >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh do you have allergy? Okay. This is also do you have a…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 11: 23 from the like original uh clean data to some random tokens right in the some random tokens right in the some random tokens right in the vocabulary. So that basically say like vocabulary. So that basic…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 12: 23 vector. Uh then basically say we have a vector. Uh then basically say we have a beta chance. This is just a some chance beta chance. This is just a some chance beta chance. This is just a some chance …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 13: 09 is why this is like one minus you know is why this is like one minus you know every everything else has one beta over every everything else has one beta over every everything else has one beta over th…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 14: 08 sentences with real vocabulary, it's not sentences with real vocabulary, it's not necessary that the noise sentence is necessary that the noise sentence is necessary that the noise sentence is actuall…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 15: 10 then you know so you just so the first then you know so you just so the first row is uh I second row is love uh third row is uh I second row is love uh third row is uh I second row is love uh third ro…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 16: 21 that by induction we can not only get like basically it's the same thing as like basically it's the same thing as like basically it's the same thing as like you know two gausians added like you know t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 17: 11 this is like literally how to jump this is like literally how to jump directly from x0 to xt is you just like directly from x0 to xt is you just like directly from x0 to xt is you just like kind of li…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 18: 24 as the beta t in uh DDPM. as the beta t in uh DDPM. >> Yeah. Don't worry. Everything's gonna be Don't worry. Everything's gonna be simplified later on in this class. right simplified later on in this …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 19: 40 >> Estimate these points and then Q >> Estimate these points and then Q transpose. transpose. transpose. >> Yes, pretty much. And then specifically, >> Yes, pretty much. And then specifically, >> Yes,…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 20: 37 because you're getting you're marovian like everything is marovian here. So you like everything is marovian here. So you like everything is marovian here. So you can ignore the x0 here. So then this c…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 21: 35 categorical distribution that is defined categorical distribution that is defined by this black font term here. >> Oh dot is like elementwise >> Oh dot is like elementwise multiplication. Anyway, hone…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 22: 42 difficult to define in this case. Right? What is the other thing? What is the other thing? What is the other thing? >> Tokens. >> Tokens. >> Tokens. >> The what >> The what >> The what >> the probabil…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 23: 30 prediction right so literally you need to only you only need to predict the to only you only need to predict the to only you only need to predict the logist of the final clean output given logist of t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 24: 18 a D3PM. All right, any questions about D3PM? Yes, Yes, >> this won't have like an equivalence to >> this won't have like an equivalence to >> this won't have like an equivalence to like score matching…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 25: 17 like discrete diffusion models >> great question and the answer is Yes. >> great question and the answer is Yes. >> great question and the answer is Yes. And we're going to talk about it next And we'r…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 26: 33 fixed length. >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is fixed length. Yeah. fixed length. Yeah. fixed length. Yeah. >> …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 27: 24 Alrighty. Okay. So h just a reminder what is a uh what is a continuous score what is a uh what is a continuous score what is a uh what is a continuous score and why is it weird to think about score an…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 28: 10 no continuous flow anymore. There's no gradient like you cannot like really do gradient like you cannot like really do gradient like you cannot like really do any gradient stuff anymore here, right? a…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 28: 56 type of thing. Uh which is not a healthy thing to do but like you know just for thing to do but like you know just for thing to do but like you know just for the sake of the class anyway um you know t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 29: 41 basically what people does is that they define something called uh the concrete define something called uh the concrete define something called uh the concrete score where it's literally just the scor…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 30: 49 apparently uh if you read this particular paper uh if you read this particular paper uh if you read this particular paper they have some connections to something they have some connections to somethin…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 31: 48 people actually call it a rate matrix. people actually call it a rate matrix. So it's a instantaneous rate to jump So it's a instantaneous rate to jump So it's a instantaneous rate to jump like just j…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 32: 43 thing, then you're actually basically if thing, then you're actually basically if you do the math then then then you're you do the math then then then you're you do the math then then then you're actu…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 33: 27 everything well you can represent everything in a well you can represent everything in a well you can represent everything in a fully connected graph and then assign fully connected graph and then ass…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 34: 22 So if you want your chain to satisfy detail balance you're going to get this detail balance you're going to get this detail balance you're going to get this thing so that your chain is reversible, thi…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 35: 19 I see everyone's confused by how this whole thing but we need basically we whole thing but we need basically we whole thing but we need basically we need a neuronet network to predict this need a neur…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 36: 48 score versus your uh you know your your score versus your uh you know your your your your your real score. And you your your your real score. And you your your your real score. And you literally just …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 37: 47 non-zero number divided by a non-zero number or like or sorry non- negative number or like or sorry non- negative number or like or sorry non- negative number divided by a non- negative number number …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 38: 35 good. Exactly. That's exactly correct. good. Exactly. That's exactly correct. Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you yo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 39: 20 like a like a like a like a like a what we call uh constant so let's just ignore we call uh constant so let's just ignore we call uh constant so let's just ignore that for now so we basically we can t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 40: 06 then the person is going to be this thing, right? So what you do is you thing, right? So what you do is you thing, right? So what you do is you essentially so K is like you you jump to essentially so …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 40: 54 then this part becomes one uh s minus r uh times the sum of all the possible all uh times the sum of all the possible all uh times the sum of all the possible all the probabilities and then this one i…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 41: 37 us something that is that has that has exponential in it and also literally exponential in it and also literally exponential in it and also literally right the the the easiest right the the the easies…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 42: 20 right? So you the the S minus the right? So you the the S minus the likelihood ratio times the log of the likelihood ratio times the log of the likelihood ratio times the log of the your S. Um so beca…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 43: 12 calculate the ground truth by literally the same trick the den noising score the same trick the den noising score the same trick the den noising score matching. This is just the noising score matching…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 44: 40 the reverse rate, right? So, you just replace this concrete score with the replace this concrete score with the replace this concrete score with the concrete score that you trained. And concrete score…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 45: 39 >> earlier how are we choosing why do we just randomly sample any token choices just randomly sample any token choices just randomly sample any token choices >> yeah based on >> yeah based on >> yeah …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 46: 38 basically like how about we can only do masking and unmasking, right? So in the masking and unmasking, right? So in the masking and unmasking, right? So in the forward process all you can do is you fo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 47: 20 like basically the only two columns that you're going to have non- negative value you're going to have non- negative value you're going to have non- negative value is the columns on the diagonal and t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 48: 06 >> But if we go all the way end of the day everything will be masked. everything will be masked. everything will be masked. >> Yeah. And it's still a distribution >> Yeah. And it's still a distributio…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 48: 55 multiplication and stuff anymore. Uh because we simplified everything, right? because we simplified everything, right? because we simplified everything, right? So basically what what we're essentially…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 49: 51 stay unmasked and then with one minus alpha t uh probability you're going to alpha t uh probability you're going to alpha t uh probability you're going to stay masked and uh basically if we if stay ma…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 51: 08 when uh xt equals x0 and x t minus one also equals x zero um like basically if also equals x zero um like basically if also equals x zero um like basically if if you have not been masked uh at time t …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 52: 02 simplified into this really really simple uh ratio. And similarly simple uh ratio. And similarly simple uh ratio. And similarly because the only other option for you is because the only other option f…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 53: 11 over. You should you should have the same value because it hasn't been same value because it hasn't been same value because it hasn't been masked. And if XT is masked, then you masked. And if XT is ma…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 54: 23 times the clean data estimation. times the clean data estimation. That's it. >> We have that problem because we'll start >> We have that problem because we'll start with full mask. So if you start wit…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 55: 26 >> So you're always starting with the full mask. mask. mask. >> That's right. >> That's right. >> That's right. >> So if that one reverse gets you to I >> So if that one reverse gets you to I >> So if…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 56: 12 actually a good point. Basically, uh the the the mercury um founder, Inception the the mercury um founder, Inception the the mercury um founder, Inception Lab founder, Stephano, who is the one of Lab …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 57: 09 you can also do like continuous time training with this. You literally just training with this. You literally just training with this. You literally just take the continuous time limit of the of take …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 57: 58 >> uh you yes kind of so it's the same thing it's uh like analogy to you choose thing it's uh like analogy to you choose thing it's uh like analogy to you choose the number of sampling steps right in …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 58: 42 this is purely depending on like the the this is purely depending on like the the model that you got trained but usually model that you got trained but usually model that you got trained but usually i…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 59: 34 basically the mask ratio is depending on this alpha t thing which you set. So you this alpha t thing which you set. So you this alpha t thing which you set. So you you can you can choose uh but like y…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 60: 34 the tokens. the tokens. >> Yeah. >> Yeah. >> But then in the last you have something >> But then in the last you have something >> But then in the last you have something like XT equal to M. So like X…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 61: 16 >> uh what do you mean by latent text well like so you mean like like so you mean like like so you mean like a region space for images and then use a region space for images and then use a region spac…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 62: 02 >> Does it produce something coherent if >> Does it produce something coherent if all the things are independently getting all the things are independently getting all the things are independently get…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 62: 49 >> It is pretty much just birth literally if you think about it. Yeah. if you think about it. Yeah. if you think about it. Yeah. >> Well, between like >> Well, between like >> Well, between like >> yo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 63: 40 when they when he said like oh we can correct hallucination but like here we correct hallucination but like here we correct hallucination but like here we cannot really it's the same as cannot really …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 64: 24 like a sort of like a like a like a like a interesting moment in science. But if a interesting moment in science. But if a interesting moment in science. But if you're one of the authors, you may get,…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 65: 08 >> very low because it's 2024. Okay. Anyway um but since like you know people show um but since like you know people show um but since like you know people show saw the potential of this they have saw…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 65: 55 say and seems like super compatible uh say and seems like super compatible uh to comparable to like large language to comparable to like large language to comparable to like large language models like…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 66: 39 Please come up and uh can get rich as well. All right, see you guys next well. All right, see you guys next well. All right, see you guys next class.
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 00: 03 Okay, so let's get started. uh so far uh Okay, so let's get started. uh so far uh our last class we have talked about you our last class we have talked about you our last class we have talked about yo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 00: 48 Okay. Uh so this is just like a quick like reminder of like like reminder of like like reminder of like what happened next last class. So what happened next last class. So what happened next last clas…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 01: 39 lastly we talked about like the easiest lastly we talked about like the easiest way to do discrete diffusion which is way to do discrete diffusion which is way to do discrete diffusion which is mass d…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 02: 46 probability path, right? to go from like some noise or like some the full mass to some noise or like some the full mass to some noise or like some the full mass to clean data, right? Yeah, basically t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 03: 32 let let's look at them in a in the discrete settings. So last time we kind discrete settings. So last time we kind discrete settings. So last time we kind of skipped through this part uh or of skipped…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 04: 23 distribution, right? So the probability the sum of the probability over all the sum of the probability over all the sum of the probability over all possible states or all possible tokens possible stat…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 05: 13 transitions should add up to zero. And this is like and this is literally just this is like and this is literally just this is like and this is literally just a con conversation uh conservation of a c…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 06: 09 likely that you're going to go basically it's like how how how likely that you're it's like how how how likely that you're it's like how how how likely that you're going to go jump from this state to …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 07: 03 It's a velocity, right? Essentially. So It's a velocity, right? Essentially. So it's basically just saying that like um it's basically just saying that like um it's basically just saying that like um …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 07: 53 this is also why we're trying to do reparameterization trick yeah but that's reparameterization trick yeah but that's reparameterization trick yeah but that's a great question. All right. Anything a g…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 08: 58 this state that's it right this is like super uh very intuitive I hope is super uh very intuitive I hope is super uh very intuitive I hope is All good. What can we observe here? Can we write What can …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 10: 17 not equal anymore. So just sum of all the z all the possible z's and then qzx the z all the possible z's and then qzx the z all the possible z's and then qzx ptz right and z could be equal to x as ptz…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 11: 21 oiler, right? So you just get you you just literally plug everything in here just literally plug everything in here just literally plug everything in here and then you'll get this thing and the and th…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 12: 15 like the the core of the the story in all the discrete diffusion papers is all the discrete diffusion papers is all the discrete diffusion papers is that like no matter what you do try to that like no…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 13: 03 just use cross entropy to learn this, right? Because we have the ground truth right? Because we have the ground truth right? Because we have the ground truth clean data. So we just need to use cross c…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 14: 01 flips right. So you start from t t flips right. So you start from t t equals z which and then you sample from equals z which and then you sample from equals z which and then you sample from your prior…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 14: 45 you just increment the time and then you do it over over again until you reach do it over over again until you reach do it over over again until you reach the end point. Okay, make sense? This this is…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 15: 42 things that we developed somehow but basically there are two discreful basically there are two discreful basically there are two discreful matching papers uh this time they are matching papers uh this…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 16: 22 know but anyway it's like really like I really hope this guy get rich and famous really hope this guy get rich and famous really hope this guy get rich and famous soon. Uh anyway and then the second o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 17: 05 of learn the rate matrix but we don't of learn the rate matrix but we don't actually learn the rate matrix directly actually learn the rate matrix directly actually learn the rate matrix directly we w…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 17: 50 your par paragraph is in order to build this model and this is just like super this model and this is just like super this model and this is just like super like not good right this is a very very lik…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 18: 36 like it's the same problem as the fixed length thing, right? Yeah. length thing, right? Yeah. length thing, right? Yeah. >> I heard this from like who's doing >> I heard this from like who's doing >> …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 19: 20 right. So let's take a look at it. Uh right. So let's take a look at it. Uh yeah. So essentially variable length you yeah. So essentially variable length you yeah. So essentially variable length you s…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 19: 57 like something that we can think about, right? And it's like now we have like right? And it's like now we have like right? And it's like now we have like two um like frameworks that like sort of two u…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 20: 56 >> Yeah. What else? >> Yeah. What else? >> Inserting at but I saw that you can do >> Inserting at but I saw that you can do >> Inserting at but I saw that you can do decoding in blocks like you can pr…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 21: 29 >> Okay, so tatting at the end. So you can >> Okay, so tatting at the end. So you can discard some of the tokens. Yeah, that discard some of the tokens. Yeah, that discard some of the tokens. Yeah, th…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 22: 15 paragraph and for the next diffusion process that paragraph is that one process that paragraph is that one process that paragraph is that one element for you. You noise the paragraph element for you. …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 23: 09 the same thing and that this is like this is corresponding to one paper that this is corresponding to one paper that this is corresponding to one paper that we're going to talk about but not now we're…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 23: 49 different tokens in between tokens, right? I don't need to insert at the right? I don't need to insert at the right? I don't need to insert at the end. Why do I have to insert at the end? end. Why do …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 24: 31 >> Does this work in the diffusion case? >> Good question. I I'm gonna talk about >> Good question. I I'm gonna talk about >> Good question. I I'm gonna talk about it. Okay. So, how do we even model t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 25: 15 you use it to to to essentially construct construct the the ground truth construct construct the the ground truth construct construct the the ground truth of like what this like what the full of like …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 25: 57 right? What is this other thing? >> The position or the whether to generate >> The position or the whether to generate >> The position or the whether to generate it between these it between these it b…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 26: 52 basically if we're missing tokens and basically if we're missing tokens and like essentially like essentially like essentially note that like it's more likely or like note that like it's more likely o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 27: 45 what you're predicting and then the second objective from your model is second objective from your model is second objective from your model is predict what tokens are missing so this predict what tok…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 28: 26 like I don't know. >> Yeah. >> Yeah. >> Yeah. But basically what Okay. So >> Yeah. But basically what Okay. So >> Yeah. But basically what Okay. So basically what happens in the model or basically wha…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 29: 10 >> Yeah. Yeah. So basically what's happening in what's hap what's happening happening in what's hap what's happening happening in what's hap what's happening in like implementation is that you are in …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 29: 55 is that is missing out of all the tokens is that is missing out of all the tokens that are missing? Right. Okay. So these that are missing? Right. Okay. So these that are missing? Right. Okay. So thes…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 30: 48 missing versus you know how many tokens you predict to be missing and then the you predict to be missing and then the you predict to be missing and then the second thing is uh like literally just seco…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 31: 48 flow And why is it a flow model? Well, it's And why is it a flow model? Well, it's And why is it a flow model? Well, it's actually because this thing like the actually because this thing like the actu…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 32: 35 another thing then you should be doing another thing then you should be doing this operation where it's a ratio or this operation where it's a ratio or this operation where it's a ratio or like a rate…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 33: 23 divergence that is basically just a generalized version of KL. Uh but this generalized version of KL. Uh but this generalized version of KL. Uh but this is like too complicated. We're not going is lik…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 34: 11 actually like already good enough for whatever you want to do. So you can whatever you want to do. So you can whatever you want to do. So you can literally just train a insertion based literally just …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 34: 56 we kind of fixed a veryable length uh problem with insertion based length uh problem with insertion based length uh problem with insertion based generation. Uh we have other things to generation. Uh w…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 35: 40 So like basically this thing is very nice because you can um like essentially nice because you can um like essentially nice because you can um like essentially you can do paralyze generation like you …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 36: 20 that you have generated like in your previous blocks. The only thing you need previous blocks. The only thing you need previous blocks. The only thing you need to like do birectional attention to is t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 37: 00 people think about this. Um but yeah, so people think about this. Um but yeah, so this is like a nice way to do you know this is like a nice way to do you know this is like a nice way to do you know t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 37: 50 >> not in the current block but but we can K catch all the previous blocks. >> No because we don't we haven't generated >> No because we don't we haven't generated that yet, right? So like the block a…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 38: 29 of like also like but but but but if you have too small of a blog then it just have too small of a blog then it just have too small of a blog then it just Yeah. It doesn't really it just is just Yeah.…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 39: 16 in I think. Yeah. in I think. Yeah. >> Do you think the second one is >> Do you think the second one is >> Do you think the second one is um like it's it's the it's a problem of um like it's it's the …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 40: 11 That's correct. Yeah. >> For the for the second case, the mass >> For the for the second case, the mass >> For the for the second case, the mass diffusion, why can't we do something diffusion, why can…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 40: 57 Anything else? Yep. >> Can you overhand why mass diffusion is >> Can you overhand why mass diffusion is >> Can you overhand why mass diffusion is lower? lower? lower? >> Uh okay. Who wants to tell who…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 41: 49 but it doesn't mean that you you cannot train a good enough diffusion model to train a good enough diffusion model to train a good enough diffusion model to blm that's not the case it's just that blm …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 42: 45 the the optimal that you're going to get the the optimal that you're going to get from whatever algorithm that they from whatever algorithm that they from whatever algorithm that they develop will loo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 43: 32 this is not image and text but like we're going to look at image and text we're going to look at image and text we're going to look at image and text later. But for those of you who are in later. But …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 44: 31 >> What are we cogenerating? >> So, we're co-generating something uh >> So, we're co-generating something uh >> So, we're co-generating something uh discreet and something uh continuous at discreet an…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 45: 15 like alternating between the two that's like alternating between the two that's pretty much that right and like pretty much that right and like pretty much that right and like basically And then if yo…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 46: 03 well right so what you can do is you can well right so what you can do is you can uh have like just like I think some uh have like just like I think some uh have like just like I think some someone al…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 46: 50 >> I mean the problem is like this is a >> I mean the problem is like this is a research project right so it's like very research project right so it's like very research project right so it's like ve…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 47: 37 and then the the the image for example like flower or like cat and then and like flower or like cat and then and like flower or like cat and then and then the the the the image are going to then the t…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 48: 13 text and like image at the same time text and like image at the same time because like you start from nothing and because like you start from nothing and because like you start from nothing and then y…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 48: 49 the way to become omni but this is like just like two two modalities I guess. just like two two modalities I guess. just like two two modalities I guess. Yeah. Yeah. Omni model just means that Yeah. Y…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 49: 50 you cannot access the ponapto anymore, it's still going to be on YouTube. Uh I it's still going to be on YouTube. Uh I it's still going to be on YouTube. Uh I kind of do some editing uh for the kind o…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 50: 25 your turn to take what you've learned into the real world so I will highly into the real world so I will highly into the real world so I will highly recommend if you are proud of what recommend if you…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 51: 05 something like that. I think it could be something like that. I think it could be a good experience if you guys are a good experience if you guys are a good experience if you guys are interested in ge…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 51: 39 research slash you know when you go to the AGI company you know make the AGI a the AGI company you know make the AGI a the AGI company you know make the AGI a diffusion model please we don't want to d…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 52: 25 >> padding tokens and tokens after the OS, right? right? >> Yeah. >> Yeah. >> Uh so one thing I implemented for >> Uh so one thing I implemented for >> Uh so one thing I implemented for homework 3 and…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 53: 06 I thought that's what they did in I thought that's what they did in refflow but maybe not. If it's not then refflow but maybe not. If it's not then refflow but maybe not. If it's not then this is a go…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 53: 42 than a 2 RF model because the only than a 2 RF model because the only errors accumulate. errors accumulate. errors accumulate. >> Yeah. No, that's your homework. Four. >> Yeah. No, that's your homewor…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 54: 19 in general I think that's a better idea than the normal the the the way that than the normal the the the way that than the normal the the the way that they they did. Okay. Any other question? they the…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 55: 03 >> Yeah. Yeah. So actually there multiple >> Yeah. Yeah. So actually there multiple uh so the most famous one in the uh so the most famous one in the uh so the most famous one in the robotics world is…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 55: 49 gradient you should apply from those buffer data to your current time to one buffer data to your current time to one buffer data to your current time to one of the time that you sampled. Um so this of…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 56: 25 in that framework but you can turn you in that framework but you can turn you can essentially turn an S OD like a flow can essentially turn an S OD like a flow can essentially turn an S OD like a flow…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 57: 06 basically is there any other ways that we can do it besides turning into a SDE we can do it besides turning into a SDE we can do it besides turning into a SDE um yeah so like let's hope we can get um …
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 57: 58 essentially you just sort of like uh they sort of extend the flow map uh uh they sort of extend the flow map uh uh they sort of extend the flow map uh uh what you call framework into the what you call…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 58: 37 Uh, so let's see if he actually post it online this week. Uh, but yeah, I don't online this week. Uh, but yeah, I don't online this week. Uh, but yeah, I don't actually know how how he did it, but act…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 59: 19 of similar to uh one of the other previous uh framework and equilibrium previous uh framework and equilibrium previous uh framework and equilibrium model. So equilibrium model is something model. So e…
cmu-10799-diffusion-flow L12: CMU 10799 S26: Lecture 13 - Discrete Flow Matching & Edit Flow - Diffusion & Flow Matching 60: 02 or the underlying uh, you know, uh, math is a little bit related, but like this is a little bit related, but like this is a little bit related, but like this is definitely something new. I think is de…