概念

Score-Entropy Discrete Diffusion (SEDD)

— 把 "score" 推广到离散空间:用比值代替梯度,用 score-entropy 代替 score matching。

很多人第一次听到 SEDD,会以为它就是"discrete diffusion 换了个名字"。这是错的。SEDD 把 "score" 这个概念本身推广到了离散空间——它不是给离散扩散加了个新损失项,而是回答了一个更基础的问题:当 $x$ 是离散 token 时,$\nabla \log p(x)$ 根本不存在(你没法对"猫"这个 token 求梯度),那连续扩散里赖以为生的 "score-based model" 这整条路,还能不能搬到文本和分子上来?

「continuous diffusion 我们有三种视角——denoising、score-based、flow matching。问题是:能不能把这三套公式映射到 discrete diffusion?这就是今天主要要回答的问题。」—— CMU 10799 L12 @06:19

讲师在 CMU 10799 Lecture 12 把这件事讲得很直白:连续空间里 score 是 $\nabla_x \log p_t(x)$,它是一个向量场,告诉反向 SDE "在 $x$ 处应该往哪个方向漂"。在离散空间里没有"方向"这种东西,只有"该不该从当前 token 跳到某个邻居 token"。换句话说,连续 score 回答的是"how much",离散 score 必须回答的是"which one"。于是 SEDD 用一个新对象——concrete score,也就是相邻状态的概率比值 $p_t(y)/p_t(x)$——来代替梯度。这个比值大于 1 意味着邻居 $y$ 比当前 $x$ 更"可能",反向链就倾向于跳过去;小于 1 意味着邻居更不可能,反向链就更愿意停留。整套框架因此被称作 Score Entropy Discrete Diffusion,缩写 SEDD,原始论文是 Lou, Meng & Ermon (ICML 2024)。

「Stanford 那篇是第一个 diffusion language model 的尝试——先 embedding,再在连续 embedding 空间跑扩散,最后 nearest neighbor 解码——但效果不如我们今天要讲的这个。」—— CMU 10799 L12 @04:55
「Anthropic 的人和我说,ICML 2024 有个观点:任何不能压进文本的信息都是无趣的、图像生成不在通往 AGI 的路上。所以离散数据生成现在变得超级重要——不光是文本,还有分子、DNA 序列,甚至 image 本身也是离散的(0-255 像素值)。」—— CMU 10799 L12 @02:00(讲师铺垫为什么要研究离散扩散)

讲师特意把 SEDD 放在 Stanford 早期"embedding + 连续扩散"那条思路的对面来介绍:那条路是把离散问题伪装成连续问题——先把 token 映到连续 embedding,再在 embedding 空间跑标准扩散,最后做 nearest neighbor 解码——它的麻烦在于训练目标和最终生成目标不对齐,连续空间里 score 很小的样本可能 nearest-neighbor 之后跳到一个完全不同的 token;SEDD 则是把 score-based 的整套语言真正改造成离散原生的,不再绕道连续空间。这件事在训练里对应的就是:用一个网络 $s_\theta(x, y, t)$ 去预测比值 $p_t(y)/p_t(x)$,损失函数叫 score entropy——它是连续 score matching 的离散类比,结构上是一个针对正比值的 Bregman divergence,最优解恰好命中真实比值。后面我们会把这件事一步步拆开。

核心对象一:concrete score(具体得分)

连续空间里 score 定义为 $\nabla_x \log p_t(x)$,它是个向量场。离散空间里 $x$ 是 token,没有梯度。SEDD 的关键替换:

$$s(x, y, t) \;:=\; \frac{p_t(y)}{p_t(x)}, \qquad y \neq x.$$

对每一对相邻状态 $(x, y)$,concrete score 是一个标量——告诉你"从 $x$ 跳到 $y$ 比待在 $x$ 概率高/低多少倍"。Why:这是 score 的"离散类比",理由可以一步推出来。注意 $\log p_t(y) - \log p_t(x) = \log \frac{p_t(y)}{p_t(x)}$,而连续 score $\nabla_x \log p_t$ 本质上就是 $\log p_t$ 在邻域的"局部差分"——在连续空间它通过求极限退化成偏导数,在离散空间它就是相邻状态的对数比值,无需取极限。SEDD 把这件事直接搬到 token 空间:邻居之间的对数比值。换句话说,"score" 这个词在两个空间里指的是同一个数学对象,只是在连续空间用梯度实现、在离散空间用比值实现,二者通过测度论的密度概念被统一起来。

关键观察:对于词汇表大小 $K$ 的离散变量,每个状态有 $K-1$ 个邻居(除了它自己),所以 concrete score 不是一个数而是一个 $(K-1)$ 维向量——每个出邻居占一个分量。神经网络 $s_\theta(x, \cdot, t)$ 一次 forward 输出整组比值。对一段长度 $L$ 的文本,输出张量形状就是 $L \times (K-1)$。和"预测下一个 token 的概率分布"形状一致,但语义完全不同:softmax 输出的是归一化概率、加起来等于 1,SEDD 输出的是未归一化的比值,可以大于 1,也不需要总和约束。这件事一旦没意识到,工程实现就会无意识地套 softmax,导致目标变形。

核心对象二:CTMC 与生成矩阵

SEDD 的前向过程是连续时间马尔可夫链(CTMC)。状态分布演化由生成矩阵 $Q_t$ 描述:

$$\dot p_t \;=\; Q_t\, p_t,$$

其中 $Q_t \in \mathbb{R}^{K \times K}$ 满足 $(Q_t)_{yx} \geq 0$($y \neq x$)且每列和为 0。$(Q_t)_{yx}$ 的物理含义是"在时间 $t$ 从 $x$ 跳到 $y$ 的瞬时速率",单位是 1/时间。Why:CTMC 把"加噪"理解成"以一定速率随机跳到别的 token"——masking 加噪对应 $Q$ 只允许跳到一个特殊的 [MASK] 状态,uniform 加噪对应 $Q$ 让所有 token 等概率互相跳,Gaussian-on-codebook 对应按 embedding 距离加权的 $Q$。三种常见的离散噪声方案都可以通过设计 $Q_t$ 在同一个框架下实现,前向加噪和反向去噪用同一套矩阵语言描述。这一点和连续扩散里"forward SDE + 反向 SDE"是完全对偶的:那边是无穷小漂移加扩散,这边是无穷小跳跃速率。

把 $Q_t$ 的对角元定义成 $-\sum_{y \neq x}(Q_t)_{yx}$ 保证概率守恒;这件事虽然小但关键,因为它让"留在原地"自动被计入,反向链不需要单独建模"自跳"。反向 CTMC 的生成矩阵 $\bar Q_t$ 由 Kolmogorov 时间反演关系给出,结果异常干净:

$$(\bar Q_t)_{xy} \;=\; \frac{p_t(y)}{p_t(x)}\,(Q_t)_{yx} \quad (x \neq y).$$

注意右边出现了 concrete score $p_t(y)/p_t(x)$——这就是为什么你只需要学比值就能反向采样。这条关系是 SEDD 整个框架的几何中心:前向矩阵 $Q_t$ 由你设计、是已知的;反向矩阵 $\bar Q_t$ 多了一个"比值因子";这个因子正是 concrete score。讲师在 L12 把它和连续扩散的反向 SDE 对照:连续情形是 $\nabla \log p$ 加进 drift 项 $f - g^2 \nabla \log p$,离散情形是比值乘进 transition rate。两个反向公式结构上完全平行。

核心对象三:score entropy 损失

要学 $s_\theta(x, y, t) \approx p_t(y)/p_t(x)$,但 $p_t(y)/p_t(x)$ 没法直接监督——分母 $p_t(x)$ 是边缘概率,不可观测。SEDD 用 score entropy:

$$\mathcal{L} \;=\; \mathbb{E}_{x_0,\, x \sim p_t(\cdot | x_0)} \!\!\left[\sum_{y \neq x} q(y|x)\!\left(s_\theta(x,y,t) - \frac{p_t(y\,|\,x_0)}{p_t(x\,|\,x_0)}\log s_\theta(x,y,t)\right)\right] + \text{const}.$$

Why(最关键的一步):这个损失初看奇怪——既有 $s_\theta$ 的线性项又有 $\log s_\theta$ 的非线性项,符号还不一致——但它实际上是一个 Bregman divergence on positive ratios。具体来说,定义凸函数 $\phi(u) = u \log u - u$(这是正实数上的标准熵函数),它生成的 Bregman 散度为 $D_\phi(r \| s) = r \log(r/s) - (r - s)$。把 SEDD 损失里 $y$ 这一项展开并丢掉与 $s_\theta$ 无关的常数,结果正比于 $D_\phi(r^\star \| s_\theta)$——所以 score entropy 不是凭空构造的,它就是"在正比值空间上做凸距离最小化"的标准形式。展开后对 $s_\theta$ 求梯度得到 $\mathbb{E}[(s_\theta - r^\star)\,\nabla \log s_\theta]$,其中 $r^\star = p_t(y)/p_t(x)$ 是真实比值;令梯度为零,最优解恰好是 $s_\theta^\star = p_t(y)/p_t(x)$。这和 Hyvärinen score matching 的思想完全一致:构造一个看起来不依赖未知分布的代理损失,但保证其最优解仍然命中真实 score。Score matching 在连续空间用分部积分消未知量,SEDD 在离散空间用"枚举邻居 + 条件化到 $x_0$"消未知量——两套机制不同,但精神相同。

类比 Hyvärinen 的 trick:连续 score matching 用分部积分消掉未知的 $\nabla \log p$,把 $\mathbb{E}_p\|s_\theta - \nabla \log p\|^2$ 改写成不含 $p$ 显式出现的 $\mathbb{E}\|s_\theta\|^2 + 2 \nabla \cdot s_\theta$。SEDD 在离散空间没有分部积分这种工具,但有一个等价的离散对应物:状态枚举 + 条件化。对每个 $x$ 枚举它的邻居 $y$,把比值 $p_t(y)/p_t(x)$ 用全概率公式展开成对 $x_0$ 的期望,得到比值 $p_t(y\,|\,x_0)/p_t(x\,|\,x_0)$——这个量在给定干净样本 $x_0$ 下是条件可计算的解析量,因为前向 transition kernel $p_t(\cdot\,|\,x_0)$ 是你自己设计的、有闭式表达。于是 score entropy 损失变成完全可监督的——这就是离散版的"消未知量"。
Concrete Score:从 x 出发,每个邻居一个比值 x = cat dog car [MASK] bat s(x,dog) = p(dog)/p(cat) s(x,car) ≈ 0.02 s(x,MASK) = 0.6 s(x,bat) ≈ 0.05 网络 $s_\theta(x, \cdot, t)$ 一次输出 K-1 个比值;反向链按比值采样下一跳。

采样:反向 CTMC 走一步

给定当前状态 $x_t$,反向链下一步的瞬时跳跃速率为 $(\bar Q_t)_{x_t \to y} = s_\theta(x_t, y, t)\,(Q_t)_{y \to x_t}$。用 tau-leaping 或 Euler 离散化采样:在小步长 $\Delta t$ 内,以概率 $\propto s_\theta(x_t, y, t) \cdot (Q_t)_{y \to x_t} \cdot \Delta t$ 跳到 $y$,剩余概率留在 $x_t$ 不动。这正是连续 score 的"WHICH transition to take"在离散空间的等价表述——比值越大的邻居越可能被选中。整个采样过程在概念上和连续扩散的 Euler-Maruyama 一模一样,只是把"沿着 score 方向漂一小步"换成"按比值采样一个跳跃",把高斯增量换成 Bernoulli/Poisson 增量。

🟡 教材外补充:和 MDLM (Masked Diffusion Language Model) 对比——MDLM 的 forward 只允许 token → MASK 一种跳跃,反向时网络只需预测"这个 MASK 位置应该填什么 token",每个位置 1 套 logits、O(1) 个监督信号。SEDD 是完全连接的 CTMC,每个位置要预测 $K-1$ 个比值——表达力更强(可以模型化任意 token 之间的转移倾向),但计算量也是 $O(K)$ 倍。在 text vocab $K \approx 50\,000$ 时这是非常显著的开销,原论文用结构化 $Q$(uniform / absorbing)让计算可以摊销到 $O(\log K)$ 或近似 $O(1)$;这也是后续工作 (MD4、RADD 等) 重点优化的方向。在 perplexity 指标上,SEDD 和 MDLM 早期版本表现接近,但 SEDD 的框架被认为更通用、可以原生支持非掩码的噪声方案,是 diffusion-as-foundation-model 的离散版起点;这也是为什么后续讨论"离散扩散基础模型"时通常以 SEDD 作为参考点,而不是更早的 D3PM。

训练 pipeline 速览

把上面的零件拼起来,SEDD 的训练循环和连续扩散惊人地相似:

  1. 采干净样本 $x_0 \sim p_{\text{data}}$,例如一句文本。
  2. 采时间 $t \sim \text{Uniform}(0, T)$。
  3. 采加噪样本 $x_t \sim p_t(\cdot\,|\,x_0)$,按设计好的前向 CTMC 跳跃。对 uniform $Q$ 而言这是闭式分布,对 absorbing $Q$ 而言每个 token 独立按概率被替换成 [MASK]。
  4. 解析算真实比值 $r^\star(y) = p_t(y\,|\,x_0)/p_t(x_t\,|\,x_0)$,对每个邻居 $y$。这一步因为 kernel 已知所以是确定性的。
  5. 前向 + 反向:网络输出 $s_\theta(x_t, \cdot, t)$,计算 score entropy 损失,反向传播。

对比 DDPM 训练循环:采 $x_0$、采 $t$、采 $\epsilon$、加噪得 $x_t = \sqrt{\bar\alpha_t} x_0 + \sqrt{1 - \bar\alpha_t}\,\epsilon$、网络预测 $\hat\epsilon$、算 MSE。结构完全一致,只是"加噪 + 解析监督量"从 $\epsilon$ 换成了"邻居比值向量"。这种深刻的结构相似性正是 SEDD 被称为"score-based 在离散空间的自然延续"的原因,而不只是又一种 discrete diffusion 变体。如果你看懂了连续扩散的 noise prediction 训练流程,把 noise vector 换成 ratio vector、把 Gaussian kernel 换成 CTMC kernel,就直接得到 SEDD。

工程上的几个细节

在实际实现里,有几个不写出来就训不收敛的细节,讲师虽然没在课堂上完全展开但论文里反复强调:第一,$s_\theta$ 必须用 $\exp$ 参数化保证正性,否则 $\log s_\theta$ 在 $s_\theta \to 0$ 时会爆炸出 $-\infty$;第二,时间 $t$ 接近 0 时比值的方差非常大(干净样本附近,邻居比值能差几个数量级)、接近 $T$ 时几乎都接近 1(噪声充分混合),所以要按 noise schedule 给不同时间段的损失加权才能保持训练稳定;第三,对 absorbing $Q$(也就是只允许 token → MASK 的情形),未 mask 的位置 concrete score 退化成常数 1,损失自动稀疏化、计算量从 $O(L \cdot K)$ 降到 $O(\text{mask count} \cdot K)$;第四,反向采样的 tau-leaping 离散化里,如果步长 $\Delta t$ 太大会导致总跳跃概率超过 1(数学上越界),工程上常用 Poisson clipping 或者干脆缩小 $\Delta t$ 来回避;第五,词汇表 $K = 50\,000$ 时 $L \times K$ 的输出张量内存压力巨大,需要 mixed precision 或分块计算。这些细节合在一起,决定了 SEDD 能不能跑出和同规模 autoregressive transformer 接近的 perplexity——论文报告的结果是在 OpenWebText 上和 GPT-2 接近,这在 2024 年是离散扩散的标志性成果,也奠定了它作为后续工作(MD4、RADD、SDTT 等)出发点的地位。

和连续 score-based 的对照表

连续 (Song & Ermon)SEDD (离散)
score$\nabla_x \log p_t(x)$$p_t(y)/p_t(x)$, $y \neq x$
前向SDE: $dx = f\,dt + g\,dW$CTMC: $\dot p_t = Q_t p_t$
反向SDE 带 score drift反向 CTMC, rate 乘比值
损失Hyvärinen (分部积分)score entropy (枚举邻居)
每步代价$O(d)$ 维度$O(K)$ 状态数

注意看不变量:随着步骤推进,公式从"我们要学的目标"出发,逐步替换出"可计算的代理"——但最优解的位置永远是 $s_\theta^\star = p_t(y)/p_t(x)$,这就是 SEDD 能 work 的核心。每一步都不改变最优解、只改变损失函数的"形状"和"可监督性",整个推导本质上是一个等价变换链。

想学的真目标(不可直接监督):
$$s_\theta(x, y, t) \;\approx\; \frac{p_t(y)}{p_t(x)}.$$
用 Bregman-on-ratios 构造代理损失:
$$\mathcal{L} = \mathbb{E}\!\left[\sum_{y \neq x} q(y|x)\!\left(s_\theta - \frac{p_t(y)}{p_t(x)}\log s_\theta\right)\right].$$
用条件分布消掉未知的 $p_t$:
$$\frac{p_t(y)}{p_t(x)} = \mathbb{E}_{x_0}\!\!\left[\frac{p_t(y\,|\,x_0)}{p_t(x\,|\,x_0)}\,\Big|\,x_t = x\right] \text{(kernel 已知,可解析算).}$$
取梯度令其为 0,最优解恰好命中真实比值:
$$s_\theta^\star(x, y, t) = \frac{p_t(y)}{p_t(x)}. \quad\checkmark$$

三 token 玩具例子

设词汇表 $\{A, B, C\}$,前向 uniform CTMC,时刻 $t$ 真实条件分布 $p_t(\cdot\,|\,x_0=A) = (0.6, 0.3, 0.1)$(也就是说,从干净样本 $A$ 加噪到时刻 $t$,有 60% 概率还是 $A$、30% 跳到了 $B$、10% 跳到了 $C$)。现在采样器观察到 $x_t = B$,要决定反向往哪儿跳。算两个 concrete score:

$$\frac{p_t(A\,|\,x_0)}{p_t(B\,|\,x_0)} = \frac{0.6}{0.3} = 2.0, \qquad \frac{p_t(C\,|\,x_0)}{p_t(B\,|\,x_0)} = \frac{0.1}{0.3} \approx 0.33.$$

反向链从 $B$ 出发,跳到 $A$ 的瞬时速率是跳到 $C$ 的 $2.0 / 0.33 \approx 6$ 倍——符合直觉:$A$ 是真实干净样本,比值应当大。score entropy 损失给网络的监督信号就是让 $s_\theta(B, A, t) \to 2.0$、$s_\theta(B, C, t) \to 0.33$。注意这两个值都是未归一化的,而且 $s_\theta(B, A, t)$ 大于 1——这是 concrete score 和 softmax 概率最容易混淆的地方。如果你硬要在最后一层套 softmax,最优解就被强行归一化到 $(2.0, 1.0, 0.33)/3.33 \approx (0.60, 0.30, 0.10)$,看似还是真实分布,但训练目标的形状变了——score entropy 关心的是比值的几何,cross-entropy 关心的是概率的单纯形几何,两者梯度方向不同、收敛轨迹也不同。这也是为什么 SEDD 原始实现一定用 exp 参数化、不用 softmax。

常见误区

  1. "SEDD 就是给 discrete diffusion 换个损失"。反例:D3PM (Austin 2021) 用 ELBO + KL 散度,目标是直接学反向转移核 $p_\theta(x_{t-1}\,|\,x_t)$;SEDD 学的对象是 比值 $p_t(y)/p_t(x)$,再通过 Kolmogorov 关系反推 $\bar Q_t$ 才得到转移概率——架构、训练目标、采样器、归一化处理全都不同。两者关系类似于连续扩散里 DDPM 和 score-based SDE:终点相似,路径完全不同。
  2. "concrete score 是一个数"。反例:对 token 化文本,每个位置每步要预测 $K-1 \approx 50\,000$ 个比值。网络输出张量形状是 $L \times (K-1)$,每个位置一个完整的"邻居比值向量",不是 $L \times 1$。这是 SEDD 比 MDLM 贵的根源——MDLM 每个位置只输出一个 $K$ 维 softmax,只回答 "MASK 应填什么",但 SEDD 要回答 "当前 token 与每个候选 token 的相对密度",信息量大得多、监督信号也密集得多。
  3. "score entropy 就是 cross-entropy"。反例:cross-entropy 是 $-\sum_i p_i \log q_i$,定义在归一化概率分布上;score entropy 是 $\sum_y q(y|x)(s - r \log s)$,定义在正比值上 ($s, r > 0$),对应的不是 KL 而是 Bregman divergence on positive reals。两者最优解形式相似 ($s^\star = r^\star$) 但代数结构和数值稳定性截然不同——$s$ 必须严格大于 0,工程上常用 $s_\theta = \exp(f_\theta)$ 参数化,且 log 项可能在 $s$ 很小时数值爆炸,需要特别处理。混淆这两者会让你以为可以套用 softmax 训练 SEDD,结果完全 train 不出来。

相关概念

来源

  1. cmu-10799-diffusion-flow · L12 · Discrete Diffusion & Masked Diffusion · (00:02)
  2. cmu-10799-diffusion-flow · L12 · 三视角 → discrete diffusion 映射 · (05:26)
  3. cmu-10799-diffusion-flow · L12 · Stanford 第一篇 diffusion LM 与 SEDD 对比 · (04:44)
查看 73 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 00: 02 All right. All right. Right. So, so far what we have learn we Right. So, so far what we have learn we Right. So, so far what we have learn we have learned about like pretty much have learned about lik…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 00: 56 to do distillation or uh you know self to do distillation or uh you know self dissolations or flow maps or consistency dissolations or flow maps or consistency dissolations or flow maps or consistency…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 01: 45 who, actually, I'll just name who, like the people from Anthropic have claimed the people from Anthropic have claimed the people from Anthropic have claimed to me that uh I I believe it's ICML 2024 to…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 02: 37 uh I mean obviously you know we have other important modalities right for other important modalities right for other important modalities right for example like molecules or like DNA example like mole…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 03: 35 >> Near space and do like a nearest neighbors type thing to match it. neighbors type thing to match it. neighbors type thing to match it. >> Uh so diffusion model on a continuous >> Uh so diffusion mo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 04: 44 then embedding is what goes through >> so you first take the text and go to >> so you first take the text and go to >> so you first take the text and go to embedding embedding existing embedding embed…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 05: 26 running out of the allergyfree options. Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy sensitive option. Okay. Anyway, you're sensitive option…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 06: 19 And then uh for flow matching is pretty And then uh for flow matching is pretty much the same thing, but instead of much the same thing, but instead of much the same thing, but instead of learning the…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 07: 03 adding noise process is the forward process and the gradually d noiseis process and the gradually d noiseis process and the gradually d noiseis process is the reverse process. Right. process is the re…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 08: 09 version of a tweet from Andre Kapathi uh so let's identify some of the noise in so let's identify some of the noise in so let's identify some of the noise in this tweet what do we see this tweet what …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 09: 15 then you get some weird emojis and you then you get some weird emojis and you get some like out of place like six, get some like out of place like six, get some like out of place like six, seven like …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 10: 13 >> Yeah. Okay. >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh do you have allergy? Okay. This is also do you have a…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 11: 23 from the like original uh clean data to some random tokens right in the some random tokens right in the some random tokens right in the vocabulary. So that basically say like vocabulary. So that basic…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 12: 23 vector. Uh then basically say we have a vector. Uh then basically say we have a beta chance. This is just a some chance beta chance. This is just a some chance beta chance. This is just a some chance …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 13: 09 is why this is like one minus you know is why this is like one minus you know every everything else has one beta over every everything else has one beta over every everything else has one beta over th…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 14: 08 sentences with real vocabulary, it's not sentences with real vocabulary, it's not necessary that the noise sentence is necessary that the noise sentence is necessary that the noise sentence is actuall…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 15: 10 then you know so you just so the first then you know so you just so the first row is uh I second row is love uh third row is uh I second row is love uh third row is uh I second row is love uh third ro…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 16: 21 that by induction we can not only get like basically it's the same thing as like basically it's the same thing as like basically it's the same thing as like you know two gausians added like you know t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 17: 11 this is like literally how to jump this is like literally how to jump directly from x0 to xt is you just like directly from x0 to xt is you just like directly from x0 to xt is you just like kind of li…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 18: 24 as the beta t in uh DDPM. as the beta t in uh DDPM. >> Yeah. Don't worry. Everything's gonna be Don't worry. Everything's gonna be simplified later on in this class. right simplified later on in this …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 19: 40 >> Estimate these points and then Q >> Estimate these points and then Q transpose. transpose. transpose. >> Yes, pretty much. And then specifically, >> Yes, pretty much. And then specifically, >> Yes,…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 20: 37 because you're getting you're marovian like everything is marovian here. So you like everything is marovian here. So you like everything is marovian here. So you can ignore the x0 here. So then this c…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 21: 35 categorical distribution that is defined categorical distribution that is defined by this black font term here. >> Oh dot is like elementwise >> Oh dot is like elementwise multiplication. Anyway, hone…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 22: 42 difficult to define in this case. Right? What is the other thing? What is the other thing? What is the other thing? >> Tokens. >> Tokens. >> Tokens. >> The what >> The what >> The what >> the probabil…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 23: 30 prediction right so literally you need to only you only need to predict the to only you only need to predict the to only you only need to predict the logist of the final clean output given logist of t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 24: 18 a D3PM. All right, any questions about D3PM? Yes, Yes, >> this won't have like an equivalence to >> this won't have like an equivalence to >> this won't have like an equivalence to like score matching…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 25: 17 like discrete diffusion models >> great question and the answer is Yes. >> great question and the answer is Yes. >> great question and the answer is Yes. And we're going to talk about it next And we'r…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 26: 33 fixed length. >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is fixed length. Yeah. fixed length. Yeah. fixed length. Yeah. >> …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 27: 24 Alrighty. Okay. So h just a reminder what is a uh what is a continuous score what is a uh what is a continuous score what is a uh what is a continuous score and why is it weird to think about score an…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 28: 10 no continuous flow anymore. There's no gradient like you cannot like really do gradient like you cannot like really do gradient like you cannot like really do any gradient stuff anymore here, right? a…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 28: 56 type of thing. Uh which is not a healthy thing to do but like you know just for thing to do but like you know just for thing to do but like you know just for the sake of the class anyway um you know t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 29: 41 basically what people does is that they define something called uh the concrete define something called uh the concrete define something called uh the concrete score where it's literally just the scor…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 30: 49 apparently uh if you read this particular paper uh if you read this particular paper uh if you read this particular paper they have some connections to something they have some connections to somethin…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 31: 48 people actually call it a rate matrix. people actually call it a rate matrix. So it's a instantaneous rate to jump So it's a instantaneous rate to jump So it's a instantaneous rate to jump like just j…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 32: 43 thing, then you're actually basically if thing, then you're actually basically if you do the math then then then you're you do the math then then then you're you do the math then then then you're actu…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 33: 27 everything well you can represent everything in a well you can represent everything in a well you can represent everything in a fully connected graph and then assign fully connected graph and then ass…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 34: 22 So if you want your chain to satisfy detail balance you're going to get this detail balance you're going to get this detail balance you're going to get this thing so that your chain is reversible, thi…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 35: 19 I see everyone's confused by how this whole thing but we need basically we whole thing but we need basically we whole thing but we need basically we need a neuronet network to predict this need a neur…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 36: 48 score versus your uh you know your your score versus your uh you know your your your your your real score. And you your your your real score. And you your your your real score. And you literally just …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 37: 47 non-zero number divided by a non-zero number or like or sorry non- negative number or like or sorry non- negative number or like or sorry non- negative number divided by a non- negative number number …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 38: 35 good. Exactly. That's exactly correct. good. Exactly. That's exactly correct. Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you yo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 39: 20 like a like a like a like a like a what we call uh constant so let's just ignore we call uh constant so let's just ignore we call uh constant so let's just ignore that for now so we basically we can t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 40: 06 then the person is going to be this thing, right? So what you do is you thing, right? So what you do is you thing, right? So what you do is you essentially so K is like you you jump to essentially so …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 40: 54 then this part becomes one uh s minus r uh times the sum of all the possible all uh times the sum of all the possible all uh times the sum of all the possible all the probabilities and then this one i…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 41: 37 us something that is that has that has exponential in it and also literally exponential in it and also literally exponential in it and also literally right the the the easiest right the the the easies…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 42: 20 right? So you the the S minus the right? So you the the S minus the likelihood ratio times the log of the likelihood ratio times the log of the likelihood ratio times the log of the your S. Um so beca…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 43: 12 calculate the ground truth by literally the same trick the den noising score the same trick the den noising score the same trick the den noising score matching. This is just the noising score matching…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 44: 40 the reverse rate, right? So, you just replace this concrete score with the replace this concrete score with the replace this concrete score with the concrete score that you trained. And concrete score…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 45: 39 >> earlier how are we choosing why do we just randomly sample any token choices just randomly sample any token choices just randomly sample any token choices >> yeah based on >> yeah based on >> yeah …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 46: 38 basically like how about we can only do masking and unmasking, right? So in the masking and unmasking, right? So in the masking and unmasking, right? So in the forward process all you can do is you fo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 47: 20 like basically the only two columns that you're going to have non- negative value you're going to have non- negative value you're going to have non- negative value is the columns on the diagonal and t…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 48: 06 >> But if we go all the way end of the day everything will be masked. everything will be masked. everything will be masked. >> Yeah. And it's still a distribution >> Yeah. And it's still a distributio…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 48: 55 multiplication and stuff anymore. Uh because we simplified everything, right? because we simplified everything, right? because we simplified everything, right? So basically what what we're essentially…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 49: 51 stay unmasked and then with one minus alpha t uh probability you're going to alpha t uh probability you're going to alpha t uh probability you're going to stay masked and uh basically if we if stay ma…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 51: 08 when uh xt equals x0 and x t minus one also equals x zero um like basically if also equals x zero um like basically if also equals x zero um like basically if if you have not been masked uh at time t …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 52: 02 simplified into this really really simple uh ratio. And similarly simple uh ratio. And similarly simple uh ratio. And similarly because the only other option for you is because the only other option f…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 53: 11 over. You should you should have the same value because it hasn't been same value because it hasn't been same value because it hasn't been masked. And if XT is masked, then you masked. And if XT is ma…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 54: 23 times the clean data estimation. times the clean data estimation. That's it. >> We have that problem because we'll start >> We have that problem because we'll start with full mask. So if you start wit…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 55: 26 >> So you're always starting with the full mask. mask. mask. >> That's right. >> That's right. >> That's right. >> So if that one reverse gets you to I >> So if that one reverse gets you to I >> So if…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 56: 12 actually a good point. Basically, uh the the the mercury um founder, Inception the the mercury um founder, Inception the the mercury um founder, Inception Lab founder, Stephano, who is the one of Lab …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 57: 09 you can also do like continuous time training with this. You literally just training with this. You literally just training with this. You literally just take the continuous time limit of the of take …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 57: 58 >> uh you yes kind of so it's the same thing it's uh like analogy to you choose thing it's uh like analogy to you choose thing it's uh like analogy to you choose the number of sampling steps right in …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 58: 42 this is purely depending on like the the this is purely depending on like the the model that you got trained but usually model that you got trained but usually model that you got trained but usually i…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 59: 34 basically the mask ratio is depending on this alpha t thing which you set. So you this alpha t thing which you set. So you this alpha t thing which you set. So you you can you can choose uh but like y…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 60: 34 the tokens. the tokens. >> Yeah. >> Yeah. >> But then in the last you have something >> But then in the last you have something >> But then in the last you have something like XT equal to M. So like X…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 61: 16 >> uh what do you mean by latent text well like so you mean like like so you mean like like so you mean like a region space for images and then use a region space for images and then use a region spac…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 62: 02 >> Does it produce something coherent if >> Does it produce something coherent if all the things are independently getting all the things are independently getting all the things are independently get…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 62: 49 >> It is pretty much just birth literally if you think about it. Yeah. if you think about it. Yeah. if you think about it. Yeah. >> Well, between like >> Well, between like >> Well, between like >> yo…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 63: 40 when they when he said like oh we can correct hallucination but like here we correct hallucination but like here we correct hallucination but like here we cannot really it's the same as cannot really …
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 64: 24 like a sort of like a like a like a like a interesting moment in science. But if a interesting moment in science. But if a interesting moment in science. But if you're one of the authors, you may get,…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 65: 08 >> very low because it's 2024. Okay. Anyway um but since like you know people show um but since like you know people show um but since like you know people show saw the potential of this they have saw…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 65: 55 say and seems like super compatible uh say and seems like super compatible uh to comparable to like large language to comparable to like large language to comparable to like large language models like…
cmu-10799-diffusion-flow L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching 66: 39 Please come up and uh can get rich as well. All right, see you guys next well. All right, see you guys next well. All right, see you guys next class.