概念 · 对比轴

AR vs Diffusion

— 文字坚守"逐 token 自回归",图像却倒向"并行迭代降噪",这是为什么。

很多人以为"图像用 diffusion、文字用 AR"是模态的天命——好像像素天然要并行、词元天然要顺序。其实不是。Vizuara 课程的 Dr. Raj Dandekar 在 Lecture 5 直接把这件事当成悬疑提出来:

「为什么扩散只能用于图像?我们脑子里是不是有偏见?Andrej Karpathy 也说过——大多数图像生成工具用的是扩散而不是自回归,只有文本被落下了。这对我和很多人都是一个谜。这种反直觉的问题,正是研究者应该问的。」—— Dr. Raj Dandekar,diffusion-lm-vizuara L5 (00:02)

这一句"反直觉问题"就是整门课的引子。如果 diffusion 在图像上效果惊艳,那它在文本上失败的原因,不该是"模态不同"这种含糊的答案,而要落到具体的归纳偏置上。Dr. Raj 给出的研究纲领是:把 AR 拆到矩阵维度级别,再看 diffusion 替换了哪三件事——其余 80% 与 AR 共享。

「事实上,扩散语言模型的架构与传统语言模型 80% 相同,只多了三个概念。所以要从零搭扩散语言模型,必须先深入 AR——而且是从每一个小模块的矩阵维度开始的那种深入。」—— Dr. Raj,L7 (00:03)

放到机器学习的训练语境里:AR 的 loss 是逐位置的 cross-entropy,target 是右移一位的输入;Diffusion 的 loss 是给所有位置同时加噪后预测被遮蔽部分。两者优化目标都是最大似然,区别在分解方式不同——这一点不要被"图像 / 文本"的表面差别带跑偏。

Dr. Raj 在 L1 (00:03) 还坦白了自己的学习曲线:「开始我也觉得扩散语言模型很难,前期花了非常长的时间才弄懂;但等到最后从零搭出来、并和 AR 做对照实验时,事情变得相当直接和简单。」这句话的潜台词是——AR 与 Diffusion 不是两个独立宇宙,它们在矩阵维度上是同一套零件的两种拼装方式。这门课最终交付的"扩散语言模型 from scratch"代码,与传统 AR Transformer 共享 tokenizer、embedding、attention 块、layer-norm、shortcut connection、feedforward 等几乎所有组件——只在训练目标、时间步条件、注意力遮蔽方式这三件事上分叉。

分解一:概率分解的方向

AR 把联合分布按时间链式法则展开:$p(x_1, \dots, x_N) = p(x_1) \cdot p(x_2 \mid x_1) \cdots p(x_N \mid x_{<N})$。Dr. Raj 在 L6 (00:03) 用最朴素的例子说明:

「语言模型是按『next token prediction』任务训练的。给一个 prompt『once upon a time』,模型每次只预测下一个 token——一次一个。推理时也一样。」—— L6

Why:这个分解的代价是"必须从左到右生成",但好处是每一步的条件分布 $p(x_i \mid x_{<i})$ 都是低维 softmax(词表大小 ≈ 5 万),且监督信号密集——句子里每个位置都贡献一个 loss。

Diffusion 反过来:不分解时间,而是按噪声等级分解。给定干净样本 $x_0$,定义 forward 过程加 $T$ 步噪声到 $x_T \sim \mathcal{N}(0, I)$,模型学反向 $p_\theta(x_{t-1} \mid x_t)$。生成时整段同时 denoise,每一步所有位置一起更新。

Why:这种分解的代价是 reverse 链长($T$ 步前向),好处是并行——一次前向同时刷新所有位置,且训练时可以随机采样 $t$,不必从头跑到尾。

把 AR 写成显式公式更直观:

$$p_\theta(x_1, \dots, x_N) = p_\theta(x_1) \cdot \prod_{i=2}^{N} p_\theta(x_i \mid x_1, \dots, x_{i-1})$$

训练时用 teacher-forcing,所有位置同时算 loss;推理时必须串行——这是 AR 的根本不对称:训练并行、推理串行。Diffusion 反过来:训练时只在一个 $t$ 上算 loss("串行"采样 $t$),但推理时所有位置并行刷新。这个不对称在硬件层面就是 GPU 利用率的差异——Diffusion 推理几乎不浪费 SM,AR 推理大量时间在等待 KV-cache 的串行依赖。

分解二:推理代价对比

这是最容易被搞混的轴。看具体数字:

AR·生成 1024 token 句子 → 1024 次前向。KV-cache 可省内存但前向次数砍不掉。
Diffusion·生成同长度 → 通常 14–75 步(DDPM 原版 1000,DDIM/DPM-Solver 降到几十)。每步并行刷新全部 1024 个位置。

Why:AR 的前向次数与序列长度线性绑定;Diffusion 的步数与序列长度解耦。这就是为什么 LLaDA、Mercury Coder、Cornell MDLM(Sahoo 2024)这类扩散语言模型能在长序列时做出 4–10× 推理加速的承诺。

🟡 教材外补充:Cornell 的 MDLM(Sahoo et al. 2024, NeurIPS)证明了 absorbing-state 掩码扩散在简化损失下能逼近 GPT-2 perplexity(Wikitext-103 约 23.4 vs GPT-2 的 22.8),把"扩散语言模型一定打不过 AR"这条经验定律打破了。它选择的步数大约是 $T = 1000 \to$ 实测 14 步即可逼近最终质量。

分解三:监督信号的密度

AR 一个 batch 的句子里,每个位置都贡献一个 cross-entropy(teacher-forcing),监督信号密集。这就是为什么 AR 在小数据上也能学得动——Vizuara 课程的 L11 (00:02) 演示了在 TinyStories 这种几百 MB 量级的小语料上,从零搭的 AR 也能学出连贯生成。

「我想先告诉你,基于传统 ARM 架构,我们完全可以从零搭出一个小语言模型。整个 pipeline 是:拿 TinyStories 数据 → tokenize → 构造输入-输出对 → 过架构 → 算 predictions 与 targets 的 loss → 反向更新参数 → 取下一个 batch,循环不息——这就是预训练。」—— Dr. Raj,L11 (00:02)

Diffusion 训练时只在一个随机采样的 $t$ 上算 loss,监督信号稀疏。换来的是任意位置都可以被遮蔽——天然支持infilling(中间补全),而 AR 必须左到右、不能从中间生成。从信息论看,AR 把 $N$ 个 token 的监督全部利用;Diffusion 在每个 $t$ 上只利用一部分(被掩码的位置)。Why:这是用监督密度换得方向自由度。MDLM 通过 simplified loss + linear schedule 在实际工程里把这种稀疏性的负面影响压到很小。

AR:一次一个 token,从左到右 x₁ x₂ x₃ x₄ … x_N 前向次数 = N(与序列长度线性绑定) Diffusion:所有位置同时去噪,T 步 前向次数 = T ≈ 14–75(与序列长度无关)

分解 3.5:context vector 是公共货币

不论 AR 还是 Diffusion,注意力的产出都叫 context vector——比原始 token embedding 含义更丰富,因为它编码了"这个位置该被多大权重关注它的邻居"。Dr. Raj 在 L9 (00:04) 给的直觉:「现在如果我看 "bright" 这个 token,我只是把它的 token embedding 与 position embedding 加起来——我完全没有线索该给 'is'、'day'、'next'、'the' 等邻居分配多少注意力。context vector 的意义就是把这些注意力关系编码进每个位置的向量里。」

关键洞见:context vector 这块"公共货币"是模态无关的——文本、图像 patch、音频帧都可以被同一套 attention 算 context vector。AR 与 Diffusion 在这一步是完全一致的;分叉只发生在如何用这个 context vector:AR 用它预测下一个 token,Diffusion 用它预测被掩码位置的原 token。这又一次印证 80%/20% 的分配。

分解四:为什么文字"卡"在 AR

Dr. Raj 在 L5、L6 给出的非正式回答有两层。第一层:文字是离散低维 token——L8 (00:02) 强调机器学习模型不能直接理解文本,必须先 tokenize 成 word ID 再 embedding,每位置一个数值索引,softmax 直接出概率,没有连续噪声的天然栖息地。第二层:图像被 VAE 编码成连续高维 latent(例如 Stable Diffusion 的 4 × 64 × 64 = 16384 维),扩散在这种空间里自然——每一步加的 Gaussian 噪声有明确语义,"加噪 ε" 在欧氏空间里就是平移、不会跳出 manifold。

把这两层组合起来:文本的预 tokenize 与图像的VAE 高维 reframing 是同一类操作,都是把原始数据搬到模型能处理的向量空间——但向量空间的拓扑很不同。文本 token 是离散码本,无法定义"在 hello 和 hella 之间走 0.3 步";图像 latent 是连续流形,可以平滑插值。这才是 diffusion 在两种模态上待遇悬殊的根因——不是模态本身,是 tokenize 之后的表示空间几何。

🟡 教材外补充——Karpathy 自己的回答(X 2024-09):「图像和视频更接近感官数据,文字更接近已经被压缩过的高熵符号。在高熵符号上做 diffusion 等于先把已经压紧的东西再松开来去噪,吃力不讨好。」这与课程里的"tokenize→高维 reframing"互为补充:MDLM 选的妥协就是不去重新连续化,而是在离散 token 空间做 absorbing-state masking,等价于 BERT 式的迭代填空——把每个位置要么保留要么替换成 [MASK],反向时预测原 token。这绕开了"在离散码本上加 Gaussian 噪声没有几何意义"这个根本难题,让扩散语言模型终于追上 AR 的 perplexity。

分解五:reversal curse 与双向访问

AR 的左到右分解有个被广泛报告的缺陷——reversal curse(Berglund et al. 2023):训练时只见过 "Tom Cruise 的妈妈是 Mary Lee Pfeiffer",模型可能答不出反过来"Mary Lee Pfeiffer 的儿子是谁"。Why:AR 的条件概率链是单向的,"$y$ 在 $x$ 之后"和"$x$ 在 $y$ 之后"是两个不同的条件分布,必须分别学。这不是参数量的问题——GPT-4 同样会犯,因为它的训练目标本身就只优化了"已知前缀预测后缀",反向条件没有被显式监督。

Diffusion 没有这个方向偏置——每个位置都可以从噪声/掩码态被任意上下文条件化恢复。这是 MDLM 论文里强调的、AR 无法天然获得的属性。在掩码扩散里,模型同时看到句子里 Mary Lee Pfeiffer 两侧的所有未掩码 token,"儿子"和"妈妈"的对应关系被双向监督。

🟡 教材外补充:reversal curse 不是死刑——可以靠数据增强(双向重写训练样本)或在推理时加 prefix scan 缓解,但训练成本会涨。Diffusion 把这个问题在架构层解决,代价是监督密度下降。这是典型的 no-free-lunch。把它和分解三对照看更清楚:AR 用密度换方向、Diffusion 用方向换密度,两者在 ELBO 上殊途同归。

分解六:共享的 80% 与分叉的 20%

这是 Dr. Raj 把 AR 拆到矩阵维度的真正回报。一个 AR Transformer block 的标准流水线(L8–L11 一字一句搭出来的)是:输入 token IDs → token embedding + position embedding → LayerNorm → 多头注意力(带 causal mask)→ Dropout → shortcut connection → LayerNorm → 前馈网络(768 → 3072 → 768 的扩张-收缩)→ Dropout → shortcut → 输出层映射到词表大小。维度始终保持 768。

「现在我们处于这样一个阶段:输入向量已经被转成了 context vector——比原始 token embedding 含义更丰富,因为它还编码了每个 token 与邻居的关系。出来之后接 dropout(借鉴自深度学习,把一些维度随机置零)、shortcut connection、第二个 layer-norm……整个 embedding size 是 768,layer-norm 不变、attention 不变、dropout 不变、shortcut 不变,全程维度守恒。」—— Dr. Raj,L10 (00:03 + 01:37)

Diffusion 语言模型替换/补足的是:(1) 输入端额外注入时间步 embedding $t$;(2) attention 的 causal mask 改成双向(或仅遮蔽未确定位置的 absorbing mask);(3) 输出层从"预测下一个 token"改为"预测被掩码位置的原 token"。这就是 Dr. Raj 说的"多三个概念"。其余 80%——tokenizer、embedding、attention block、layer-norm、shortcut、feedforward——一行代码不改。

🟡 教材外补充:dropout 与 shortcut 不是装饰。Dr. Raj 用"四人小组项目里有两个划水的,如果劳模请假其他人不得不顶上"来类比 dropout 提升泛化(L10, 00:55);shortcut 是为了"给梯度一条额外通路,防止反向传播时梯度连乘后消失"(L10, 02:26)。这些 AR 时代的工程技巧在 diffusion 语言模型里被原样继承——这正是"80% 共享"在工程上的具体含义。

下面按钮按顺序点击,注意看不变量:两种分解最终都在最大化 $\log p(x)$——只是路径不同。AR 把对数似然按时间拆,Diffusion 按噪声等级拆。

起点:$p_\theta(x_1, x_2, \dots, x_N)$ —— 我们要建模长度为 $N$ 的样本的联合分布。
AR 路线:$\log p_\theta(x) = \sum_{i=1}^{N} \log p_\theta(x_i \mid x_{<i})$ —— 时间链式法则,每位置一个 softmax。
Diffusion 路线:$\log p_\theta(x_0) \ge \mathbb{E}_{q}\!\left[-\sum_{t=1}^{T} D_{\mathrm{KL}}\!\big(q(x_{t-1}|x_t,x_0) \,\|\, p_\theta(x_{t-1}|x_t)\big)\right] + \text{const}$ —— ELBO,按噪声等级拆。
共同目标:两种都在最大化 $\log p_\theta(x)$。AR 取等号,Diffusion 用下界——但实操中 ELBO 已足够紧。

手算:生成 4 token 句子的代价

设序列长度 $N = 4$,扩散步数 $T = 14$(MDLM 在 Wikitext-103 上的实测值),词表大小 $V = 50000$,模型隐藏维度 $d = 768$(沿用 Dr. Raj 课程的设定)。

AR:前向次数 = $N = 4$。每次输出一个长度为 $V$ 的 logits 向量。总 logits 计算 = $N \cdot V = 4 \cdot 50000 = 200{,}000$ 次 softmax 比较。生成必须串行——第 $i$ 个 token 需要 $x_{<i}$ 已经被采出来才能算,无法并行。

Diffusion (MDLM):前向次数 = $T = 14$。每次同时刷新所有 4 个位置。总 logits 计算 = $T \cdot N \cdot V = 14 \cdot 4 \cdot 50000 = 2{,}800{,}000$ 次——更多。但因为同一步内 4 个位置并行,GPU wall-clock 时间 ≈ $T \cdot t_\text{fwd} = 14 \cdot t_\text{fwd}$,AR 是 $N \cdot t_\text{fwd} = 4 \cdot t_\text{fwd}$。$N=4$ 时 AR 反而快 3.5 倍。

把 $N$ 拉到 1024:AR = $1024 \cdot t_\text{fwd}$,Diffusion 仍然 = $14 \cdot t_\text{fwd}$。交叉点满足 $N = T$,即 $N \ge 14$ 后 Diffusion 占优。这就是 MDLM、Mercury Coder 这类扩散语言模型主打长序列代码 / 长文档场景的根本原因——短回答场景反而是 AR 的舒适区。

等式链给出理论上限:speedup $= N / T = 1024 / 14 \approx 73.1$ 倍。实测 MDLM 的端到端加速大约在 4–10 倍,原因有三:(1) 每步前向需要额外的 timestep embedding 和并行更新带来的 attention pattern 调整;(2) AR 可以靠 KV-cache 把每步的实际计算量降到 O($d^2$) 而非 O($N \cdot d^2$);(3) batch 大小不同时 GPU 利用率差别巨大。把这三项算回去后,理论 73× 缩到 4–10× 是合理的——这条估算 Dr. Raj 在课程末尾用具体 PyTorch 代码跑出来的数字也吻合。

常见误区

  1. "diffusion 总是比 AR 慢,因为要 T 步"。反例:$N = 1024$、$T = 14$ 时,Diffusion 前向次数只有 AR 的 1/73。慢/快取决于 $N$ 与 $T$ 的相对大小,不是绝对的。常被搞混的根本原因是把图像 diffusion 早期 $T = 1000$ 的经验外推到了语言场景——但 MDLM 论文用 absorbing-state 把有效步数压到 14,与 GPT-2 接近 perplexity。一旦记住"步数 $T$ 不一定大于序列长度 $N$",这条误区就破。
  2. "AR 模型不能做中间填空"。反例:可以——但只能靠特殊训练(如 FIM, fill-in-the-middle,OpenAI 2022;StarCoder、Code Llama 也都用了这个技巧)+ 重新组织 token 顺序:把句子从 `prefix + middle + suffix` 改写成 `prefix + suffix + <FIM> + middle`,本质是把"中间"重排成"末尾"绕过分解方向。原生、零样本支持中间生成是 Diffusion 的天然优势——任意位置都可以从掩码态被同一套权重恢复,不需要重排训练数据。
  3. "扩散语言模型必然比 GPT 弱"。反例:Cornell MDLM(Sahoo et al. NeurIPS 2024)在 Wikitext-103 上 perplexity 23.4,与同规模 GPT-2(22.8)几乎齐平;Inception Labs 的 Mercury Coder 在 HumanEval 上接近 GPT-3.5,推理速度 5× 提升;Google DeepMind 的 LLaDA 系列也证明了同等参数下 zero-shot 任务可以与 LLaMA-2 拼。这条经验定律已经在 2024 年被证伪。它过去成立是因为早期连续扩散在离散文本上有几何 mismatch,而不是因为扩散本身有理论上限——一旦切换到 absorbing-state 离散扩散,这条定律就失效了。

相关概念

  • Diffusion Language Model —— 本对比的"另一极";具体看 MDLM 用什么训练目标补足 AR 在双向访问与并行解码上的短板。

来源

  1. diffusion-lm-vizuara · L5 · Motivation behind Language Diffusion Models · (00:02) —— Dr. Raj 抛出"为什么文本被落下"的悬疑;引用 Karpathy。
  2. diffusion-lm-vizuara · L6 · ARMs Introduction · (00:03) —— 「next token prediction」白板讲解,"once upon a time" 例子。
  3. diffusion-lm-vizuara · L7 · ARM Architecture Intro · (00:02) —— 「扩散语言模型 80% 与 AR 相同,只多三个概念」的纲领句。
查看 114 条课程原文出处
CourseLectureTimeExcerpt
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 00: 03 So in attention as I mentioned the main So in attention as I mentioned the main idea is to have these token embeddings idea is to have these token embeddings idea is to have these token embeddings or …
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 00: 52 to capture if I'm looking at a particular token if I'm looking at a particular token if I'm looking at a particular token like brightite, how much attention I want to capture all of that I want to cap…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 01: 50 vectors, value vectors, etc. So what is vectors, value vectors, etc. So what is done is that we have this input matrix, done is that we have this input matrix, done is that we have this input matrix, …
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 02: 33 with query key and values in the attention block. Right? attention block. Right? attention block. Right? Once we have the queries and the keys Once we have the queries and the keys Once we have the qu…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 03: 16 by 5 right 1 2 3 4 5 and let me duplicate this duplicate this duplicate this 1 2 3 1 2 3 1 2 3 4 and 5. 4 and 5. 4 and 5. And And And this is the this is the this is the next next next day is bright. …
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 04: 19 statements like if I'm looking at bright, bright, bright, I want to make statements like give 10% I want to make statements like give 10% I want to make statements like give 10% attention to the, give…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 05: 10 to or give 50% importance to bright. Um there is also the scaling by square Um there is also the scaling by square Um there is also the scaling by square root of this square root of keys root of this …
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 05: 48 multiply it with the value matrix and this gives me my context vector matrix. this gives me my context vector matrix. this gives me my context vector matrix. That's my final output from the That's my …
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 06: 45 So what I'll do is that I'll take the So what I'll do is that I'll take the values vector for each of these and values vector for each of these and values vector for each of these and multiply it with…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 07: 37 tokens. But now what I'm saying is that when we look at next I have information when we look at next I have information when we look at next I have information of how next relates to the other tokens.…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 08: 30 Hogwarts and he Hogwarts and he now I have encoded how he relates to now I have encoded how he relates to now I have encoded how he relates to these different tokens. these different tokens. these dif…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 09: 30 So, we are doing the next token prediction task here. Um, so if next is prediction task here. Um, so if next is prediction task here. Um, so if next is the input, I want to predict what comes the inpu…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 10: 11 is no need of finding the attention score between is and bright. But for score between is and bright. But for score between is and bright. But for bright it's fine because all the other bright it's fi…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 10: 59 essentially, I only need to find the attention scores of what comes in this attention scores of what comes in this attention scores of what comes in this triangle. I don't need these attention triangl…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 11: 42 these attention score values are these attention score values are essentially set to zero because we don't essentially set to zero because we don't essentially set to zero because we don't need them a…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 12: 28 attention mass does not exist for attention mass does not exist for diffusion models. In multi head diffusion models. In multi head diffusion models. In multi head attention what's actually done is th…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 13: 19 that causal attention mask here right we don't need these attention score so we don't need these attention score so we don't need these attention score so we put them to zero. That's why it's called p…
diffusion-lm-vizuara L10: Lecture 9: Auto Regressive Models (ARM) Attention Block 14: 01 explanation which I've tried to give explanation which I've tried to give you. If you want to go and drill down you. If you want to go and drill down you. If you want to go and drill down into the det…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 00: 03 So once we come out of the attention So once we come out of the attention block, we have a dropout layer. Uh let block, we have a dropout layer. Uh let block, we have a dropout layer. Uh let me me me …
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 00: 55 these vectors to zero. And the reason dropout is implemented is to improve the dropout is implemented is to improve the dropout is implemented is to improve the generalization performance. I always ge…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 01: 37 the generalization performance, right? U so after dropout we again have these U so after dropout we again have these U so after dropout we again have these context vectors. Now take a look at the cont…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 02: 26 we want to avoid the problem of vanishing gradients. vanishing gradients. vanishing gradients. The thing is if multiple gradients The thing is if multiple gradients The thing is if multiple gradients …
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 03: 23 So we have to go from here to now here So we have to go from here to now here which is the second layer normalization. which is the second layer normalization. which is the second layer normalization.…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 04: 08 The activation layer here which is generally preferred is JLU activation. generally preferred is JLU activation. generally preferred is JLU activation. The way it differs from RLU is that uh The way i…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 04: 58 been borrowed and which show up repeatedly. So which are these elements? repeatedly. So which are these elements? repeatedly. So which are these elements? You'll see that the dropout layer is You'll s…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 05: 44 output to the transformer block. The output of the transformer block is this. output of the transformer block is this. output of the transformer block is this. Again, we have these tokens with the Aga…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 06: 55 build neural networks from scratch, I build neural networks from scratch, I think. Let's see. Or build LLM from think. Let's see. Or build LLM from think. Let's see. Or build LLM from scratch new, I t…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 08: 24 block goes to the second the output of the second goes to the third. The output the second goes to the third. The output the second goes to the third. The output of the third goes to the fourth to the…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 09: 05 say transformer block output it's the say transformer block output it's the output from those 12 24 or 96 output from those 12 24 or 96 output from those 12 24 or 96 transformer blocks that output goe…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 10: 04 moves you right. This has as we have seen this is one input sequence but it seen this is one input sequence but it seen this is one input sequence but it actually has four input output actually has fo…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 11: 01 training. Right? Initially I would have some random random guesses. So this is some random random guesses. So this is some random random guesses. So this is the target and I need to have the target an…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 11: 47 and the predictions. Okay. So, how do we get to the predictions? Currently we get to the predictions? Currently we get to the predictions? Currently we just have uh the output of this just have uh the…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 12: 34 that's the vocabulary size which will be that's the vocabulary size which will be the number of columns. the number of columns. the number of columns. So you have something like every effort moves U a…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 13: 41 vector it will make sure that each value vector it will make sure that each value lies between 0 to one and they sum up to lies between 0 to one and they sum up to lies between 0 to one and they sum u…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 14: 36 maybe token ID number 40,000 which is a boy So these are my predictions now and then So these are my predictions now and then I'll find the loss function between the I'll find the loss function betwee…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 15: 34 each other. As the target and the prediction start to approximate each prediction start to approximate each prediction start to approximate each other, what will happen is that the other, what will ha…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 16: 27 model is trained properly the model is trained properly the probability at token ID number three probability at token ID number three probability at token ID number three should be equal to one should…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 17: 29 my loss will be zero which is what we want. But in this case the loss will be want. But in this case the loss will be want. But in this case the loss will be negative log of2. negative log of2. negati…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 19: 33 probability which should actually be one but maybe it's.3 if every effort moves but maybe it's.3 if every effort moves but maybe it's.3 if every effort moves is the input U should be the output. is th…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 21: 26 if we take negative of this it will be if we take negative of this it will be equal to 2.619 619 that's the loss for equal to 2.619 619 that's the loss for equal to 2.619 619 that's the loss for this …
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 22: 14 models are trained. I hope all of you now see the end to end details of why now see the end to end details of why now see the end to end details of why they are called as auto reggressive they are cal…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 22: 55 All right. So if you take a look at the parameters now I hope all of you can parameters now I hope all of you can parameters now I hope all of you can visualize where all there are parameters visualiz…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 23: 50 is the positional embeddings also a is the positional embeddings also a parameter? Because we have the token parameter? Because we have the token parameter? Because we have the token embedding matrix …
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 24: 49 attention mechanism. So that's also a parameter. If you scroll about to the parameter. If you scroll about to the parameter. If you scroll about to the attention mechanism, you'll see uh WQ, attention…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 25: 56 this is the feed forward neural network. So we have 768 dimensions going to 4 So we have 768 dimensions going to 4 So we have 768 dimensions going to 4 into 768 and then that again going to into 768 a…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 26: 44 parameters. We have parameters in the final layer norm. And in the output final layer norm. And in the output final layer norm. And in the output layer, why do we have parameters here? layer, why do w…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 27: 46 contributes to the parameters in the output layer. So all of these parameters output layer. So all of these parameters output layer. So all of these parameters termed as P are the trainable parameters…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 28: 31 start of this that to truly understand start of this that to truly understand diffusion language models, we really diffusion language models, we really diffusion language models, we really need to fir…
diffusion-lm-vizuara L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks 29: 20 we'll see how the ARM architecture is the backbone of this predicting noise the backbone of this predicting noise the backbone of this predicting noise step. So let's start looking at that step. So le…
diffusion-lm-vizuara L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference 00: 02 Before moving on to the fusion model Before moving on to the fusion model architecture architecture architecture and uh assembling it from scratch, and uh assembling it from scratch, and uh assembling…
diffusion-lm-vizuara L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference 00: 59 continues. That's pre-training of continues. That's pre-training of language models or traditional language language models or traditional language language models or traditional language model pre-tr…
diffusion-lm-vizuara L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference 01: 50 we get the token embeddings we get the we get the token embeddings we get the position embeddings we add the token position embeddings we add the token position embeddings we add the token embeddings …
diffusion-lm-vizuara L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference 02: 40 function like the way we have um looked at on the whiteboard at on the whiteboard at on the whiteboard right we saw that once we pass the input right we saw that once we pass the input right we saw th…
diffusion-lm-vizuara L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference 03: 24 essentially the context length or the sequence length. That's the number of sequence length. That's the number of sequence length. That's the number of tokens in one tokens in one tokens in one input …
diffusion-lm-vizuara L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference 04: 17 not. Right? The whole idea when we started this exercise was started this exercise was started this exercise was um we wanted to train the model so that um we wanted to train the model so that um we w…
diffusion-lm-vizuara L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference 04: 58 completely correct here. There are some mistakes here and there, but overall the mistakes here and there, but overall the mistakes here and there, but overall the grammar makes sense. The meaning also…
diffusion-lm-vizuara L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference 05: 40 exactly, but mostly it's pretty cool. So auto reggressive models definitely work auto reggressive models definitely work auto reggressive models definitely work and they give good results. Right now w…
diffusion-lm-vizuara L2: Lecture 1: Introduction to Diffusion Language Models 00: 03 Hello everyone. This is Dr. Raj Dandeek Hello everyone. This is Dr. Raj Dandeek and uh today I'm very excited to present and uh today I'm very excited to present and uh today I'm very excited to prese…
diffusion-lm-vizuara L2: Lecture 1: Introduction to Diffusion Language Models 01: 01 models from scratch. But that's not how models from scratch. But that's not how I want this course or this learning I want this course or this learning I want this course or this learning material to …
diffusion-lm-vizuara L2: Lecture 1: Introduction to Diffusion Language Models 01: 49 scared of them, then how I learned the scared of them, then how I learned the subject, how I mastered the subject now subject, how I mastered the subject now subject, how I mastered the subject now so…
diffusion-lm-vizuara L2: Lecture 1: Introduction to Diffusion Language Models 02: 27 model fully from scratch. You'll compare it with the auto reggressive language it with the auto reggressive language it with the auto reggressive language models. So it's an amazing exercise or models…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 00: 02 So I hope all of us are on the same page So I hope all of us are on the same page now with respect to the motivation now with respect to the motivation now with respect to the motivation behind this l…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 00: 50 questions which researchers should ask that leads to the best research. that leads to the best research. that leads to the best research. If you ask this question, you'll slowly If you ask this questi…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 01: 49 models. This so this paper I believe is a very important one mask diffusion is a very important one mask diffusion is a very important one mask diffusion language models this is where many language mo…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 02: 51 introduction section itself uh that why should images and video uh that why should images and video uh that why should images and video generation models have all the fun why generation models have al…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 03: 39 images might lie and then sampling from it. So why can't this same thing be it. So why can't this same thing be it. So why can't this same thing be applied to text as well? applied to text as well? ap…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 04: 58 when we talked about images, first we broke down the images into pixels, broke down the images into pixels, broke down the images into pixels, right? Similarly, text can be broken right? Similarly, te…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 05: 54 So one way to look at generating text is So one way to look at generating text is just one word at a time. Another way of just one word at a time. Another way of just one word at a time. Another way o…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 06: 57 in high dimensional spaces? What does it mean probability distribution of mean probability distribution of mean probability distribution of meaningful text? Let's see. So text is trickier than images.…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 08: 12 sequence of words based on two things. sequence of words based on two things. Grammar and logic both need to be Grammar and logic both need to be Grammar and logic both need to be correct, right? And …
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 09: 12 can be carried forward to languages. can be carried forward to languages. What these people suggested is auto What these people suggested is auto What these people suggested is auto reggressive models…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 00: 03 To understand the concept of auto To understand the concept of auto reggressive models, reggressive models, reggressive models, let me go to the whiteboard and uh let me go to the whiteboard and uh le…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 01: 37 upon a time and I want the language model to complete this prompt. model to complete this prompt. model to complete this prompt. What it will do is that it will not What it will do is that it will not…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 02: 23 sequence then the next token is predicted this is during the training predicted this is during the training predicted this is during the training during inference also one new token is during inferenc…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 03: 06 see multiple tokens here, it's one token being generated at a time and the next being generated at a time and the next being generated at a time and the next token which is generated depends on the to…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 03: 58 one thing which works. But why can't diffusion be the default option? This is diffusion be the default option? This is diffusion be the default option? This is what Andre Karpati also mentions here, w…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 04: 46 predicts one token at a time. So if you see to generate the same amount of text see to generate the same amount of text see to generate the same amount of text the right hand side which is diffusion t…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 05: 34 it's fine because it's one new token generated at a time but this reverse generated at a time but this reverse generated at a time but this reverse reasoning becomes very difficult for the reasoning b…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 06: 21 mentioned is that if the true language distribution lies somewhere here can we distribution lies somewhere here can we distribution lies somewhere here can we find the model can we make a diffusion fi…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 07: 13 what's the need of auto reggressive what's the need of auto reggressive models to achieve this to make the models to achieve this to make the models to achieve this to make the probability distributio…
diffusion-lm-vizuara L7: Lecture 6: Auto Regressive Models (ARMs) Introduction 08: 02 ure of diffusion models differs from ure of diffusion models differs from auto reggressive models and then we are auto reggressive models and then we are auto reggressive models and then we are going …
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 00: 02 To truly understand To truly understand diffusion language models, we actually diffusion language models, we actually diffusion language models, we actually really need to understand how auto really n…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 01: 07 small aspect like the feed forward small aspect like the feed forward neural network block here etc. So this neural network block here etc. So this neural network block here etc. So this is going to b…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 02: 05 model. This data set essentially consists of around 2 million stories consists of around 2 million stories consists of around 2 million stories which are such that 3 to four year old which are such th…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 02: 51 so that I can predict or I can generate language. language. language. And what does what does it mean? So the And what does what does it mean? So the And what does what does it mean? So the goal is to…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 03: 58 want to do. And what does it mean learn patterns from the data? Well, the model patterns from the data? Well, the model patterns from the data? Well, the model needs to understand about the English ne…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 04: 46 the kicked. That means it has not understand not understood the correct understand not understood the correct understand not understood the correct grammatical constructions. And the model grammatical…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 05: 40 Yeah. So how do we get started with this task? how do we get started with this task? how do we get started with this task? The first step is to look at our data The first step is to look at our data T…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 06: 49 size. It defines how many tokens or how size. It defines how many tokens or how many words the model can pay attention many words the model can pay attention many words the model can pay attention to …
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 07: 38 if this is my first input sequence, right? I shift it to the right hand side right? I shift it to the right hand side right? I shift it to the right hand side by one. If this is my second input by one…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 08: 38 model to learn that given any token it model to learn that given any token it has to predict the next token. So if one has to predict the next token. So if one has to predict the next token. So if one…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 09: 24 input, little will be the output. And if one day a little is the input, girl will one day a little is the input, girl will one day a little is the input, girl will be the output. be the output. be the…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 10: 24 so imagine the huge data set which we had 2 million rows that's but that's had 2 million rows that's but that's had 2 million rows that's but that's coupled into in let's say input sequence coupled in…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 11: 19 and the true answer, I have my loss function. Right? Once I have my loss function. Right? Once I have my loss function. Right? Once I have my loss function, I find the gradient of the function, I find…
diffusion-lm-vizuara L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro 12: 10 Um, then what I'll do is that I go through my second batch, I get the loss, through my second batch, I get the loss, through my second batch, I get the loss, I update the parameters, then I go I updat…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 00: 02 So the way a batch is constructed is as So the way a batch is constructed is as follows. Let's say for now we just follows. Let's say for now we just follows. Let's say for now we just assume the batc…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 00: 50 the third stage is the output stage. When we look at diffusion language When we look at diffusion language When we look at diffusion language models, there are changes in all of models, there are chan…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 01: 29 first is that since the model cannot understand text these tokens these are understand text these tokens these are understand text these tokens these are called as tokens they are converted called as …
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 02: 27 is given a token ID of 1 then this is 12 is given a token ID of 1 then this is 12 then this is 50 and this is 80. then this is 50 and this is 80. then this is 50 and this is 80. These token ids are th…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 03: 12 model is not trained the vectors for cat and kitten might lie further apart from and kitten might lie further apart from and kitten might lie further apart from each other. But once the language model…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 04: 01 This is my whole vocabulary set. I assign token ids to each of them. Let's assign token ids to each of them. Let's assign token ids to each of them. Let's say this goes up till 100,000. This is say th…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 04: 55 If I want to retrieve the embedding vectors for these, I just go to my token vectors for these, I just go to my token vectors for these, I just go to my token embedding matrix. I look at that embeddin…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 05: 52 the same words. So their token embedding vectors will be the same but they come vectors will be the same but they come vectors will be the same but they come at different positions, right? So their at…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 06: 45 length. So this goes from one to right up to the sequence length. And I have a up to the sequence length. And I have a up to the sequence length. And I have a 768 dimensional vector for my position 76…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 07: 44 corresponding to position one this is the vector vector corresponding to the vector vector corresponding to the vector vector corresponding to position two, vector corresponding to position two, vecto…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 08: 28 block is where all the magic happens. So initially we started out by uh thinking initially we started out by uh thinking initially we started out by uh thinking that how will the model learn the that …
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 09: 13 are also calling processor here. are also calling processor here. This transformer block consists of all This transformer block consists of all This transformer block consists of all of these componen…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 09: 58 by another layer normalization followed by feed forward followed by dropout. The by feed forward followed by dropout. The by feed forward followed by dropout. The good thing is that the transformer go…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 10: 46 that so now the input tokens let's say so ideally we should we should have so ideally we should we should have so ideally we should we should have continued with the same one day a little continued wi…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 11: 28 root of variance. Why do we do this? root of variance. Why do we do this? Because normalization really helps the Because normalization really helps the Because normalization really helps the back prop…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 12: 17 normalization. The second block which I normalization. The second block which I want to emphasize a lot over here. This want to emphasize a lot over here. This want to emphasize a lot over here. This …
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 13: 11 Harry Potter daughter or Harry lived or Harry lived or Harry lived at Hogwarts and he so let's say this is my input and he so let's say this is my input sequence if I want to complete this sequence if…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 14: 05 have not linked one token to another have not linked one token to another token anywhere. Even this uh layer token anywhere. Even this uh layer token anywhere. Even this uh layer normalization is an o…
diffusion-lm-vizuara L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks 14: 58 the mask diffusion models as well. So let's understand this attention let's understand this attention let's understand this attention mechanism and what it's actually doing mechanism and what it's act…