概念

Diffusion Language Model

— 用离散扩散过程替代自回归 next-token;并行去噪生成整段文本。

很多人第一眼看到「diffusion language model」会以为:那不就是 BERT 训练完拿来生成嘛? 不是。BERT 只在一个固定的 15% mask ratio 上学了一次预测,它无法构造一个连续的 $t=1\to 0$ 反向过程,因为反向链条上每一步对应的 mask 比例都不一样——从 100% 一路降到 0%,BERT 只见过其中一个切片。而 DLM 在所有 mask 比例上联合训练,因此能像图像扩散那样从「整段全是 [MASK]」的纯噪声出发,多步迭代去噪出完整句子。Karpathy 2025 年那条触发了整波研究热潮的推文戳的就是这件事——大多数你见到的 LLM 都是从左到右预测 token 的「克隆体」,扩散不一样,它不从左到右,它一次性把整段 token 流去噪出来。

「大多数 LLM 在核心建模上是克隆体——都是 auto-regressive、从左到右预测 token。Diffusion 不一样,它不从左到右,它一次性全部出。你从噪声开始,逐步去噪成 token 流。」—— Vizuara《Diffusion LLM》L2 @0:02(Karpathy 推文转述)

Vizuara 的 Dr. Raj 在 L5 抛出了贯穿整门课的灵魂提问。他把这个问题当成研究者应该养成的「反直觉」思维训练例:明明扩散在图像、视频上已经几乎取代了所有自回归方法,为什么唯独文本死守 next-token prediction?

「为什么图像偏好 diffusion,文本偏好 auto-regression?这件事对我和很多研究者来说至今是个谜。文本是唯一被甩在后面的模态。我们脑子里是不是有某种偏见?这正是研究者应该问的那种反直觉问题。」—— Vizuara《Diffusion LLM》L5 @0:02

《Principles of Diffusion Models》课程从图像扩散的成功一路推过来,到这门姊妹课 DLM 时把这个问题正式拎出来作为切入点——课程开篇的逻辑链条很清晰:2022 年 Midjourney 和 Stable Diffusion 让 generative AI 在图像上彻底完成 paradigm shift,而文本的对应转变直到 2024–2025 年才借 MDLM、SEDD、LLaDA 这一波工作真正发生。

把概念定位回 ML 上下文:在训练里,DLM 对应一种全新的语言模型范式——decoder 不再是 causal Transformer,而是双向 attention的 Transformer,输入是带 [MASK] 的整段序列,输出是每个被 mask 位置上的 vocabulary 分布。这把第 1 章那个「AR vs Diffusion」的悬念在语言模态上画了句号:2025 年 LLaDA-8B 出现以后,语言开始追上图像——它证明 diffusion 在 8B 参数、双向 attention 的设定下能匹配同 size 的 AR LLaMA-3,不再只是 toy 规模的概念演示。

从图像扩散迁移:三件套

Dr. Raj 反复强调 DLM 沿用图像扩散的「三件套」结构,只是把高斯噪声换成「mask 噪声」:

「我们要整合扩散模型的三个主要特性。第一,noising process;第二,用某种模型去预测被加上的噪声;第三,denoising process。在图像里 noising 是按 schedule 给图加高斯噪声,在文本里它变成把 token 替换成 [MASK]。」—— Vizuara《Diffusion LLM》L12 @0:03

1. 前向(加噪)过程:absorbing-state 链

最主流的 MDLM (Sahoo et al., Cornell 2024) 和 LLaDA (Nie et al. 2025) 走的是 absorbing-state 离散扩散:每个 token 以与 $t$ 相关的概率独立被替换为一个特殊的「吸收态」[MASK]。如果 noise schedule 是 $\alpha_t$($t=0$ 时 $\alpha_0=1$ 意味着不加噪,$t=1$ 时 $\alpha_1=0$ 意味着全部被 mask 掉),则单个位置的前向条件分布是:

$$q(x_t^i \mid x_0^i) = \alpha_t \cdot \mathbb{1}[x_t^i = x_0^i] + (1-\alpha_t) \cdot \mathbb{1}[x_t^i = \mathtt{[MASK]}].$$

整段序列的前向是按位置独立做的——这一点很重要,它让前向过程不需要存储巨大的转移矩阵,每个位置只是一次伯努利采样。

Why:吸收态有个非常好的性质——一旦一个位置变成 [MASK],它就「锁死」了,不会再跳到别的 token,也不会跳回原 token。这让反向过程只需要决定该不该把这个 mask 揭开、揭开成哪个 token,不用考虑「token A → token B」这种状态间跳转。转移矩阵从 $V\times V$ 退化为 $V\times 1$——这是工程上能把 vocabulary 开到 32k+ 的关键,也是 D3PM 早期 uniform transition 方案做不大的根本原因。

2. 噪声预测器:双向 Transformer

给定带 mask 的中间状态 $\mathbf{x}_t$,模型只在被 mask 的位置上预测原始 token:

$$p_\theta(x_0^i \mid \mathbf{x}_t), \quad i\in\{j: x_t^j = \mathtt{[MASK]}\}.$$

没被 mask 的位置 loss 是 0——它们已经是 ground truth,无需再预测。这跟图像扩散里「每个像素都要预测 noise」很不一样:文本扩散的 loss 只在「被破坏」的那部分位置上算。

Why:注意这里 attention 是双向的——模型能同时看到 mask 位置左边和右边的 context。这是 DLM 和 AR LM 最根本的差异,也是它「能做填空、能 bidirectional 推理」的物理基础。AR LM 的 causal mask 锁死了「每个位置只能看左边」,所以处理「Tom Cruise 的妈妈是谁」这类反向问题时会出现著名的 reversal curse——模型学了「Tom Cruise's mother is Mary Lee Pfeiffer」,却答不出「Mary Lee Pfeiffer's son is who」。DLM 双向看,训练时正反方向同等学习,这个 curse 在原理上就被消除了。

Vizuara L13 把这件事和图像的 noise prediction 类比得很直接:「就像图像扩散里你要预测每一步加了多少高斯噪声,这里你要预测每个被破坏的位置原本该是什么 token」。换句话说,DLM 的「噪声预测器」其实是个token reconstructor,本质上还是同一个扩散思想——给我中间态,告诉我起点。

3. 反向(去噪)过程:并行解码

推理从 $\mathbf{x}_T = (\mathtt{[MASK]}, \dots, \mathtt{[MASK]})$ 开始,按 schedule 把时间从 $t=1$ 推到 $t=0$。每一步做三件事:

  1. 模型对所有还是 [MASK] 的位置同时预测 $\hat x_0^i$ 以及对应的置信度(通常用 softmax 概率的最大值或 entropy 倒数表征);
  2. 按某个策略——top-k 置信、随机抽样、或 entropy-based 选择——挑出一批位置揭开,把它们从 [MASK] 替换成预测出的真实 token;
  3. 剩下的位置保持 [MASK],进入下一个时间步重新算 attention。

这套循环是 DLM 在推理阶段和 AR LM 区别最大的地方:AR 的每个 forward 只产出序列末尾一个新 token,而 DLM 的每个 forward 在整段序列上同时更新一批位置。揭开顺序通常并不固定,而是由模型对每个位置的置信度动态决定,这意味着「先确定 token」和「后确定 token」之间没有固定的语法依赖方向。Vizuara 的 L14 课程把这件事讲得很直白:

「Denoising 的意思是从纯噪声出发,慢慢回到那个『真实概率分布』所在的地方。在文本里,纯噪声就是全 [MASK] 的序列,真实分布就是有意义的句子聚集的那个 manifold。」—— Vizuara《Diffusion LLM》L14 @0:03
选揭开多少 token:每步揭开的位置数由 schedule 决定。最常见的 cosine schedule 让前几步揭开少量「最确定的 token」,中段加速,最后几步剩下的低置信位置才被强行决议。这种渐进解码避免了「先随便填,后面再纠错」的不一致问题——一旦揭开就锁定,绝不回头。
AR vs Diffusion Language Model AR The cat sat on ? ? 单向 causal · 一次出 1 token · KV-cache 友好 DLM t=1 [M] [M] [M] [M] [M] [M] t=0.5 The [M] sat [M] the [M] t=0 The cat sat on the mat 并行去噪 双向 attention

训练目标的细节

把 ELBO 在 absorbing-state 链上一路推到底(中间需要利用「每位置独立 + 吸收态不再跳出」这两个性质做大量约简),可以得到 MDLM 给出的简化损失:

$$\mathcal{L}_{\text{MDLM}} = \mathbb{E}_{t\sim\mathcal{U}(0,1)}\,\mathbb{E}_{\mathbf{x}_t\sim q(\cdot|\mathbf{x}_0)}\left[\frac{1}{t}\sum_{i}\mathbb{1}[x_t^i=\mathtt{[M]}]\cdot(-\log p_\theta(x_0^i\mid\mathbf{x}_t))\right].$$

注意这个目标在形式上就是一个加权 cross-entropy——这也是为什么 DLM 实现起来出乎意料地简单:只需要在标准 LLM 训练代码里加一步「按 $t$ 采样 mask 比例并替换 token」、把 causal attention 改成双向、再给 loss 乘 $\frac{1}{t}$ 权重即可。Dr. Raj 在 L13 也强调过这种「概念复杂但实现简单」的反差:「初学时觉得很难,真正从零搭起来才发现和 AR 比并不复杂」。

Why $\frac{1}{t}$:当 $t$ 很小(几乎没 mask,只有 5% 位置被破坏)时,模型几乎不用动脑就能凭已有上下文猜对——这种 easy sample 在 ELBO 里贡献的「信息量」本应该低;而 $t$ 很大(80% 都是 mask)时,每个位置都几乎无 context,预测难度极大、信息量也大。$\frac{1}{t}$ 等价于把损失重写成「平均每被 mask 一个 token 贡献的 NLL」,让难易样本的 gradient 量级在不同 $t$ 上几乎不变——这是 MDLM 在论文里强调让训练稳定的关键 trick,没有这个权重时 loss 会被 easy sample 主导,模型在难样本上欠拟合。

同期还有另一个流派 SEDD (Lou et al. 2024) 走 score-based 路线,预测 token 比率的对数差(concrete score),数学上更接近连续扩散的 score-matching,但实现复杂度更高、训练更慢,因此 LLaDA / DiffuLLaMA 这一代实用工作都走 MDLM 这条吸收态简化路线。

两门姊妹课的取景差异

本概念的两个源课在切入角度上差异明显,值得对照看:

《Diffusion LLM》专项课

从工程实现切入——L1 直接宣告「我要带你从零搭一个 diffusion language model」,路线是 motivation → noising → 训练 pipeline → inference pipeline → 编码实现 part 1/2。强调可动手、可 ablation。L17 结尾 Dr. Raj 说目标是「打开黑盒、让你能在上面做研究,甚至自己搞个开源 DLM 跟 Qwen 比一比」。

《Principles of Diffusion Models》

从原理切入——把图像扩散的数学基础(VAE → DDPM → score → flow matching)讲透后,把 DLM 当成「同一套原理在离散数据上的实例化」。所以这门课里 DLM 不是主菜,而是证明扩散框架普适性的收官章节。

建议初学者把后者当成「为什么这样做」的 prerequisite,把前者当成「如何动手做」的 capstone。两条路线合起来才完整覆盖 DLM 这个 2024–2025 年正在快速演化的方向。值得注意的是:两门课对同一件事的命名也有微妙差异——专项课更愿意叫「diffusion language model」,原理课偶尔混用「discrete diffusion」「language diffusion」「text diffusion」。研究文献里这几个词目前还在并行使用,没有彻底定型。

🟡 教材外补充:LLaDA (Nie et al. 2025) 把这个训练目标推到 8B 参数、2.3T token 规模,得到一个完全用 diffusion 训出来的 LLM,在 instruction following、math、code 三个 benchmark 上都能和同 size 的 LLaMA-3 8B 打平——这是 diffusion 第一次在 8B 规模上证明可用而不只是可行。DiffuLLaMA (Gong et al. 2025) 则换了条更省钱的路:直接拿预训练好的 LLaMA 续训成 diffusion,不用从头跑 trillion token,把 diffusion 当成 AR 之上的「fine-tuning 范式切换」。两条路线并行说明 DLM 已经不只是学术 toy 了——一条证明从零训得通,另一条证明能廉价改造现有大模型。

为什么要这么麻烦:DLM 的三个收益

  • 消除 reversal curse:AR LM 学过「Tom Cruise's mother is Mary Lee Pfeiffer」并不能答出反向问题「Mary Lee Pfeiffer's son is who」——因为训练时 loss 只在「从左 token 预测右 token」上算,反向方向的条件概率从未被显式学过。DLM 用双向 attention + 任意 mask 位置预测,正反方向同等学习,原理上就消除了这个问题。这是 Karpathy 在那条推文里最看重的优势之一。
  • 原生 infilling 能力:「给开头和结尾,补中间」这件事对 AR LM 是个棘手的 prompt 工程问题(需要 fill-in-the-middle 特殊训练),但对 DLM 来说是训练时就在做的事——把中间一段 mask 掉再恢复,本身就是一个标准训练样本。DLM 的 infilling 不需要任何额外架构改动。
  • 并行解码带来吞吐量优势:理论上一步能出 $K$ 个 token,$T=20$ 步就能生成 256 个 token。但 Vizuara 在 L16 课程里也明说实际工程没理论那么乐观:
「Diffusion language model 训练起来比 AR 慢。所以你应该有 Colab Pro,最好是 A100 或者 H100。」—— Vizuara《Diffusion LLM》L16 @0:02
🟡 教材外补充:「并行解码」的真实速度并不像理论账面那么美。AR 的 KV-cache 在 batch=1、长 prefix 的场景下几乎只算当前一个 token的 attention,单 forward 极快,单流速度被压到几乎纯 memory-bound;DLM 每一步都要重算整段序列的 bidirectional attention,而 KV-cache 在去噪步之间几乎不能复用——因为每步揭开新位置后,未来步骤的 mask 模式都变了,需要重新算 key/value。结果就是 DLM 的速度优势主要体现在 large batch + 高 throughput 服务 场景,或者「必须并行 infilling」的任务上;单条 latency 反而经常比 AR 慢。这也解释了为什么 LLaDA paper 在和 LLaMA-3 比对时都强调 throughput-per-GPU,不直接比单流 latency。

下面这个 schedule 演示 4 步并行解码。注意看一个不变量:已经被揭开的 token(fixed)不会在后续步骤里再被改回 [MASK]——absorbing chain 保证这件事。

[M][M][M][M][M][M][M][M]
起点:长度 8 的纯噪声序列。所有位置都是吸收态。

手算一个 mini DLM 训练步

设 vocab $V=$ {The, cat, sat, [M]},原始序列 $\mathbf{x}_0=$ [The, cat, sat]。第一步采样时间 $t = 0.6$;按 $1-\alpha_t = 0.6$ 概率独立 mask 每个位置——假设抛三次伯努利后结果是 $\mathbf{x}_t=$ [The, [M], sat],只有中间位置被破坏。把 $\mathbf{x}_t$ 送进双向 Transformer,模型对位置 2 输出 softmax 概率 $p_\theta = [0.1, \mathbf{0.7}, 0.15, 0.05]$(对应 The/cat/sat/[M] 四个 vocab 项)。

该位置的 NLL 是:

$$-\log p_\theta(\mathrm{cat}\mid \mathbf{x}_t) = -\log 0.7 \approx 0.357.$$

注意位置 1 和 3 不参与 loss——它们没被 mask,模型不在那里预测。带 $\frac{1}{t}$ 权重的 per-sample 损失:

$$\mathcal{L} = \frac{1}{t}\cdot 0.357 = \frac{0.357}{0.6} \approx 0.595.$$

对比另一个 $t$ 取值:若 $t=0.2$(只 mask 约 1/5 位置)时同样以 0.7 的概率预测对 cat,损失会被放大到 $0.357/0.2 \approx 1.79$——这正是 MDLM 想要的效果:低 mask 比例下的预测虽然容易,但被加权得更重,让难易样本在 gradient 上贡献相当,loss landscape 更平。这一行权重看似不起眼,却是 MDLM 论文里反复 ablation 验证过的关键改进。

常见误区

  1. 「DLM 就是 BERT 改成多步迭代」。反例:BERT 在固定 15% mask ratio 上训练,从 $\mathbf{x}_T=$ 全 [M] 反向推回去时,第一步等价于要在 100% mask 的输入上做预测——这个分布 BERT 根本没见过,结果会全是高频通用词,无法产生连贯句子。DLM 必须在 $t\sim\mathcal{U}(0,1)$ 上联合训练所有 mask 比例,让 $t\to 1$ 极端噪声情况也是 in-distribution,反向链条才有意义。
  2. 「并行解码一定比 AR 快」。反例:单条 256-token 生成场景下,LLaMA-3 8B 在 H100 上配 KV-cache 大约能跑 80 tok/s,每步只算 1 个 token 的 attention;LLaDA-8B 跑 20 步去噪要重算 20 次整段 bidirectional attention(每段 256 token),实测单流速度反而慢 1.5–3 倍。DLM 的速度收益要在 batch 大、prefill 长、或硬性需要并行 infilling 的场景下才真正显现。
  3. 「转移矩阵可以随便选」。反例:原始 D3PM (Austin et al. 2021) 尝试过 uniform transition——任意 token 可以以小概率跳到任意其他 token——这种设定下转移矩阵尺寸是 $V\times V$。当 $V=32000$(LLaMA tokenizer 常见尺寸)时,单步存储约 $V\times V\approx 10^9$ float,内存直接爆炸。MDLM/LLaDA 切换到 absorbing-state(只允许跳到 [M])后矩阵退化为 $V\times 1$,存储压力立刻消失,这才让 32k+ vocab 的 LLM 规模训练成为可能。Tokenizer 越大、这个 trick 收益越显著。

相关概念

来源

  1. diffusion-lm-vizuara · L1 · Build a Diffusion Language Model from Scratch Workshop · (0:02)
  2. diffusion-lm-vizuara · L2 · How I began my Diffusion LLM Journey(Karpathy 推文转述)· (0:02)
  3. diffusion-lm-vizuara · L5 · Motivation behind Language Diffusion Models · (0:02)
  4. diffusion-lm-vizuara · L12 · Diffusion LLM Noising Schedule · (0:03)
  5. diffusion-lm-vizuara · L13 · Diffusion LLM Training Pipeline · (0:04)
  6. diffusion-lm-vizuara · L14 · Diffusion LLM Inference Pipeline · (0:03)
  7. diffusion-lm-vizuara · L16 · Diffusion LLM Coded from Scratch Part 1 · (0:02)
  8. diffusion-principles-vizuara · L11 · The Principles of Diffusion Models New Course · (0:03)
查看 169 条课程原文出处
CourseLectureTimeExcerpt
diffusion-lm-vizuara L1: Build a Diffusion Language Model from Scratch Workshop 00: 02 All of you love the build large language All of you love the build large language models from scratch cars which we have models from scratch cars which we have models from scratch cars which we have o…
diffusion-lm-vizuara L1: Build a Diffusion Language Model from Scratch Workshop 01: 11 Then there was a flurry of diffusion based models which were released. based models which were released. based models which were released. Essentially think of diffusion as Essentially think of diffus…
diffusion-lm-vizuara L1: Build a Diffusion Language Model from Scratch Workshop 02: 16 GitHub repositories or online videos which teach me everything I want to which teach me everything I want to which teach me everything I want to know. I did not want a 5 minute tutorial know. I did no…
diffusion-lm-vizuara L1: Build a Diffusion Language Model from Scratch Workshop 03: 14 pipeline, the training pipeline, and the inference pipeline. And on day four, inference pipeline. And on day four, inference pipeline. And on day four, we'll code the full diffusion model from we'll c…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 00: 03 So what we aim to do is as follows. We So what we aim to do is as follows. We have a true probability distribution, have a true probability distribution, have a true probability distribution, right? S…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 00: 57 an image using a noising schedule maybe an image using a noising schedule maybe a goshian schedule. The predicting noise a goshian schedule. The predicting noise a goshian schedule. The predicting noi…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 01: 41 very crucial role. Right? So let's see how noising is included. how noising is included. how noising is included. What is the noising process in the text? What is the noising process in the text? What…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 02: 32 that noise and then when we start with that noise and then when we start with noise, we can predict the original text noise, we can predict the original text noise, we can predict the original text ju…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 03: 32 embedding. So this is a 384 dimensional embedding. So this is a 384 dimensional vector. vector. vector. This is a 384 dimensional vector. This is a 384 dimensional vector. This is a 384 dimensional ve…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 04: 44 this is the true sentences probability distribution. sentences probability distribution. sentences probability distribution. This and this sentence the next day is This and this sentence the next day …
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 05: 25 a portion of this, this is no longer a a portion of this, this is no longer a true sentence. It becomes a cor, it true sentence. It becomes a cor, it true sentence. It becomes a cor, it becomes a corr…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 06: 33 schedule. Right? So first I define a time step. Let's say I go from time time step. Let's say I go from time time step. Let's say I go from time equal to 1 till time equal to 6. equal to 1 till time e…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 07: 42 going haywire. So maybe initially when only one mask is added, I go to a lower only one mask is added, I go to a lower only one mask is added, I go to a lower probability. When two masks are added, I …
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 08: 56 way it happens in text is that uh at time equal to 1, you almost have an time equal to 1, you almost have an time equal to 1, you almost have an entire clean text with maybe some mask entire clean tex…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 09: 40 it's a Bernoli distribution. So the probability of masking is way higher. probability of masking is way higher. probability of masking is way higher. Probability of masking all tokens is way Probabili…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 10: 44 3,000. Right? Now these are masked. So this is masked and this is masked. So this is masked and this is masked. So this is masked and this is masked. So this will be replaced with 1 0 0 0 1 this will …
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 11: 35 a noisy input with mask tokens. Okay. a noisy input with mask tokens. Okay. Everything else remains exactly the Everything else remains exactly the Everything else remains exactly the same. Now we'll …
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 12: 23 architecture if you compare them side by architecture if you compare them side by side is the input which goes into this side is the input which goes into this side is the input which goes into this i…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 13: 03 targets then we sorry we get the predictions then we sorry we get the predictions then we sorry we get the predictions then we get the loss then we get the loss then we get the loss then we update the…
diffusion-lm-vizuara L13: Lecture 12: Diffusion LLM Noising Schedule 13: 51 what is the noising process in the case of diffusion language models. The next of diffusion language models. The next of diffusion language models. The next step which we are going to see is that step…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 00: 04 Okay. So the next step in these key Okay. So the next step in these key characteristics which we mentioned and characteristics which we mentioned and characteristics which we mentioned and probably th…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 01: 01 You take a noisy image. You take a noisy image. Let's say this is a noisy image of a Let's say this is a noisy image of a Let's say this is a noisy image of a panda which is the example which we have …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 01: 44 function, right? We need to predict the function, right? We need to predict the noise and then we need to compare it noise and then we need to compare it noise and then we need to compare it with the …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 02: 44 How do I predict the true values here? So what I want to do what does it mean So what I want to do what does it mean So what I want to do what does it mean prediction of noise? I somehow want to predi…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 03: 41 at this position? And how do you get the loss function? loss function? loss function? Well, the answer is extremely simple, Well, the answer is extremely simple, Well, the answer is extremely simple, …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 04: 32 Right. So the next token which is predicted at this position I'll compare it with the true next token I'll compare it with the true next token which should be next. which should be next. which should …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 05: 25 predictions, right? For the next, these predictions, right? For the next, these are my predictions. are my predictions. are my predictions. These are my predictions. I'm just going These are my predic…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 06: 11 um second characteristics the model which predicts the noise which was a which predicts the noise which was a which predicts the noise which was a unit in the case of diffusion for unit in the case of…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 07: 14 are. Right? In the ARM architecture, we are. Right? In the ARM architecture, we remember we had the uh token embedding remember we had the uh token embedding remember we had the uh token embedding and…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 08: 06 we add the token embedding plus position embedding the one major difference in embedding the one major difference in embedding the one major difference in diffusion language models is that along diffu…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 08: 51 time embedding vector is added to all my tokens. So now the input embedding is tokens. So now the input embedding is tokens. So now the input embedding is not just the token embedding plus not just th…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 09: 35 change. So now if you see if you compare these block diagrams these block diagrams these block diagrams in the ARM architecture, we only had the in the ARM architecture, we only had the in the ARM arc…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 10: 25 exactly the same. The difference is in the attention mechanism. the attention mechanism. the attention mechanism. Now think about it right here. We no Now think about it right here. We no Now think ab…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 11: 08 means that when I'm calculating the attention so the mask the mask day the mask day mask bright right which means that if mask bright right which means that if mask bright right which means that if I'…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 12: 28 I essentially have this entire causal attention matrix which is available to attention matrix which is available to attention matrix which is available to me. I don't have to set these values to me. I…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 13: 15 major difference? The third major difference comes in the output token difference comes in the output token difference comes in the output token prediction, right? Uh let's say here we prediction, rig…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 13: 56 these losses. That's my loss. I don't have to find the loss for the I don't have to find the loss for the I don't have to find the loss for the other positions at all. other positions at all. other po…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 15: 04 So if this is let's say 0.1 this probability and this is2 my loss in this probability and this is2 my loss in this probability and this is2 my loss in this case will be - 1 by2 log of.1 log of.1 plus …
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 16: 15 we have a time embedding now which was we have a time embedding now which was not there auto reggressive model in not there auto reggressive model in not there auto reggressive model in multi attentio…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 17: 02 on u the time step. More the time step, on u the time step. More the time step, more is the masking. Less the time step, more is the masking. Less the time step, more is the masking. Less the time ste…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 17: 43 transformer block. So I have my layer normalization multi head attention I normalization multi head attention I normalization multi head attention I have my dropout I have my skip have my dropout I ha…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 18: 31 batch size. We choose the time step, right? Let's say batch size is 128. We right? Let's say batch size is 128. We right? Let's say batch size is 128. We assemble an input sequence of assemble an inpu…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 19: 30 tokens. Based on this loss, we find the partial derivative of the loss with partial derivative of the loss with partial derivative of the loss with respect to all the parameters. And respect to all th…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 20: 27 case is the better we pre-train a language diffusion model the better we language diffusion model the better we language diffusion model the better we are able to discover what noise is added are able…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 21: 18 Start with all masks and then we uncover Start with all masks and then we uncover tokens correct tokens at that position. tokens correct tokens at that position. tokens correct tokens at that position…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 22: 02 predict is that the probability of actually recovering the probability of actually recovering the probability of actually recovering the true true distribution or the the true true distribution or the…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 22: 48 proved to be an upper bound on the proved to be an upper bound on the negative log likelihood making it a negative log likelihood making it a negative log likelihood making it a principled objective f…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 23: 34 the masked positions. Okay. So, until now we have seen the first two So, until now we have seen the first two So, until now we have seen the first two steps of diffusion based language steps of diffus…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 24: 22 the third difference. Noisy input here the third difference. Noisy input here that's the four difference. There are that's the four difference. There are that's the four difference. There are four maj…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 25: 09 embedding matrix and we construct a time embedding matrix also. And I show you embedding matrix also. And I show you embedding matrix also. And I show you how everything is done. So at each how everyt…
diffusion-lm-vizuara L14: Lecture 13: Diffusion LLM Training Pipeline 25: 54 model can be summarized here. We have model can be summarized here. We have token ids convert them into token token ids convert them into token token ids convert them into token embedding vectors add …
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 00: 03 The last step which we need to do is the The last step which we need to do is the actual step of dnoising. actual step of dnoising. actual step of dnoising. What is dnoising? Well, dnoising is What is…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 00: 51 yeah, at each step of the training yeah, at each step of the training process, if we are able to predict how process, if we are able to predict how process, if we are able to predict how much noise ha…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 01: 35 recover the clean image. So we start from a complete noisy image and then we from a complete noisy image and then we from a complete noisy image and then we recover Chinese characters as the recover C…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 02: 15 parameters of the model will not change in the dnoising process. The model is in the dnoising process. The model is in the dnoising process. The model is fully trained. Um so all these fully trained. …
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 03: 02 time steps, for example, in the D noising process, we have to go backwards noising process, we have to go backwards noising process, we have to go backwards from four. So, we have to go 4 3 2 1, from …
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 03: 49 let's take an example. Okay. So if this let's take an example. Okay. So if this is the input sequence we start at time is the input sequence we start at time is the input sequence we start at time equ…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 04: 42 What is the predicted logits matrix? Um so let's say we have position number Um so let's say we have position number Um so let's say we have position number we have three main positions which we have …
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 05: 58 then we have a b and c right so let's then we have a b and c right so let's actually actually actually take some different values of my let me take some different values of my let me take some differe…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 07: 16 change this again. Sorry for that. So let's say this 336 Sorry for that. So let's say this 336 Sorry for that. So let's say this 336 and this 336. Let's say everything is and this 336. Let's say every…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 08: 07 we need three masks here. So again we'll mask all these three. So these three mask all these three. So these three mask all these three. So these three stay in masks. That's the output from stay in ma…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 08: 52 of this step we have to keep three masks at the end of this step we have to keep at the end of this step we have to keep at the end of this step we have to keep two masks at the end of this step which…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 09: 31 or I will keep so which what what will I or I will keep so which what what will I unmask? So I have to keep two masks here unmask? So I have to keep two masks here unmask? So I have to keep two masks …
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 10: 09 confidence of 3175. At this we have a confidence of 336 and at this we have a confidence of 336 and at this we have a confidence of 336 and at this we have a confidence of 3175. Now here the rule is c…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 10: 54 So at first position I sample A with confidence of 3175. At second position I confidence of 3175. At second position I confidence of 3175. At second position I sample C with confidence 336. At third s…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 11: 45 input architecture and wherever there is input architecture and wherever there is mask that slowly unmasked right. So here mask that slowly unmasked right. So here mask that slowly unmasked right. So …
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 12: 32 reggressive which means it's not one token at a time. Tokens masks can be token at a time. Tokens masks can be token at a time. Tokens masks can be unmasked here, masks can be unmasked at unmasked her…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 13: 17 what happens is that let's say you fully trained the model as we have seen trained the model as we have seen trained the model as we have seen before. Then you start with a sequence before. Then you s…
diffusion-lm-vizuara L15: Lecture 14: Diffusion LLM Inference Pipeline 14: 06 unmasking one after the other. So this unmasking one after the other. So this video or this GIF which Andre Karpati video or this GIF which Andre Karpati video or this GIF which Andre Karpati shared. …
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 00: 02 Before we go to building the small Before we go to building the small language diffusion model from scratch language diffusion model from scratch language diffusion model from scratch and we are going…
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 00: 48 uh cluster of meaningful sentences lie what language models are doing is that what language models are doing is that what language models are doing is that we are trying to find this probability we ar…
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 01: 26 noise the input sequence. You corrupt the input sequence. You pass this the input sequence. You pass this the input sequence. You pass this corrupted input sequence uh you pass corrupted input sequenc…
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 02: 14 One key difference is the time embedding which is not there in ARM but it's there which is not there in ARM but it's there which is not there in ARM but it's there in diffusion language model. Second …
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 02: 58 Fourth thing of course is that we have Fourth thing of course is that we have noisy input with mass tokens. Whereas in noisy input with mass tokens. Whereas in noisy input with mass tokens. Whereas in…
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 03: 45 unmasked? The most confident tokens are unmasked? The most confident tokens are essentially unmasked. So we unmask the k essentially unmasked. So we unmask the k essentially unmasked. So we unmask the…
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 04: 29 get coherent text towards the end. Right? So this is our whole diffusion Right? So this is our whole diffusion Right? So this is our whole diffusion pipeline diffusion language model pipeline diffusio…
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 05: 20 front of it. So we don't have causal attention. Then we generate by unmasking attention. Then we generate by unmasking attention. Then we generate by unmasking tokens one at a time. Right? That's the …
diffusion-lm-vizuara L16: Lecture 15: Diffusion LLM Entire Pipeline Summary 06: 14 so if you zoom into here I'm going to share this code file with all of you and share this code file with all of you and share this code file with all of you and at the end of this code file we are at …
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 00: 03 So here you can see the file which has So here you can see the file which has been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was getting som…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 00: 56 iterations you'll definitely see samples which are as co inference samples which which are as co inference samples which which are as co inference samples which are as coherent as what you are seeing …
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 01: 37 you can even chain different GPUs together to do distributed computing if together to do distributed computing if together to do distributed computing if possible possible possible so let's actually p…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 02: 36 before. Um yeah if you train for 200,000 iterations you'll see an amazing iterations you'll see an amazing iterations you'll see an amazing coherent language diffusion language coherent language diffu…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 03: 23 all the necessary things which you will need to learn about language diffusion need to learn about language diffusion need to learn about language diffusion models. All right. So yeah we have this All…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 04: 19 shown on the screen right now serves as shown on the screen right now serves as a reference for you to build multiple a reference for you to build multiple a reference for you to build multiple such s…
diffusion-lm-vizuara L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 05: 08 fraction of the cost which might be awesome for the future. But now you know awesome for the future. But now you know awesome for the future. But now you know how to build this fully yourself. Um, oka…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 00: 02 Okay. So now I'm going to take you Okay. So now I'm going to take you through the code of assembling this through the code of assembling this through the code of assembling this diffusion language mod…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 01: 00 here. The H100 GPU is much faster. If you don't have access to the Google If you don't have access to the Google If you don't have access to the Google Collab Pro, you can start with the T4 Collab Pro…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 01: 43 number of training steps is only 2,000. Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 which is if you have compute budget or w…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 02: 34 take a look at the whiteboard you'll see that whiteboard you'll see that whiteboard you'll see that this dimension which I keep talking this dimension which I keep talking this dimension which I keep …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 03: 20 essentially here in these examples we have a sequence length of four tokens, have a sequence length of four tokens, have a sequence length of four tokens, right? Whereas in actual code we have right? …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 04: 13 chained to each other. So these are the uh this variable is the number of uh this variable is the number of uh this variable is the number of layers. The number of heads is layers. The number of heads…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 05: 11 that's this uh four times the D model. that's this uh four times the D model. So DFF is the feed forward neural So DFF is the feed forward neural So DFF is the feed forward neural network hidden layer…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 06: 01 optimizer. Okay. So you run this initial configuration and then you move forward. configuration and then you move forward. configuration and then you move forward. You install a bunch of dependencies …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 06: 49 go to this uh hugging face repository and I load my data. I load my training and I load my data. I load my training and I load my data. I load my training data set. I load my validation data set data …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 07: 36 we I'm connected to an L4 GPU at the moment. See that's fine. So let's see. Um Um yeah so here you'll see that I'm yeah so here you'll see that I'm yeah so here you'll see that I'm connected to an L4 …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 08: 32 token, beginning of sequence, end of token, beginning of sequence, end of sequence and a mask token. This is very sequence and a mask token. This is very sequence and a mask token. This is very import…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 09: 20 with a tokenizer directly borrowed from with a tokenizer directly borrowed from tick token. I I I think the code should tick token. I I I think the code should tick token. I I I think the code should …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 10: 21 because in the noising process we are because in the noising process we are going to replace the tokens with masks going to replace the tokens with masks going to replace the tokens with masks and the…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 11: 13 becomes clearer to all of you. Yeah. And if you see that you first define a And if you see that you first define a And if you see that you first define a token embedding, position embedding and token …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 12: 20 a transformer um we are going to do a transformer um we are going to do a transformer um we are going to do a transformer encoder layer over here and uh let's see and attention mask equal to none. Thi…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 13: 11 embedding. We add the time embedding to it. The resultant input embedding we it. The resultant input embedding we it. The resultant input embedding we pass through the transformer block. Then pass thr…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 13: 59 at the time embedding matrix at the time embedding matrix if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need to know at what tim…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 14: 47 to based on the time step we have to add to based on the time step we have to add the mask tokens. This is what is done the mask tokens. This is what is done the mask tokens. This is what is done here…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 15: 29 done because we have already seen this on the whiteboard right to calculate the on the whiteboard right to calculate the on the whiteboard right to calculate the loss I only need to look at the mask l…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 16: 13 Well, it's included over here where we Well, it's included over here where we pass an attention mask equal to none. pass an attention mask equal to none. pass an attention mask equal to none. These ar…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 16: 56 So this is the cross entropy loss. So you can run this right now. Once we have you can run this right now. Once we have you can run this right now. Once we have this defined, the last step is just to …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 17: 35 time step right so when you go to the diffusion loss function you'll see that diffusion loss function you'll see that diffusion loss function you'll see that uh where is the diffusion loss yeah when u…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 18: 19 Now this thing which I'm telling right now the corrupted input sequence is now the corrupted input sequence is now the corrupted input sequence is passed through the model is exactly passed through th…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 19: 11 Yeah, see the training process has started. Now I'm on the L I think let's started. Now I'm on the L I think let's started. Now I'm on the L I think let's see which GPU I'm on. I'm on the L4 GPU see w…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 19: 57 in u the convergence speed so if you see the auto reggressive model in 15,000 the auto reggressive model in 15,000 the auto reggressive model in 15,000 epochs I go from a loss of 8 to a loss epochs I …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 20: 44 can see the training has started over here and we can print the loss we can here and we can print the loss we can here and we can print the loss we can print the learning rate etc. While the print the…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 21: 34 here. I am not going to go through this here. I am not going to go through this code in detail because it just code in detail because it just code in detail because it just implements the same thing, …
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 22: 19 Uh so this is run right now and uh here Uh so this is run right now and uh here I'm just seeing a sample inference. Once I'm just seeing a sample inference. Once I'm just seeing a sample inference. On…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 23: 27 iterations did not finish. But if you actually run um actually run um actually run um 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iteratio…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 24: 09 Lily's mom asked her to help clean up Lily's mom asked her to help clean up her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom p…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 24: 50 we can also do this on runpod. So if you have credentials or if you have some have credentials or if you have some have credentials or if you have some budget even $6 to $7 what I'll suggest budget ev…
diffusion-lm-vizuara L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 25: 45 I'll set overrides. Oops. I think it was I'll set overrides. Oops. I think it was 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set overrides over here. A…
diffusion-lm-vizuara L19: Lecture 18: Diffusion LLM Series Conclusion 00: 02 All right, everyone. So, this brings us All right, everyone. So, this brings us to the end of this diffusion language to the end of this diffusion language to the end of this diffusion language model …
diffusion-lm-vizuara L19: Lecture 18: Diffusion LLM Series Conclusion 00: 51 I've not shown supervised fine-tuning, I've not shown supervised fine-tuning, which you can do on your own since at which you can do on your own since at which you can do on your own since at differen…
diffusion-lm-vizuara L19: Lecture 18: Diffusion LLM Series Conclusion 01: 40 an open-source way. So, you can be the first one to do this. The same pipeline first one to do this. The same pipeline first one to do this. The same pipeline which we have built for small language wh…
diffusion-lm-vizuara L2: Lecture 1: Introduction to Diffusion Language Models 00: 03 Hello everyone. This is Dr. Raj Dandeek Hello everyone. This is Dr. Raj Dandeek and uh today I'm very excited to present and uh today I'm very excited to present and uh today I'm very excited to prese…
diffusion-lm-vizuara L2: Lecture 1: Introduction to Diffusion Language Models 01: 01 models from scratch. But that's not how models from scratch. But that's not how I want this course or this learning I want this course or this learning I want this course or this learning material to …
diffusion-lm-vizuara L2: Lecture 1: Introduction to Diffusion Language Models 01: 49 scared of them, then how I learned the scared of them, then how I learned the subject, how I mastered the subject now subject, how I mastered the subject now subject, how I mastered the subject now so…
diffusion-lm-vizuara L2: Lecture 1: Introduction to Diffusion Language Models 02: 27 model fully from scratch. You'll compare it with the auto reggressive language it with the auto reggressive language it with the auto reggressive language models. So it's an amazing exercise or models…
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 00: 02 Let's get started. Let's get started. Initially, as I mentioned, I'll be Initially, as I mentioned, I'll be Initially, as I mentioned, I'll be taking you through my journey, right? taking you through …
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 00: 51 thought that okay this is something thought that okay this is something cool. I had thought about diffusion only cool. I had thought about diffusion only cool. I had thought about diffusion only for i…
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 01: 33 over here. So take a look here first only on the left hand side. Okay. And only on the left hand side. Okay. And only on the left hand side. Okay. And now when we restart the video then take now when …
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 02: 18 you learn how much noise is added. So you learn how much noise is added. So later you are able to take noise and later you are able to take noise and later you are able to take noise and then build im…
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 03: 07 diffusion model which fills the screen like this and it's not one token at a like this and it's not one token at a like this and it's not one token at a time. So when I saw this GIF, I time. So when I…
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 03: 59 at a time. Um this has now become very at a time. Um this has now become very popular on YouTube and it's one of the popular on YouTube and it's one of the popular on YouTube and it's one of the go-to…
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 04: 47 And uh I'm going to first walk you through diffusion in an intuitive way through diffusion in an intuitive way through diffusion in an intuitive way because I I strongly think that all of because I I …
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 05: 36 formulations will which I'll not so I'll not take that route when I explain this not take that route when I explain this not take that route when I explain this to you I'll explain this to you in an t…
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 06: 23 fully from scratch. You'll be doing this fully from scratch. You'll be doing this if you follow you follow along with me if you follow you follow along with me if you follow you follow along with me t…
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 07: 43 take an image you keep on adding noise to it. Right? You keep on adding noise to it. Right? You keep on adding noise to it. Right? You keep on adding noise to an image until it becomes fully to an ima…
diffusion-lm-vizuara L3: Lecture 2: How I began my Diffusion LLM Journey 08: 43 clean image. So now you'll see that as we are going So now you'll see that as we are going So now you'll see that as we are going in the reverse direction slowly the dye in the reverse direction slowl…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 00: 02 So I hope all of us are on the same page So I hope all of us are on the same page now with respect to the motivation now with respect to the motivation now with respect to the motivation behind this l…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 00: 50 questions which researchers should ask that leads to the best research. that leads to the best research. that leads to the best research. If you ask this question, you'll slowly If you ask this questi…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 01: 49 models. This so this paper I believe is a very important one mask diffusion is a very important one mask diffusion is a very important one mask diffusion language models this is where many language mo…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 02: 51 introduction section itself uh that why should images and video uh that why should images and video uh that why should images and video generation models have all the fun why generation models have al…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 03: 39 images might lie and then sampling from it. So why can't this same thing be it. So why can't this same thing be it. So why can't this same thing be applied to text as well? applied to text as well? ap…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 04: 58 when we talked about images, first we broke down the images into pixels, broke down the images into pixels, broke down the images into pixels, right? Similarly, text can be broken right? Similarly, te…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 05: 54 So one way to look at generating text is So one way to look at generating text is just one word at a time. Another way of just one word at a time. Another way of just one word at a time. Another way o…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 06: 57 in high dimensional spaces? What does it mean probability distribution of mean probability distribution of mean probability distribution of meaningful text? Let's see. So text is trickier than images.…
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 08: 12 sequence of words based on two things. sequence of words based on two things. Grammar and logic both need to be Grammar and logic both need to be Grammar and logic both need to be correct, right? And …
diffusion-lm-vizuara L6: Lecture 5: Motivation behind Language Diffusion Models 09: 12 can be carried forward to languages. can be carried forward to languages. What these people suggested is auto What these people suggested is auto What these people suggested is auto reggressive models…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 00: 03 Hello everyone. I'm very excited to be Hello everyone. I'm very excited to be recording this video today because uh recording this video today because uh recording this video today because uh I'm goin…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 01: 00 focus on the technology behind uh midjourney or stable diffusion but I uh midjourney or stable diffusion but I uh midjourney or stable diffusion but I was so excited to try this out that I was so exci…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 01: 49 related to the diffusion which I know about I know about diffusion where let's about I know about diffusion where let's about I know about diffusion where let's say I spray a perfume in one corner of …
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 02: 45 these diffusion models have these diffusion models have lasted for 10 to 15 years and every lasted for 10 to 15 years and every lasted for 10 to 15 years and every development has used breakthroughs m…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 03: 43 So this series has been made from that So this series has been made from that point of view. I want to explain point of view. I want to explain point of view. I want to explain diffusion models to you…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 04: 36 I have structured this course and how I plan to teach it. It'll be very similar plan to teach it. It'll be very similar plan to teach it. It'll be very similar to Vijuara's other teaching material and…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 05: 43 discuss is that of protein discuss is that of protein structure prediction. structure prediction. structure prediction. This has been a problem which has This has been a problem which has This has bee…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 06: 42 seconds, 5 seconds, 10 seconds and now you can generate up to 5 minute videos you can generate up to 5 minute videos you can generate up to 5 minute videos using u generative AI and a lot of these usi…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 07: 42 this advancement. Recently we are seeing this advancement. Recently we are seeing a lot of foundational robotics models a lot of foundational robotics models a lot of foundational robotics models bein…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 08: 43 wonder is there something intrinsic which is happening in diffusion that is which is happening in diffusion that is which is happening in diffusion that is common in all these applications. But common…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 09: 42 images and videos but also audio images and videos but also audio synthesis. Now that we have established that Now that we have established that diffusion models have a huge number of diffusion models…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 10: 55 So you see there are these small videos So you see there are these small videos 15 minutes, 30 minutes, 12 minutes and 15 minutes, 30 minutes, 12 minutes and 15 minutes, 30 minutes, 12 minutes and som…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 11: 48 minutes maximum 30 minutes long. So very little is known about how they work and little is known about how they work and little is known about how they work and I think that diffusion models should be…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 12: 47 Now the reason I'm coming out with this Now the reason I'm coming out with this course right now is because I received a course right now is because I received a course right now is because I received…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 13: 42 and one thing uh I realized after and one thing uh I realized after reading this book and these thoughts reading this book and these thoughts reading this book and these thoughts were also circling in…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 14: 31 have divided the curriculum according to this book. Uh and I I will not go very this book. Uh and I I will not go very this book. Uh and I I will not go very theoretical in this section. So it might t…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 15: 20 the first place? And what is the broader the first place? And what is the broader objective we are trying to achieve here? objective we are trying to achieve here? objective we are trying to achieve h…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 16: 08 lot in the last 10 years. As you can see one of the first methods actually came one of the first methods actually came one of the first methods actually came 40 years back which was energy based 40 ye…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 17: 05 we will cover all of these different we will cover all of these different techniques individually. techniques individually. techniques individually. So broadly speaking there are three So broadly spea…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 17: 59 variational view then we will cover score based view. Then we will cover score based view. Then we will cover score based view. Then we will cover flow-based view and then after that we flow-based vie…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 18: 59 finally in part three we are going to understand how to sample from these understand how to sample from these understand how to sample from these diffusion models. How to steer the diffusion models. H…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 19: 51 actually attracted me to this field. I want I I I wanted to study it in in in want I I I wanted to study it in in in want I I I wanted to study it in in in more depth. more depth. more depth. People w…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 20: 40 reinforcement learning course just finished a while back and uh after that finished a while back and uh after that finished a while back and uh after that I have come up with this diffusion I have com…
diffusion-principles-vizuara L11: The Principles of Diffusion Models - New Course Launch 21: 32 feeling you will not get by looking at these concepts from a very surface level these concepts from a very surface level these concepts from a very surface level to get that you need to understand the…