概念

Variational Lower Bound(ELBO)

— $\log p(x)$ 这把天花板算不动,ELBO 是你能算的地板。

很多人第一次读到 ELBO 都会想:「这不就是把一个算不动的对数似然换成一个能算的下界,凑一个能反向传播的损失而已嘛——一个数学小聪明。」这个理解从一开始就错了。ELBO 不是为了让数学更容易而发明的技巧,也不是某种近似公式——它是 $\log p(x)$ 与你实际能计算的量之间那条客观存在的缝隙本身。换句话说,ELBO 不是"我们决定用什么来代替 $\log p(x)$",而是"$\log p(x)$ 减去 ELBO,刚好等于你选错后验的代价"。你不是在「逼近」一个似然,你是在抬升一块叫做下界的地板,直到它顶到了那个叫真实对数似然的天花板。

Vizuara 的 Dr. Raj 在 Lecture 2 用了一个极朴素的物理比喻——「猫被瞬移走了」。你想求一个完整的 $\log p(x) = \log \int p(x, z) dz$,但这个积分在高维(比如一张 32×32 图所有可能的潜变量 $z$,几千维的连续空间)上根本算不出来——就像猫从你眼前被瞬移走,你抓不到它本身,只能在它原本的位置周围设法探出它的轮廓。怎么办?找一个能算的下界,然后想办法把这个下界一步步往上推。下界推得越高,离真正的猫越近;如果你能把下界抬到与天花板贴齐,那等价于你已经知道了真后验。

Dr. Raj(Vizuara L2):「积分算不动?那就找一个能算的 lower bound,然后想办法把这个 bound 往上 push。push 得越高,你就越接近真实的 log-likelihood。」

CMU 的 Kelly Hu 则把同样的事情称作「有史以来最伟大的数学小把戏」——在那个让人窒息的积分里乘上、再除以同一个 $q(z|x)$。这个看上去毫无意义、像是数学课上小学生才会做的「乘 1」操作,恰恰是整套变分推断的引擎:它把一个不可计算的高维积分,变成了一个对 $q$ 的期望;只要你能从 $q$ 采样、能算 $\log p(x,z)$ 和 $\log q(z|x)$,整件事就变成一个普通的可微目标函数。再配合 Jensen 不等式,地板就出现了。

Kelly Hu(CMU 10-799 L1):「The greatest mathematical trick of all time — multiply and divide by $q$. 你什么都没改变,但你刚刚把一个高维积分变成了一个可以蒙特卡洛估计的期望。」

这两位讲师用了截然不同的比喻——一个偏物理直觉(瞬移的猫、抬地板),一个偏代数手法(乘 1 除 1 的把戏)——但它们指向同一件事:对数似然不能直接算,但可以围攻。在训练里,这件事对应:VAE 不是直接极大化 $\log p(x)$(不可能),而是极大化 ELBO;扩散模型把这个 ELBO 沿着 $T$ 步展开成 $\sum_t L_t$,每一步都是一个高斯之间的 KL;流匹配则更进一步,连 $q$ 都不显式定义、直接学速度场。它们都在做同一件事——把一块叫做 ELBO 的地板,朝那块叫 $\log p(x)$ 的天花板抬升。理解这一点,你就理解了 VAE、扩散、流匹配在数学上为何是同一族算法,理解了为何"重建 + KL"会从一个看似无关的不等式里自然冒出来,理解了为何 DDPM 那行 $\|\epsilon - \epsilon_\theta\|^2$ 不是工程妥协而是原则导出。这是整个生成模型理论的入口。

从「算不动」到「能算」:Jensen 风格推导

下面这四步推导,是 1990 年代以来变分推断的核心、是 VAE 论文(Kingma & Welling 2013)半页内容、是 DDPM 论文附录 A 的展开起点。请一行一行跟着读,不要跳——每一步都附了一段 Why 注解,告诉你这一步在干什么、为什么合法、它把你带到了哪里。

潜变量模型把数据看作 $x \sim p_\theta(x) = \int p_\theta(x, z) \, dz$。$z$ 是你设想的「秘密配方」——Dr. Raj 在 Lecture 2 的术语是笔压、倾斜度、字宽、书写速度、整洁度等隐藏特征——它们组合在一起决定了"这位同学写的 hello 长什么样"。问题在于:每个数据点 $x$ 背后都对应着无穷多种合理的 $z$,所以你必须把所有 $z$ 积出来才能得到 $x$ 的边缘概率。高维下这个积分是不可解析的,朴素蒙特卡洛也会方差爆炸(绝大多数 $z$ 对当前 $x$ 的贡献近似为零,采到一个高贡献样本是天文级低概率)。下面的四步推导是整个变分推断的发动机,VAE、扩散、流匹配的损失都从这里长出来。请仔细看每一步的 Why 注解——前两步是恒等变形(值不变),第三步是 Jensen(出现 $\geq$,这是唯一的不等号来源),第四步是合并整理。

① 引入任意 $q(z|x)$("乘 1 除 1"的把戏) $$\log p(x) = \log \int p(x, z)\, dz = \log \int q(z|x) \cdot \frac{p(x, z)}{q(z|x)}\, dz$$ Why:$q(z|x)$ 是任意满足 $\int q\, dz = 1$ 且支撑集覆盖 $p(x,z)$ 支撑集的分布——你完全没有改变积分的值,只是凭空乘了一个 1。但这个 1 是关键:它把"对 $z$ 的盲目积分"换成了"对 $q$ 期望"的形式,等式右边已经长得像 $\log \mathbb{E}_q[\cdot]$ 了。这一步是 Kelly Hu 所说"最伟大的小把戏"的全部内容——它没有给你新东西,但它给了你新视角。
② 改写为期望 $$\log p(x) = \log \mathbb{E}_{q(z|x)}\!\left[\frac{p(x, z)}{q(z|x)}\right]$$ Why:$\int q(z|x) f(z) dz$ 就是期望的定义——形式没变,意义变了:现在它是一个"采 $z$、算比值、取平均"的过程。
③ Jensen 不等式($\log$ 是凹函数) $$\log \mathbb{E}_q\!\left[\frac{p(x, z)}{q(z|x)}\right] \geq \mathbb{E}_q\!\left[\log \frac{p(x, z)}{q(z|x)}\right] \triangleq \mathcal{L}(x)$$ Why:$\log$ 是凹函数,对凹函数 Jensen 给出 $\log \mathbb{E}[X] \geq \mathbb{E}[\log X]$(凸函数则相反)。直观上,"先取期望再取 $\log$"会比"先取 $\log$ 再取期望"高一截,差出来的那一截就是 ELBO 与对数似然之间的 gap。等号成立的条件——也是整个变分推断的目标——是 $X$ 退化为常数,即 $\frac{p(x,z)}{q(z|x)}$ 对所有 $z$ 都相等;换言之 $q(z|x) \propto p(x,z)$,即 $q$ 等于真后验 $p(z|x)$。
④ 重建 / KL 分解 $$\mathcal{L}(x) = \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{\mathrm{KL}}\big(q(z|x)\,\|\,p(z)\big)$$ Why:用 $p(x,z) = p(x|z)p(z)$ 把联合密度拆开,$\log$ 把乘法变成加法:$\log\frac{p(x|z)p(z)}{q(z|x)} = \log p(x|z) + \log p(z) - \log q(z|x)$。对 $q$ 取期望后,第一项就是重建对数似然;后两项 $\mathbb{E}_q[\log p(z) - \log q(z|x)] = -\mathbb{E}_q[\log\frac{q(z|x)}{p(z)}] = -D_{\mathrm{KL}}(q\|p(z))$,正好是 KL 散度的负数。这就是 VAE loss 的真身:重建项 − KL 正则项。两项各有清晰物理含义:重建项要求 $z$ 必须保留足够信息以复原 $x$;KL 项要求 $q(z|x)$ 不能离先验太远,保证 latent space 的连续性。

那条「缝隙」到底是什么?

到现在你已经知道:ELBO 是 $\log p(x)$ 的下界,下界什么时候紧、什么时候松?把 ELBO 与 $\log p(x)$ 之间的差直接展开,会发现一个惊艳的等式:

$$\log p(x) - \mathcal{L}(x) = D_{\mathrm{KL}}\big(q(z|x)\,\|\,p(z|x)\big)$$

这是整个变分框架最重要的一行公式,请记住它。它说:地板和天花板之间的距离,恰好等于你选的 $q(z|x)$ 离真后验 $p(z|x)$ 的 KL 距离。这条等式让"近似推断"这件事第一次有了量纲——你不再是模糊地"逼近"什么,而是在定量地把 $q$ 推向 $p(z|x)$,每减少一个 nat 的 KL,地板就升高一个 nat。它也解释了为什么 ELBO 是下界而不是上界——KL 散度永远非负,所以缝隙永远朝同一个方向,永远是 ELBO 在下、$\log p(x)$ 在上。

三个直接推论:
  • $q$ = 真后验 ⇒ KL = 0 ⇒ ELBO = $\log p(x)$(地板顶到天花板,bound is tight)。在数学上等价于"你已经解决了推断问题"——但真后验在高维下通常算不出来,所以这是一个理想极限。
  • 极大化 ELBO 同时做两件事:(a) 把生成模型参数 $\theta$ 推向"让数据 $x$ 更可能"的方向(间接提高真实 $\log p(x)$),(b) 把变分参数 $\phi$($q$ 的参数)推向真后验 $p(z|x)$。一个目标函数,两组参数,同时优化——这是 VAE 训练能 work 的核心。
  • 用过度受限的 $q$(比如 mean-field 高斯:假设各维度独立、单峰、各向同性)得到 loose ELBO——这是一个有偏的似然估计,可能永远顶不到天花板,但工程上可训、可微、可批量化。这就是经典的"训练可行性 vs 紧性"权衡。

抬地板的几何图景

不妨用一张图把这件事彻底固化。横轴是训练步数,纵轴是对数似然的取值。天花板 $\log p(x)$ 是一条水平直线——它由数据和模型族 $\{p_\theta\}$ 一起决定,对当前 $\theta$ 来说是固定值,是你"理论上能达到的最佳分数"。地板 ELBO 是一条向上凸起的曲线,初始时压得很低(因为随机初始化的 $q$ 离真后验十万八千里、KL 散度爆炸),随着训练它一步步向天花板靠拢。两者之间的缝就是 $D_{\mathrm{KL}}(q\|p(z|x))$——你可以从 TensorBoard 的 ELBO 曲线斜率读出"今天你的近似后验质量改善了多少"。

但要注意一个微妙的点:极大化 ELBO 时,天花板本身也在动。每一步 SGD 同时更新生成模型参数 $\theta$(影响天花板的高度)和变分参数 $\phi$(影响地板和缝的大小)。这是 EM 算法在连续优化里的推广——经典 EM 把这两步分开:E-step 把 $q$ 推近 $p(z|x)$(地板向上贴),M-step 把 $\theta$ 推到让 ELBO(也即下界)更大(天花板和地板一起向上抬)。VAE 的训练把两步混在同一个梯度里,一次反传同时改 $\theta$ 和 $\phi$。这种"amortized inference"是 Kingma & Welling 2013 的核心贡献,它让原本要每个数据点跑一次推断的代价,被一个共享的 encoder 网络摊销掉。

扩散模型里的 ELBO:T 步链式分解

把 VAE 推广到 $T$ 步的 Markov chain $x_0 \to x_1 \to \cdots \to x_T$(其中 $x_T \approx \mathcal{N}(0, I)$ 是纯噪声),完全相同的 Jensen 推导给出:

$$\log p_\theta(x_0) \geq \mathbb{E}_q\!\left[\log p(x_T) + \sum_{t \geq 1} \log \frac{p_\theta(x_{t-1}|x_t)}{q(x_t|x_{t-1})}\right] = -\underbrace{L_0}_\text{recon} - \sum_{t=1}^{T-1} \underbrace{L_t}_\text{KL} - \underbrace{L_T}_\text{prior}$$

每一项 $L_t$ 都是两个高斯之间的 KL,可以闭式写出(不需要采样近似)。Ho et al. 2020 在 DDPM 论文里把它进一步化简成"预测噪声的 MSE"——所有归一化常数、$\beta_t$ 权重在适当的参数化下要么消失、要么被吸收到 loss 权重里,剩下的就是那行简单到不可思议的损失 $\|\epsilon - \epsilon_\theta\|^2$。从这个视角看,DDPM 的"简单 loss" 并不是工程上的妥协,而是一个完整 T 步 ELBO 在合适参数化下的天然结果。这也是为什么扩散模型有真正的概率解释——它的训练目标是一个严格的对数似然下界,不是某种启发式重建误差。

tight 与 loose:训练可行性的代价

有两个极端值得记住。极端一:$q$ 等于真后验 $p(z|x)$。此时 ELBO = $\log p(x)$,下界紧到顶。但真后验在高维下通常无法解析,所以这种"紧 ELBO"只在玩具问题里见过。极端二:$q$ 取 mean-field 高斯(各维独立、单峰、参数仅由 $x$ 通过一个神经网络决定)。这种 $q$ 表达力很弱,永远无法贴合真后验,所以 ELBO 永远低于 $\log p(x)$——你得到的是一个有偏的对数似然估计。但好处是:可微、可批量、可 GPU 加速。这就是工程取舍。

现代研究的整条进步轨迹,几乎就是在把 $q$ 做得更像真后验:normalizing flow 让 $q$ 拥有任意复杂的密度;IWAE 用多采样 + log-sum-exp 拉紧下界;hierarchical VAE 把 $z$ 分层,每层条件高斯,整体表达力远超单层;diffusion 干脆放弃显式 $q$,而用一个固定的 forward 噪声过程构造它——而由于 forward 是可控的,KL 项总能闭式算出来。所有这些技巧都在做同一件事:抬高地板。当你下次看到一篇论文宣称"我们改进了 ELBO"时,几乎总可以翻译成"我们让 $q$ 更接近真后验"。这是判断生成模型工作是否扎实的快速 sanity check。

$\log p(x)$(天花板) ELBO 地板(训练中抬升) gap = $D_{\mathrm{KL}}(q\|p(z|x))$ gap → 0 训练步数 → 对数似然 起步:loose tight

下面把 ELBO 的四步推导拆成按钮——点击按顺序展开。这里有一个特别重要的不变量需要你盯住:从 ① 到 ②,每一步等号两边的值都没有变,变的只是"这件事看起来像什么"——直到第 ③ 步才出现真正的不等号 $\geq$,那一步是整套理论里唯一的"近似"。换句话说,ELBO 这块地板与 $\log p(x)$ 天花板之间的缝隙,完全来自 Jensen 不等式那一步。理解了这一点,你就理解了"为什么改进 $q$ 等于抬高 ELBO"。

$\log p(x) = \log \int q(z|x) \cdot \dfrac{p(x,z)}{q(z|x)}\, dz$ — 引入任意 $q$,等式严格相等。

手算:一个最小的 ELBO

来做一个能拿铅笔跟着算的例子,把抽象公式落到具体数字。设 $z$ 是一维标量;真模型 $p(x|z) = \mathcal{N}(z, 1)$(给定 $z$,数据是单位方差高斯),先验 $p(z) = \mathcal{N}(0, 1)$;变分后验 $q(z|x) = \mathcal{N}(\mu, \sigma^2)$,取 $\mu = 0.5, \sigma = 1.2$。给定一个数据点 $x = 1$,用 reparameterization 采到一次 $\epsilon = 0.1$。下面把这个数据点上的单样本 ELBO 完整算出来——总共三步,每一步都是小学算术加一个对数。

重建项 $\mathbb{E}_q[\log p(x|z)]$ $z = \mu + \sigma\epsilon = 0.5 + 1.2 \cdot 0.1 = 0.62$;代入高斯对数密度: $\log p(x|z) = -\tfrac{1}{2}(x - z)^2 - \tfrac{1}{2}\log 2\pi = -\tfrac{1}{2}(1 - 0.62)^2 - 0.919 = -\tfrac{1}{2}(0.144) - 0.919 = -0.072 - 0.919 = -0.991$。
KL 项(两个 1D 高斯之间,闭式) $D_{\mathrm{KL}}\big(\mathcal{N}(\mu, \sigma^2) \| \mathcal{N}(0, 1)\big) = \tfrac{1}{2}\big(\mu^2 + \sigma^2 - 1 - \log\sigma^2\big) = \tfrac{1}{2}(0.25 + 1.44 - 1 - \log 1.44) = \tfrac{1}{2}(0.69 - 0.365) = \tfrac{1}{2}(0.325) = 0.163$。
ELBO $\mathcal{L}(x) = -0.991 - 0.163 = -1.154$。地板就在 $-1.154$ 这里;天花板 $\log p(x)$ 至少 $\geq -1.154$。差距是多少?等于 $D_{\mathrm{KL}}(q\|p(z|x))$——你选的 $q$ 离真后验越远,缝隙越大。训练(即极大化 ELBO)就是同时把这个数往上推、把 $q$ 推向真后验。

三个反直觉的常见误区

  1. "ELBO 就是 $\log p(x)$ 的近似"——错。反例:取 $q(z|x) = \mathcal{N}(0, 100)$(一个荒唐的"平坦"后验,几乎不依赖 $x$),ELBO 可能是 $-50$ 而真实 $\log p(x) = -2$——差出 48 个 nats。ELBO 是严格的下界,不是无偏估计;只有当 $q$ 接近真后验 $p(z|x)$ 时它才会"接近"对数似然。报告"我的 VAE ELBO 是 -120"≠"我的对数似然是 -120",后者可能高得多。
  2. "只要重建好,ELBO 就高"——错。反例:把变分后验的 $\sigma$ 压到 $10^{-6}$,重建误差几乎完美 $\approx 0$ 没错,但 KL 项 $\tfrac{1}{2}(\mu^2 + 10^{-12} - 1 - \log 10^{-12}) \approx \tfrac{1}{2}(- 1 + 27.6) = 13.3$ 直接爆炸——ELBO 反而骤降,远不如让 $\sigma$ 稍大、重建稍差的方案。两项必须协同优化,KL 不是"装饰性正则",它是 ELBO 公式里不可拆解的一半。$\beta$-VAE 故意把 KL 权重调大调小,就是在这条曲线上选不同的点。
  3. "loose ELBO 也是无偏似然估计,多采样平均就好了"——错。反例:用 mean-field 高斯(各维独立)作为 $q$,它永远捕捉不到双峰真后验的形状;任何单峰 $q$ 在双峰 $p(z|x)$ 下都有非零 KL,ELBO 永远低于 $\log p(x)$。Burda et al. 2016(IWAE 论文)证明,单样本 ELBO 系统性低估 $\log p(x)$,即使训到收敛、平均无穷次样本,结果依然有偏——这是偏差(bias)不是方差(variance)。要修,需要 importance-weighted ELBO(多采样、log-sum-exp,等价于把 bound 拉紧)、normalizing flow(让 $q$ 表达力变强),或者干脆放弃 $q$ 走 score-based 路线。

相关概念

  • VAE Encoder — ELBO 中 $q(z|x)$ 的具体参数化网络,输出 $\mu(x), \sigma(x)$ 让你能从 $q$ 采样并算 KL。
  • Reparameterization Trick — 让 ELBO 里那个对 $q$ 的期望对 $\phi$ 可微的工程把戏,是 ELBO 能用 SGD 训练的前提。

来源

  1. diffusion-principles-vizuara · L2 · Variational Autoencoders Explained From Scratch · (00:05)
  2. diffusion-principles-vizuara · L2 · 「猫被瞬移走 / secret recipe / push the bound up」 · (06:45)
  3. cmu-10799-diffusion-flow · L1 · "the greatest mathematical trick of all time" · (00:02)
查看 240 条课程原文出处
CourseLectureTimeExcerpt
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 00: 02 All right, class. Class, shall we begin? All right, class. Class, shall we begin? Shall we begin? Shall we begin? Shall we begin? Welcome to 10799. Welcome to 10799. Welcome to 10799. My name is Kelly…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 01: 24 2025 will be the Italian brain rods. Uh so for those of you guys who don't know so for those of you guys who don't know so for those of you guys who don't know this is this is this is this is bombado …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 02: 23 the same underlying technology and that the same underlying technology and that is is is hello and that is we got stuck. Oh and that is and that is we got stuck. Oh and that is this uh diffuser um and…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 03: 15 implement these things um like in Python implement these things um like in Python which is the language that we use every which is the language that we use every which is the language that we use ever…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 04: 06 probably need to first answer what is a probably need to first answer what is a good generation uh image generation good generation uh image generation good generation uh image generation model right …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 04: 52 they like have a lot of like weird artifacts? Um so you probably don't artifacts? Um so you probably don't artifacts? Um so you probably don't really understand what what weird really understand what …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 05: 34 or can I use another image to you know or can I use another image to you know to specify what I want to generate? For to specify what I want to generate? For to specify what I want to generate? For ex…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 06: 09 model. Can we do that? Um so that's the second aspect. And then the third aspect second aspect. And then the third aspect second aspect. And then the third aspect is basically how fast is the generati…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 06: 45 I reduce war clock time with the same amount uh same number of network amount uh same number of network amount uh same number of network evaluation which means that like you evaluation which means tha…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 07: 31 beginnings village and where you you you beginnings village and where you you you get to like fail pretty fast and safely get to like fail pretty fast and safely get to like fail pretty fast and safel…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 08: 07 bosses throughout the way. And uh, these bosses throughout the way. And uh, these ones are not going to kill you and ones are not going to kill you and ones are not going to kill you and they're just …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 08: 52 environment try to get yourself familiar environment try to get yourself familiar with GPU training and also just build with GPU training and also just build with GPU training and also just build two …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 09: 42 Okay. And uh at the end of the class uh Okay. And uh at the end of the class uh which is the last week uh of class, it's which is the last week uh of class, it's which is the last week uh of class, it…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 10: 19 to actually present your poster in class you know tell your classmate what you you know tell your classmate what you you know tell your classmate what you did and then the third thing is you did and t…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 10: 58 right. Uh so a couple of things to remind you guys is that uh this class is remind you guys is that uh this class is remind you guys is that uh this class is completely AI friendly and open completely…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 11: 53 people. However, uh because we're so people. However, uh because we're so flexible on our you know just like open flexible on our you know just like open flexible on our you know just like open source…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 12: 33 like it's totally fine and yeah just in like it's totally fine and yeah just in general do not plagiarize. It's you know general do not plagiarize. It's you know general do not plagiarize. It's you kn…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 13: 20 basically um so uh myself and the TAs are going to so uh myself and the TAs are going to so uh myself and the TAs are going to start hosting office hours um next week. start hosting office hours um ne…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 14: 00 they do know um you know they they have done research in terms of well Douglas done research in terms of well Douglas done research in terms of well Douglas has done research in terms of flow has done…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 14: 44 probably your classmate are going to figure like a lot of things out on your figure like a lot of things out on your figure like a lot of things out on your own through ex own through ex own through e…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 15: 27 by anything I mean you're not allowed to by anything I mean you're not allowed to use your laptop at all. So just going to use your laptop at all. So just going to use your laptop at all. So just goin…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 16: 07 website. Basically, they're just like website. Basically, they're just like easy conceptual questions. Uh, and they easy conceptual questions. Uh, and they easy conceptual questions. Uh, and they shou…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 16: 54 with me uh to collect feedback on your learning experience in this class. And learning experience in this class. And learning experience in this class. And uh yeah basically it's just going to be uh y…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 17: 31 talk about like how people have been trying to improve upon those uh basic trying to improve upon those uh basic trying to improve upon those uh basic algorithms throughout the years. Uh and algorithm…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 18: 17 then week six is going to be we're going then week six is going to be we're going to be talking about probably a lot of to be talking about probably a lot of to be talking about probably a lot of you …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 19: 06 post it on there. Um and uh post it on there. Um and uh the gray scope, which I have not add the gray scope, which I have not add the gray scope, which I have not add anyone to, by the way, so don't p…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 19: 56 have announcement thank you um forum uh have announcement thank you um forum uh uh uh discussions you and feel free to uh uh discussions you and feel free to uh uh discussions you and feel free to sha…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 20: 39 constant basis? Raise your hand if you do. All right. Amazing. All right. Amazing. Actually uh a lot of people do. For Actually uh a lot of people do. For Actually uh a lot of people do. For those of …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 21: 30 free. Uh so in total you should be able to get around like 280 GP hours on L4DS. to get around like 280 GP hours on L4DS. to get around like 280 GP hours on L4DS. Um and that should be plenty and shou…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 22: 19 AWS is, but it's just like another uh AWS is, but it's just like another uh cloud comput service that is very very cloud comput service that is very very cloud comput service that is very very popular…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 23: 05 you are a MLDD student or LTI student or probably any SCES student at this point probably any SCES student at this point probably any SCES student at this point uh you can probably get access to the u…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 23: 45 any lab clusters, uh then uh I have listed a bunch of uh freeish and listed a bunch of uh freeish and listed a bunch of uh freeish and cheapish uh cloud comput service that cheapish uh cloud comput se…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 24: 40 you're not using it. Do not go bankrupt you're not using it. Do not go bankrupt uh when you're training models. This is uh when you're training models. This is uh when you're training models. This is …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 25: 23 finalize the list of registered student and uh yeah but like whether or not you and uh yeah but like whether or not you and uh yeah but like whether or not you are going to be able to register please …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 26: 02 And uh oh yeah, one other thing that I need to mention there are a couple of need to mention there are a couple of need to mention there are a couple of people already done that but which you people a…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 26: 43 streamed on Zoom like we have seen here. Thank you Jacob for doing that. Um so Thank you Jacob for doing that. Um so Thank you Jacob for doing that. Um so yeah so basically what you can do here yeah s…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 27: 27 Ponupto I don't know uh like the the the Ponupto I don't know uh like the the the thing that I also shared on Discord and thing that I also shared on Discord and thing that I also shared on Discord an…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 28: 09 uh feel free to read it especially this one feel free to read it especially this one feel free to read it especially this one book uh so this book is written by a book uh so this book is written by a …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 28: 50 going to be the same um yeah so feel free to read this book this is amazing free to read this book this is amazing free to read this book this is amazing uh and besides this class if you want to uh an…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 29: 37 and diffusion and flow matching models. and diffusion and flow matching models. Uh so nothing is important, nothing else Uh so nothing is important, nothing else Uh so nothing is important, nothing el…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 30: 14 in the class if you don't want to. Uh I in the class if you don't want to. Uh I know I don't like taking classes uh when know I don't like taking classes uh when know I don't like taking classes uh wh…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 30: 51 some of my survey you know if you need some of my survey you know if you need some extra credits. Um but anyway um some extra credits. Um but anyway um some extra credits. Um but anyway um yeah but ju…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 32: 01 given Y. Is that what you said? Okay. So, but what is P? >> I mean, I guess I could model X. >> I mean, I guess I could model X. >> Okay. Yeah. Yeah. Yeah. Pretty much. >> Okay. Yeah. Yeah. Yeah. Pret…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 33: 04 of Y right what are they? So X and Y's of Y right what are they? So X and Y's are random variables. So basically just are random variables. So basically just are random variables. So basically just th…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 33: 47 know which one is more unfortunate actually. Um but uh basically we usually actually. Um but uh basically we usually actually. Um but uh basically we usually denote it as P of X. denote it as P of X. …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 34: 38 consider X or only consider Y. And the consider X or only consider Y. And the conditional distribution is when you are conditional distribution is when you are conditional distribution is when you are…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 35: 21 just do a little bit of algebra you're just do a little bit of algebra you're going to be able to get this very nice going to be able to get this very nice going to be able to get this very nice formu…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 36: 11 y tells you nothing about x or knowing x tells you nothing about y. So you uh tells you nothing about y. So you uh tells you nothing about y. So you uh just do not change your posterior. Oh just do no…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 37: 01 person distribution we have like the person distribution we have like the average rate as our uh parameter. average rate as our uh parameter. average rate as our uh parameter. Uh basically it's just l…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 37: 57 we call the probability of the data we call the probability of the data given our learn the model parameter as given our learn the model parameter as given our learn the model parameter as the likelih…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 39: 28 Richard Feman. Uh apparently people found it on his blackboard after he died found it on his blackboard after he died found it on his blackboard after he died or something. I don't know if that after …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 40: 06 bunch of images of a bedroom uh and then the label of the bedroom is whether or the label of the bedroom is whether or the label of the bedroom is whether or not it's a luxurious bedroom. I've been no…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 40: 51 we're trying to learn is what a bedroom should look like or what a luxurious or should look like or what a luxurious or should look like or what a luxurious or what a cheap I guess uh bedroom should w…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 41: 36 should be? So basically do we believe this is a gausian? Do we believe this is this is a gausian? Do we believe this is this is a gausian? Do we believe this is a mixture of gausian? Uh what kind of a…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 42: 26 necessarily able to uh be able to tell necessarily able to uh be able to tell like how likely are your uh like is some like how likely are your uh like is some like how likely are your uh like is some…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 43: 06 labeling or like uh annotations and stuff like that. Okay, stuff like that. Okay, stuff like that. Okay, cool. So how to train your generated cool. So how to train your generated cool. So how to train…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 44: 05 if my um parameters are correct then the if my um parameters are correct then the real data point should have high real data point should have high real data point should have high likelihood. Uh whic…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 44: 52 image is composed by a bunch of um pixels pixels pixels uh then um by the probability chain rule uh then um by the probability chain rule uh then um by the probability chain rule which just means that…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 45: 39 able to get this thing that is very nice where the log likely the conditional log where the log likely the conditional log where the log likely the conditional log likelihood of your case element is o…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 46: 23 LLMs. Yes, we got LM already. Yeah, LLMs. Yes, we got LM already. Yeah, super easy. This is why people love it. super easy. This is why people love it. super easy. This is why people love it. Um, yeah…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 47: 07 we cannot see directly. Uh so for example how a person look is largely example how a person look is largely example how a person look is largely determined by their genes which is not determined by th…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 47: 52 such a beautiful paper. I was like oh my god this this thing is like who who god this this thing is like who who god this this thing is like who who thought of this thing? This is like thought of this…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 48: 37 maximizing the likelihood of the data we maximizing the likelihood of the data we also need to do another thing which is also need to do another thing which is also need to do another thing which is t…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 49: 19 two distributions. So we're trying to two distributions. So we're trying to minimize the difference between the minimize the difference between the minimize the difference between the distribution of …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 50: 04 measure it by like so all the divergence that you output should be greater than that you output should be greater than that you output should be greater than zero and it should only be equal greater z…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 50: 53 kind of the intuition. Uh but basically what's going to happen here is that what's going to happen here is that what's going to happen here is that we're going to be using the KL we're going to be usi…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 51: 41 when you're trying to decode it given when you're trying to decode it given unz and then the other thing is what we unz and then the other thing is what we unz and then the other thing is what we call…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 52: 29 a expectation of this thing and then like By bas rule again you are going to like By bas rule again you are going to like By bas rule again you are going to be able to get from the log px is be able t…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 53: 16 part is the the elbow the elbow. The second part is the kale the elbow. The second part is the kale the elbow. The second part is the kale divergence that we're trying to uh divergence that we're tryi…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 54: 07 distribution uh like then it's just distribution uh like then it's just going to be like the the the the going to be like the the the the going to be like the the the the integral of the the joint uh …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 55: 03 >> yeah because the because log is concave. Yeah, that's right. Uh so yeah, Jensen Yeah, that's right. Uh so yeah, Jensen Yeah, that's right. Uh so yeah, Jensen inequality here and we get an inequalit…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 55: 51 about today. It's called uh Catch Me If about today. It's called uh Catch Me If You Can. Uh basically, I don't know if You Can. Uh basically, I don't know if You Can. Uh basically, I don't know if you…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 56: 27 he was wearing a suit or something like he was wearing a suit or something like that and people were laughing me hah and that and people were laughing me hah and that and people were laughing me hah a…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 57: 08 but because he's getting like really good at like faking things so he's able good at like faking things so he's able good at like faking things so he's able to like fool the detective in into to like …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 57: 45 uh the model that we're going to be talking about. So you have a generator talking about. So you have a generator talking about. So you have a generator uh and then you have a discriminator. uh and th…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 58: 28 network or GAN. Uh this was also developed in I guess 2014. So you see developed in I guess 2014. So you see developed in I guess 2014. So you see how like a decade ago I don't know how like a decade …
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 59: 02 first just like sample something um from a easy to sample distribution such as a easy to sample distribution such as a easy to sample distribution such as like gausian and then you have a like gausian…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 59: 48 that uh basically this is just a regular you know classification loss right like you know classification loss right like you know classification loss right like binary classification law for the for b…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 60: 59 something from it I guess why does the well actually this is not exactly the well actually this is not exactly the well actually this is not exactly the case we're going to be talking about how case w…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 61: 37 we don't even have P of Z given X in our final loss. Actually our final loss is final loss. Actually our final loss is final loss. Actually our final loss is this thing. So our final loss is this this…
cmu-10799-diffusion-flow L4: CMU 10799 S26: Diffusion & Flow Matching - Lecture 1 - Basics of Probabilistic & Generative Modeling 62: 30 understand how diffusion works. Uh so in understand how diffusion works. Uh so in uh general next class we're going to be uh general next class we're going to be uh general next class we're going to b…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 00: 03 To really understand about how this To really understand about how this video links to image diffusion, I video links to image diffusion, I video links to image diffusion, I believe that we first need…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 00: 49 two. So this is my vector. That's how you represent this vector. That's how you represent this vector. That's how you represent this vector. But how would you represent an entire But how would you rep…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 01: 31 this is 128x 128 pixels, I will arrange these 128x 128 pixels in a in a these 128x 128 pixels in a in a these 128x 128 pixels in a in a u column. So imagine that you have this u column. So imagine tha…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 02: 23 somewhere here in a pixel space. This is another image. It may live somewhere another image. It may live somewhere another image. It may live somewhere here in pixel space. So pixel space is here in p…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 03: 17 you really need to understand. How is the concept of probability How is the concept of probability How is the concept of probability distribution linked to that of an image distribution linked to that…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 04: 03 and this is what I would have explained to you if chat GPT or Gemini did not to you if chat GPT or Gemini did not to you if chat GPT or Gemini did not exist but now let me actually take you exist but …
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 06: 01 Let's see. So the reason I'm showing Let's see. So the reason I'm showing this to you is because I want this this to you is because I want this this to you is because I want this learning material to …
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 06: 46 right? It wouldn't even look like a cat or a tree. It would look like random or a tree. It would look like random or a tree. It would look like random noise. So 99% of points in the pixel noise. So 99…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 07: 43 where all Chinese symbols are living. where all Chinese symbols are living. That's the probability distribution of That's the probability distribution of That's the probability distribution of this im…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 08: 29 higher probability of finding the Chinese symbol. Right? So now if you Chinese symbol. Right? So now if you Chinese symbol. Right? So now if you take a look at this 3D plot, right? If I take a look at…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 09: 19 pixels. Pixels are converted into flat vectors. Those are represented in higher vectors. Those are represented in higher vectors. Those are represented in higher dimensional space. dimensional space. …
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 10: 13 saying that we are finding this this probability distribution. If you ask if probability distribution. If you ask if probability distribution. If you ask if you go to chat GPT right now or let's you g…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 11: 26 finding the true probability distribution or getting as close to it distribution or getting as close to it distribution or getting as close to it as possible. Right? So as possible. Right? So as possi…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 12: 25 Right? So if I the red line is my true Right? So if I the red line is my true probability distribution probability distribution probability distribution and the purple line the purple line is and the …
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 13: 22 definitely do look like Chinese symbols definitely do look like Chinese symbols because this purple probability because this purple probability because this purple probability distribution is close to…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 14: 16 sampled that purple probability distribution. distribution. Sampled from that purple probability Sampled from that purple probability Sampled from that purple probability distribution and created this…
diffusion-lm-vizuara L4: Lecture 3: Generative AI through a probabilistic lens 15: 26 underlying probability distribution. Let me write it here. Purpose Purpose of a diffusion model is to find Purpose of a diffusion model is to find the underlying probability distribution. That is the …
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 00: 03 Hello everyone and welcome to the first Hello everyone and welcome to the first lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion mo…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 01: 40 the heights of maybe a few friends of yours and uh you have noted them down in yours and uh you have noted them down in yours and uh you have noted them down in your diary. your diary. your diary. So …
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 02: 47 your school and uh you want to make a your school and uh you want to make a guess a reasonable guess about what guess a reasonable guess about what guess a reasonable guess about what would be the hei…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 03: 50 will be that I think the new student will have uh you know a height of maybe will have uh you know a height of maybe will have uh you know a height of maybe 5' 5 in. 5' 5 in. 5' 5 in. So your predicti…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 05: 15 are very less to be seen. Now if we have this distribution with Now if we have this distribution with Now if we have this distribution with us, we can easily say that us, we can easily say that us, we…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 06: 29 curve is the true distribution. So this is an example of a decent match. So this is an example of a decent match. So this is an example of a decent match. I can see that the model distribution I can s…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 07: 24 height of the new student what I see is that I'll make a guess that okay fine that I'll make a guess that okay fine that I'll make a guess that okay fine the new student will have a height of the new …
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 08: 24 Uh true distribution. So true distribution is something which is uh in distribution is something which is uh in distribution is something which is uh in in supervised learning you're trying to in supe…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 09: 34 models are trained to do. They are trained so that the predicted trained so that the predicted trained so that the predicted distribution matches as close as distribution matches as close as distribut…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 10: 35 see something visually it stays in our mind much longer than reading a text or mind much longer than reading a text or mind much longer than reading a text or hearing from someone. So let's look at he…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 12: 11 So visually if you just keep this So visually if you just keep this uh video in mind and uh video in mind and uh video in mind and you saw how the initial curve slowly you saw how the initial curve sl…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 13: 37 probably not thought of it as a case probably not thought of it as a case study which comes under the domain of study which comes under the domain of study which comes under the domain of deep generat…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 15: 00 and the deep generative model outputs a trained neural network that distribution. So there are three main points here. The So there are three main points here. The So there are three main points here.…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 16: 03 So there are two main goals of deep generative modeling. generative modeling. generative modeling. The first goal is that we always aim for The first goal is that we always aim for The first goal is t…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 17: 04 generative modeling. Realistic generation and controllable generation. generation and controllable generation. generation and controllable generation. Now let's move to the goal of deep Now let's move…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 18: 09 named as generative AI. But where does the name generative really come from? It the name generative really come from? It the name generative really come from? It it comes from the development of it co…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 19: 10 to generate new images of cats in this to generate new images of cats in this case. Okay. So now let's formally look at a Okay. So now let's formally look at a more complex example more complex exampl…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 20: 21 gives gives that to us beforehand. We get a small sneak peek into the true We get a small sneak peek into the true We get a small sneak peek into the true distribution through the data samples distrib…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 21: 24 represent a probability distribution. So orange contour is the true So orange contour is the true So orange contour is the true probability distribution and the black probability distribution and the …
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 22: 33 they developed they developed models which are trained to approximate models which are trained to approximate models which are trained to approximate the true data distribution of cats and the true da…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 23: 30 whenever we look at a cat in our mind we know that this is a cat because we have know that this is a cat because we have know that this is a cat because we have formed some assumption of the formed so…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 24: 36 if we take two numbers I can clearly see that the the discrepancy is the that the the discrepancy is the that the the discrepancy is the difference between the two numbers. difference between the two …
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 25: 38 screen. I'm initially going to just run this video and then um I'm going to this video and then um I'm going to this video and then um I'm going to explain different parts of it. Okay. So let's try to…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 28: 10 difference is a great start but it doesn't capture everything. doesn't capture everything. doesn't capture everything. For example, if we take the absolute For example, if we take the absolute For exa…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 29: 13 between the true distribution and the model distribution is actually less than model distribution is actually less than model distribution is actually less than one. which means that our model has one…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 30: 13 So we take the log of the ratio for all So we take the log of the ratio for all these different points and then just add these different points and then just add these different points and then just a…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 31: 16 into account by multiplying by a factor which is dependent on how likely the which is dependent on how likely the which is dependent on how likely the case is under the true distribution. case is unde…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 32: 24 distribution with a model distribution Q. It is zero when P and Q are identical Q. It is zero when P and Q are identical Q. It is zero when P and Q are identical and grows as Q becomes worse. Especial…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 33: 33 how do we start to train these deep generative models. So let's understand generative models. So let's understand generative models. So let's understand that. objectives of objectives of deep generati…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 35: 07 should always add up to one. should always add up to one. Take an example of a coin toss. Take an example of a coin toss. Take an example of a coin toss. Probability of heads is 0.5. Probability Proba…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 36: 05 of these operations will make sure that of these operations will make sure that the output is positive. A very common the output is positive. A very common the output is positive. A very common choice…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 37: 04 you divide all of these numbers by 90. So you get Alice, Bob and Charlie as 44, So you get Alice, Bob and Charlie as 44, So you get Alice, Bob and Charlie as 44, 33 and 22. And these numbers add up 33…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 38: 10 finding the scores for every single sample in this contour is going to be sample in this contour is going to be sample in this contour is going to be challenging because we do not even know challengin…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 39: 11 uh we are now going to briefly discuss uh we are now going to briefly discuss the different families of deep the different families of deep the different families of deep generative models. generative…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 40: 17 more probable and higher energy to data points which are less probable. points which are less probable. points which are less probable. So imagine that we collect the exam So imagine that we collect t…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 41: 17 Now what happens is that we want a model that can understand this we want a model that can understand this we want a model that can understand this pattern. So energy based models do not pattern. So e…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 42: 15 numbers around 80 have a lower energy. numbers around 60 have a lower energy. numbers around 60 have a lower energy. numbers around 60 have a lower energy. Numbers around 30 which are not very Numbers…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 43: 12 should receive a higher score. Once we understand this simple rule, the understand this simple rule, the understand this simple rule, the objective of energy based models is to objective of energy bas…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 44: 48 So energy based models uh was the first starting point for deep was the first starting point for deep was the first starting point for deep generative modeling. generative modeling. Then we moved on t…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 46: 02 After this context is seen by the LLM, After this context is seen by the LLM, it predicts the next token it predicts the next token it predicts the next token based on the token which gets the based o…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 47: 01 know take a look at this video first know take a look at this video first which explains variational autoenccoders which explains variational autoenccoders which explains variational autoenccoders at …
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 47: 59 image flowing through a autoenccoder you can see that the latin space the image can see that the latin space the image can see that the latin space the image is blurred because we are reducing the is …
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 48: 54 which is devoted exactly on this. The next prominent milestone is called as next prominent milestone is called as next prominent milestone is called as normalizing flows. normalizing flows. normalizin…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 50: 01 some simple distribution and you twist it, bend it, turn it to reach the true it, bend it, turn it to reach the true it, bend it, turn it to reach the true distribution which is exactly what the distr…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 51: 24 investigate investigate the difference between generated art and the difference between generated art and the difference between generated art and deep fix. We basically have to identify We basically …
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 52: 45 there is a generator and a discriminator. The generator discriminator. The generator discriminator. The generator generates images and the role of the generates images and the role of the generates im…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 54: 19 not be able to recognize this as a fake not be able to recognize this as a fake image. image. image. So see here real image detected the the So see here real image detected the the So see here real im…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 55: 15 student heights. But we only had a few student heights. But we only had a few data samples. So we had to use those data samples. So we had to use those data samples. So we had to use those data sample…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 56: 10 distribution mainly we looked at finding the normalizing factor is very difficult the normalizing factor is very difficult the normalizing factor is very difficult and uh that's why we need more and u…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 57: 04 families uh illustrate complimentary strategies for modeling complex strategies for modeling complex strategies for modeling complex distribution and they also provide distribution and they also provi…
diffusion-principles-vizuara L10: Lecture 1 - Deep Generative Modeling | Principles of Diffusion Models 57: 56 look at very practical examples as as we move along. move along. move along. Thank you very much everyone and uh I Thank you very much everyone and uh I Thank you very much everyone and uh I hope this…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 00: 05 Hello everyone and welcome to the second Hello everyone and welcome to the second lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 01: 17 from scratch. from scratch. And this lecture is And this lecture is And this lecture is going to be very important for the going to be very important for the going to be very important for the subsequ…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 02: 19 with the understanding of VAS. with the understanding of VAS. So first we'll take a simple example. So first we'll take a simple example. So first we'll take a simple example. Imagine that you have co…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 03: 19 in cursive. Some of us u don't write in cursive. So there are a u don't write in cursive. So there are a u don't write in cursive. So there are a lot of different ways in which people lot of different…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 04: 26 Then another time you press a button. Another time you get another hello. This Another time you get another hello. This Another time you get another hello. This time it might be some different student…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 05: 35 such that or generate a machine such that the samples drawn from that machine that the samples drawn from that machine that the samples drawn from that machine are from the probability distribution ar…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 06: 45 factors that determine the style of the factors that determine the style of the handwriting. handwriting. handwriting. So each student handwriting depends on So each student handwriting depends on So …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 08: 03 So my first intuition is that okay fine So my first intuition is that okay fine my first job is to figure out all these my first job is to figure out all these my first job is to figure out all these …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 09: 02 final shape of the handwriting. final shape of the handwriting. So once you capture the secret recipe So once you capture the secret recipe So once you capture the secret recipe for for for a student …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 09: 57 machine looks as follows. machine looks as follows. Secret recipe is given as an input to Secret recipe is given as an input to Secret recipe is given as an input to the machine. The machine takes thi…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 10: 52 interesting, right? It it it looks like okay fine this this it it makes a lot of okay fine this this it it makes a lot of okay fine this this it it makes a lot of sense. [snorts] sense. [snorts] sense…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 11: 53 now um in the rest of this lecture we We now um in the rest of this lecture we We are going to assume that are going to assume that are going to assume that for the handwriting example that we took fo…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 12: 54 not know these factors beforehand. That is why they are called as latent is why they are called as latent is why they are called as latent variables. variables. variables. So So intuitively we think t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 13: 51 y-axis is represented by the symbol zed 2. Zed1 stands for slantness of the 2. Zed1 stands for slantness of the 2. Zed1 stands for slantness of the handwriting and zed2 stands for neatness handwriting…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 14: 48 below it's it's not neat. So I have made some spelling mistake So I have made some spelling mistake So I have made some spelling mistake here. But what I want to show is I want here. But what I want t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 15: 33 mapping which I was talking about. For mapping which I was talking about. For example, if you pick a point here, you example, if you pick a point here, you example, if you pick a point here, you know …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 16: 31 in your class. in your class. So this is also called as the So this is also called as the So this is also called as the distribution in the latent space of the distribution in the latent space of the …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 17: 37 typical handwriting typical handwriting let's say it is divided into 28x 28 let's say it is divided into 28x 28 let's say it is divided into 28x 28 pixels. pixels. pixels. So there are 784 numbers rig…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 18: 39 latent space is has a mean of zero and a latent space is has a mean of zero and a standard deviation of one or any other standard deviation of one or any other standard deviation of one or any other v…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 19: 27 So for now what we have seen is that we have this general architecture for the have this general architecture for the have this general architecture for the machine which takes the secret recipe as ma…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 20: 34 another word for this machine which is called as decoder. called as decoder. called as decoder. Let us quickly summarize what all we Let us quickly summarize what all we Let us quickly summarize what …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 21: 44 vary. So here you can see that as the yellow dot moves around the Latin space, yellow dot moves around the Latin space, yellow dot moves around the Latin space, the handwriting on the right hand side …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 22: 44 next. So far we have used the decoder to generate samples from the latent generate samples from the latent generate samples from the latent variables. But what is this decoder variables. But what is t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 23: 55 we create a latent space with the latent variables. So just as before we assume variables. So just as before we assume variables. So just as before we assume that there are two latent variables zed th…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 24: 54 know. [snorts] Now let's say we have chosen a point in Now let's say we have chosen a point in Now let's say we have chosen a point in the Latin space which corresponds to the the Latin space which co…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 25: 56 we can represent digit 5 as a bunch of we can represent digit 5 as a bunch of pixels pixels pixels which are divided as 28 into 28 pixels where some pixels have a value of where some pixels have a val…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 27: 08 five, if you closely look at it, it it does look like a five. It's it's it's does look like a five. It's it's it's does look like a five. It's it's it's almost something like this. Okay. So it looks l…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 28: 08 is that with this approach we will get a fixed shape of five every time but we fixed shape of five every time but we fixed shape of five every time but we will not get variations of it and what will n…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 29: 10 restricted to zero and one. You are open to the possibility of to the possibility of to the possibility of a color which appears in between white a color which appears in between white a color which a…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 30: 08 the decoder does not give a single value but for every pixel the decoder gives but for every pixel the decoder gives but for every pixel the decoder gives the mean the mean the mean and the standard d…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 31: 13 deterministic approach only. So now let us So now let us So now let us have a simple example to understand how have a simple example to understand how have a simple example to understand how does the …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 32: 16 So the decoder is nothing but a neural network which is trained to take inputs network which is trained to take inputs network which is trained to take inputs from the latin space and generate from th…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 33: 11 somewhere towards the right, which means that it has to be a bit bright and the that it has to be a bit bright and the that it has to be a bit bright and the pixel value is high. Bright pixels have pi…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 34: 10 now we have covered now we have covered one part of the story which explains the one part of the story which explains the one part of the story which explains the variational autoenccoder. variational…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 35: 10 part, you will understand that we have made a major assumption. made a major assumption. made a major assumption. Remember when we talked about the Remember when we talked about the Remember when we t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 36: 14 recipe and then you need to map all these handwriting styles to the secret these handwriting styles to the secret these handwriting styles to the secret recipe and make all those points and recipe and…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 37: 13 the latin space to access to generate the latin space to access to generate digit five. One option is to access all digit five. One option is to access all digit five. One option is to access all poss…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 38: 04 Wouldn't it be better if we knew which part of the Latin space to access for part of the Latin space to access for part of the Latin space to access for the type of image we want to generate? the type…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 39: 00 decoder. So the overall architecture looks like So the overall architecture looks like So the overall architecture looks like this. this. this. First I get let's say you go to any First I get let's sa…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 40: 00 this architecture is very similar to a plain autoenccoder where you have an plain autoenccoder where you have an plain autoenccoder where you have an encoder which compresses the image into encoder wh…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 41: 03 a dis or or a region which is the most a dis or or a region which is the most probable region of finding the digit probable region of finding the digit probable region of finding the digit five instea…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 42: 01 we are going to look at a nice visual we are going to look at a nice visual description to understand this much description to understand this much description to understand this much better. better. …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 43: 12 individual points. So 99% of these points, they they produce garbage. So points, they they produce garbage. So points, they they produce garbage. So your your space your latent space is not your your …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 44: 14 regions which tell you that this is the region corresponding to the dog. This is region corresponding to the dog. This is region corresponding to the dog. This is the region corresponding to the cat e…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 45: 08 space which is continuous and the space which is continuous and the garbage gaps are now filled with garbage gaps are now filled with garbage gaps are now filled with distributions. So now you can saf…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 46: 07 So the two stories put together form the variational autoenccoder. variational autoenccoder. We first looked at the decoder where we We first looked at the decoder where we We first looked at the deco…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 47: 03 the areas in the latin space? How do you know which levers to pull in the know which levers to pull in the know which levers to pull in the decoder? This is given to you by the decoder? This is given …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 48: 08 uh styles of the students when they uh styles of the students when they wrote the word hello or the handwritten wrote the word hello or the handwritten wrote the word hello or the handwritten digits a…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 49: 09 going to give you the latent space distribution given the image. So that's distribution given the image. So that's distribution given the image. So that's why you see the notation kind of flipped why …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 50: 09 the encoder like this and the decoder the encoder like this and the decoder going from a smaller line here to a going from a smaller line here to a going from a smaller line here to a bigger line. In …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 51: 16 these variational autoenccoders trained. How do you train these to generate How do you train these to generate How do you train these to generate images? Let's say you give a sample of images? Let's s…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 52: 05 and then sample from it. So we are going to take a real example in the next class to take a real example in the next class to take a real example in the next class where you will see exactly how deep …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 52: 56 is not deterministic but rather it is is not deterministic but rather it is probabilistic. probabilistic. probabilistic. And finally in your mind I want you to And finally in your mind I want you to A…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 53: 48 image which is generated at the end. This is all that is happening in the This is all that is happening in the This is all that is happening in the variation autoenccoders. variation autoenccoders. va…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 55: 04 tells you the hidden factors of variation behind that image. variation behind that image. variation behind that image. [snorts] The machine tells you the hidden factors The machine tells you the hidde…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 56: 10 can lie. So it appears to be a very can lie. So it appears to be a very smart and a clever machine. smart and a clever machine. smart and a clever machine. Okay. Now you have these hidden factors Okay…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 57: 11 the final image should be as close as possible to the original image. So you possible to the original image. So you possible to the original image. So you can see that there has to be some kind can se…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 58: 10 encode and the second neural network is the machine which learns to decode. the machine which learns to decode. the machine which learns to decode. So somehow we have to train the So somehow we have t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 59: 20 a dot like this and you pass it to the a dot like this and you pass it to the decoder and you get a value of zero decoder and you get a value of zero decoder and you get a value of zero which which me…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 60: 07 our predicted distribution which should match as close as possible to the true match as close as possible to the true match as close as possible to the true distribution which we do not really distrib…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 61: 19 points in the latin space which is also given in this formula. given in this formula. given in this formula. I I don't want to stress too much on how I I don't want to stress too much on how I I don't…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 62: 21 the visualization before where we saw that it's like finding or that it's like finding or that it's like finding or sampling for from a very giant ocean and sampling for from a very giant ocean and sa…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 63: 17 that point being real? And this actually does not even make use And this actually does not even make use And this actually does not even make use of the encoder. This is all decoder of the encoder. Th…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 64: 26 Now the question is that what is the Now the question is that what is the elbow objective? [snorts] elbow objective? [snorts] elbow objective? [snorts] Well to understand this let's understand Well to…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 65: 25 I will not get very mathematical here and I will not focus too much on the and I will not focus too much on the and I will not focus too much on the derivation but I will provide a link in derivation …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 66: 24 So this essentially says that whatever image you're looking at the end whatever image you're looking at the end whatever image you're looking at the end is that image is that image is that image uh as…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 67: 33 in any fashion but in more realistic settings the in more realistic settings the in more realistic settings the distribution is more likely to be distribution is more likely to be distribution is more…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 68: 30 will be average in terms of the will be average in terms of the slantness, in terms of the neatness etc. slantness, in terms of the neatness etc. slantness, in terms of the neatness etc. So we are So …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 69: 32 how likely is the reconstructed um output. um output. um output. What is the probability of that? And we What is the probability of that? And we What is the probability of that? And we want to maximiz…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 70: 35 our encoder distribution to stay as close as possible to a gshian which close as possible to a gshian which close as possible to a gshian which comes from our human intuition that comes from our human…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 71: 38 want to compress the earth cat into something which captures the essence of something which captures the essence of something which captures the essence of the cat. We want to send a secret recipe the…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 72: 30 objective and we try to maximize the elbow. elbow. elbow. So since the true objective is always So since the true objective is always So since the true objective is always greater than the elbow, maxi…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 73: 27 want to move our code to the yellow distribution. We want to move it as distribution. We want to move it as distribution. We want to move it as close as possible to the normal close as possible to the…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 74: 23 what remains and it makes sense from an what remains and it makes sense from an intuitive standpoint because later intuitive standpoint because later intuitive standpoint because later people forget t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 75: 20 going to come together very nicely in this last practical example. Our task is this last practical example. Our task is this last practical example. Our task is to train a variation autoenccoder to to…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 76: 21 hidden factors which we have absolutely hidden factors which we have absolutely no clue what they are but the encoder is no clue what they are but the encoder is no clue what they are but the encoder …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 77: 13 So remember our decoder setup looks like this. It's it it needs to take the this. It's it it needs to take the this. It's it it needs to take the sample from the latent space and then it sample from t…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 78: 10 So it should take the latent vector and So it should take the latent vector and it should generate the reconstructed it should generate the reconstructed it should generate the reconstructed image. Ri…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 79: 07 first machine do? The first machine actually does the reverse. It takes the actually does the reverse. It takes the actually does the reverse. It takes the image as the input, the real image, and imag…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 80: 13 region which has the mu of.5 comma.5 and region which has the mu of.5 comma.5 and the sigma of 2a 2. Right? So you need the sigma of 2a 2. Right? So you need the sigma of 2a 2. Right? So you need two …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 81: 01 the GitHub repo in the description the GitHub repo in the description section. section. section. [snorts] [snorts] Okay. So overall the encoder decoder Okay. So overall the encoder decoder Okay. So ov…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 81: 56 So this is our entire encoder decoder So this is our entire encoder decoder architecture whatever we discussed as architecture whatever we discussed as architecture whatever we discussed as this machi…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 82: 51 This compares every pixel of the input with the output. If the original pixel with the output. If the original pixel with the output. If the original pixel is black and the VA predicts white, the is b…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 83: 46 zero and a standard deviation of one. To ensure that the mean is zero, we add a ensure that the mean is zero, we add a ensure that the mean is zero, we add a penalty if the mean deviates from zero. pe…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 84: 40 because log of a small value is a big uh number and there is a negative sign number and there is a negative sign number and there is a negative sign associated with it. So this will always associated …
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 85: 34 the uh description. But first I just want to want to want to describe everything that we have describe everything that we have describe everything that we have conceptually described in a mathematical…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 86: 26 for longer duration maybe we'll get an for longer duration maybe we'll get an even better distribution which is even better distribution which is even better distribution which is centered at zero and…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 87: 21 this training proceeds and this is the this training proceeds and this is the quality of the reconstructions because quality of the reconstructions because quality of the reconstructions because we ha…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 88: 15 latin space uh you see that the it's it's kind of blur in some cases which it's kind of blur in some cases which it's kind of blur in some cases which we'll come to in in in some time but you we'll co…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 89: 05 is that it it produces blurry outputs and there is a very nice reason for it. and there is a very nice reason for it. and there is a very nice reason for it. I will share the link in the description I…
diffusion-principles-vizuara L9: Lecture 2 - Variational Autoencoders Explained From Scratch | Principles of Diffusion Models 89: 57 applications I have seen and uh there are encoders and distribute and and are encoders and distribute and and are encoders and distribute and and decoders right so you also have decoders right so you …