cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 00:
02 | All right. All right.
Right. So, so far what we have learn we Right. So, so far what we have learn we Right. So, so far what we have learn we
have learned about like pretty much have learned about lik… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 00:
56 | to do distillation or uh you know self to do distillation or uh you know self
dissolations or flow maps or consistency dissolations or flow maps or consistency dissolations or flow maps or consistency… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 01:
45 | who, actually, I'll just name who, like
the people from Anthropic have claimed the people from Anthropic have claimed the people from Anthropic have claimed
to me that uh I I believe it's ICML 2024 to… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 02:
37 | uh I mean obviously you know we have
other important modalities right for other important modalities right for other important modalities right for
example like molecules or like DNA example like mole… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 03:
35 | >> Near space and do like a nearest
neighbors type thing to match it. neighbors type thing to match it. neighbors type thing to match it.
>> Uh so diffusion model on a continuous >> Uh so diffusion mo… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 04:
44 | then embedding is what goes through
>> so you first take the text and go to >> so you first take the text and go to >> so you first take the text and go to
embedding embedding existing embedding embed… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 05:
26 | running out of the allergyfree options.
Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy
sensitive option. Okay. Anyway, you're sensitive option… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 06:
19 | And then uh for flow matching is pretty And then uh for flow matching is pretty
much the same thing, but instead of much the same thing, but instead of much the same thing, but instead of
learning the… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 07:
03 | adding noise process is the forward
process and the gradually d noiseis process and the gradually d noiseis process and the gradually d noiseis
process is the reverse process. Right. process is the re… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 08:
09 | version of a tweet from Andre Kapathi uh
so let's identify some of the noise in so let's identify some of the noise in so let's identify some of the noise in
this tweet what do we see this tweet what … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 09:
15 | then you get some weird emojis and you then you get some weird emojis and you
get some like out of place like six, get some like out of place like six, get some like out of place like six,
seven like … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 10:
13 | >> Yeah. Okay.
>> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh
do you have allergy? Okay. This is also do you have a… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 11:
23 | from the like original uh clean data to
some random tokens right in the some random tokens right in the some random tokens right in the
vocabulary. So that basically say like vocabulary. So that basic… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 12:
23 | vector. Uh then basically say we have a vector. Uh then basically say we have a
beta chance. This is just a some chance beta chance. This is just a some chance beta chance. This is just a some chance
… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 13:
09 | is why this is like one minus you know is why this is like one minus you know
every everything else has one beta over every everything else has one beta over every everything else has one beta over
th… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 14:
08 | sentences with real vocabulary, it's not sentences with real vocabulary, it's not
necessary that the noise sentence is necessary that the noise sentence is necessary that the noise sentence is
actuall… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 15:
10 | then you know so you just so the first then you know so you just so the first
row is uh I second row is love uh third row is uh I second row is love uh third row is uh I second row is love uh third
ro… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 16:
21 | that by induction we can not only get
like basically it's the same thing as like basically it's the same thing as like basically it's the same thing as
like you know two gausians added like you know t… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 17:
11 | this is like literally how to jump this is like literally how to jump
directly from x0 to xt is you just like directly from x0 to xt is you just like directly from x0 to xt is you just like
kind of li… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 18:
24 | as the beta t in uh DDPM. as the beta t in uh DDPM.
>> Yeah. Don't worry. Everything's gonna be Don't worry. Everything's gonna be
simplified later on in this class. right simplified later on in this … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 19:
40 | >> Estimate these points and then Q >> Estimate these points and then Q
transpose. transpose. transpose.
>> Yes, pretty much. And then specifically, >> Yes, pretty much. And then specifically, >> Yes,… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 20:
37 | because you're getting you're marovian
like everything is marovian here. So you like everything is marovian here. So you like everything is marovian here. So you
can ignore the x0 here. So then this c… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 21:
35 | categorical distribution that is defined categorical distribution that is defined
by this black font term here. >> Oh dot is like elementwise >> Oh dot is like elementwise
multiplication. Anyway, hone… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 22:
42 | difficult to define in this case. Right?
What is the other thing? What is the other thing? What is the other thing?
>> Tokens. >> Tokens. >> Tokens.
>> The what >> The what >> The what
>> the probabil… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 23:
30 | prediction right so literally you need
to only you only need to predict the to only you only need to predict the to only you only need to predict the
logist of the final clean output given logist of t… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 24:
18 | a D3PM.
All right, any questions about D3PM? Yes, Yes,
>> this won't have like an equivalence to >> this won't have like an equivalence to >> this won't have like an equivalence to
like score matching… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 25:
17 | like discrete diffusion models
>> great question and the answer is Yes. >> great question and the answer is Yes. >> great question and the answer is Yes.
And we're going to talk about it next And we'r… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 26:
33 | fixed length.
>> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is
fixed length. Yeah. fixed length. Yeah. fixed length. Yeah.
>> … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 27:
24 | Alrighty. Okay. So h just a reminder
what is a uh what is a continuous score what is a uh what is a continuous score what is a uh what is a continuous score
and why is it weird to think about score an… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 28:
10 | no continuous flow anymore. There's no
gradient like you cannot like really do gradient like you cannot like really do gradient like you cannot like really do
any gradient stuff anymore here, right? a… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 28:
56 | type of thing. Uh which is not a healthy
thing to do but like you know just for thing to do but like you know just for thing to do but like you know just for
the sake of the class anyway um you know t… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 29:
41 | basically what people does is that they
define something called uh the concrete define something called uh the concrete define something called uh the concrete
score where it's literally just the scor… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 30:
49 | apparently
uh if you read this particular paper uh if you read this particular paper uh if you read this particular paper
they have some connections to something they have some connections to somethin… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 31:
48 | people actually call it a rate matrix. people actually call it a rate matrix.
So it's a instantaneous rate to jump So it's a instantaneous rate to jump So it's a instantaneous rate to jump
like just j… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 32:
43 | thing, then you're actually basically if thing, then you're actually basically if
you do the math then then then you're you do the math then then then you're you do the math then then then you're
actu… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 33:
27 | everything
well you can represent everything in a well you can represent everything in a well you can represent everything in a
fully connected graph and then assign fully connected graph and then ass… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 34:
22 | So if you want your chain to satisfy
detail balance you're going to get this detail balance you're going to get this detail balance you're going to get this
thing so that your chain is reversible, thi… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 35:
19 | I see everyone's confused by how this
whole thing but we need basically we whole thing but we need basically we whole thing but we need basically we
need a neuronet network to predict this need a neur… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 36:
48 | score versus your uh you know your your score versus your uh you know your your
your your your real score. And you your your your real score. And you your your your real score. And you
literally just … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 37:
47 | non-zero number divided by a non-zero
number or like or sorry non- negative number or like or sorry non- negative number or like or sorry non- negative
number divided by a non- negative number number … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 38:
35 | good. Exactly. That's exactly correct. good. Exactly. That's exactly correct.
Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you
yo… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 39:
20 | like a like a like a like a like a what
we call uh constant so let's just ignore we call uh constant so let's just ignore we call uh constant so let's just ignore
that for now so we basically we can t… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 40:
06 | then the person is going to be this
thing, right? So what you do is you thing, right? So what you do is you thing, right? So what you do is you
essentially so K is like you you jump to essentially so … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 40:
54 | then this part becomes one uh s minus r
uh times the sum of all the possible all uh times the sum of all the possible all uh times the sum of all the possible all
the probabilities and then this one i… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 41:
37 | us something that is that has that has
exponential in it and also literally exponential in it and also literally exponential in it and also literally
right the the the easiest right the the the easies… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 42:
20 | right? So you the the S minus the right? So you the the S minus the
likelihood ratio times the log of the likelihood ratio times the log of the likelihood ratio times the log of the
your S. Um so beca… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 43:
12 | calculate the ground truth by literally
the same trick the den noising score the same trick the den noising score the same trick the den noising score
matching. This is just the noising score matching… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 44:
40 | the reverse rate, right? So, you just
replace this concrete score with the replace this concrete score with the replace this concrete score with the
concrete score that you trained. And concrete score… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 45:
39 | >> earlier how are we choosing why do we
just randomly sample any token choices just randomly sample any token choices just randomly sample any token choices
>> yeah based on >> yeah based on >> yeah … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 46:
38 | basically like how about we can only do
masking and unmasking, right? So in the masking and unmasking, right? So in the masking and unmasking, right? So in the
forward process all you can do is you fo… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 47:
20 | like basically the only two columns that
you're going to have non- negative value you're going to have non- negative value you're going to have non- negative value
is the columns on the diagonal and t… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 48:
06 | >> But if we go all the way end of the day
everything will be masked. everything will be masked. everything will be masked.
>> Yeah. And it's still a distribution >> Yeah. And it's still a distributio… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 48:
55 | multiplication and stuff anymore. Uh
because we simplified everything, right? because we simplified everything, right? because we simplified everything, right?
So basically what what we're essentially… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 49:
51 | stay unmasked and then with one minus
alpha t uh probability you're going to alpha t uh probability you're going to alpha t uh probability you're going to
stay masked and uh basically if we if stay ma… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 51:
08 | when uh xt equals x0 and x t minus one
also equals x zero um like basically if also equals x zero um like basically if also equals x zero um like basically if
if you have not been masked uh at time t … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 52:
02 | simplified into this really really
simple uh ratio. And similarly simple uh ratio. And similarly simple uh ratio. And similarly
because the only other option for you is because the only other option f… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 53:
11 | over. You should you should have the
same value because it hasn't been same value because it hasn't been same value because it hasn't been
masked. And if XT is masked, then you masked. And if XT is ma… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 54:
23 | times the clean data estimation. times the clean data estimation.
That's it. >> We have that problem because we'll start >> We have that problem because we'll start
with full mask. So if you start wit… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 55:
26 | >> So you're always starting with the full
mask. mask. mask.
>> That's right. >> That's right. >> That's right.
>> So if that one reverse gets you to I >> So if that one reverse gets you to I >> So if… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 56:
12 | actually a good point. Basically, uh the
the the mercury um founder, Inception the the mercury um founder, Inception the the mercury um founder, Inception
Lab founder, Stephano, who is the one of Lab … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 57:
09 | you can also do like continuous time
training with this. You literally just training with this. You literally just training with this. You literally just
take the continuous time limit of the of take … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 57:
58 | >> uh you yes kind of so it's the same
thing it's uh like analogy to you choose thing it's uh like analogy to you choose thing it's uh like analogy to you choose
the number of sampling steps right in … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 58:
42 | this is purely depending on like the the this is purely depending on like the the
model that you got trained but usually model that you got trained but usually model that you got trained but usually
i… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 59:
34 | basically the mask ratio is depending on
this alpha t thing which you set. So you this alpha t thing which you set. So you this alpha t thing which you set. So you
you can you can choose uh but like y… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 60:
34 | the tokens. the tokens.
>> Yeah. >> Yeah.
>> But then in the last you have something >> But then in the last you have something >> But then in the last you have something
like XT equal to M. So like X… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 61:
16 | >> uh what do you mean by latent text well
like so you mean like like so you mean like like so you mean like
a region space for images and then use a region space for images and then use a region spac… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 62:
02 | >> Does it produce something coherent if >> Does it produce something coherent if
all the things are independently getting all the things are independently getting all the things are independently get… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 62:
49 | >> It is pretty much just birth literally
if you think about it. Yeah. if you think about it. Yeah. if you think about it. Yeah.
>> Well, between like >> Well, between like >> Well, between like
>> yo… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 63:
40 | when they when he said like oh we can
correct hallucination but like here we correct hallucination but like here we correct hallucination but like here we
cannot really it's the same as cannot really … |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 64:
24 | like a sort of like a like a like a like
a interesting moment in science. But if a interesting moment in science. But if a interesting moment in science. But if
you're one of the authors, you may get,… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 65:
08 | >> very low because it's 2024. Okay. Anyway
um but since like you know people show um but since like you know people show um but since like you know people show
saw the potential of this they have saw… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 65:
55 | say and seems like super compatible uh say and seems like super compatible uh
to comparable to like large language to comparable to like large language to comparable to like large language
models like… |
cmu-10799-diffusion-flow |
L11: CMU 10799 S26: Lecture 12 - Discrete Diffusion & Masked Diffusion - Diffusion & Flow Matching | 66:
39 | Please come up and uh can get rich as
well. All right, see you guys next well. All right, see you guys next well. All right, see you guys next
class. |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 00:
03 | So what we aim to do is as follows. We So what we aim to do is as follows. We
have a true probability distribution, have a true probability distribution, have a true probability distribution,
right? S… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 00:
57 | an image using a noising schedule maybe an image using a noising schedule maybe
a goshian schedule. The predicting noise a goshian schedule. The predicting noise a goshian schedule. The predicting noi… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 01:
41 | very crucial role. Right? So let's see
how noising is included. how noising is included. how noising is included.
What is the noising process in the text? What is the noising process in the text? What… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 02:
32 | that noise and then when we start with that noise and then when we start with
noise, we can predict the original text noise, we can predict the original text noise, we can predict the original text
ju… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 03:
32 | embedding. So this is a 384 dimensional embedding. So this is a 384 dimensional
vector. vector. vector.
This is a 384 dimensional vector. This is a 384 dimensional vector. This is a 384 dimensional ve… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 04:
44 | this is the true
sentences probability distribution. sentences probability distribution. sentences probability distribution.
This and this sentence the next day is This and this sentence the next day … |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 05:
25 | a portion of this, this is no longer a a portion of this, this is no longer a
true sentence. It becomes a cor, it true sentence. It becomes a cor, it true sentence. It becomes a cor, it
becomes a corr… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 06:
33 | schedule. Right? So first I define a
time step. Let's say I go from time time step. Let's say I go from time time step. Let's say I go from time
equal to 1 till time equal to 6. equal to 1 till time e… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 07:
42 | going haywire. So maybe initially when
only one mask is added, I go to a lower only one mask is added, I go to a lower only one mask is added, I go to a lower
probability. When two masks are added, I … |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 08:
56 | way it happens in text is that uh at
time equal to 1, you almost have an time equal to 1, you almost have an time equal to 1, you almost have an
entire clean text with maybe some mask entire clean tex… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 09:
40 | it's a Bernoli distribution. So the
probability of masking is way higher. probability of masking is way higher. probability of masking is way higher.
Probability of masking all tokens is way Probabili… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 10:
44 | 3,000. Right? Now these are masked. So
this is masked and this is masked. So this is masked and this is masked. So this is masked and this is masked. So
this will be replaced with 1 0 0 0 1 this will … |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 11:
35 | a noisy input with mask tokens. Okay. a noisy input with mask tokens. Okay.
Everything else remains exactly the Everything else remains exactly the Everything else remains exactly the
same. Now we'll … |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 12:
23 | architecture if you compare them side by architecture if you compare them side by
side is the input which goes into this side is the input which goes into this side is the input which goes into this
i… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 13:
03 | targets
then we sorry we get the predictions then we sorry we get the predictions then we sorry we get the predictions
then we get the loss then we get the loss then we get the loss
then we update the… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 13:
51 | what is the noising process in the case
of diffusion language models. The next of diffusion language models. The next of diffusion language models. The next
step which we are going to see is that step… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 00:
04 | Okay. So the next step in these key Okay. So the next step in these key
characteristics which we mentioned and characteristics which we mentioned and characteristics which we mentioned and
probably th… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 01:
01 | You take a noisy image. You take a noisy image.
Let's say this is a noisy image of a Let's say this is a noisy image of a Let's say this is a noisy image of a
panda which is the example which we have … |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 01:
44 | function, right? We need to predict the function, right? We need to predict the
noise and then we need to compare it noise and then we need to compare it noise and then we need to compare it
with the … |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 02:
44 | How do I predict the true values here?
So what I want to do what does it mean So what I want to do what does it mean So what I want to do what does it mean
prediction of noise? I somehow want to predi… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 03:
41 | at this position? And how do you get the
loss function? loss function? loss function?
Well, the answer is extremely simple, Well, the answer is extremely simple, Well, the answer is extremely simple,
… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 04:
32 | Right. So the next token which is
predicted at this position I'll compare it with the true next token I'll compare it with the true next token
which should be next. which should be next. which should … |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 05:
25 | predictions, right? For the next, these predictions, right? For the next, these
are my predictions. are my predictions. are my predictions.
These are my predictions. I'm just going These are my predic… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 06:
11 | um second characteristics the model
which predicts the noise which was a which predicts the noise which was a which predicts the noise which was a
unit in the case of diffusion for unit in the case of… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 07:
14 | are. Right? In the ARM architecture, we are. Right? In the ARM architecture, we
remember we had the uh token embedding remember we had the uh token embedding remember we had the uh token embedding
and… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 08:
06 | we add the token embedding plus position
embedding the one major difference in embedding the one major difference in embedding the one major difference in
diffusion language models is that along diffu… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 08:
51 | time embedding vector is added to all my
tokens. So now the input embedding is tokens. So now the input embedding is tokens. So now the input embedding is
not just the token embedding plus not just th… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 09:
35 | change. So now if you see if you compare
these block diagrams these block diagrams these block diagrams
in the ARM architecture, we only had the in the ARM architecture, we only had the in the ARM arc… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 10:
25 | exactly the same. The difference is in
the attention mechanism. the attention mechanism. the attention mechanism.
Now think about it right here. We no Now think about it right here. We no Now think ab… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 11:
08 | means that when I'm calculating the
attention so the mask the mask day the mask day
mask bright right which means that if mask bright right which means that if mask bright right which means that if
I'… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 12:
28 | I essentially have this entire causal
attention matrix which is available to attention matrix which is available to attention matrix which is available to
me. I don't have to set these values to me. I… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 13:
15 | major difference? The third major
difference comes in the output token difference comes in the output token difference comes in the output token
prediction, right? Uh let's say here we prediction, rig… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 13:
56 | these losses. That's my loss.
I don't have to find the loss for the I don't have to find the loss for the I don't have to find the loss for the
other positions at all. other positions at all. other po… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 15:
04 | So if this is let's say 0.1 this
probability and this is2 my loss in this probability and this is2 my loss in this probability and this is2 my loss in this
case will be - 1 by2 log of.1 log of.1
plus … |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 16:
15 | we have a time embedding now which was we have a time embedding now which was
not there auto reggressive model in not there auto reggressive model in not there auto reggressive model in
multi attentio… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 17:
02 | on u the time step. More the time step, on u the time step. More the time step,
more is the masking. Less the time step, more is the masking. Less the time step, more is the masking. Less the time ste… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 17:
43 | transformer block. So I have my layer
normalization multi head attention I normalization multi head attention I normalization multi head attention I
have my dropout I have my skip have my dropout I ha… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 18:
31 | batch size. We choose the time step,
right? Let's say batch size is 128. We right? Let's say batch size is 128. We right? Let's say batch size is 128. We
assemble an input sequence of assemble an inpu… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 19:
30 | tokens. Based on this loss, we find the
partial derivative of the loss with partial derivative of the loss with partial derivative of the loss with
respect to all the parameters. And respect to all th… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 20:
27 | case is the better we pre-train a
language diffusion model the better we language diffusion model the better we language diffusion model the better we
are able to discover what noise is added are able… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 21:
18 | Start with all masks and then we uncover Start with all masks and then we uncover
tokens correct tokens at that position. tokens correct tokens at that position. tokens correct tokens at that position… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 22:
02 | predict is that
the probability of actually recovering the probability of actually recovering the probability of actually recovering
the true true distribution or the the true true distribution or the… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 22:
48 | proved to be an upper bound on the proved to be an upper bound on the
negative log likelihood making it a negative log likelihood making it a negative log likelihood making it a
principled objective f… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 23:
34 | the masked positions. Okay.
So, until now we have seen the first two So, until now we have seen the first two So, until now we have seen the first two
steps of diffusion based language steps of diffus… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 24:
22 | the third difference. Noisy input here the third difference. Noisy input here
that's the four difference. There are that's the four difference. There are that's the four difference. There are
four maj… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 25:
09 | embedding matrix and we construct a time
embedding matrix also. And I show you embedding matrix also. And I show you embedding matrix also. And I show you
how everything is done. So at each how everyt… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 25:
54 | model can be summarized here. We have model can be summarized here. We have
token ids convert them into token token ids convert them into token token ids convert them into token
embedding vectors add … |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 00:
03 | So here you can see the file which has So here you can see the file which has
been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was
getting som… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 00:
56 | iterations you'll definitely see samples
which are as co inference samples which which are as co inference samples which which are as co inference samples which
are as coherent as what you are seeing … |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 01:
37 | you can even chain different GPUs
together to do distributed computing if together to do distributed computing if together to do distributed computing if
possible possible possible
so let's actually p… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 02:
36 | before. Um yeah if you train for 200,000
iterations you'll see an amazing iterations you'll see an amazing iterations you'll see an amazing
coherent language diffusion language coherent language diffu… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 03:
23 | all the necessary things which you will
need to learn about language diffusion need to learn about language diffusion need to learn about language diffusion
models. All right. So yeah we have this All… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 04:
19 | shown on the screen right now serves as shown on the screen right now serves as
a reference for you to build multiple a reference for you to build multiple a reference for you to build multiple
such s… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 05:
08 | fraction of the cost which might be
awesome for the future. But now you know awesome for the future. But now you know awesome for the future. But now you know
how to build this fully yourself. Um, oka… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 00:
02 | Okay. So now I'm going to take you Okay. So now I'm going to take you
through the code of assembling this through the code of assembling this through the code of assembling this
diffusion language mod… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 01:
00 | here. The H100 GPU is much faster.
If you don't have access to the Google If you don't have access to the Google If you don't have access to the Google
Collab Pro, you can start with the T4 Collab Pro… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 01:
43 | number of training steps is only 2,000.
Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100
which is if you have compute budget or w… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 02:
34 | take a look at the
whiteboard you'll see that whiteboard you'll see that whiteboard you'll see that
this dimension which I keep talking this dimension which I keep talking this dimension which I keep … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 03:
20 | essentially here in these examples we
have a sequence length of four tokens, have a sequence length of four tokens, have a sequence length of four tokens,
right? Whereas in actual code we have right? … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 04:
13 | chained to each other. So these are the
uh this variable is the number of uh this variable is the number of uh this variable is the number of
layers. The number of heads is layers. The number of heads… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 05:
11 | that's this uh four times the D model. that's this uh four times the D model.
So DFF is the feed forward neural So DFF is the feed forward neural So DFF is the feed forward neural
network hidden layer… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 06:
01 | optimizer. Okay. So you run this initial
configuration and then you move forward. configuration and then you move forward. configuration and then you move forward.
You install a bunch of dependencies … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 06:
49 | go to this uh hugging face repository
and I load my data. I load my training and I load my data. I load my training and I load my data. I load my training
data set. I load my validation data set data … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 07:
36 | we I'm connected to an L4 GPU at the
moment. See that's fine. So let's see. Um Um
yeah so here you'll see that I'm yeah so here you'll see that I'm yeah so here you'll see that I'm
connected to an L4 … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 08:
32 | token, beginning of sequence, end of token, beginning of sequence, end of
sequence and a mask token. This is very sequence and a mask token. This is very sequence and a mask token. This is very
import… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 09:
20 | with a tokenizer directly borrowed from with a tokenizer directly borrowed from
tick token. I I I think the code should tick token. I I I think the code should tick token. I I I think the code should
… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 10:
21 | because in the noising process we are because in the noising process we are
going to replace the tokens with masks going to replace the tokens with masks going to replace the tokens with masks
and the… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 11:
13 | becomes clearer to all of you. Yeah.
And if you see that you first define a And if you see that you first define a And if you see that you first define a
token embedding, position embedding and token … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 12:
20 | a transformer
um we are going to do a transformer um we are going to do a transformer um we are going to do a transformer
encoder layer over here and uh let's see and attention mask equal to none. Thi… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 13:
11 | embedding. We add the time embedding to
it. The resultant input embedding we it. The resultant input embedding we it. The resultant input embedding we
pass through the transformer block. Then pass thr… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 13:
59 | at the time embedding matrix at the time embedding matrix
if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need
to know at what tim… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 14:
47 | to based on the time step we have to add to based on the time step we have to add
the mask tokens. This is what is done the mask tokens. This is what is done the mask tokens. This is what is done
here… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 15:
29 | done because we have already seen this
on the whiteboard right to calculate the on the whiteboard right to calculate the on the whiteboard right to calculate the
loss I only need to look at the mask l… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 16:
13 | Well, it's included over here where we Well, it's included over here where we
pass an attention mask equal to none. pass an attention mask equal to none. pass an attention mask equal to none.
These ar… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 16:
56 | So this is the cross entropy loss. So
you can run this right now. Once we have you can run this right now. Once we have you can run this right now. Once we have
this defined, the last step is just to … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 17:
35 | time step right so when you go to the
diffusion loss function you'll see that diffusion loss function you'll see that diffusion loss function you'll see that
uh where is the diffusion loss yeah when u… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 18:
19 | Now this thing which I'm telling right
now the corrupted input sequence is now the corrupted input sequence is now the corrupted input sequence is
passed through the model is exactly passed through th… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 19:
11 | Yeah, see the training process has
started. Now I'm on the L I think let's started. Now I'm on the L I think let's started. Now I'm on the L I think let's
see which GPU I'm on. I'm on the L4 GPU see w… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 19:
57 | in u the convergence speed so if you see
the auto reggressive model in 15,000 the auto reggressive model in 15,000 the auto reggressive model in 15,000
epochs I go from a loss of 8 to a loss epochs I … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 20:
44 | can see the training has started over
here and we can print the loss we can here and we can print the loss we can here and we can print the loss we can
print the learning rate etc. While the print the… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 21:
34 | here. I am not going to go through this here. I am not going to go through this
code in detail because it just code in detail because it just code in detail because it just
implements the same thing, … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 22:
19 | Uh so this is run right now and uh here Uh so this is run right now and uh here
I'm just seeing a sample inference. Once I'm just seeing a sample inference. Once I'm just seeing a sample inference. On… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 23:
27 | iterations did not finish. But if you
actually run um actually run um actually run um
60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iteratio… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 24:
09 | Lily's mom asked her to help clean up Lily's mom asked her to help clean up
her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom
p… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 24:
50 | we can also do this on runpod. So if you
have credentials or if you have some have credentials or if you have some have credentials or if you have some
budget even $6 to $7 what I'll suggest budget ev… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 25:
45 | I'll set overrides. Oops. I think it was I'll set overrides. Oops. I think it was
187. Let's do 200. And I'll set 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set
overrides over here. A… |