diffusion-lm-vizuara |
L1: Build a Diffusion Language Model from Scratch Workshop | 00:
02 | All of you love the build large language All of you love the build large language
models from scratch cars which we have models from scratch cars which we have models from scratch cars which we have
o… |
diffusion-lm-vizuara |
L1: Build a Diffusion Language Model from Scratch Workshop | 01:
11 | Then there was a flurry of diffusion
based models which were released. based models which were released. based models which were released.
Essentially think of diffusion as Essentially think of diffus… |
diffusion-lm-vizuara |
L1: Build a Diffusion Language Model from Scratch Workshop | 02:
16 | GitHub repositories or online videos
which teach me everything I want to which teach me everything I want to which teach me everything I want to
know. I did not want a 5 minute tutorial know. I did no… |
diffusion-lm-vizuara |
L1: Build a Diffusion Language Model from Scratch Workshop | 03:
14 | pipeline, the training pipeline, and the
inference pipeline. And on day four, inference pipeline. And on day four, inference pipeline. And on day four,
we'll code the full diffusion model from we'll c… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 00:
03 | So what we aim to do is as follows. We So what we aim to do is as follows. We
have a true probability distribution, have a true probability distribution, have a true probability distribution,
right? S… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 00:
57 | an image using a noising schedule maybe an image using a noising schedule maybe
a goshian schedule. The predicting noise a goshian schedule. The predicting noise a goshian schedule. The predicting noi… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 01:
41 | very crucial role. Right? So let's see
how noising is included. how noising is included. how noising is included.
What is the noising process in the text? What is the noising process in the text? What… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 02:
32 | that noise and then when we start with that noise and then when we start with
noise, we can predict the original text noise, we can predict the original text noise, we can predict the original text
ju… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 03:
32 | embedding. So this is a 384 dimensional embedding. So this is a 384 dimensional
vector. vector. vector.
This is a 384 dimensional vector. This is a 384 dimensional vector. This is a 384 dimensional ve… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 04:
44 | this is the true
sentences probability distribution. sentences probability distribution. sentences probability distribution.
This and this sentence the next day is This and this sentence the next day … |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 05:
25 | a portion of this, this is no longer a a portion of this, this is no longer a
true sentence. It becomes a cor, it true sentence. It becomes a cor, it true sentence. It becomes a cor, it
becomes a corr… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 06:
33 | schedule. Right? So first I define a
time step. Let's say I go from time time step. Let's say I go from time time step. Let's say I go from time
equal to 1 till time equal to 6. equal to 1 till time e… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 07:
42 | going haywire. So maybe initially when
only one mask is added, I go to a lower only one mask is added, I go to a lower only one mask is added, I go to a lower
probability. When two masks are added, I … |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 08:
56 | way it happens in text is that uh at
time equal to 1, you almost have an time equal to 1, you almost have an time equal to 1, you almost have an
entire clean text with maybe some mask entire clean tex… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 09:
40 | it's a Bernoli distribution. So the
probability of masking is way higher. probability of masking is way higher. probability of masking is way higher.
Probability of masking all tokens is way Probabili… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 10:
44 | 3,000. Right? Now these are masked. So
this is masked and this is masked. So this is masked and this is masked. So this is masked and this is masked. So
this will be replaced with 1 0 0 0 1 this will … |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 11:
35 | a noisy input with mask tokens. Okay. a noisy input with mask tokens. Okay.
Everything else remains exactly the Everything else remains exactly the Everything else remains exactly the
same. Now we'll … |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 12:
23 | architecture if you compare them side by architecture if you compare them side by
side is the input which goes into this side is the input which goes into this side is the input which goes into this
i… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 13:
03 | targets
then we sorry we get the predictions then we sorry we get the predictions then we sorry we get the predictions
then we get the loss then we get the loss then we get the loss
then we update the… |
diffusion-lm-vizuara |
L13: Lecture 12: Diffusion LLM Noising Schedule | 13:
51 | what is the noising process in the case
of diffusion language models. The next of diffusion language models. The next of diffusion language models. The next
step which we are going to see is that step… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 00:
04 | Okay. So the next step in these key Okay. So the next step in these key
characteristics which we mentioned and characteristics which we mentioned and characteristics which we mentioned and
probably th… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 01:
01 | You take a noisy image. You take a noisy image.
Let's say this is a noisy image of a Let's say this is a noisy image of a Let's say this is a noisy image of a
panda which is the example which we have … |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 01:
44 | function, right? We need to predict the function, right? We need to predict the
noise and then we need to compare it noise and then we need to compare it noise and then we need to compare it
with the … |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 02:
44 | How do I predict the true values here?
So what I want to do what does it mean So what I want to do what does it mean So what I want to do what does it mean
prediction of noise? I somehow want to predi… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 03:
41 | at this position? And how do you get the
loss function? loss function? loss function?
Well, the answer is extremely simple, Well, the answer is extremely simple, Well, the answer is extremely simple,
… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 04:
32 | Right. So the next token which is
predicted at this position I'll compare it with the true next token I'll compare it with the true next token
which should be next. which should be next. which should … |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 05:
25 | predictions, right? For the next, these predictions, right? For the next, these
are my predictions. are my predictions. are my predictions.
These are my predictions. I'm just going These are my predic… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 06:
11 | um second characteristics the model
which predicts the noise which was a which predicts the noise which was a which predicts the noise which was a
unit in the case of diffusion for unit in the case of… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 07:
14 | are. Right? In the ARM architecture, we are. Right? In the ARM architecture, we
remember we had the uh token embedding remember we had the uh token embedding remember we had the uh token embedding
and… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 08:
06 | we add the token embedding plus position
embedding the one major difference in embedding the one major difference in embedding the one major difference in
diffusion language models is that along diffu… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 08:
51 | time embedding vector is added to all my
tokens. So now the input embedding is tokens. So now the input embedding is tokens. So now the input embedding is
not just the token embedding plus not just th… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 09:
35 | change. So now if you see if you compare
these block diagrams these block diagrams these block diagrams
in the ARM architecture, we only had the in the ARM architecture, we only had the in the ARM arc… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 10:
25 | exactly the same. The difference is in
the attention mechanism. the attention mechanism. the attention mechanism.
Now think about it right here. We no Now think about it right here. We no Now think ab… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 11:
08 | means that when I'm calculating the
attention so the mask the mask day the mask day
mask bright right which means that if mask bright right which means that if mask bright right which means that if
I'… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 12:
28 | I essentially have this entire causal
attention matrix which is available to attention matrix which is available to attention matrix which is available to
me. I don't have to set these values to me. I… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 13:
15 | major difference? The third major
difference comes in the output token difference comes in the output token difference comes in the output token
prediction, right? Uh let's say here we prediction, rig… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 13:
56 | these losses. That's my loss.
I don't have to find the loss for the I don't have to find the loss for the I don't have to find the loss for the
other positions at all. other positions at all. other po… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 15:
04 | So if this is let's say 0.1 this
probability and this is2 my loss in this probability and this is2 my loss in this probability and this is2 my loss in this
case will be - 1 by2 log of.1 log of.1
plus … |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 16:
15 | we have a time embedding now which was we have a time embedding now which was
not there auto reggressive model in not there auto reggressive model in not there auto reggressive model in
multi attentio… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 17:
02 | on u the time step. More the time step, on u the time step. More the time step,
more is the masking. Less the time step, more is the masking. Less the time step, more is the masking. Less the time ste… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 17:
43 | transformer block. So I have my layer
normalization multi head attention I normalization multi head attention I normalization multi head attention I
have my dropout I have my skip have my dropout I ha… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 18:
31 | batch size. We choose the time step,
right? Let's say batch size is 128. We right? Let's say batch size is 128. We right? Let's say batch size is 128. We
assemble an input sequence of assemble an inpu… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 19:
30 | tokens. Based on this loss, we find the
partial derivative of the loss with partial derivative of the loss with partial derivative of the loss with
respect to all the parameters. And respect to all th… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 20:
27 | case is the better we pre-train a
language diffusion model the better we language diffusion model the better we language diffusion model the better we
are able to discover what noise is added are able… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 21:
18 | Start with all masks and then we uncover Start with all masks and then we uncover
tokens correct tokens at that position. tokens correct tokens at that position. tokens correct tokens at that position… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 22:
02 | predict is that
the probability of actually recovering the probability of actually recovering the probability of actually recovering
the true true distribution or the the true true distribution or the… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 22:
48 | proved to be an upper bound on the proved to be an upper bound on the
negative log likelihood making it a negative log likelihood making it a negative log likelihood making it a
principled objective f… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 23:
34 | the masked positions. Okay.
So, until now we have seen the first two So, until now we have seen the first two So, until now we have seen the first two
steps of diffusion based language steps of diffus… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 24:
22 | the third difference. Noisy input here the third difference. Noisy input here
that's the four difference. There are that's the four difference. There are that's the four difference. There are
four maj… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 25:
09 | embedding matrix and we construct a time
embedding matrix also. And I show you embedding matrix also. And I show you embedding matrix also. And I show you
how everything is done. So at each how everyt… |
diffusion-lm-vizuara |
L14: Lecture 13: Diffusion LLM Training Pipeline | 25:
54 | model can be summarized here. We have model can be summarized here. We have
token ids convert them into token token ids convert them into token token ids convert them into token
embedding vectors add … |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 00:
03 | The last step which we need to do is the The last step which we need to do is the
actual step of dnoising. actual step of dnoising. actual step of dnoising.
What is dnoising? Well, dnoising is What is… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 00:
51 | yeah, at each step of the training yeah, at each step of the training
process, if we are able to predict how process, if we are able to predict how process, if we are able to predict how
much noise ha… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 01:
35 | recover the clean image. So we start
from a complete noisy image and then we from a complete noisy image and then we from a complete noisy image and then we
recover Chinese characters as the recover C… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 02:
15 | parameters of the model will not change
in the dnoising process. The model is in the dnoising process. The model is in the dnoising process. The model is
fully trained. Um so all these fully trained. … |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 03:
02 | time steps, for example, in the D
noising process, we have to go backwards noising process, we have to go backwards noising process, we have to go backwards
from four. So, we have to go 4 3 2 1, from … |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 03:
49 | let's take an example. Okay. So if this let's take an example. Okay. So if this
is the input sequence we start at time is the input sequence we start at time is the input sequence we start at time
equ… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 04:
42 | What is the predicted logits matrix?
Um so let's say we have position number Um so let's say we have position number Um so let's say we have position number
we have three main positions which we have … |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 05:
58 | then we have a b and c right so let's then we have a b and c right so let's
actually actually actually
take some different values of my let me take some different values of my let me take some differe… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 07:
16 | change this again.
Sorry for that. So let's say this 336 Sorry for that. So let's say this 336 Sorry for that. So let's say this 336
and this 336. Let's say everything is and this 336. Let's say every… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 08:
07 | we need three masks here. So again we'll
mask all these three. So these three mask all these three. So these three mask all these three. So these three
stay in masks. That's the output from stay in ma… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 08:
52 | of this step we have to keep three masks
at the end of this step we have to keep at the end of this step we have to keep at the end of this step we have to keep
two masks at the end of this step which… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 09:
31 | or I will keep so which what what will I or I will keep so which what what will I
unmask? So I have to keep two masks here unmask? So I have to keep two masks here unmask? So I have to keep two masks … |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 10:
09 | confidence of 3175. At this we have a
confidence of 336 and at this we have a confidence of 336 and at this we have a confidence of 336 and at this we have a
confidence of 3175. Now here the rule is c… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 10:
54 | So at first position I sample A with
confidence of 3175. At second position I confidence of 3175. At second position I confidence of 3175. At second position I
sample C with confidence 336. At third s… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 11:
45 | input architecture and wherever there is input architecture and wherever there is
mask that slowly unmasked right. So here mask that slowly unmasked right. So here mask that slowly unmasked right. So … |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 12:
32 | reggressive which means it's not one
token at a time. Tokens masks can be token at a time. Tokens masks can be token at a time. Tokens masks can be
unmasked here, masks can be unmasked at unmasked her… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 13:
17 | what happens is that let's say you fully
trained the model as we have seen trained the model as we have seen trained the model as we have seen
before. Then you start with a sequence before. Then you s… |
diffusion-lm-vizuara |
L15: Lecture 14: Diffusion LLM Inference Pipeline | 14:
06 | unmasking one after the other. So this unmasking one after the other. So this
video or this GIF which Andre Karpati video or this GIF which Andre Karpati video or this GIF which Andre Karpati
shared. … |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 00:
02 | Before we go to building the small Before we go to building the small
language diffusion model from scratch language diffusion model from scratch language diffusion model from scratch
and we are going… |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 00:
48 | uh cluster of meaningful sentences lie
what language models are doing is that what language models are doing is that what language models are doing is that
we are trying to find this probability we ar… |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 01:
26 | noise the input sequence. You corrupt
the input sequence. You pass this the input sequence. You pass this the input sequence. You pass this
corrupted input sequence uh you pass corrupted input sequenc… |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 02:
14 | One key difference is the time embedding
which is not there in ARM but it's there which is not there in ARM but it's there which is not there in ARM but it's there
in diffusion language model. Second … |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 02:
58 | Fourth thing of course is that we have Fourth thing of course is that we have
noisy input with mass tokens. Whereas in noisy input with mass tokens. Whereas in noisy input with mass tokens. Whereas in… |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 03:
45 | unmasked? The most confident tokens are unmasked? The most confident tokens are
essentially unmasked. So we unmask the k essentially unmasked. So we unmask the k essentially unmasked. So we unmask the… |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 04:
29 | get coherent text towards the end.
Right? So this is our whole diffusion Right? So this is our whole diffusion Right? So this is our whole diffusion
pipeline diffusion language model pipeline diffusio… |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 05:
20 | front of it. So we don't have causal
attention. Then we generate by unmasking attention. Then we generate by unmasking attention. Then we generate by unmasking
tokens one at a time. Right? That's the … |
diffusion-lm-vizuara |
L16: Lecture 15: Diffusion LLM Entire Pipeline Summary | 06:
14 | so if you zoom into here I'm going to
share this code file with all of you and share this code file with all of you and share this code file with all of you and
at the end of this code file we are at … |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 00:
03 | So here you can see the file which has So here you can see the file which has
been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was
getting som… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 00:
56 | iterations you'll definitely see samples
which are as co inference samples which which are as co inference samples which which are as co inference samples which
are as coherent as what you are seeing … |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 01:
37 | you can even chain different GPUs
together to do distributed computing if together to do distributed computing if together to do distributed computing if
possible possible possible
so let's actually p… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 02:
36 | before. Um yeah if you train for 200,000
iterations you'll see an amazing iterations you'll see an amazing iterations you'll see an amazing
coherent language diffusion language coherent language diffu… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 03:
23 | all the necessary things which you will
need to learn about language diffusion need to learn about language diffusion need to learn about language diffusion
models. All right. So yeah we have this All… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 04:
19 | shown on the screen right now serves as shown on the screen right now serves as
a reference for you to build multiple a reference for you to build multiple a reference for you to build multiple
such s… |
diffusion-lm-vizuara |
L18: Lecture 17: Diffusion LLM Coded from Scratch Part 2 | 05:
08 | fraction of the cost which might be
awesome for the future. But now you know awesome for the future. But now you know awesome for the future. But now you know
how to build this fully yourself. Um, oka… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 00:
02 | Okay. So now I'm going to take you Okay. So now I'm going to take you
through the code of assembling this through the code of assembling this through the code of assembling this
diffusion language mod… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 01:
00 | here. The H100 GPU is much faster.
If you don't have access to the Google If you don't have access to the Google If you don't have access to the Google
Collab Pro, you can start with the T4 Collab Pro… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 01:
43 | number of training steps is only 2,000.
Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100
which is if you have compute budget or w… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 02:
34 | take a look at the
whiteboard you'll see that whiteboard you'll see that whiteboard you'll see that
this dimension which I keep talking this dimension which I keep talking this dimension which I keep … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 03:
20 | essentially here in these examples we
have a sequence length of four tokens, have a sequence length of four tokens, have a sequence length of four tokens,
right? Whereas in actual code we have right? … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 04:
13 | chained to each other. So these are the
uh this variable is the number of uh this variable is the number of uh this variable is the number of
layers. The number of heads is layers. The number of heads… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 05:
11 | that's this uh four times the D model. that's this uh four times the D model.
So DFF is the feed forward neural So DFF is the feed forward neural So DFF is the feed forward neural
network hidden layer… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 06:
01 | optimizer. Okay. So you run this initial
configuration and then you move forward. configuration and then you move forward. configuration and then you move forward.
You install a bunch of dependencies … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 06:
49 | go to this uh hugging face repository
and I load my data. I load my training and I load my data. I load my training and I load my data. I load my training
data set. I load my validation data set data … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 07:
36 | we I'm connected to an L4 GPU at the
moment. See that's fine. So let's see. Um Um
yeah so here you'll see that I'm yeah so here you'll see that I'm yeah so here you'll see that I'm
connected to an L4 … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 08:
32 | token, beginning of sequence, end of token, beginning of sequence, end of
sequence and a mask token. This is very sequence and a mask token. This is very sequence and a mask token. This is very
import… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 09:
20 | with a tokenizer directly borrowed from with a tokenizer directly borrowed from
tick token. I I I think the code should tick token. I I I think the code should tick token. I I I think the code should
… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 10:
21 | because in the noising process we are because in the noising process we are
going to replace the tokens with masks going to replace the tokens with masks going to replace the tokens with masks
and the… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 11:
13 | becomes clearer to all of you. Yeah.
And if you see that you first define a And if you see that you first define a And if you see that you first define a
token embedding, position embedding and token … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 12:
20 | a transformer
um we are going to do a transformer um we are going to do a transformer um we are going to do a transformer
encoder layer over here and uh let's see and attention mask equal to none. Thi… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 13:
11 | embedding. We add the time embedding to
it. The resultant input embedding we it. The resultant input embedding we it. The resultant input embedding we
pass through the transformer block. Then pass thr… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 13:
59 | at the time embedding matrix at the time embedding matrix
if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need
to know at what tim… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 14:
47 | to based on the time step we have to add to based on the time step we have to add
the mask tokens. This is what is done the mask tokens. This is what is done the mask tokens. This is what is done
here… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 15:
29 | done because we have already seen this
on the whiteboard right to calculate the on the whiteboard right to calculate the on the whiteboard right to calculate the
loss I only need to look at the mask l… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 16:
13 | Well, it's included over here where we Well, it's included over here where we
pass an attention mask equal to none. pass an attention mask equal to none. pass an attention mask equal to none.
These ar… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 16:
56 | So this is the cross entropy loss. So
you can run this right now. Once we have you can run this right now. Once we have you can run this right now. Once we have
this defined, the last step is just to … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 17:
35 | time step right so when you go to the
diffusion loss function you'll see that diffusion loss function you'll see that diffusion loss function you'll see that
uh where is the diffusion loss yeah when u… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 18:
19 | Now this thing which I'm telling right
now the corrupted input sequence is now the corrupted input sequence is now the corrupted input sequence is
passed through the model is exactly passed through th… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 19:
11 | Yeah, see the training process has
started. Now I'm on the L I think let's started. Now I'm on the L I think let's started. Now I'm on the L I think let's
see which GPU I'm on. I'm on the L4 GPU see w… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 19:
57 | in u the convergence speed so if you see
the auto reggressive model in 15,000 the auto reggressive model in 15,000 the auto reggressive model in 15,000
epochs I go from a loss of 8 to a loss epochs I … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 20:
44 | can see the training has started over
here and we can print the loss we can here and we can print the loss we can here and we can print the loss we can
print the learning rate etc. While the print the… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 21:
34 | here. I am not going to go through this here. I am not going to go through this
code in detail because it just code in detail because it just code in detail because it just
implements the same thing, … |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 22:
19 | Uh so this is run right now and uh here Uh so this is run right now and uh here
I'm just seeing a sample inference. Once I'm just seeing a sample inference. Once I'm just seeing a sample inference. On… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 23:
27 | iterations did not finish. But if you
actually run um actually run um actually run um
60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iteratio… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 24:
09 | Lily's mom asked her to help clean up Lily's mom asked her to help clean up
her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom
p… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 24:
50 | we can also do this on runpod. So if you
have credentials or if you have some have credentials or if you have some have credentials or if you have some
budget even $6 to $7 what I'll suggest budget ev… |
diffusion-lm-vizuara |
L17: Lecture 16: Diffusion LLM Coded from Scratch Part 1 | 25:
45 | I'll set overrides. Oops. I think it was I'll set overrides. Oops. I think it was
187. Let's do 200. And I'll set 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set
overrides over here. A… |
diffusion-lm-vizuara |
L19: Lecture 18: Diffusion LLM Series Conclusion | 00:
02 | All right, everyone. So, this brings us All right, everyone. So, this brings us
to the end of this diffusion language to the end of this diffusion language to the end of this diffusion language
model … |
diffusion-lm-vizuara |
L19: Lecture 18: Diffusion LLM Series Conclusion | 00:
51 | I've not shown supervised fine-tuning, I've not shown supervised fine-tuning,
which you can do on your own since at which you can do on your own since at which you can do on your own since at
differen… |
diffusion-lm-vizuara |
L19: Lecture 18: Diffusion LLM Series Conclusion | 01:
40 | an open-source way. So, you can be the
first one to do this. The same pipeline first one to do this. The same pipeline first one to do this. The same pipeline
which we have built for small language wh… |
diffusion-lm-vizuara |
L2: Lecture 1: Introduction to Diffusion Language Models | 00:
03 | Hello everyone. This is Dr. Raj Dandeek Hello everyone. This is Dr. Raj Dandeek
and uh today I'm very excited to present and uh today I'm very excited to present and uh today I'm very excited to prese… |
diffusion-lm-vizuara |
L2: Lecture 1: Introduction to Diffusion Language Models | 01:
01 | models from scratch. But that's not how models from scratch. But that's not how
I want this course or this learning I want this course or this learning I want this course or this learning
material to … |
diffusion-lm-vizuara |
L2: Lecture 1: Introduction to Diffusion Language Models | 01:
49 | scared of them, then how I learned the scared of them, then how I learned the
subject, how I mastered the subject now subject, how I mastered the subject now subject, how I mastered the subject now
so… |
diffusion-lm-vizuara |
L2: Lecture 1: Introduction to Diffusion Language Models | 02:
27 | model fully from scratch. You'll compare
it with the auto reggressive language it with the auto reggressive language it with the auto reggressive language
models. So it's an amazing exercise or models… |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 00:
02 | Let's get started. Let's get started.
Initially, as I mentioned, I'll be Initially, as I mentioned, I'll be Initially, as I mentioned, I'll be
taking you through my journey, right? taking you through … |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 00:
51 | thought that okay this is something thought that okay this is something
cool. I had thought about diffusion only cool. I had thought about diffusion only cool. I had thought about diffusion only
for i… |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 01:
33 | over here. So take a look here first
only on the left hand side. Okay. And only on the left hand side. Okay. And only on the left hand side. Okay. And
now when we restart the video then take now when … |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 02:
18 | you learn how much noise is added. So you learn how much noise is added. So
later you are able to take noise and later you are able to take noise and later you are able to take noise and
then build im… |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 03:
07 | diffusion model which fills the screen
like this and it's not one token at a like this and it's not one token at a like this and it's not one token at a
time. So when I saw this GIF, I time. So when I… |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 03:
59 | at a time. Um this has now become very at a time. Um this has now become very
popular on YouTube and it's one of the popular on YouTube and it's one of the popular on YouTube and it's one of the
go-to… |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 04:
47 | And uh I'm going to first walk you
through diffusion in an intuitive way through diffusion in an intuitive way through diffusion in an intuitive way
because I I strongly think that all of because I I … |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 05:
36 | formulations will which I'll not so I'll
not take that route when I explain this not take that route when I explain this not take that route when I explain this
to you I'll explain this to you in an t… |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 06:
23 | fully from scratch. You'll be doing this fully from scratch. You'll be doing this
if you follow you follow along with me if you follow you follow along with me if you follow you follow along with me
t… |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 07:
43 | take an image you keep on adding noise
to it. Right? You keep on adding noise to it. Right? You keep on adding noise to it. Right? You keep on adding noise
to an image until it becomes fully to an ima… |
diffusion-lm-vizuara |
L3: Lecture 2: How I began my Diffusion LLM Journey | 08:
43 | clean image.
So now you'll see that as we are going So now you'll see that as we are going So now you'll see that as we are going
in the reverse direction slowly the dye in the reverse direction slowl… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 00:
02 | So I hope all of us are on the same page So I hope all of us are on the same page
now with respect to the motivation now with respect to the motivation now with respect to the motivation
behind this l… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 00:
50 | questions which researchers should ask
that leads to the best research. that leads to the best research. that leads to the best research.
If you ask this question, you'll slowly If you ask this questi… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 01:
49 | models. This so this paper I believe
is a very important one mask diffusion is a very important one mask diffusion is a very important one mask diffusion
language models this is where many language mo… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 02:
51 | introduction section itself
uh that why should images and video uh that why should images and video uh that why should images and video
generation models have all the fun why generation models have al… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 03:
39 | images might lie and then sampling from
it. So why can't this same thing be it. So why can't this same thing be it. So why can't this same thing be
applied to text as well? applied to text as well? ap… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 04:
58 | when we talked about images, first we
broke down the images into pixels, broke down the images into pixels, broke down the images into pixels,
right? Similarly, text can be broken right? Similarly, te… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 05:
54 | So one way to look at generating text is So one way to look at generating text is
just one word at a time. Another way of just one word at a time. Another way of just one word at a time. Another way o… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 06:
57 | in high dimensional spaces? What does it
mean probability distribution of mean probability distribution of mean probability distribution of
meaningful text? Let's see. So text is trickier than images.… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 08:
12 | sequence of words based on two things. sequence of words based on two things.
Grammar and logic both need to be Grammar and logic both need to be Grammar and logic both need to be
correct, right? And … |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 09:
12 | can be carried forward to languages. can be carried forward to languages.
What these people suggested is auto What these people suggested is auto What these people suggested is auto
reggressive models… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 00:
03 | Hello everyone. I'm very excited to be Hello everyone. I'm very excited to be
recording this video today because uh recording this video today because uh recording this video today because uh
I'm goin… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 01:
00 | focus on the technology behind
uh midjourney or stable diffusion but I uh midjourney or stable diffusion but I uh midjourney or stable diffusion but I
was so excited to try this out that I was so exci… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 01:
49 | related to the diffusion which I know
about I know about diffusion where let's about I know about diffusion where let's about I know about diffusion where let's
say I spray a perfume in one corner of … |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 02:
45 | these diffusion models have these diffusion models have
lasted for 10 to 15 years and every lasted for 10 to 15 years and every lasted for 10 to 15 years and every
development has used breakthroughs m… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 03:
43 | So this series has been made from that So this series has been made from that
point of view. I want to explain point of view. I want to explain point of view. I want to explain
diffusion models to you… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 04:
36 | I have structured this course and how I
plan to teach it. It'll be very similar plan to teach it. It'll be very similar plan to teach it. It'll be very similar
to Vijuara's other teaching material and… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 05:
43 | discuss is that of protein discuss is that of protein
structure prediction. structure prediction. structure prediction.
This has been a problem which has This has been a problem which has This has bee… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 06:
42 | seconds, 5 seconds, 10 seconds and now
you can generate up to 5 minute videos you can generate up to 5 minute videos you can generate up to 5 minute videos
using u generative AI and a lot of these usi… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 07:
42 | this advancement. Recently we are seeing this advancement. Recently we are seeing
a lot of foundational robotics models a lot of foundational robotics models a lot of foundational robotics models
bein… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 08:
43 | wonder is there something intrinsic
which is happening in diffusion that is which is happening in diffusion that is which is happening in diffusion that is
common in all these applications. But common… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 09:
42 | images and videos but also audio images and videos but also audio
synthesis. Now that we have established that Now that we have established that
diffusion models have a huge number of diffusion models… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 10:
55 | So you see there are these small videos So you see there are these small videos
15 minutes, 30 minutes, 12 minutes and 15 minutes, 30 minutes, 12 minutes and 15 minutes, 30 minutes, 12 minutes and
som… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 11:
48 | minutes maximum 30 minutes long. So very
little is known about how they work and little is known about how they work and little is known about how they work and
I think that diffusion models should be… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 12:
47 | Now the reason I'm coming out with this Now the reason I'm coming out with this
course right now is because I received a course right now is because I received a course right now is because I received… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 13:
42 | and one thing uh I realized after and one thing uh I realized after
reading this book and these thoughts reading this book and these thoughts reading this book and these thoughts
were also circling in… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 14:
31 | have divided the curriculum according to
this book. Uh and I I will not go very this book. Uh and I I will not go very this book. Uh and I I will not go very
theoretical in this section. So it might t… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 15:
20 | the first place? And what is the broader the first place? And what is the broader
objective we are trying to achieve here? objective we are trying to achieve here? objective we are trying to achieve h… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 16:
08 | lot in the last 10 years. As you can see
one of the first methods actually came one of the first methods actually came one of the first methods actually came
40 years back which was energy based 40 ye… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 17:
05 | we will cover all of these different we will cover all of these different
techniques individually. techniques individually. techniques individually.
So broadly speaking there are three So broadly spea… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 17:
59 | variational view then we will cover
score based view. Then we will cover score based view. Then we will cover score based view. Then we will cover
flow-based view and then after that we flow-based vie… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 18:
59 | finally in part three we are going to
understand how to sample from these understand how to sample from these understand how to sample from these
diffusion models. How to steer the diffusion models. H… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 19:
51 | actually attracted me to this field. I
want I I I wanted to study it in in in want I I I wanted to study it in in in want I I I wanted to study it in in in
more depth. more depth. more depth.
People w… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 20:
40 | reinforcement learning course just
finished a while back and uh after that finished a while back and uh after that finished a while back and uh after that
I have come up with this diffusion I have com… |
diffusion-principles-vizuara |
L11: The Principles of Diffusion Models - New Course Launch | 21:
32 | feeling you will not get by looking at
these concepts from a very surface level these concepts from a very surface level these concepts from a very surface level
to get that you need to understand the… |