Diffusion Language Model — across courses
diffusion-lm-vizuara
- L1 @ 2s
All of you love the build large language All of you love the build large language models from scratch cars which we have models from scratch cars which we have models from scratch cars which we have on YouTube. on YouTube. on YouTube. Those were auto reggressive models which Those were auto reggressive models which Those were auto reggressive models which predict one token at a time. Now I'm predict one token at a time. Now I'm predict one token at a time. Now I'm coming up with a new series of lectures coming up with a new series of lectures coming up with a new series of lectures in which I'm going to teach you how to in which I'm going to teach you how to in which I'm going to teach you how to build a diffusion language model build a diffusion language model build a diffusion language model entirely from scratch. Hello everyone. This is Dr. Raj Gandhar, Hello everyone. This is Dr. Raj Gandhar, one of the three co-founders of Vijara one of the three co-founders of Vijara one of the three co-founders of Vijara AI Labs. Today I'm very excited to AI Labs. Today I'm very excited to AI Labs. Today I'm very excited to announce the launch of our new workshop, announce the launch of our new workshop, announce the launch of our new workshop, build a diffusion language model from build a diffusion language model from build a diffusion language model from scratch. I have been working on this for scratch. I have been working on this for scratch. I have been working on this for the past five to 6 months and I'm the past five to 6 months and I'm the past five to 6 months and I'm finally ready with this uh series of finally ready with this uh series of finally ready with this uh series of lectures. lectures. lectures. So in 2025 So in 2025 So in 2025 the state-ofthe-art for language models the state-ofthe-art for language models the state-ofthe-art for language models was auto reggressive language models was auto reggressive language models was auto reggressive language models which predicted one new token at a time. which predicted one new token at a time. which predicted one new token at a time. Then there was a flurry of diffusion Then there was a flurry of diffusion
- L1 @ 71s
Then there was a flurry of diffusion based models which were released. based models which were released. based models which were released. Essentially think of diffusion as Essentially think of diffusion as Essentially think of diffusion as instead of one token at a time, it instead of one token at a time, it instead of one token at a time, it predicts all the text at once. Like in predicts all the text at once. Like in predicts all the text at once. Like in the video which you see here. So when we when I saw this video or when So when we when I saw this video or when I saw this GIF, it looked amazing to me. I saw this GIF, it looked amazing to me. I saw this GIF, it looked amazing to me. Yeah, why should X be produced one token Yeah, why should X be produced one token Yeah, why should X be produced one token at a time? Why cannot it just appear to at a time? Why cannot it just appear to at a time? Why cannot it just appear to me like diffusion based models for me like diffusion based models for me like diffusion based models for images? That leads to much faster images? That leads to much faster images? That leads to much faster inference times, right? And then I went inference times, right? And then I went inference times, right? And then I went on a journey to learn how to build this on a journey to learn how to build this on a journey to learn how to build this diffusion language model fully from diffusion language model fully from diffusion language model fully from scratch. scratch. scratch. How can I understand the nuts and bolts How can I understand the nuts and bolts How can I understand the nuts and bolts of how the entire architecture of the of how the entire architecture of the of how the entire architecture of the diffusion language model was assembled? diffusion language model was assembled? diffusion language model was assembled? There was very little information There was very little information There was very little information online. It was very difficult to find online. It was very difficult to find online. It was very difficult to find GitHub repositories or online videos GitHub repositories or online videos
- L1 @ 136s
GitHub repositories or online videos which teach me everything I want to which teach me everything I want to which teach me everything I want to know. I did not want a 5 minute tutorial know. I did not want a 5 minute tutorial know. I did not want a 5 minute tutorial or a 10-minute tutorial. or a 10-minute tutorial. or a 10-minute tutorial. As I went through this journey, now I'm As I went through this journey, now I'm As I went through this journey, now I'm ready with the whole uh pipeline which I ready with the whole uh pipeline which I ready with the whole uh pipeline which I have built and I'm ready to teach it all have built and I'm ready to teach it all have built and I'm ready to teach it all to you. Initially this workshop was to you. Initially this workshop was to you. Initially this workshop was planned as uh intensive boot camp for planned as uh intensive boot camp for planned as uh intensive boot camp for industry professionals but now we are industry professionals but now we are industry professionals but now we are opening it up um for everyone. opening it up um for everyone. opening it up um for everyone. Here is the content which we will cover through this uh boot camp or through through this uh boot camp or through this workshop. On each day we will cover this workshop. On each day we will cover this workshop. On each day we will cover something different. On day one, we will something different. On day one, we will something different. On day one, we will understand the introduction to diffusion understand the introduction to diffusion understand the introduction to diffusion models and looking at generative AI models and looking at generative AI models and looking at generative AI through a probabilistic lens. On day through a probabilistic lens. On day through a probabilistic lens. On day two, we'll understand auto reggressive two, we'll understand auto reggressive two, we'll understand auto reggressive models, their limitations. On day three, models, their limitations. On day three, models, their limitations. On day three, we'll build the diffusion language model we'll build the diffusion language model we'll build the diffusion language model pipeline, the training pipeline, and the pipeline, the training pipeline, and the
- L1 @ 194s
pipeline, the training pipeline, and the inference pipeline. And on day four, inference pipeline. And on day four, inference pipeline. And on day four, we'll code the full diffusion model from we'll code the full diffusion model from we'll code the full diffusion model from scratch. I'll even give you research scratch. I'll even give you research scratch. I'll even give you research projects at the end of this workshop projects at the end of this workshop projects at the end of this workshop which you can do on your own and you can which you can do on your own and you can which you can do on your own and you can then convert it into a publication. I then convert it into a publication. I then convert it into a publication. I think this is the first workshop in the think this is the first workshop in the think this is the first workshop in the world which teaches you how to build a world which teaches you how to build a world which teaches you how to build a diffusion language model fully from diffusion language model fully from scratch. The workshop starts from the scratch. The workshop starts from the scratch. The workshop starts from the 2nd of February and it will finish on 2nd of February and it will finish on 2nd of February and it will finish on the 6th of February. I'm really looking the 6th of February. I'm really looking the 6th of February. I'm really looking forward to seeing you all there. So, forward to seeing you all there. So, forward to seeing you all there. So, thank you for watching and uh I hope you thank you for watching and uh I hope you thank you for watching and uh I hope you find the workshop very valuable.
- L13 @ 3s
So what we aim to do is as follows. We So what we aim to do is as follows. We have a true probability distribution, have a true probability distribution, have a true probability distribution, right? So let's say the meaningful right? So let's say the meaningful right? So let's say the meaningful sentences look like this in probability sentences look like this in probability sentences look like this in probability distribution space. We want to get a distribution space. We want to get a distribution space. We want to get a probability distribution probability distribution probability distribution or we want to predict a probability or we want to predict a probability or we want to predict a probability distribution which is as close to this distribution which is as close to this distribution which is as close to this true probability distribution. true probability distribution. true probability distribution. But we want to predict this not with an But we want to predict this not with an But we want to predict this not with an auto reggressive model but we want to auto reggressive model but we want to auto reggressive model but we want to use a diffusion model use a diffusion model use a diffusion model for getting this probability for getting this probability for getting this probability distribution of the predicted sentences. distribution of the predicted sentences. distribution of the predicted sentences. Right? Right? Right? So we will integrate the three main So we will integrate the three main So we will integrate the three main characteristics of diffusion models. characteristics of diffusion models. characteristics of diffusion models. First we need a noising process. Second First we need a noising process. Second First we need a noising process. Second we need to predict the noise using some we need to predict the noise using some we need to predict the noise using some sort of a model. And third we need a sort of a model. And third we need a sort of a model. And third we need a dnoising process. The noising process in dnoising process. The noising process in dnoising process. The noising process in the case of images was adding noise to the case of images was adding noise to the case of images was adding noise to an image using a noising schedule maybe
- L13 @ 57s
an image using a noising schedule maybe an image using a noising schedule maybe a goshian schedule. The predicting noise a goshian schedule. The predicting noise a goshian schedule. The predicting noise was the ML model called unit in the case was the ML model called unit in the case was the ML model called unit in the case of images and the dnoising process which of images and the dnoising process which of images and the dnoising process which they had was just uh reversing uh the they had was just uh reversing uh the they had was just uh reversing uh the noising process backwards so that we can noising process backwards so that we can noising process backwards so that we can generate images from noise. Again I want generate images from noise. Again I want generate images from noise. Again I want to bring your attention to this visual to bring your attention to this visual to bring your attention to this visual which we saw in the dnoising process. which we saw in the dnoising process. which we saw in the dnoising process. What happens is we start with complete What happens is we start with complete What happens is we start with complete noise and then we bring it back to the noise and then we bring it back to the noise and then we bring it back to the original image. Now let's start original image. Now let's start original image. Now let's start implementing these characteristics step implementing these characteristics step implementing these characteristics step by step. So how do we first of all let's by step. So how do we first of all let's by step. So how do we first of all let's start with noising process. How do we start with noising process. How do we start with noising process. How do we integrate noising process in the text? integrate noising process in the text? integrate noising process in the text? Okay, this is where our understanding of Okay, this is where our understanding of Okay, this is where our understanding of the ARM model is going to play a very the ARM model is going to play a very the ARM model is going to play a very very crucial role. Right? So let's see very crucial role. Right? So let's see
- L13 @ 101s
very crucial role. Right? So let's see how noising is included. how noising is included. how noising is included. What is the noising process in the text? What is the noising process in the text? What is the noising process in the text? Right? Right? And for this the first thing which is And for this the first thing which is And for this the first thing which is going to be extremely crucial for us is going to be extremely crucial for us is going to be extremely crucial for us is that in the case of images we had an that in the case of images we had an that in the case of images we had an image right which we want to noise or image right which we want to noise or image right which we want to noise or which we want to subsequently add noise which we want to subsequently add noise which we want to subsequently add noise to. Um let's say the image was that of a to. Um let's say the image was that of a to. Um let's say the image was that of a Chinese character. That was an image. We Chinese character. That was an image. We Chinese character. That was an image. We can successfully go on adding noise to can successfully go on adding noise to can successfully go on adding noise to this image. this image. this image. We add noise. So this is for images. So this is for images. What about text? How do we go about What about text? How do we go about What about text? How do we go about adding noise to a piece of text? So how adding noise to a piece of text? So how adding noise to a piece of text? So how do we corrupt a piece of text? So the do we corrupt a piece of text? So the do we corrupt a piece of text? So the whole idea would be to take a bunch of whole idea would be to take a bunch of whole idea would be to take a bunch of text to corrupt it with noise to predict text to corrupt it with noise to predict text to corrupt it with noise to predict that noise and then when we start with
- L13 @ 152s
that noise and then when we start with that noise and then when we start with noise, we can predict the original text noise, we can predict the original text noise, we can predict the original text just like how we did in images. So first just like how we did in images. So first just like how we did in images. So first is how do we add noise to text? is how do we add noise to text? is how do we add noise to text? So let's think about ARMs, right? What So let's think about ARMs, right? What So let's think about ARMs, right? What happens in ARMs? ARMs have this uh ARM have this architecture, right? And ARM have this architecture, right? And here I'm again it's the exact same here I'm again it's the exact same here I'm again it's the exact same architecture but just a different uh architecture but just a different uh architecture but just a different uh visual here. So what ARMs do is that you visual here. So what ARMs do is that you visual here. So what ARMs do is that you have this piece of text which is passed have this piece of text which is passed have this piece of text which is passed into this architecture. Right? Let's say into this architecture. Right? Let's say into this architecture. Right? Let's say I have this piece of text called the I have this piece of text called the I have this piece of text called the next day is bright. next day is bright. next day is bright. Okay, I have tokenized it. So let's say Okay, I have tokenized it. So let's say Okay, I have tokenized it. So let's say the token ids are 1 11 20,000 the token ids are 1 11 20,000 the token ids are 1 11 20,000 55 and 3,000. 55 and 3,000. 55 and 3,000. And then what I have done is that I have And then what I have done is that I have And then what I have done is that I have converted it into embedding vectors. converted it into embedding vectors. converted it into embedding vectors. That's the next step. So token That's the next step. So token That's the next step. So token embedding. So this is a 384 dimensional
- L13 @ 212s
embedding. So this is a 384 dimensional embedding. So this is a 384 dimensional vector. vector. vector. This is a 384 dimensional vector. This is a 384 dimensional vector. This is a 384 dimensional vector. I'll just mute my discord. This is a I'll just mute my discord. This is a I'll just mute my discord. This is a 384dimensional vector. 384dimensional vector. 384dimensional vector. This is a 384dimensional vector. This is a 384dimensional vector. This is a 384dimensional vector. And this is again a 384dimensional And this is again a 384dimensional And this is again a 384dimensional vector. Right? This is how it works for vector. Right? This is how it works for vector. Right? This is how it works for auto reggressive models. Now the auto reggressive models. Now the auto reggressive models. Now the question which I'm asking to all of you question which I'm asking to all of you question which I'm asking to all of you is that is that is that I want to start with the first step of I want to start with the first step of I want to start with the first step of these three these three these three three characteristics which is the three characteristics which is the three characteristics which is the noising process. How do I corrupt the noising process. How do I corrupt the noising process. How do I corrupt the given piece of text by adding noise? given piece of text by adding noise? given piece of text by adding noise? What can I do over here? So that noise What can I do over here? So that noise What can I do over here? So that noise is added. Now what does noise mean? is added. Now what does noise mean? is added. Now what does noise mean? Noise essentially means that this piece Noise essentially means that this piece Noise essentially means that this piece of text right now let's say I have the of text right now let's say I have the of text right now let's say I have the actual distribution of true sentences. actual distribution of true sentences. actual distribution of true sentences. Right? This is the true true sentences Right? This is the true true sentences Right? This is the true true sentences probability distribution. Let me Let me this is the true this is the true
- L13 @ 284s
this is the true sentences probability distribution. sentences probability distribution. sentences probability distribution. This and this sentence the next day is This and this sentence the next day is This and this sentence the next day is bright belongs somewhere here. The next bright belongs somewhere here. The next bright belongs somewhere here. The next day is bright lies somewhere here. So day is bright lies somewhere here. So day is bright lies somewhere here. So the probability associated with this. So the probability associated with this. So the probability associated with this. So if this sentence is X, probability if this sentence is X, probability if this sentence is X, probability associated with X is very high because associated with X is very high because associated with X is very high because this is a good sentence. Now I want to this is a good sentence. Now I want to this is a good sentence. Now I want to corrupt this text which means I want to corrupt this text which means I want to corrupt this text which means I want to take it to spaces. So if this is my take it to spaces. So if this is my take it to spaces. So if this is my probability distribution space, I want probability distribution space, I want probability distribution space, I want to take it to spaces here where the to take it to spaces here where the to take it to spaces here where the probability is very low of it being a probability is very low of it being a probability is very low of it being a true sentence. How do I do that? How do true sentence. How do I do that? How do true sentence. How do I do that? How do I corrupt a sentence? The best way to I corrupt a sentence? The best way to I corrupt a sentence? The best way to corrupt a sentence is to just remove corrupt a sentence is to just remove corrupt a sentence is to just remove portions of it. Right? portions of it. Right? portions of it. Right? If I remove portion of this, if I remove If I remove portion of this, if I remove If I remove portion of this, if I remove a portion of this, this is no longer a
- L13 @ 325s
a portion of this, this is no longer a a portion of this, this is no longer a true sentence. It becomes a cor, it true sentence. It becomes a cor, it true sentence. It becomes a cor, it becomes a corrupt text or it becomes a becomes a corrupt text or it becomes a becomes a corrupt text or it becomes a noisy text. Um, this is exactly how noisy text. Um, this is exactly how noisy text. Um, this is exactly how noising is done at the sentence level. noising is done at the sentence level. noising is done at the sentence level. So we have input sequences or we have So we have input sequences or we have So we have input sequences or we have sequences of tokens, right? sequences of tokens, right? sequences of tokens, right? In diffusion language models, we corrupt In diffusion language models, we corrupt In diffusion language models, we corrupt this text by introducing masks. this text by introducing masks. this text by introducing masks. What this means is that if I have the What this means is that if I have the What this means is that if I have the next day is bright, next day is bright, next day is bright, I will mask this input sequence. Which I will mask this input sequence. Which I will mask this input sequence. Which means I will let I will say that the means I will let I will say that the means I will let I will say that the input sequence is this. The mask input sequence is this. The mask input sequence is this. The mask day mask mask bright. bright. bright. This is basically corrupting the input This is basically corrupting the input This is basically corrupting the input sequence by adding masks. sequence by adding masks. sequence by adding masks. Right? Uh now how do I add these masks? Right? Uh now how do I add these masks? Right? Uh now how do I add these masks? The the way I add these masks is by The the way I add these masks is by The the way I add these masks is by defining something like a noising defining something like a noising defining something like a noising schedule. Right? So first I define a schedule. Right? So first I define a
- L13 @ 393s
schedule. Right? So first I define a time step. Let's say I go from time time step. Let's say I go from time time step. Let's say I go from time equal to 1 till time equal to 6. equal to 1 till time equal to 6. equal to 1 till time equal to 6. And I define the number of tokens uh proportional to uh proportional to let's say my time let's say t divided by let's say my time let's say t divided by let's say my time let's say t divided by so if 6 is capital t divided by capital so if 6 is capital t divided by capital so if 6 is capital t divided by capital t so if my t is equal to 1 t so if my t is equal to 1 t so if my t is equal to 1 um I will approximately mask um I will approximately mask um I will approximately mask 16th of the tokens If my t is equal to 2, I will If my t is equal to 2, I will approximately mask 26 of the tokens. If approximately mask 26 of the tokens. If approximately mask 26 of the tokens. If my t is equal to 6, I will mask all of my t is equal to 6, I will mask all of my t is equal to 6, I will mask all of the tokens. So you see at t equal to0, I have my So you see at t equal to0, I have my entire entire entire clean sequence which is not masked. So clean sequence which is not masked. So clean sequence which is not masked. So that's the correct sequence. Think of it that's the correct sequence. Think of it that's the correct sequence. Think of it like lying in the correct probability like lying in the correct probability like lying in the correct probability distribution space. As more and more distribution space. As more and more distribution space. As more and more noise gets added, I move out of this noise gets added, I move out of this noise gets added, I move out of this probability distribution space and start probability distribution space and start probability distribution space and start going haywire. So maybe initially when going haywire. So maybe initially when
- L13 @ 462s
going haywire. So maybe initially when only one mask is added, I go to a lower only one mask is added, I go to a lower only one mask is added, I go to a lower probability. When two masks are added, I probability. When two masks are added, I probability. When two masks are added, I go to even lower probability. When all go to even lower probability. When all go to even lower probability. When all the masks are added, it's complete the masks are added, it's complete the masks are added, it's complete noise. So I go to a probability noise. So I go to a probability noise. So I go to a probability distribution or I go to a space in my distribution or I go to a space in my distribution or I go to a space in my probability distribution space which probability distribution space which probability distribution space which contains no meaningful text at all. I go contains no meaningful text at all. I go contains no meaningful text at all. I go to a random location basically. to a random location basically. to a random location basically. Uh so if you see Uh so if you see Uh so if you see diffusion noise schedule images diffusion noise schedule images diffusion noise schedule images and maybe let's see a gif. Noising schedule. Noising schedule Noising schedule. Noising schedule diffusion. Yeah, take a look at this. This is a Yeah, take a look at this. This is a noising schedule, right? I have a dog noising schedule, right? I have a dog noising schedule, right? I have a dog image on the left. I keep on adding image on the left. I keep on adding image on the left. I keep on adding noise and then it becomes complete noise and then it becomes complete noise and then it becomes complete noise. The way it happens in text is can noise. The way it happens in text is can noise. The way it happens in text is can be represented by be represented by be represented by uh this kind of a thing. Yeah. So the uh this kind of a thing. Yeah. So the uh this kind of a thing. Yeah. So the way it happens in text is that uh at way it happens in text is that uh at
- L13 @ 536s
way it happens in text is that uh at time equal to 1, you almost have an time equal to 1, you almost have an time equal to 1, you almost have an entire clean text with maybe some mask entire clean text with maybe some mask entire clean text with maybe some mask over here. But as time increases, over here. But as time increases, over here. But as time increases, as time becomes two, I have two masks. as time becomes two, I have two masks. as time becomes two, I have two masks. At time as time becomes three, I have At time as time becomes three, I have At time as time becomes three, I have three masks. As time increases almost three masks. As time increases almost three masks. As time increases almost everything becomes masked. So if I watch everything becomes masked. So if I watch everything becomes masked. So if I watch the full noising you'll see that you the full noising you'll see that you the full noising you'll see that you start with lower masks and then you start with lower masks and then you start with lower masks and then you increase the number of masks. That's increase the number of masks. That's increase the number of masks. That's basically what is meant by a noising basically what is meant by a noising basically what is meant by a noising schedule. In the case of diffusion LLMs schedule. In the case of diffusion LLMs schedule. In the case of diffusion LLMs it's also called as a masking schedule it's also called as a masking schedule it's also called as a masking schedule and the masking schedule is basically and the masking schedule is basically and the masking schedule is basically proportional to the time. The further proportional to the time. The further proportional to the time. The further along we are in the noising schedule the along we are in the noising schedule the along we are in the noising schedule the more masks there will be. Usually there more masks there will be. Usually there more masks there will be. Usually there is a probability associated with it. So is a probability associated with it. So is a probability associated with it. So it's a Bernoli distribution. So the it's a Bernoli distribution. So the
- L13 @ 580s
it's a Bernoli distribution. So the probability of masking is way higher. probability of masking is way higher. probability of masking is way higher. Probability of masking all tokens is way Probability of masking all tokens is way Probability of masking all tokens is way higher as time is increased. But this higher as time is increased. But this higher as time is increased. But this animation suffices to understand how animation suffices to understand how animation suffices to understand how masking works for text. Okay. masking works for text. Okay. masking works for text. Okay. Um Um Um yeah. So yeah. So yeah. So what does it mean by masking essentially what does it mean by masking essentially what does it mean by masking essentially right? What it essentially means is that right? What it essentially means is that right? What it essentially means is that we just in our vocabulary. So if we have we just in our vocabulary. So if we have we just in our vocabulary. So if we have a vocabulary of 100,000 a vocabulary of 100,000 a vocabulary of 100,000 100,000 100,000 100,000 and the vocabulary is uh the boy etc. We and the vocabulary is uh the boy etc. We and the vocabulary is uh the boy etc. We add one more token to this vocabulary add one more token to this vocabulary add one more token to this vocabulary that's the mask token right. So now that's the mask token right. So now that's the mask token right. So now wherever there is a mask so here we have wherever there is a mask so here we have wherever there is a mask so here we have a mask right or here there is a mask and a mask right or here there is a mask and a mask right or here there is a mask and here there is a mask here there is a mask here there is a mask right or here there is a mask. So the right or here there is a mask. So the right or here there is a mask. So the original tokens were 1 11 20,000 55 and original tokens were 1 11 20,000 55 and original tokens were 1 11 20,000 55 and 3,000. Right? Now these are masked. So 3,000. Right? Now these are masked. So
- L13 @ 644s
3,000. Right? Now these are masked. So this is masked and this is masked. So this is masked and this is masked. So this is masked and this is masked. So this will be replaced with 1 0 0 0 1 this will be replaced with 1 0 0 0 1 this will be replaced with 1 0 0 0 1 which is the token ID for the mask. And which is the token ID for the mask. And which is the token ID for the mask. And this will also be replaced with 10 0 01 this will also be replaced with 10 0 01 this will also be replaced with 10 0 01 which is the token ID for the mask. And which is the token ID for the mask. And then this input sequence will go into then this input sequence will go into then this input sequence will go into the transformer architecture. Right? the transformer architecture. Right? the transformer architecture. Right? This input sequence goes into the This input sequence goes into the This input sequence goes into the transformer architecture transformer architecture transformer architecture or when I say transformer architecture or when I say transformer architecture or when I say transformer architecture actually this is the input sequence actually this is the input sequence actually this is the input sequence which goes over here which goes through which goes over here which goes through which goes over here which goes through the input the processor and the output. the input the processor and the output. the input the processor and the output. So we start with a noisy input with mask So we start with a noisy input with mask So we start with a noisy input with mask tokens whereas on the left hand side is tokens whereas on the left hand side is tokens whereas on the left hand side is the ARM. In the ARM we start with the the ARM. In the ARM we start with the the ARM. In the ARM we start with the entire sequence whereas in the case of entire sequence whereas in the case of entire sequence whereas in the case of diffusion language models we start with diffusion language models we start with diffusion language models we start with a noisy input with mask tokens. Okay.
- L13 @ 695s
a noisy input with mask tokens. Okay. a noisy input with mask tokens. Okay. Everything else remains exactly the Everything else remains exactly the Everything else remains exactly the same. Now we'll see what are the changes same. Now we'll see what are the changes same. Now we'll see what are the changes when we go through this IPO block. What when we go through this IPO block. What when we go through this IPO block. What are the changes when we go through the I are the changes when we go through the I are the changes when we go through the I um um um I P and the O block in the next step. I P and the O block in the next step. I P and the O block in the next step. But the first thing which I wanted all But the first thing which I wanted all But the first thing which I wanted all of you to understand is how the noising of you to understand is how the noising of you to understand is how the noising process actually works. So to give a process actually works. So to give a process actually works. So to give a more concrete feel to it, let's say if more concrete feel to it, let's say if more concrete feel to it, let's say if we have four time steps, t= 1, t=2, we have four time steps, t= 1, t=2, we have four time steps, t= 1, t=2, uh here we have t= 3 and here we have t uh here we have t= 3 and here we have t uh here we have t= 3 and here we have t equal to 4. Right? Initially nothing is equal to 4. Right? Initially nothing is equal to 4. Right? Initially nothing is masked but as the time increases you'll masked but as the time increases you'll masked but as the time increases you'll see more and more masks to the input see more and more masks to the input see more and more masks to the input tokens. Um yeah so the first difference between Um yeah so the first difference between the ARM architecture and the diffusion the ARM architecture and the diffusion the ARM architecture and the diffusion architecture if you compare them side by
- L13 @ 743s
architecture if you compare them side by architecture if you compare them side by side is the input which goes into this side is the input which goes into this side is the input which goes into this input block the processor block and the input block the processor block and the input block the processor block and the output block this input itself is output block this input itself is output block this input itself is different here we take a bunch of token different here we take a bunch of token different here we take a bunch of token ids and we add masks right so when the ids and we add masks right so when the ids and we add masks right so when the forward pass is done first a random time forward pass is done first a random time forward pass is done first a random time step is chosen based on that time step step is chosen based on that time step step is chosen based on that time step we decide how much noise is added how do we decide how much noise is added how do we decide how much noise is added how do we know how much noise to add because we we know how much noise to add because we we know how much noise to add because we a noise schedule which is defined. So a noise schedule which is defined. So a noise schedule which is defined. So let's say if a time step equal to three let's say if a time step equal to three let's say if a time step equal to three is chosen. The time step is chosen is chosen. The time step is chosen is chosen. The time step is chosen randomly in each forward pass. So here randomly in each forward pass. So here randomly in each forward pass. So here what happens is that in ARM architecture what happens is that in ARM architecture what happens is that in ARM architecture I told you that a batch is selected I told you that a batch is selected I told you that a batch is selected right? batch is selected then we get the right? batch is selected then we get the right? batch is selected then we get the targets targets
- L13 @ 783s
targets then we sorry we get the predictions then we sorry we get the predictions then we sorry we get the predictions then we get the loss then we get the loss then we get the loss then we update the parameters then we update the parameters then we update the parameters and then we choose a new batch this and then we choose a new batch this and then we choose a new batch this process stays the same for the diffusion process stays the same for the diffusion process stays the same for the diffusion LLM but it's just that whenever a new LLM but it's just that whenever a new LLM but it's just that whenever a new batch is selected a new time step is batch is selected a new time step is batch is selected a new time step is also randomly selected also randomly selected also randomly selected so if you have sim six time steps a new so if you have sim six time steps a new so if you have sim six time steps a new time step will be selected from 1 to time step will be selected from 1 to time step will be selected from 1 to six. So if time step number three is six. So if time step number three is six. So if time step number three is selected, selected, selected, if time step number three is se if time step number three is se if time step number three is se selected, roughly 50% of the input token selected, roughly 50% of the input token selected, roughly 50% of the input token sequence will be masked and they'll be sequence will be masked and they'll be sequence will be masked and they'll be fed into the architecture. The input, fed into the architecture. The input, fed into the architecture. The input, the processor and the output, the input, the processor and the output, the input, the processor and the output, the input, the processor and the output. Okay. the processor and the output. Okay. the processor and the output. Okay. So until now what we have seen is that So until now what we have seen is that So until now what we have seen is that what is the noising process in the case what is the noising process in the case
- L13 @ 831s
what is the noising process in the case of diffusion language models. The next of diffusion language models. The next of diffusion language models. The next step which we are going to see is that step which we are going to see is that step which we are going to see is that how do we predict the noise. So in the how do we predict the noise. So in the how do we predict the noise. So in the case of uh images the noise is predicted case of uh images the noise is predicted case of uh images the noise is predicted by unit. But let's see in the case of by unit. But let's see in the case of by unit. But let's see in the case of diffusion language models how do we diffusion language models how do we diffusion language models how do we predict the noise? What kind of a model predict the noise? What kind of a model predict the noise? What kind of a model do we need to predict the noise? Um and do we need to predict the noise? Um and do we need to predict the noise? Um and the answer is much simpler than you the answer is much simpler than you the answer is much simpler than you think. So, let's get into that right
- L14 @ 4s
Okay. So the next step in these key Okay. So the next step in these key characteristics which we mentioned and characteristics which we mentioned and characteristics which we mentioned and probably that's the most important step probably that's the most important step probably that's the most important step is this noise prediction. Right. is this noise prediction. Right. is this noise prediction. Right. Basically we need a model Basically we need a model Basically we need a model which can help us predict the noise in which can help us predict the noise in which can help us predict the noise in the noising process the noising process the noising process because the whole idea is as follows. because the whole idea is as follows. because the whole idea is as follows. Right? If you take a look at this video, if you take a look at this video again, if you take a look at this video again, what we want to do is that when we start what we want to do is that when we start what we want to do is that when we start rotating the fluid slowly, we want to rotating the fluid slowly, we want to rotating the fluid slowly, we want to predict how much noise is added at each predict how much noise is added at each predict how much noise is added at each step. So that during dinoising, it will step. So that during dinoising, it will step. So that during dinoising, it will it will help us recover the original it will help us recover the original it will help us recover the original state. state. state. So at each step of the noising process, So at each step of the noising process, So at each step of the noising process, how can we predict the amount of noise how can we predict the amount of noise how can we predict the amount of noise which is added? The way this is done in which is added? The way this is done in which is added? The way this is done in in the case of images is that in the case of images is that in the case of images is that you take a noisy image. you take a noisy image. you take a noisy image. You take a noisy image.
- L14 @ 61s
You take a noisy image. You take a noisy image. Let's say this is a noisy image of a Let's say this is a noisy image of a Let's say this is a noisy image of a panda which is the example which we have panda which is the example which we have panda which is the example which we have been seeing. You take a noisy image and been seeing. You take a noisy image and been seeing. You take a noisy image and then you use a noise prediction model then you use a noise prediction model then you use a noise prediction model which is a unit model to predict the which is a unit model to predict the which is a unit model to predict the noise. How does the unit model work? noise. How does the unit model work? noise. How does the unit model work? Well, it's a combination of Well, it's a combination of Well, it's a combination of CNN which is convolutional neural CNN which is convolutional neural CNN which is convolutional neural network upsampling and down sampling network upsampling and down sampling network upsampling and down sampling layer. So it can extract features from layer. So it can extract features from layer. So it can extract features from the image. So it can identify what's the image. So it can identify what's the image. So it can identify what's noise and what's not. We'll also we also noise and what's not. We'll also we also noise and what's not. We'll also we also know the actual noise right during know the actual noise right during know the actual noise right during training. So we can predict whether the training. So we can predict whether the training. So we can predict whether the predicted noise is actually close to the predicted noise is actually close to the predicted noise is actually close to the actual noise. So loss function is actual noise. So loss function is actual noise. So loss function is defined based on the predicted noise and defined based on the predicted noise and defined based on the predicted noise and the actual noise. So we need a loss the actual noise. So we need a loss the actual noise. So we need a loss function, right? We need to predict the
- L14 @ 104s
function, right? We need to predict the function, right? We need to predict the noise and then we need to compare it noise and then we need to compare it noise and then we need to compare it with the actual noise. Essentially we we with the actual noise. Essentially we we with the actual noise. Essentially we we need a model which can predict the noise need a model which can predict the noise need a model which can predict the noise added during the noising process and added during the noising process and added during the noising process and compare it with some true value. What is compare it with some true value. What is compare it with some true value. What is such model in the case of text? I want such model in the case of text? I want such model in the case of text? I want you all to take a pause at this moment you all to take a pause at this moment you all to take a pause at this moment and think about it yourself. Let's say and think about it yourself. Let's say and think about it yourself. Let's say you have this noisy input text, right? you have this noisy input text, right? you have this noisy input text, right? which is uh which is uh which is uh uh uh uh the the the so the next day is bright and we are so the next day is bright and we are so the next day is bright and we are masking next and is so it's the masking next and is so it's the masking next and is so it's the mask mask mask day day day mask and bright mask and bright mask and bright now that's my input sequence right now now that's my input sequence right now now that's my input sequence right now okay okay okay and uh so this is my input sequence this and uh so this is my input sequence this and uh so this is my input sequence this is mask and uh this is mask. Okay. is mask and uh this is mask. Okay. is mask and uh this is mask. Okay. How do I predict the true values here? How do I predict the true values here?
- L14 @ 164s
How do I predict the true values here? So what I want to do what does it mean So what I want to do what does it mean So what I want to do what does it mean prediction of noise? I somehow want to prediction of noise? I somehow want to prediction of noise? I somehow want to predict predict predict the true tokens the true tokens the true tokens at the places This is exactly what is done in images This is exactly what is done in images also. Right? What what does it mean to also. Right? What what does it mean to also. Right? What what does it mean to predict the noise from a noisy image? It predict the noise from a noisy image? It predict the noise from a noisy image? It just means that we need to predict how just means that we need to predict how just means that we need to predict how we go from the noisy image to the clean we go from the noisy image to the clean we go from the noisy image to the clean image. This is exactly what we are doing over This is exactly what we are doing over here. We need to predict how to go from here. We need to predict how to go from here. We need to predict how to go from this noisy text to the clean text. So we this noisy text to the clean text. So we this noisy text to the clean text. So we need to predict what's the text which need to predict what's the text which need to predict what's the text which was here in the first place. was here in the first place. was here in the first place. Again, I invite you to pause this video Again, I invite you to pause this video Again, I invite you to pause this video at this moment and think about it. at this moment and think about it. at this moment and think about it. How do you know what the true token was How do you know what the true token was How do you know what the true token was at this position? And how do you get the at this position? And how do you get the
- L14 @ 221s
at this position? And how do you get the loss function? loss function? loss function? Well, the answer is extremely simple, Well, the answer is extremely simple, Well, the answer is extremely simple, right? Because we we already know what right? Because we we already know what right? Because we we already know what the true tokens are. We know that here the true tokens are. We know that here the true tokens are. We know that here the true token is next and we know that the true token is next and we know that the true token is next and we know that here the true token is is, here the true token is is, here the true token is is, right? Then how do we get the loss right? Then how do we get the loss right? Then how do we get the loss function? We have exactly seen how to function? We have exactly seen how to function? We have exactly seen how to propagate this input sequence through my propagate this input sequence through my propagate this input sequence through my model model model and I'll get my predictions right. I'll and I'll get my predictions right. I'll and I'll get my predictions right. I'll get my predictions get my predictions get my predictions for the next I'll get my predictions for for the next I'll get my predictions for for the next I'll get my predictions for each of these. each of these. each of these. So I'll get the next token prediction So I'll get the next token prediction So I'll get the next token prediction for the I'll get the next token for the I'll get the next token for the I'll get the next token prediction for this mask. I'll get the prediction for this mask. I'll get the prediction for this mask. I'll get the next token prediction for day. I'll get next token prediction for day. I'll get next token prediction for day. I'll get the next token for mask and I'll get the the next token for mask and I'll get the the next token for mask and I'll get the next token prediction for bride. next token prediction for bride. next token prediction for bride. Right. So the next token which is Right. So the next token which is
- L14 @ 272s
Right. So the next token which is predicted at this position I'll compare it with the true next token I'll compare it with the true next token which should be next. which should be next. which should be next. The next token which is predicted at The next token which is predicted at The next token which is predicted at this position. I'll compare it with the this position. I'll compare it with the this position. I'll compare it with the true next token which is bright. true next token which is bright. true next token which is bright. And I'll use this to compute my loss And I'll use this to compute my loss And I'll use this to compute my loss function. So this is L1 and this is L2. function. So this is L1 and this is L2. function. So this is L1 and this is L2. So my loss function will then be L1 + L2 So my loss function will then be L1 + L2 So my loss function will then be L1 + L2 / 2. Think about this for a moment, right? We Think about this for a moment, right? We are corrupting the text. We are masking are corrupting the text. We are masking are corrupting the text. We are masking it. I'm just passing the whole input it. I'm just passing the whole input it. I'm just passing the whole input sequence with the masks through the sequence with the masks through the sequence with the masks through the architecture and I'm getting the architecture and I'm getting the architecture and I'm getting the predictions. Why? How do I know I get predictions. Why? How do I know I get predictions. Why? How do I know I get the predictions? Because this is exactly the predictions? Because this is exactly the predictions? Because this is exactly what we saw in ARMs. If you see, if you what we saw in ARMs. If you see, if you what we saw in ARMs. If you see, if you remember what we saw in ARMs, remember what we saw in ARMs, remember what we saw in ARMs, in ARMs, for every input, we are getting in ARMs, for every input, we are getting in ARMs, for every input, we are getting predictions, right? For the next, these
- L14 @ 325s
predictions, right? For the next, these predictions, right? For the next, these are my predictions. are my predictions. are my predictions. These are my predictions. I'm just going These are my predictions. I'm just going These are my predictions. I'm just going to compare the predictions for my mask to compare the predictions for my mask to compare the predictions for my mask with the true tokens with the true tokens with the true tokens and I'm going to get the loss function. and I'm going to get the loss function. and I'm going to get the loss function. It's as simple as that. So I'll compare It's as simple as that. So I'll compare It's as simple as that. So I'll compare the predicted token at this position. the predicted token at this position. the predicted token at this position. I'll compare it with the true token at I'll compare it with the true token at I'll compare it with the true token at this position. I'll get the predicted this position. I'll get the predicted this position. I'll get the predicted token at this position. I'll compare it token at this position. I'll compare it token at this position. I'll compare it with the true token at this position. with the true token at this position. with the true token at this position. And then I'll get the loss function And then I'll get the loss function And then I'll get the loss function based on the predicted token and the based on the predicted token and the based on the predicted token and the true token. And then I'll take an true token. And then I'll take an true token. And then I'll take an average of these losses. That's the average of these losses. That's the average of these losses. That's the model which I'm going to use. So in model which I'm going to use. So in model which I'm going to use. So in simple way the model which predicts the simple way the model which predicts the simple way the model which predicts the this model which predicts the noise this model which predicts the noise this model which predicts the noise right which is the second right which is the second right which is the second um second characteristics the model um second characteristics the model
- L14 @ 371s
um second characteristics the model which predicts the noise which was a which predicts the noise which was a which predicts the noise which was a unit in the case of diffusion for unit in the case of diffusion for unit in the case of diffusion for images. This model is nothing but a images. This model is nothing but a images. This model is nothing but a transformer. is the or I should say the language is the or I should say the language model architecture itself. We are going to see this architecture We are going to see this architecture now just in a moment. But this now just in a moment. But this now just in a moment. But this architecture which I'm showing on on architecture which I'm showing on on architecture which I'm showing on on this side that is the model which this side that is the model which this side that is the model which predicts the noise. Okay. So now I'm predicts the noise. Okay. So now I'm predicts the noise. Okay. So now I'm just going to just going to just going to uh take you through this entire model uh take you through this entire model uh take you through this entire model because you'll see that most of it because you'll see that most of it because you'll see that most of it actually remains exactly the same. Here actually remains exactly the same. Here actually remains exactly the same. Here we have the input in the ARM we have the input in the ARM we have the input in the ARM architecture. We saw that there is a architecture. We saw that there is a architecture. We saw that there is a input, processor and output. Here also input, processor and output. Here also input, processor and output. Here also there will be an input, processor and there will be an input, processor and there will be an input, processor and output. Here also there is a transformer output. Here also there is a transformer output. Here also there is a transformer block. Many things remain exactly the block. Many things remain exactly the block. Many things remain exactly the same. Let's see what the differences same. Let's see what the differences same. Let's see what the differences are. Right? In the ARM architecture, we
- L14 @ 434s
are. Right? In the ARM architecture, we are. Right? In the ARM architecture, we remember we had the uh token embedding remember we had the uh token embedding remember we had the uh token embedding and position embedding. We added them and position embedding. We added them and position embedding. We added them for the input sequences. Right? In the for the input sequences. Right? In the for the input sequences. Right? In the case of uh case of uh case of uh uh diffusion models, the first change uh diffusion models, the first change uh diffusion models, the first change which happens is that in these token which happens is that in these token which happens is that in these token embeddings, one or more of these tokens embeddings, one or more of these tokens embeddings, one or more of these tokens are masked. So this is masked. are masked. So this is masked. are masked. So this is masked. That's what we have already seen. So That's what we have already seen. So That's what we have already seen. So token embeddings, it's masked. Position token embeddings, it's masked. Position token embeddings, it's masked. Position embeddings, these remain the same. Here embeddings, these remain the same. Here embeddings, these remain the same. Here we don't have anything masked but token we don't have anything masked but token we don't have anything masked but token embedding we have a mask token here and embedding we have a mask token here and embedding we have a mask token here and which tokens to mask are is decided which tokens to mask are is decided which tokens to mask are is decided probabilistically but how many tokens to probabilistically but how many tokens to probabilistically but how many tokens to mask is decided by that noise schedule mask is decided by that noise schedule mask is decided by that noise schedule which we have. So this this noise should which we have. So this this noise should which we have. So this this noise should so let's say we get the token embedding so let's say we get the token embedding so let's say we get the token embedding we get the position embedding these we get the position embedding these we get the position embedding these tokens are now masked we add the token embedding plus position
- L14 @ 486s
we add the token embedding plus position embedding the one major difference in embedding the one major difference in embedding the one major difference in diffusion language models is that along diffusion language models is that along diffusion language models is that along with adding token embedding plus with adding token embedding plus with adding token embedding plus position embedding we also add time position embedding we also add time position embedding we also add time embeddings right so we have time steps embeddings right so we have time steps embeddings right so we have time steps for the noise schedule if those are from for the noise schedule if those are from for the noise schedule if those are from 1 to 4 we have a 768 embed 768 create 1 to 4 we have a 768 embed 768 create 1 to 4 we have a 768 embed 768 create dimensional vector for each of these dimensional vector for each of these dimensional vector for each of these time steps. time steps. time steps. And remember what I mentioned for And remember what I mentioned for And remember what I mentioned for diffusion models. In diffusion models, diffusion models. In diffusion models, diffusion models. In diffusion models, whenever we are doing a forward pass, we whenever we are doing a forward pass, we whenever we are doing a forward pass, we not only choose a batch, we also choose not only choose a batch, we also choose not only choose a batch, we also choose the time step as well. the time step as well. the time step as well. So if you have chosen let's say time So if you have chosen let's say time So if you have chosen let's say time step equal to two, the time embedding, step equal to two, the time embedding, step equal to two, the time embedding, we'll go to the time embedding matrix we'll go to the time embedding matrix we'll go to the time embedding matrix and we'll see the vector corresponding and we'll see the vector corresponding and we'll see the vector corresponding to time step equal to two and that same to time step equal to two and that same to time step equal to two and that same time embedding vector is added to all my time embedding vector is added to all my
- L14 @ 531s
time embedding vector is added to all my tokens. So now the input embedding is tokens. So now the input embedding is tokens. So now the input embedding is not just the token embedding plus not just the token embedding plus not just the token embedding plus position embedding. My input embedding position embedding. My input embedding position embedding. My input embedding is token embedding plus position is token embedding plus position is token embedding plus position embedding plus time embedding. There are embedding plus time embedding. There are embedding plus time embedding. There are three embedding vectors which are added three embedding vectors which are added three embedding vectors which are added together. Why do we consider time together. Why do we consider time together. Why do we consider time embedding here? Because the time at embedding here? Because the time at embedding here? Because the time at which we are predicting the noise, it's which we are predicting the noise, it's which we are predicting the noise, it's also important. also important. also important. While dnoising, we need to know the time While dnoising, we need to know the time While dnoising, we need to know the time step at which we are denoising. Right? step at which we are denoising. Right? step at which we are denoising. Right? If we are way if it's extremely if it's If we are way if it's extremely if it's If we are way if it's extremely if it's extremely noisy image, it's further extremely noisy image, it's further extremely noisy image, it's further along in time. If the image is not that along in time. If the image is not that along in time. If the image is not that noisy, it's earlier in time. This is one noisy, it's earlier in time. This is one noisy, it's earlier in time. This is one major change. We add the time embedding major change. We add the time embedding major change. We add the time embedding also along with the token embedding plus also along with the token embedding plus also along with the token embedding plus the position embedding. That's the first the position embedding. That's the first the position embedding. That's the first change. So now if you see if you compare change. So now if you see if you compare
- L14 @ 575s
change. So now if you see if you compare these block diagrams these block diagrams these block diagrams in the ARM architecture, we only had the in the ARM architecture, we only had the in the ARM architecture, we only had the token embedding and position embedding. token embedding and position embedding. token embedding and position embedding. In the uh diffusion model architecture, In the uh diffusion model architecture, In the uh diffusion model architecture, we have token embedding, we have we have token embedding, we have we have token embedding, we have position embedding, and we have time position embedding, and we have time position embedding, and we have time embedding. And all these are added to embedding. And all these are added to embedding. And all these are added to each other. This resulting input each other. This resulting input each other. This resulting input embedding is then passed through through embedding is then passed through through embedding is then passed through through the transformer block. Okay. Now the the transformer block. Okay. Now the the transformer block. Okay. Now the transformer block remains exactly the transformer block remains exactly the transformer block remains exactly the same. same. same. The transformer block has no change. We The transformer block has no change. We The transformer block has no change. We have the layer norm, we have the multi have the layer norm, we have the multi have the layer norm, we have the multi head attention, we have dropout, head attention, we have dropout, head attention, we have dropout, shortcut, layer norm, feed forward shortcut, layer norm, feed forward shortcut, layer norm, feed forward network and dropout. So the whole flow network and dropout. So the whole flow network and dropout. So the whole flow which we have seen layer norm, multi which we have seen layer norm, multi which we have seen layer norm, multi head attention, dropout, shortcut head attention, dropout, shortcut head attention, dropout, shortcut connection here. then another layer norm connection here. then another layer norm connection here. then another layer norm feed forward neural network dropout and feed forward neural network dropout and feed forward neural network dropout and another shortcut connection. This stays another shortcut connection. This stays another shortcut connection. This stays exactly the same. The difference is in exactly the same. The difference is in
- L14 @ 625s
exactly the same. The difference is in the attention mechanism. the attention mechanism. the attention mechanism. Now think about it right here. We no Now think about it right here. We no Now think about it right here. We no longer have a next token prediction longer have a next token prediction longer have a next token prediction task. What we have to do is we just have task. What we have to do is we just have task. What we have to do is we just have to predict the masks right? We just have to predict the masks right? We just have to predict the masks right? We just have to predict the masks. So if we are to predict the masks. So if we are to predict the masks. So if we are looking at this mask, I can look at looking at this mask, I can look at looking at this mask, I can look at tokens before this mask and I can look tokens before this mask and I can look tokens before this mask and I can look at tokens after this mask also. I get at tokens after this mask also. I get at tokens after this mask also. I get information of tokens behind me and I information of tokens behind me and I information of tokens behind me and I get information of tokens in front of me get information of tokens in front of me get information of tokens in front of me also. This is very different than ARMs. also. This is very different than ARMs. also. This is very different than ARMs. In ARM, if I'm looking at this position, In ARM, if I'm looking at this position, In ARM, if I'm looking at this position, I only have access to tokens which come I only have access to tokens which come I only have access to tokens which come before this position. But in diffusion before this position. But in diffusion before this position. But in diffusion models, for one token, I have access to models, for one token, I have access to models, for one token, I have access to tokens before it and after it. Which tokens before it and after it. Which tokens before it and after it. Which means that when I'm calculating the means that when I'm calculating the
- L14 @ 668s
means that when I'm calculating the attention so the mask the mask day the mask day mask bright right which means that if mask bright right which means that if mask bright right which means that if I'm calculating the attention score for I'm calculating the attention score for I'm calculating the attention score for any token I can calculate the attention any token I can calculate the attention any token I can calculate the attention score for tokens before that token and score for tokens before that token and score for tokens before that token and also after that token right so if this also after that token right so if this also after that token right so if this is my attention scores matrix this will is my attention scores matrix this will is my attention scores matrix this will be a 5x5 matrix 1 2 3 4 5 1 2 3 4 5 1 2 be a 5x5 matrix 1 2 3 4 5 1 2 3 4 5 1 2 be a 5x5 matrix 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 In the case of auto reggressive models I In the case of auto reggressive models I will not I will not have so these will not I will not have so these will not I will not have so these attention scores right which I'm marking attention scores right which I'm marking attention scores right which I'm marking right now these attention scores will right now these attention scores will right now these attention scores will not be there I will only have these not be there I will only have these not be there I will only have these attention scores for an auto reggressive attention scores for an auto reggressive attention scores for an auto reggressive model model whereas whereas whereas for a diffusion model I can look at a for a diffusion model I can look at a for a diffusion model I can look at a token from all sides. So if I just rub token from all sides. So if I just rub token from all sides. So if I just rub this right now this right now this right now if I look at I essentially have this entire causal
- L14 @ 748s
I essentially have this entire causal attention matrix which is available to attention matrix which is available to attention matrix which is available to me. I don't have to set these values to me. I don't have to set these values to me. I don't have to set these values to zero like in the ARM block. So one major zero like in the ARM block. So one major zero like in the ARM block. So one major difference takes place in this attention difference takes place in this attention difference takes place in this attention module. If you look here in the auto module. If you look here in the auto module. If you look here in the auto reggressive modeling the causal reggressive modeling the causal reggressive modeling the causal attention exists which means that all attention exists which means that all attention exists which means that all the attention scores above this diagonal the attention scores above this diagonal the attention scores above this diagonal are set to zero. Whereas this does not are set to zero. Whereas this does not are set to zero. Whereas this does not happen in the case of diffusion models. happen in the case of diffusion models. happen in the case of diffusion models. We don't have such a causal attention We don't have such a causal attention We don't have such a causal attention mask. We have a biirectional attention mask. We have a biirectional attention mask. We have a biirectional attention mechanism. So all of the attention mechanism. So all of the attention mechanism. So all of the attention scores remain valid. scores remain valid. scores remain valid. That's one major difference and I want That's one major difference and I want That's one major difference and I want all of you to keep this in mind. That's all of you to keep this in mind. That's all of you to keep this in mind. That's why we went into detail in the attention why we went into detail in the attention why we went into detail in the attention mechanism. That is the second major mechanism. That is the second major mechanism. That is the second major difference, right? Uh what's the third difference, right? Uh what's the third difference, right? Uh what's the third major difference? The third major major difference? The third major
- L14 @ 795s
major difference? The third major difference comes in the output token difference comes in the output token difference comes in the output token prediction, right? Uh let's say here we prediction, right? Uh let's say here we prediction, right? Uh let's say here we have every effort moves you, right? and have every effort moves you, right? and have every effort moves you, right? and we have predictions and we have targets we have predictions and we have targets we have predictions and we have targets and we compute the loss for all of these and we compute the loss for all of these and we compute the loss for all of these and we add all of these losses together and we add all of these losses together and we add all of these losses together for diffusion based modeling it's very for diffusion based modeling it's very for diffusion based modeling it's very different for diffusion based models different for diffusion based models different for diffusion based models let's say this is my input sequence I let's say this is my input sequence I let's say this is my input sequence I have to only consider the prediction at have to only consider the prediction at have to only consider the prediction at this position and I have to consider the this position and I have to consider the this position and I have to consider the prediction at this position and I have prediction at this position and I have prediction at this position and I have to compare it with the actual value to compare it with the actual value to compare it with the actual value right so I just have to find the cross right so I just have to find the cross right so I just have to find the cross entropy loss for my mask one I have to entropy loss for my mask one I have to entropy loss for my mask one I have to find the cross entropy loss for my mask find the cross entropy loss for my mask find the cross entropy loss for my mask two and I have to take the average of two and I have to take the average of two and I have to take the average of these losses. That's my loss. these losses. That's my loss.
- L14 @ 836s
these losses. That's my loss. I don't have to find the loss for the I don't have to find the loss for the I don't have to find the loss for the other positions at all. other positions at all. other positions at all. Even in the example which we have chosen Even in the example which we have chosen Even in the example which we have chosen over here um the mask day. So this over here um the mask day. So this over here um the mask day. So this example right the example right the example right the mask day mask and bright right. So here I only have to care. So let's So here I only have to care. So let's say I have my prediction here and the say I have my prediction here and the say I have my prediction here and the prediction here prediction here prediction here the true value is next and that comes at the true value is next and that comes at the true value is next and that comes at ID equal to 3. I just have to take ID equal to 3. I just have to take ID equal to 3. I just have to take negative log of probability at this negative log of probability at this negative log of probability at this position position position and here the true value is let's say is and here the true value is let's say is and here the true value is let's say is comes at ID equal to 500. So then I just comes at ID equal to 500. So then I just comes at ID equal to 500. So then I just have to take negative log of probability have to take negative log of probability have to take negative log of probability of 500 and the loss of 500 and the loss of 500 and the loss and the loss is just then equal to and the loss is just then equal to and the loss is just then equal to negative of 1 by 2 log of P3 plus log of negative of 1 by 2 log of P3 plus log of negative of 1 by 2 log of P3 plus log of P500. P500. P500. So if this is let's say 0.1 this So if this is let's say 0.1 this
- L14 @ 904s
So if this is let's say 0.1 this probability and this is2 my loss in this probability and this is2 my loss in this probability and this is2 my loss in this case will be - 1 by2 log of.1 log of.1 plus log of 2. plus log of 2. plus log of 2. So let's see how much that is. -1 by 2 So let's see how much that is. -1 by 2 So let's see how much that is. -1 by 2 into log of.1 into log of.1 into log of.1 plus log of plus log of plus log of that's 84. So the loss which I've that's 84. So the loss which I've that's 84. So the loss which I've obtained in this case is obtained in this case is obtained in this case is um um um so the loss which I've obtained in this so the loss which I've obtained in this so the loss which I've obtained in this case is 84 case is 84 case is 84 and then I do back propagation. So the and then I do back propagation. So the and then I do back propagation. So the way this entire way this entire way this entire um um noising process happens now I've tried noising process happens now I've tried noising process happens now I've tried to explain it not the entire noising to explain it not the entire noising to explain it not the entire noising process but the entire flow we first process but the entire flow we first process but the entire flow we first decide a time point let's say time equal decide a time point let's say time equal decide a time point let's say time equal to three right I asked 50% of the input to three right I asked 50% of the input to three right I asked 50% of the input sequences the masked input then goes sequences the masked input then goes sequences the masked input then goes through this whole architecture what are through this whole architecture what are through this whole architecture what are the changes in this architecture the changes in this architecture the changes in this architecture compared to the auto reggressive model compared to the auto reggressive model compared to the auto reggressive model we have a time embedding now which was
- L14 @ 975s
we have a time embedding now which was we have a time embedding now which was not there auto reggressive model in not there auto reggressive model in not there auto reggressive model in multi attention block uh we don't have multi attention block uh we don't have multi attention block uh we don't have the causal attention now we can take the the causal attention now we can take the the causal attention now we can take the whole whole attention attention scores whole whole attention attention scores whole whole attention attention scores matrix because for each token we can matrix because for each token we can matrix because for each token we can look behind and we can look ahead also look behind and we can look ahead also look behind and we can look ahead also okay that's why generally b models are okay that's why generally b models are okay that's why generally b models are good at sentiment analysis right because good at sentiment analysis right because good at sentiment analysis right because bird is also biirectional in bird models bird is also biirectional in bird models bird is also biirectional in bird models we look at a token from behind and ahead we look at a token from behind and ahead we look at a token from behind and ahead so you can look at an entire sentence so you can look at an entire sentence so you can look at an entire sentence and predict the sentiment of the and predict the sentiment of the and predict the sentiment of the sentence right It's kind of similar sentence right It's kind of similar sentence right It's kind of similar here. The only difference from the BERT here. The only difference from the BERT here. The only difference from the BERT architecture is that in BERT the masking architecture is that in BERT the masking architecture is that in BERT the masking ratio is fixed. But here the masking ratio is fixed. But here the masking ratio is fixed. But here the masking ratio is different. Right? The masking ratio is different. Right? The masking ratio is different. Right? The masking ratio uh in this case actually depends ratio uh in this case actually depends ratio uh in this case actually depends on u the time step. More the time step,
- L14 @ 1022s
on u the time step. More the time step, on u the time step. More the time step, more is the masking. Less the time step, more is the masking. Less the time step, more is the masking. Less the time step, less the masking. That's the difference less the masking. That's the difference less the masking. That's the difference between BERT and uh language diffusion. between BERT and uh language diffusion. between BERT and uh language diffusion. So we have chosen time equal to three. So we have chosen time equal to three. So we have chosen time equal to three. 50% is masked that input sequence. we 50% is masked that input sequence. we 50% is masked that input sequence. we choose a batch. So if the batch size is choose a batch. So if the batch size is choose a batch. So if the batch size is 128, we choose a batch of 128 sequences. 128, we choose a batch of 128 sequences. 128, we choose a batch of 128 sequences. We mask 50% of those and we pass it We mask 50% of those and we pass it We mask 50% of those and we pass it through this entire architecture. So through this entire architecture. So through this entire architecture. So this is let's say the input sequence, it this is let's say the input sequence, it this is let's say the input sequence, it first goes through the token embedding. first goes through the token embedding. first goes through the token embedding. So every token is converted into an So every token is converted into an So every token is converted into an embedding vector. We add the positional embedding vector. We add the positional embedding vector. We add the positional embedding, we add the time embedding and embedding, we add the time embedding and embedding, we add the time embedding and uh so token embedding plus position uh so token embedding plus position uh so token embedding plus position embedding plus time embedding is my embedding plus time embedding is my embedding plus time embedding is my final input vector that goes through my final input vector that goes through my final input vector that goes through my transformer block. So I have my layer transformer block. So I have my layer
- L14 @ 1063s
transformer block. So I have my layer normalization multi head attention I normalization multi head attention I normalization multi head attention I have my dropout I have my skip have my dropout I have my skip have my dropout I have my skip connection I have second layer connection I have second layer connection I have second layer normalization I have my feed forward normalization I have my feed forward normalization I have my feed forward neural network I have dropout and neural network I have dropout and neural network I have dropout and another shortcut connection on the another shortcut connection on the another shortcut connection on the outset which means at the end I have my outset which means at the end I have my outset which means at the end I have my final normalization I have this logits final normalization I have this logits final normalization I have this logits matrix and I have the cross entropy loss matrix and I have the cross entropy loss matrix and I have the cross entropy loss which is only computed for the mask which is only computed for the mask which is only computed for the mask tokens that's the second major last tokens that's the second major last tokens that's the second major last major difference between language major difference between language major difference between language diffusion and auto reggressive model diffusion and auto reggressive model diffusion and auto reggressive model diffusion here the Cross entropy loss is diffusion here the Cross entropy loss is diffusion here the Cross entropy loss is only calculated for the mask tokens and only calculated for the mask tokens and only calculated for the mask tokens and for nothing else. for nothing else. for nothing else. So to summarize the entire noising So to summarize the entire noising So to summarize the entire noising process looks like this. We have the process looks like this. We have the process looks like this. We have the let's say we choose the batch size. We let's say we choose the batch size. We let's say we choose the batch size. We choose two things, right? We choose the choose two things, right? We choose the choose two things, right? We choose the batch size. We choose the time step, batch size. We choose the time step,
- L14 @ 1111s
batch size. We choose the time step, right? Let's say batch size is 128. We right? Let's say batch size is 128. We right? Let's say batch size is 128. We assemble an input sequence of assemble an input sequence of assemble an input sequence of 128 tokens uh and we choose a batch size. So we uh and we choose a batch size. So we randomly mask let's say 50. We choose a randomly mask let's say 50. We choose a randomly mask let's say 50. We choose a time step. If the time step is half of time step. If the time step is half of time step. If the time step is half of the total time, we randomly mask 50% of the total time, we randomly mask 50% of the total time, we randomly mask 50% of these. So these will be replaced with these. So these will be replaced with these. So these will be replaced with mask. Okay. Now before now these this input Okay. Now before now these this input sequence before going through the sequence before going through the sequence before going through the transformer architecture we add the transformer architecture we add the transformer architecture we add the token embeddings plus the position token embeddings plus the position token embeddings plus the position embeddings plus the time embeddings. embeddings plus the time embeddings. embeddings plus the time embeddings. So this vector which I have shown is So this vector which I have shown is So this vector which I have shown is token embedding plus position plus time. token embedding plus position plus time. token embedding plus position plus time. And this this input vectors then go into And this this input vectors then go into And this this input vectors then go into the transformer architecture. When they the transformer architecture. When they the transformer architecture. When they come out of the transformer come out of the transformer come out of the transformer architecture, we get the categorical architecture, we get the categorical architecture, we get the categorical cross entropy loss only on the mask cross entropy loss only on the mask cross entropy loss only on the mask tokens. Based on this loss, we find the tokens. Based on this loss, we find the
- L14 @ 1170s
tokens. Based on this loss, we find the partial derivative of the loss with partial derivative of the loss with partial derivative of the loss with respect to all the parameters. And respect to all the parameters. And respect to all the parameters. And remember we have seen what what do I remember we have seen what what do I remember we have seen what what do I mean by all the parameters? Uh we have mean by all the parameters? Uh we have mean by all the parameters? Uh we have seen everything marked P over here are seen everything marked P over here are seen everything marked P over here are all the parameters. We have seen that in all the parameters. We have seen that in all the parameters. We have seen that in one of the previous lectures. This one of the previous lectures. This one of the previous lectures. This remains the same. All trainable remains the same. All trainable remains the same. All trainable parameters remain the same in ARM and in parameters remain the same in ARM and in parameters remain the same in ARM and in the language diffusion. the language diffusion. the language diffusion. Right? So we get the partial derivative Right? So we get the partial derivative Right? So we get the partial derivative of loss with respect to all the of loss with respect to all the of loss with respect to all the parameters. Then we update parameters. Then we update parameters. Then we update then we update the parameters then we update the parameters then we update the parameters according to ADAM optimizer or any other according to ADAM optimizer or any other according to ADAM optimizer or any other optimizer which you want. And then this optimizer which you want. And then this optimizer which you want. And then this entire thing goes into a loop. entire thing goes into a loop. entire thing goes into a loop. This is how I pre-train. This is how I pre-train. This is how I pre-train. This is how I pre-train a language This is how I pre-train a language This is how I pre-train a language diffusion model. What is implicitly happening in this What is implicitly happening in this case is the better we pre-train a case is the better we pre-train a
- L14 @ 1227s
case is the better we pre-train a language diffusion model the better we language diffusion model the better we language diffusion model the better we are able to discover what noise is added are able to discover what noise is added are able to discover what noise is added and during the generation process or and during the generation process or and during the generation process or during the dnoising the better we'll be during the dnoising the better we'll be during the dnoising the better we'll be able to find the underlying probability able to find the underlying probability able to find the underlying probability distribution right so if you look at distribution right so if you look at distribution right so if you look at this analogy over here. Yeah. So what we are doing right now is Yeah. So what we are doing right now is the noising process, right? And we are the noising process, right? And we are the noising process, right? And we are training the model to recover the noise. training the model to recover the noise. training the model to recover the noise. And the better we train this model to And the better we train this model to And the better we train this model to recover the noise, the better we'll be recover the noise, the better we'll be recover the noise, the better we'll be at denoising. at denoising. at denoising. So the better we so in dnoising what we So the better we so in dnoising what we So the better we so in dnoising what we are going to do is that we are going to are going to do is that we are going to are going to do is that we are going to start with all masks and then we we are start with all masks and then we we are start with all masks and then we we are slowly going to uncover the masks and slowly going to uncover the masks and slowly going to uncover the masks and recover the actual tokens in that recover the actual tokens in that recover the actual tokens in that position. position. position. That's why we get gifs such as this. That's why we get gifs such as this. That's why we get gifs such as this. Start with all masks and then we uncover
- L14 @ 1278s
Start with all masks and then we uncover Start with all masks and then we uncover tokens correct tokens at that position. tokens correct tokens at that position. tokens correct tokens at that position. But dnoising will only work if the loss But dnoising will only work if the loss But dnoising will only work if the loss function during the pre-training or function during the pre-training or function during the pre-training or understanding the noise is good. Which understanding the noise is good. Which understanding the noise is good. Which means the loss function becomes lower means the loss function becomes lower means the loss function becomes lower and lower and lower. and lower and lower. and lower and lower. In this large language diffusion models In this large language diffusion models In this large language diffusion models paper, they have mentioned a theoretical paper, they have mentioned a theoretical paper, they have mentioned a theoretical result here which I just want to result here which I just want to result here which I just want to mention. So this is the loss function mention. So this is the loss function mention. So this is the loss function during pre-training. All which I have during pre-training. All which I have during pre-training. All which I have explained right now. Everything which I explained right now. Everything which I explained right now. Everything which I have explained on the screen this this have explained on the screen this this have explained on the screen this this loss function which I have explained on loss function which I have explained on loss function which I have explained on the screen until now is explained in one the screen until now is explained in one the screen until now is explained in one simple formula over here. That's why simple formula over here. That's why simple formula over here. That's why it's very difficult to understand it's very difficult to understand it's very difficult to understand research papers until you have a firm research papers until you have a firm research papers until you have a firm background of the foundations. So this background of the foundations. So this background of the foundations. So this is the loss function. What these people is the loss function. What these people is the loss function. What these people predict is that predict is that
- L14 @ 1322s
predict is that the probability of actually recovering the probability of actually recovering the probability of actually recovering the true true distribution or the the true true distribution or the the true true distribution or the chances of recovering the true chances of recovering the true chances of recovering the true distribution are directly dependent on distribution are directly dependent on distribution are directly dependent on the loss function. The lower the loss the loss function. The lower the loss the loss function. The lower the loss function, the better we can recover the function, the better we can recover the function, the better we can recover the true distribution during dinoising. This true distribution during dinoising. This true distribution during dinoising. This is what is meant by this second formula. is what is meant by this second formula. is what is meant by this second formula. So dnoising is very directly linked with So dnoising is very directly linked with So dnoising is very directly linked with this loss function over here. So there this loss function over here. So there this loss function over here. So there is a link between the noising process is a link between the noising process is a link between the noising process and the dinoising. In the noising and the dinoising. In the noising and the dinoising. In the noising process, we are adding masks and we're process, we are adding masks and we're process, we are adding masks and we're recovering the true value at those recovering the true value at those recovering the true value at those masks. The better we do this, the better masks. The better we do this, the better masks. The better we do this, the better will be the dnoising also. I'm not going will be the dnoising also. I'm not going will be the dnoising also. I'm not going into the mathematical details of this into the mathematical details of this into the mathematical details of this but they have mentioned that but they have mentioned that but they have mentioned that um this has been proved the loss um this has been proved the loss um this has been proved the loss function in equation three has been function in equation three has been function in equation three has been proved to be an upper bound on the
- L14 @ 1368s
proved to be an upper bound on the proved to be an upper bound on the negative log likelihood making it a negative log likelihood making it a negative log likelihood making it a principled objective for generative principled objective for generative principled objective for generative modeling. So what they are saying is modeling. So what they are saying is modeling. So what they are saying is that the lower the loss function is the that the lower the loss function is the that the lower the loss function is the better we'll be at this uh true distri better we'll be at this uh true distri better we'll be at this uh true distri better better we'll be at approximating better better we'll be at approximating better better we'll be at approximating this true distribution. this true distribution. this true distribution. So if the green is the true distribution So if the green is the true distribution So if the green is the true distribution and the orange is the predicted, the and the orange is the predicted, the and the orange is the predicted, the lower the loss function in pre-training, lower the loss function in pre-training, lower the loss function in pre-training, the closer these two will lie. the closer these two will lie. the closer these two will lie. So the dotted red line will lie much So the dotted red line will lie much So the dotted red line will lie much closer to the orange. If the loss closer to the orange. If the loss closer to the orange. If the loss function during pre-training is lower, function during pre-training is lower, function during pre-training is lower, what is the loss function? It's exactly what is the loss function? It's exactly what is the loss function? It's exactly what we saw saw over here. what we saw saw over here. what we saw saw over here. The loss function essentially quantifies The loss function essentially quantifies The loss function essentially quantifies what is there at the masked masked what is there at the masked masked what is there at the masked masked positions versus what should be there at positions versus what should be there at positions versus what should be there at the masked positions. Okay. the masked positions. Okay.
- L14 @ 1414s
the masked positions. Okay. So, until now we have seen the first two So, until now we have seen the first two So, until now we have seen the first two steps of diffusion based language steps of diffusion based language steps of diffusion based language models. First is what is the noising models. First is what is the noising models. First is what is the noising process and second is how do we predict process and second is how do we predict process and second is how do we predict the noise? We predict the noise through the noise? We predict the noise through the noise? We predict the noise through the language model architecture itself the language model architecture itself the language model architecture itself which I've depicted which I've depicted which I've depicted using this diagram. using this diagram. using this diagram. So, anytime you are confused, you can So, anytime you are confused, you can So, anytime you are confused, you can take a look at take a look at take a look at yeah this diagram. This is the diagram yeah this diagram. This is the diagram yeah this diagram. This is the diagram which is essentially used for predicting which is essentially used for predicting which is essentially used for predicting the noise in the case of diffusion the noise in the case of diffusion the noise in the case of diffusion models. And I hope all of you have seen models. And I hope all of you have seen models. And I hope all of you have seen the differences between the diffusion the differences between the diffusion the differences between the diffusion architecture and the uh ARM or the reg architecture and the uh ARM or the reg architecture and the uh ARM or the reg auto reggressive architecture. Time auto reggressive architecture. Time auto reggressive architecture. Time embedding is the first difference. Then embedding is the first difference. Then embedding is the first difference. Then in multi attention the causality is the in multi attention the causality is the in multi attention the causality is the second difference. Then the cross second difference. Then the cross second difference. Then the cross entropy only for the mask tokens that is entropy only for the mask tokens that is entropy only for the mask tokens that is the third difference. Noisy input here
- L14 @ 1462s
the third difference. Noisy input here the third difference. Noisy input here that's the four difference. There are that's the four difference. There are that's the four difference. There are four major differences. Everything else four major differences. Everything else four major differences. Everything else essentially remains the same. That's why essentially remains the same. That's why essentially remains the same. That's why I mentioned earlier that understanding I mentioned earlier that understanding I mentioned earlier that understanding the auto reggressive model architecture the auto reggressive model architecture the auto reggressive model architecture is so crucial for understanding the is so crucial for understanding the is so crucial for understanding the language diffusion model architecture. language diffusion model architecture. language diffusion model architecture. Okay. Now when I'm going to share this Okay. Now when I'm going to share this Okay. Now when I'm going to share this resources with you, there is also a resources with you, there is also a resources with you, there is also a website in which I have explained all of website in which I have explained all of website in which I have explained all of these calculations from scratch. So if these calculations from scratch. So if these calculations from scratch. So if you scroll down below, I have explained you scroll down below, I have explained you scroll down below, I have explained there is a section called vector flow. there is a section called vector flow. there is a section called vector flow. So what we do over here is that we take So what we do over here is that we take So what we do over here is that we take a toy model configuration where we a toy model configuration where we a toy model configuration where we assume vocabulary size is equal to six assume vocabulary size is equal to six assume vocabulary size is equal to six sequence length is equal to 5. Uh we sequence length is equal to 5. Uh we sequence length is equal to 5. Uh we this is a we construct a token embedding this is a we construct a token embedding this is a we construct a token embedding matrix. We construct a position matrix. We construct a position matrix. We construct a position embedding matrix and we construct a time embedding matrix and we construct a time
- L14 @ 1509s
embedding matrix and we construct a time embedding matrix also. And I show you embedding matrix also. And I show you embedding matrix also. And I show you how everything is done. So at each how everything is done. So at each how everything is done. So at each position we add the token embedding, position we add the token embedding, position we add the token embedding, position embedding and the time position embedding and the time position embedding and the time embedding. Then we do layer embedding. Then we do layer embedding. Then we do layer normalization vector wise. So if you normalization vector wise. So if you normalization vector wise. So if you want to go into the mathematical details want to go into the mathematical details want to go into the mathematical details of how the dimensions are retained, I'll of how the dimensions are retained, I'll of how the dimensions are retained, I'll share this link with you where even I've share this link with you where even I've share this link with you where even I've shown everything from the attention shown everything from the attention shown everything from the attention mechanism to the shortcut connections mechanism to the shortcut connections mechanism to the shortcut connections which are also called residuals to the which are also called residuals to the which are also called residuals to the second layer normalization feed forward second layer normalization feed forward second layer normalization feed forward neural network final layer normalization neural network final layer normalization neural network final layer normalization final logits matrix and the final cross final logits matrix and the final cross final logits matrix and the final cross entropy loss. I have already shown this entropy loss. I have already shown this entropy loss. I have already shown this to you on the whiteboard but just for to you on the whiteboard but just for to you on the whiteboard but just for the sake of simplicity I've added this the sake of simplicity I've added this the sake of simplicity I've added this hole over here. So the entire forward hole over here. So the entire forward hole over here. So the entire forward pass pipeline for diffusion language pass pipeline for diffusion language pass pipeline for diffusion language model can be summarized here. We have
- L14 @ 1554s
model can be summarized here. We have model can be summarized here. We have token ids convert them into token token ids convert them into token token ids convert them into token embedding vectors add position embedding embedding vectors add position embedding embedding vectors add position embedding vectors add the time embedding vectors vectors add the time embedding vectors vectors add the time embedding vectors since you have to decide a time step at since you have to decide a time step at since you have to decide a time step at the start of the forward pass. Then we the start of the forward pass. Then we the start of the forward pass. Then we go into the transformer architecture. We go into the transformer architecture. We go into the transformer architecture. We pass through the first layer norm, pass through the first layer norm, pass through the first layer norm, attention mechanism, shortcut attention mechanism, shortcut attention mechanism, shortcut connection, second layer norm, feed connection, second layer norm, feed connection, second layer norm, feed forward neural network, another shortcut forward neural network, another shortcut forward neural network, another shortcut connection. Final then we come to the connection. Final then we come to the connection. Final then we come to the output header. We have the final layer output header. We have the final layer output header. We have the final layer normalization. We have the logits matrix normalization. We have the logits matrix normalization. We have the logits matrix and we have the cross entropy loss on and we have the cross entropy loss on and we have the cross entropy loss on the mask positions. the mask positions. the mask positions. All right, cool. And now we are ready to All right, cool. And now we are ready to All right, cool. And now we are ready to move to the final step which is move to the final step which is move to the final step which is essentially the dnoising process. So essentially the dnoising process. So essentially the dnoising process. So let's get to that right now.
- L15 @ 3s
The last step which we need to do is the The last step which we need to do is the actual step of dnoising. actual step of dnoising. actual step of dnoising. What is dnoising? Well, dnoising is What is dnoising? Well, dnoising is What is dnoising? Well, dnoising is essentially starting from noise. essentially starting from noise. essentially starting from noise. Starting from complete noise like in the Starting from complete noise like in the Starting from complete noise like in the case of images, we start with noisy case of images, we start with noisy case of images, we start with noisy images and slowly our aim is to get to images and slowly our aim is to get to images and slowly our aim is to get to that probability distribution where the that probability distribution where the that probability distribution where the true image actually lives. If the true image actually lives. If the true image actually lives. If the training has been done correctly, then training has been done correctly, then training has been done correctly, then dn noising should actually help us to dn noising should actually help us to dn noising should actually help us to recover images in that exact or in the recover images in that exact or in the recover images in that exact or in the true probability distribution space or true probability distribution space or true probability distribution space or as close as possible to the true as close as possible to the true as close as possible to the true probability distribution space. What probability distribution space. What probability distribution space. What does it mean if training has been done does it mean if training has been done does it mean if training has been done correctly? Well, what it means is that correctly? Well, what it means is that correctly? Well, what it means is that if at each step of the training process, if at each step of the training process, if at each step of the training process, we are able to predict the noise which we are able to predict the noise which we are able to predict the noise which has been added. So at each step, has been added. So at each step, has been added. So at each step, yeah, at each step of the training
- L15 @ 51s
yeah, at each step of the training yeah, at each step of the training process, if we are able to predict how process, if we are able to predict how process, if we are able to predict how much noise has been added in that step, much noise has been added in that step, much noise has been added in that step, essentially when we reverse when we essentially when we reverse when we essentially when we reverse when we reverse it or when we start from noise, reverse it or when we start from noise, reverse it or when we start from noise, we will be able to get back to the true we will be able to get back to the true we will be able to get back to the true image. That's the whole idea. So if we image. That's the whole idea. So if we image. That's the whole idea. So if we at each step if we are able to predict at each step if we are able to predict at each step if we are able to predict how much noise is added to make a noisy how much noise is added to make a noisy how much noise is added to make a noisy image if we start from the noisy image image if we start from the noisy image image if we start from the noisy image we will be able to go back in the we will be able to go back in the we will be able to go back in the reverse direction. reverse direction. reverse direction. So in the case of images it's fine right So in the case of images it's fine right So in the case of images it's fine right because in the case of images usually because in the case of images usually because in the case of images usually the dnoising might look something like the dnoising might look something like the dnoising might look something like this. Yeah like this we start from a this. Yeah like this we start from a this. Yeah like this we start from a completely noisy image and then we completely noisy image and then we completely noisy image and then we recover the clean image. So we start recover the clean image. So we start
- L15 @ 95s
recover the clean image. So we start from a complete noisy image and then we from a complete noisy image and then we from a complete noisy image and then we recover Chinese characters as the recover Chinese characters as the recover Chinese characters as the dnoising proceeds. Right? But in the dnoising proceeds. Right? But in the dnoising proceeds. Right? But in the case of language, how does it happen? case of language, how does it happen? case of language, how does it happen? What does it mean by completely noisy What does it mean by completely noisy What does it mean by completely noisy image? We have already seen that a image? We have already seen that a image? We have already seen that a completely noisy or what does it mean by completely noisy or what does it mean by completely noisy or what does it mean by completely noisy text? It means that we completely noisy text? It means that we completely noisy text? It means that we start with a text in which everything is start with a text in which everything is start with a text in which everything is fully masked. So let's say this is the fully masked. So let's say this is the fully masked. So let's say this is the text in which everything is fully text in which everything is fully text in which everything is fully masked. We have a beginning of sequence masked. We have a beginning of sequence masked. We have a beginning of sequence and an end of sequence. Let's ignore and an end of sequence. Let's ignore and an end of sequence. Let's ignore that for a moment. But we have three that for a moment. But we have three that for a moment. But we have three masks over here. That's how the dnoising masks over here. That's how the dnoising masks over here. That's how the dnoising process actually starts. What we have to process actually starts. What we have to process actually starts. What we have to do is that now the model is trained. The do is that now the model is trained. The do is that now the model is trained. The parameters of the model will not change parameters of the model will not change
- L15 @ 135s
parameters of the model will not change in the dnoising process. The model is in the dnoising process. The model is in the dnoising process. The model is fully trained. Um so all these fully trained. Um so all these fully trained. Um so all these parameters which we have seen over here, parameters which we have seen over here, parameters which we have seen over here, those parameters will not change. Okay. those parameters will not change. Okay. those parameters will not change. Okay. What happens in the dnoising is that we What happens in the dnoising is that we What happens in the dnoising is that we have to slowly uncover these masks and have to slowly uncover these masks and have to slowly uncover these masks and predict what was there here in the first predict what was there here in the first predict what was there here in the first place. Okay. place. Okay. place. Okay. So let's say we want to d noiseise from So let's say we want to d noiseise from So let's say we want to d noiseise from a fully noisy text which is this. The a fully noisy text which is this. The a fully noisy text which is this. The way the dnoising works in the case of way the dnoising works in the case of way the dnoising works in the case of language diffusion is through four language diffusion is through four language diffusion is through four steps. First, if we have in the forward steps. First, if we have in the forward steps. First, if we have in the forward pass, if we have time steps 1 2 3 4 or pass, if we have time steps 1 2 3 4 or pass, if we have time steps 1 2 3 4 or in this case, we had six time steps, in this case, we had six time steps, in this case, we had six time steps, right? In the forward pass, we had six right? In the forward pass, we had six right? In the forward pass, we had six time steps over here. If we had four time steps over here. If we had four time steps over here. If we had four time steps, for example, in the D time steps, for example, in the D
- L15 @ 182s
time steps, for example, in the D noising process, we have to go backwards noising process, we have to go backwards noising process, we have to go backwards from four. So, we have to go 4 3 2 1, from four. So, we have to go 4 3 2 1, from four. So, we have to go 4 3 2 1, right? So, we have to start with four right? So, we have to start with four right? So, we have to start with four and then we have to sample the tokens and then we have to sample the tokens and then we have to sample the tokens for each of these masks. So we have to for each of these masks. So we have to for each of these masks. So we have to pass this sequence through the trained pass this sequence through the trained pass this sequence through the trained model and then see what the token model and then see what the token model and then see what the token prediction is at each of these masks prediction is at each of these masks prediction is at each of these masks at each of these positions. Then what we at each of these positions. Then what we at each of these positions. Then what we have to see is that which position is have to see is that which position is have to see is that which position is predicted most confidently predicted most confidently predicted most confidently and we have to uncover these masks and we have to uncover these masks and we have to uncover these masks sequentially. Right? So in the first sequentially. Right? So in the first sequentially. Right? So in the first iteration we only uncover one mask and iteration we only uncover one mask and iteration we only uncover one mask and keep all the other masks. keep all the other masks. keep all the other masks. Okay. Okay. Okay. Which mask is uncovered? the mass for Which mask is uncovered? the mass for Which mask is uncovered? the mass for which the prediction has the highest which the prediction has the highest which the prediction has the highest confidence of course. So let me let confidence of course. So let me let confidence of course. So let me let let's take an example. Okay. So if this
- L15 @ 229s
let's take an example. Okay. So if this let's take an example. Okay. So if this is the input sequence we start at time is the input sequence we start at time is the input sequence we start at time equal to 4. So we start in the reverse equal to 4. So we start in the reverse equal to 4. So we start in the reverse manner. We pass this input sequence manner. We pass this input sequence manner. We pass this input sequence through my entire architecture which is through my entire architecture which is through my entire architecture which is now trained. When I say my entire now trained. When I say my entire now trained. When I say my entire architecture, this input sequence passed architecture, this input sequence passed architecture, this input sequence passed through is passed through this whole through is passed through this whole through is passed through this whole architecture. architecture. architecture. So let me bring it over here. This input So let me bring it over here. This input So let me bring it over here. This input sequence is passed sequence is passed sequence is passed um this input sequence is passed through um this input sequence is passed through um this input sequence is passed through this whole architecture. this whole architecture. this whole architecture. And if I clean this up a bit be easier And if I clean this up a bit be easier And if I clean this up a bit be easier for you. Yeah, this input sequence is for you. Yeah, this input sequence is for you. Yeah, this input sequence is passed through this whole architecture. passed through this whole architecture. passed through this whole architecture. And here we don't compute the loss And here we don't compute the loss And here we don't compute the loss because the training has already been because the training has already been because the training has already been done, right? But what we do compute here done, right? But what we do compute here done, right? But what we do compute here is what's the predicted uh logits is what's the predicted uh logits is what's the predicted uh logits matrix? matrix? matrix? What is the predicted logits matrix? What is the predicted logits matrix?
- L15 @ 282s
What is the predicted logits matrix? Um so let's say we have position number Um so let's say we have position number Um so let's say we have position number we have three main positions which we have three main positions which we have three main positions which matter right because the beginning of matter right because the beginning of matter right because the beginning of sequence and end of sequence does not sequence and end of sequence does not sequence and end of sequence does not matter. We have position one position 2 matter. We have position one position 2 matter. We have position one position 2 and position three. So let's say the and position three. So let's say the and position three. So let's say the logits matrix is something like this. logits matrix is something like this. logits matrix is something like this. The logit's matrix for position one is The logit's matrix for position one is The logit's matrix for position one is 122 06.22 06.22 and 336. 0 0 let's say and point 2 here. Uh for 0 0 let's say and point 2 here. Uh for position three it is 0.1 let's say same position three it is 0.1 let's say same position three it is 0.1 let's say same 0 0. Now the way it works is that um 0 0. Now the way it works is that um 0 0. Now the way it works is that um let's see now for position number one this is now for position number one this is position number one we see the maximum position number one we see the maximum position number one we see the maximum confidence is here and uh our dictionary confidence is here and uh our dictionary confidence is here and uh our dictionary actually is a actually is a actually is a uh our dictionary is uh our dictionary is uh our dictionary is let's see what our dictionary is I think let's see what our dictionary is I think let's see what our dictionary is I think the dictionary is beginning of sequence the dictionary is beginning of sequence the dictionary is beginning of sequence end of sequence sequence mask end of sequence sequence mask end of sequence sequence mask then we have a b and c right so let's
- L15 @ 358s
then we have a b and c right so let's then we have a b and c right so let's actually actually actually take some different values of my let me take some different values of my let me take some different values of my let me refresh this what I want to do here is that I want to what I want to do here is that I want to take uh different um I want to take different values of my um I want to take different values of my now it's working so let me rub this a now it's working so let me rub this a now it's working so let me rub this a bit bit bit okay okay okay and let's say the values here are such and let's say the values here are such and let's say the values here are such that it's 1 0 08 that it's 1 0 08 that it's 1 0 08 and.1 and.1 and.1 and this is also 0.100 08 and.1 okay so and this is also 0.100 08 and.1 okay so and this is also 0.100 08 and.1 okay so for this position for position number for this position for position number for this position for position number one the maximum confidence is for token one the maximum confidence is for token one the maximum confidence is for token C. So we'll predict C with this C. So we'll predict C with this C. So we'll predict C with this confidence for position number. So let's confidence for position number. So let's confidence for position number. So let's say this is.1 and8 and this is 1 and8. say this is.1 and8 and this is 1 and8. say this is.1 and8 and this is 1 and8. So for position two also it's C and for So for position two also it's C and for So for position two also it's C and for position three also it's C. position three also it's C. position three also it's C. Um so for position and actually the Um so for position and actually the Um so for position and actually the confidence let's say it's 336 here. So confidence let's say it's 336 here. So confidence let's say it's 336 here. So let's let's let's change this again. change this again.
- L15 @ 436s
change this again. Sorry for that. So let's say this 336 Sorry for that. So let's say this 336 Sorry for that. So let's say this 336 and this 336. Let's say everything is and this 336. Let's say everything is and this 336. Let's say everything is the same for all. So I'll just uh the same for all. So I'll just uh the same for all. So I'll just uh take this and copy paste for all. take this and copy paste for all. take this and copy paste for all. Yeah. Yeah. So then I look at the token with Yeah. So then I look at the token with the maximum confidence and that's C the maximum confidence and that's C the maximum confidence and that's C which is the last entry year for all my which is the last entry year for all my which is the last entry year for all my positions. Right. So after sampling my positions. Right. So after sampling my positions. Right. So after sampling my prediction should be CCC, right? But prediction should be CCC, right? But prediction should be CCC, right? But since we have just started the since we have just started the since we have just started the unmasking, unmasking, unmasking, we must end step three. So this first we must end step three. So this first we must end step three. So this first step we should end with three masks. step we should end with three masks. step we should end with three masks. That's the rule. So when we start unmask That's the rule. So when we start unmask That's the rule. So when we start unmask unmasking, first we'll end this step unmasking, first we'll end this step unmasking, first we'll end this step with three masks. Then we'll have two with three masks. Then we'll have two with three masks. Then we'll have two masks. Then we'll have one mask and then masks. Then we'll have one mask and then masks. Then we'll have one mask and then we'll have no mask. That's the idea. So we'll have no mask. That's the idea. So we'll have no mask. That's the idea. So we need three masks here. So again we'll we need three masks here. So again we'll
- L15 @ 487s
we need three masks here. So again we'll mask all these three. So these three mask all these three. So these three mask all these three. So these three stay in masks. That's the output from stay in masks. That's the output from stay in masks. That's the output from the first dinoising step. Then we move the first dinoising step. Then we move the first dinoising step. Then we move to the second dinoising step which is to the second dinoising step which is to the second dinoising step which is time equal to three. Again let's say time equal to three. Again let's say time equal to three. Again let's say position number one from the logits position number one from the logits position number one from the logits matrix we have that it samples A with matrix we have that it samples A with matrix we have that it samples A with maximum confidence. Position two we have maximum confidence. Position two we have maximum confidence. Position two we have sample C with maximum confidence and sample C with maximum confidence and sample C with maximum confidence and position three we have sample C with position three we have sample C with position three we have sample C with maximum confidence. So after sampling we maximum confidence. So after sampling we maximum confidence. So after sampling we should get beginning of sequence A C and should get beginning of sequence A C and should get beginning of sequence A C and end of sequence. Now after the second D end of sequence. Now after the second D end of sequence. Now after the second D noising step we need to keep two masks. noising step we need to keep two masks. noising step we need to keep two masks. So the whole idea is that so I'll tell So the whole idea is that so I'll tell So the whole idea is that so I'll tell you here step step number four step you here step step number four step you here step step number four step three step two and step one at the end three step two and step one at the end three step two and step one at the end of this step we have to keep three masks of this step we have to keep three masks
- L15 @ 532s
of this step we have to keep three masks at the end of this step we have to keep at the end of this step we have to keep at the end of this step we have to keep two masks at the end of this step which two masks at the end of this step which two masks at the end of this step which we have to keep one mask and at the end we have to keep one mask and at the end we have to keep one mask and at the end of the final step zero mask so that's my of the final step zero mask so that's my of the final step zero mask so that's my final thing which will appear on the final thing which will appear on the final thing which will appear on the screen this is exactly what's happening screen this is exactly what's happening screen this is exactly what's happening here we start with entire all masks and here we start with entire all masks and here we start with entire all masks and then we remove the masks one after one then we remove the masks one after one then we remove the masks one after one during the noising process I'm just during the noising process I'm just during the noising process I'm just showing you how what's the logic to be showing you how what's the logic to be showing you how what's the logic to be followed for demasking. followed for demasking. followed for demasking. So now here we have step three right and So now here we have step three right and So now here we have step three right and I have a c and c but I have to keep two I have a c and c but I have to keep two I have a c and c but I have to keep two masks. So which are the two masks I will masks. So which are the two masks I will masks. So which are the two masks I will keep I will keep the two masks with the keep I will keep the two masks with the keep I will keep the two masks with the highest highest highest um um um or I will keep so which what what will I
- L15 @ 571s
or I will keep so which what what will I or I will keep so which what what will I unmask? So I have to keep two masks here unmask? So I have to keep two masks here unmask? So I have to keep two masks here right? So I have to unmask one thing. right? So I have to unmask one thing. right? So I have to unmask one thing. What's the thing I'll unmask? I'll What's the thing I'll unmask? I'll What's the thing I'll unmask? I'll unmask the thing with the highest unmask the thing with the highest unmask the thing with the highest confidence. Right? So either position confidence. Right? So either position confidence. Right? So either position two or three because position one has two or three because position one has two or three because position one has slightly lower confidence. So by default slightly lower confidence. So by default slightly lower confidence. So by default let's say unmask this position which is let's say unmask this position which is let's say unmask this position which is position number two. I could have position number two. I could have position number two. I could have unmasked position three also but let's unmasked position three also but let's unmasked position three also but let's say I unmask position two here. So then say I unmask position two here. So then say I unmask position two here. So then the input at the next step is Bos mask C the input at the next step is Bos mask C the input at the next step is Bos mask C mask and EOS. I pass it through the mask and EOS. I pass it through the mask and EOS. I pass it through the entire architecture. I get the logits entire architecture. I get the logits entire architecture. I get the logits and let's say again I sample A C and A. and let's say again I sample A C and A. and let's say again I sample A C and A. At this position we have a probab At this position we have a probab At this position we have a probab confidence of 3175. At this we have a confidence of 3175. At this we have a
- L15 @ 609s
confidence of 3175. At this we have a confidence of 336 and at this we have a confidence of 336 and at this we have a confidence of 336 and at this we have a confidence of 3175. Now here the rule is confidence of 3175. Now here the rule is confidence of 3175. Now here the rule is that we have to unmask we have to keep that we have to unmask we have to keep that we have to unmask we have to keep one mask. So we have to unmask two which one mask. So we have to unmask two which one mask. So we have to unmask two which are the two I will unmask. Again I'll are the two I will unmask. Again I'll are the two I will unmask. Again I'll unmask those ones with the maximum unmask those ones with the maximum unmask those ones with the maximum confidence which are these two. confidence which are these two. confidence which are these two. Uh I could have unmasked this also but Uh I could have unmasked this also but Uh I could have unmasked this also but by default let's say I unmasked this. So by default let's say I unmasked this. So by default let's say I unmasked this. So I unmask position one and I unmask I unmask position one and I unmask I unmask position one and I unmask position two. position two. position two. Then at the last step I have this input Then at the last step I have this input Then at the last step I have this input sequence B ac and only one mask and then sequence B ac and only one mask and then sequence B ac and only one mask and then I pass it through the input I pass it through the input I pass it through the input architecture. I get that the after architecture. I get that the after architecture. I get that the after sampling we have BOS ACC and EOS. sampling we have BOS ACC and EOS. sampling we have BOS ACC and EOS. Um and this is the logits matrix right. Um and this is the logits matrix right. Um and this is the logits matrix right. So at first position I sample A with So at first position I sample A with
- L15 @ 654s
So at first position I sample A with confidence of 3175. At second position I confidence of 3175. At second position I confidence of 3175. At second position I sample C with confidence 336. At third sample C with confidence 336. At third sample C with confidence 336. At third position I sample C with confidence.336. So here I have to not keep any mask. So So here I have to not keep any mask. So I have to unmask everything. So the I have to unmask everything. So the I have to unmask everything. So the answer will now this everything will be answer will now this everything will be answer will now this everything will be unmasked. So the answer final answer unmasked. So the answer final answer unmasked. So the answer final answer final generated sequence is beginning of final generated sequence is beginning of final generated sequence is beginning of sequence A C C and end of sequence. sequence A C C and end of sequence. sequence A C C and end of sequence. That's the sequence which is generated That's the sequence which is generated That's the sequence which is generated by the diffusion language model. Now by the diffusion language model. Now by the diffusion language model. Now that's the end of it. We did the that's the end of it. We did the that's the end of it. We did the dnoising and that's the generated dnoising and that's the generated dnoising and that's the generated sequence produced by the dnoising sequence produced by the dnoising sequence produced by the dnoising process. This is what it looks like in process. This is what it looks like in process. This is what it looks like in action. So here you see we start with action. So here you see we start with action. So here you see we start with all the masks. So if you give a prompt all the masks. So if you give a prompt all the masks. So if you give a prompt the prompt along with these masks along the prompt along with these masks along the prompt along with these masks along with all the masks are passed to the with all the masks are passed to the with all the masks are passed to the input architecture and wherever there is
- L15 @ 705s
input architecture and wherever there is input architecture and wherever there is mask that slowly unmasked right. So here mask that slowly unmasked right. So here mask that slowly unmasked right. So here we have 128 steps. So right now I just we have 128 steps. So right now I just we have 128 steps. So right now I just showed you four steps for 128 steps we showed you four steps for 128 steps we showed you four steps for 128 steps we slowly unmask everything one after slowly unmask everything one after slowly unmask everything one after another. Right? So just see how fast the another. Right? So just see how fast the another. Right? So just see how fast the inference happens here compared to the inference happens here compared to the inference happens here compared to the next token prediction. Right. So let's next token prediction. Right. So let's next token prediction. Right. So let's play this GIF if it opens. So see everything is masked right now. So see everything is masked right now. Right? Nothing is unmasked. We have 128 Right? Nothing is unmasked. We have 128 Right? Nothing is unmasked. We have 128 masks here. Now things are slowly masks here. Now things are slowly masks here. Now things are slowly getting unmasked here one after the getting unmasked here one after the getting unmasked here one after the other. Right? For some reason there is a other. Right? For some reason there is a other. Right? For some reason there is a weird black white contrast over here. So weird black white contrast over here. So weird black white contrast over here. So I can just bring this GIF over here and I can just bring this GIF over here and I can just bring this GIF over here and then just uh refresh it maybe. Yeah. See then just uh refresh it maybe. Yeah. See then just uh refresh it maybe. Yeah. See the masks are being removed one after the masks are being removed one after the masks are being removed one after the other. And uh it's not auto the other. And uh it's not auto the other. And uh it's not auto reggressive which means it's not one reggressive which means it's not one
- L15 @ 752s
reggressive which means it's not one token at a time. Tokens masks can be token at a time. Tokens masks can be token at a time. Tokens masks can be unmasked here, masks can be unmasked at unmasked here, masks can be unmasked at unmasked here, masks can be unmasked at the top also. And everything happens the top also. And everything happens the top also. And everything happens very fast. That's the advantage of very fast. That's the advantage of very fast. That's the advantage of diffusion models. Right? This is the diffusion models. Right? This is the diffusion models. Right? This is the generation process or this is the generation process or this is the generation process or this is the dnoising process. This dinoising process dnoising process. This dinoising process dnoising process. This dinoising process is again if you have done the noising is again if you have done the noising is again if you have done the noising process correctly which means if you process correctly which means if you process correctly which means if you have obtained a very low loss in the have obtained a very low loss in the have obtained a very low loss in the noising process dn noising is kind of noising process dn noising is kind of noising process dn noising is kind of guaranteed to work even in diffusion guaranteed to work even in diffusion guaranteed to work even in diffusion language models. So even theoretically language models. So even theoretically language models. So even theoretically this is a safe method but it just as this is a safe method but it just as this is a safe method but it just as we'll see it takes a bit of time to we'll see it takes a bit of time to we'll see it takes a bit of time to train compared to train compared to train compared to auto reggressive language models. Okay, auto reggressive language models. Okay, auto reggressive language models. Okay, I hope all of you have understood the I hope all of you have understood the I hope all of you have understood the dnoising process, right? In dnoising, dnoising process, right? In dnoising, dnoising process, right? In dnoising, what happens is that let's say you fully what happens is that let's say you fully
- L15 @ 797s
what happens is that let's say you fully trained the model as we have seen trained the model as we have seen trained the model as we have seen before. Then you start with a sequence before. Then you start with a sequence before. Then you start with a sequence of masks. of masks. of masks. Let's say if you want to predict a Let's say if you want to predict a Let's say if you want to predict a certain number of tokens, right? certain number of tokens, right? certain number of tokens, right? Uh you want to predict a certain number Uh you want to predict a certain number Uh you want to predict a certain number of tokens. All of them will be masked of tokens. All of them will be masked of tokens. All of them will be masked initially. initially. initially. Let's say you want to predict 50 tokens. Let's say you want to predict 50 tokens. Let's say you want to predict 50 tokens. All of them will be masked. You pass All of them will be masked. You pass All of them will be masked. You pass those 50 tokens into the architecture those 50 tokens into the architecture those 50 tokens into the architecture and you slowly go on unmasking the and you slowly go on unmasking the and you slowly go on unmasking the tokens one after the other. tokens one after the other. tokens one after the other. Uh that's the whole idea of unmasking. Uh that's the whole idea of unmasking. Uh that's the whole idea of unmasking. It's a very simple process but there is It's a very simple process but there is It's a very simple process but there is just one rule that uh you unmask those just one rule that uh you unmask those just one rule that uh you unmask those tokens which have the maximum confidence tokens which have the maximum confidence tokens which have the maximum confidence at every unmasking step. You unmask only at every unmasking step. You unmask only at every unmasking step. You unmask only those tokens which have maximum those tokens which have maximum those tokens which have maximum confidence and slowly you keep on confidence and slowly you keep on confidence and slowly you keep on unmasking one after the other. So this
- L15 @ 846s
unmasking one after the other. So this unmasking one after the other. So this video or this GIF which Andre Karpati video or this GIF which Andre Karpati video or this GIF which Andre Karpati shared. Now if you look at the right shared. Now if you look at the right shared. Now if you look at the right hand side the diffusion you should have hand side the diffusion you should have hand side the diffusion you should have a much better understanding. a much better understanding. a much better understanding. We start with masks and what is shown on We start with masks and what is shown on We start with masks and what is shown on the right hand side is just unmasking the right hand side is just unmasking the right hand side is just unmasking one step at a time. Whereas on the left hand side this is Whereas on the left hand side this is auto reggressive. It's one token at a auto reggressive. It's one token at a auto reggressive. It's one token at a time. Whereas on the right hand side time. Whereas on the right hand side time. Whereas on the right hand side it's masked get it's masks getting it's masked get it's masks getting it's masked get it's masks getting unmasked one after the other. unmasked one after the other. unmasked one after the other. Okay. Now we'll see a summary of the Okay. Now we'll see a summary of the Okay. Now we'll see a summary of the three characteristics of the diffusion three characteristics of the diffusion three characteristics of the diffusion language models.
- L16 @ 2s
Before we go to building the small Before we go to building the small language diffusion model from scratch language diffusion model from scratch language diffusion model from scratch and we are going to do that. Uh here is and we are going to do that. Uh here is and we are going to do that. Uh here is a file on which we are going to build a file on which we are going to build a file on which we are going to build the entire diffusion language model the entire diffusion language model the entire diffusion language model fully from scratch. Um following the fully from scratch. Um following the fully from scratch. Um following the same things which we have seen it seen same things which we have seen it seen same things which we have seen it seen on the whiteboard. on the whiteboard. on the whiteboard. Um I have created this website so that I Um I have created this website so that I Um I have created this website so that I can provide all of you a quick summary can provide all of you a quick summary can provide all of you a quick summary of what all we have learned so far. So of what all we have learned so far. So of what all we have learned so far. So we started out with this goal of we started out with this goal of we started out with this goal of building a small language diffusion building a small language diffusion building a small language diffusion model from scratch, right? And the idea model from scratch, right? And the idea model from scratch, right? And the idea was that well language models what they was that well language models what they was that well language models what they are essentially doing one way to view it are essentially doing one way to view it are essentially doing one way to view it probabilistically is that we have this probabilistically is that we have this probabilistically is that we have this uh underlying space let's say this uh underlying space let's say this uh underlying space let's say this orange probability distribution where orange probability distribution where orange probability distribution where the the the uh cluster of meaningful sentences lie uh cluster of meaningful sentences lie
- L16 @ 48s
uh cluster of meaningful sentences lie what language models are doing is that what language models are doing is that what language models are doing is that we are trying to find this probability we are trying to find this probability we are trying to find this probability distribution which is shown by this red distribution which is shown by this red distribution which is shown by this red dotted line. One way to do that is dotted line. One way to do that is dotted line. One way to do that is through arms which is auto reggressive through arms which is auto reggressive through arms which is auto reggressive models. But why do we have to stay models. But why do we have to stay models. But why do we have to stay limited to ARMs? Let's try diffusion. If limited to ARMs? Let's try diffusion. If limited to ARMs? Let's try diffusion. If we have to try diffusion, we need to we have to try diffusion, we need to we have to try diffusion, we need to follow three things. We need to have a follow three things. We need to have a follow three things. We need to have a noising process. We need to have a model noising process. We need to have a model noising process. We need to have a model which predicts the noise. And we need a which predicts the noise. And we need a which predicts the noise. And we need a dinoising process. What's the noising dinoising process. What's the noising dinoising process. What's the noising noising process which we are going to noising process which we are going to noising process which we are going to use? Well, the simple idea is that you use? Well, the simple idea is that you use? Well, the simple idea is that you just take an input sequence, you add just take an input sequence, you add just take an input sequence, you add masks to it sequentially as you go masks to it sequentially as you go masks to it sequentially as you go deeper into the noising schedule and you deeper into the noising schedule and you deeper into the noising schedule and you noise the input sequence. You corrupt noise the input sequence. You corrupt
- L16 @ 86s
noise the input sequence. You corrupt the input sequence. You pass this the input sequence. You pass this the input sequence. You pass this corrupted input sequence uh you pass corrupted input sequence uh you pass corrupted input sequence uh you pass this corrupted input sequence into this this corrupted input sequence into this this corrupted input sequence into this uh diffusion language model uh diffusion language model uh diffusion language model architecture. This whole architecture architecture. This whole architecture architecture. This whole architecture serves as the model which is a second serves as the model which is a second serves as the model which is a second building block for predicting the noise. building block for predicting the noise. building block for predicting the noise. That is this whole diffusion language That is this whole diffusion language That is this whole diffusion language model architecture which uh uh which I'm model architecture which uh uh which I'm model architecture which uh uh which I'm showing you right now. So the way we showing you right now. So the way we showing you right now. So the way we predict the noise in diffusion language predict the noise in diffusion language predict the noise in diffusion language models is that we pass this noisy input models is that we pass this noisy input models is that we pass this noisy input through the input block, the processor through the input block, the processor through the input block, the processor block and the output. we get the loss block and the output. we get the loss block and the output. we get the loss only for the mask tokens. only for the mask tokens. only for the mask tokens. There are some key differences between There are some key differences between There are some key differences between the um diffusion language model the um diffusion language model the um diffusion language model architecture and the ARM architecture architecture and the ARM architecture architecture and the ARM architecture which are highlighted in this two which are highlighted in this two which are highlighted in this two schematics if you see them side by side. schematics if you see them side by side. schematics if you see them side by side. One key difference is the time embedding One key difference is the time embedding
- L16 @ 134s
One key difference is the time embedding which is not there in ARM but it's there which is not there in ARM but it's there which is not there in ARM but it's there in diffusion language model. Second is in diffusion language model. Second is in diffusion language model. Second is that in multi attention we don't have that in multi attention we don't have that in multi attention we don't have causality causality causality um in the attention scores matrix. In um in the attention scores matrix. In um in the attention scores matrix. In fact, every token can look at the tokens fact, every token can look at the tokens fact, every token can look at the tokens before it and also after it. Right? So, before it and also after it. Right? So, before it and also after it. Right? So, we don't have this. These things are we don't have this. These things are we don't have this. These things are don't need to be set to zero. Whereas in don't need to be set to zero. Whereas in don't need to be set to zero. Whereas in auto reggressive models or traditional auto reggressive models or traditional auto reggressive models or traditional language models, everything above the language models, everything above the language models, everything above the diagonal, all the attention scores above diagonal, all the attention scores above diagonal, all the attention scores above the diagonal set to zero. That's not the diagonal set to zero. That's not the diagonal set to zero. That's not needed in the case of diffusion language needed in the case of diffusion language needed in the case of diffusion language models. models. models. Third thing is that we take the cross Third thing is that we take the cross Third thing is that we take the cross entropy loss only for the mask tokens. entropy loss only for the mask tokens. entropy loss only for the mask tokens. Whereas in the case of traditional Whereas in the case of traditional Whereas in the case of traditional language models, we do it for all the language models, we do it for all the language models, we do it for all the tokens in the input sequence. tokens in the input sequence. tokens in the input sequence. Fourth thing of course is that we have
- L16 @ 178s
Fourth thing of course is that we have Fourth thing of course is that we have noisy input with mass tokens. Whereas in noisy input with mass tokens. Whereas in noisy input with mass tokens. Whereas in the case of traditional language models, the case of traditional language models, the case of traditional language models, we have noisy input and then we try to we have noisy input and then we try to we have noisy input and then we try to minimize this loss as much as possible. minimize this loss as much as possible. minimize this loss as much as possible. The more this loss is minimized, the The more this loss is minimized, the The more this loss is minimized, the more we are able to predict the noise. more we are able to predict the noise. more we are able to predict the noise. And theoretically, the more this loss is And theoretically, the more this loss is And theoretically, the more this loss is minimized, the better we can recover the minimized, the better we can recover the minimized, the better we can recover the underlying data distribution in the underlying data distribution in the underlying data distribution in the denoising or the reverse diffusion denoising or the reverse diffusion denoising or the reverse diffusion process. process. process. The third step is the dnoising process The third step is the dnoising process The third step is the dnoising process or the reverse diffusion process. What or the reverse diffusion process. What or the reverse diffusion process. What essentially happens in this step is that essentially happens in this step is that essentially happens in this step is that in the dnoising process in the dnoising process in the dnoising process uh we start with masks and then we uh we start with masks and then we uh we start with masks and then we slowly unmask it one at a time. So one slowly unmask it one at a time. So one slowly unmask it one at a time. So one mask is unmasked in each iteration and mask is unmasked in each iteration and mask is unmasked in each iteration and then we go backwards in time. What is then we go backwards in time. What is then we go backwards in time. What is unmasked? The most confident tokens are
- L16 @ 225s
unmasked? The most confident tokens are unmasked? The most confident tokens are essentially unmasked. So we unmask the k essentially unmasked. So we unmask the k essentially unmasked. So we unmask the k tokens with the highest confidence tokens with the highest confidence tokens with the highest confidence and at the end when we reach the end and at the end when we reach the end and at the end when we reach the end there should be no masks which is there should be no masks which is there should be no masks which is remaining. So here's this animation remaining. So here's this animation remaining. So here's this animation actually shows dinoising in a good actually shows dinoising in a good actually shows dinoising in a good amount of detail and good amount of amount of detail and good amount of amount of detail and good amount of depth. This is the generation process. depth. This is the generation process. depth. This is the generation process. When we do the dnoising, we don't train When we do the dnoising, we don't train When we do the dnoising, we don't train the parameters. The parameters are the parameters. The parameters are the parameters. The parameters are trained and updated in the second part trained and updated in the second part trained and updated in the second part which is the noise prediction process. which is the noise prediction process. which is the noise prediction process. The cross entropy loss, the parameters The cross entropy loss, the parameters The cross entropy loss, the parameters are updated, the model is trained. are updated, the model is trained. are updated, the model is trained. That's it. The dnoising process we just That's it. The dnoising process we just That's it. The dnoising process we just generate. generate. generate. Here is a good gif which shows the Here is a good gif which shows the Here is a good gif which shows the dnoising. So we start with all masks and dnoising. So we start with all masks and dnoising. So we start with all masks and slowly unmask one at a time and then we slowly unmask one at a time and then we slowly unmask one at a time and then we get coherent text towards the end. get coherent text towards the end.
- L16 @ 269s
get coherent text towards the end. Right? So this is our whole diffusion Right? So this is our whole diffusion Right? So this is our whole diffusion pipeline diffusion language model pipeline diffusion language model pipeline diffusion language model pipeline built entirely from scratch. If pipeline built entirely from scratch. If pipeline built entirely from scratch. If you want the key takeaways we have you want the key takeaways we have you want the key takeaways we have in image in image in image in images the way we inject noise is in images the way we inject noise is in images the way we inject noise is through a gshian process right whereas through a gshian process right whereas through a gshian process right whereas in language models we replace it with in language models we replace it with in language models we replace it with token masking. So it's also called as token masking. So it's also called as token masking. So it's also called as discrete diffusion. The papers which I discrete diffusion. The papers which I discrete diffusion. The papers which I showed you earlier showed this word showed you earlier showed this word showed you earlier showed this word discrete diffusion. Right? discrete diffusion. Right? discrete diffusion. Right? Discrete diffusion modeling. The reason Discrete diffusion modeling. The reason Discrete diffusion modeling. The reason it's called discrete diffusion is that it's called discrete diffusion is that it's called discrete diffusion is that we just unmask tokens. So these are we just unmask tokens. So these are we just unmask tokens. So these are discrete tokens which are masked rather discrete tokens which are masked rather discrete tokens which are masked rather than applying a gshian distribution to than applying a gshian distribution to than applying a gshian distribution to the whole image which is done for the whole image which is done for the whole image which is done for images. images. images. The second is biirectional context. The second is biirectional context. The second is biirectional context. Unlike auto reggressive models a token Unlike auto reggressive models a token Unlike auto reggressive models a token can live look both behind it and in can live look both behind it and in can live look both behind it and in front of it. So we don't have causal front of it. So we don't have causal
- L16 @ 320s
front of it. So we don't have causal attention. Then we generate by unmasking attention. Then we generate by unmasking attention. Then we generate by unmasking tokens one at a time. Right? That's the tokens one at a time. Right? That's the tokens one at a time. Right? That's the So the whole complete pipeline is uh we So the whole complete pipeline is uh we So the whole complete pipeline is uh we do the we add the noise, we do the do the we add the noise, we do the do the we add the noise, we do the forward pass where we do the training. forward pass where we do the training. forward pass where we do the training. All the parameters are fixed. Then we do All the parameters are fixed. Then we do All the parameters are fixed. Then we do the generation in the dnoising process the generation in the dnoising process the generation in the dnoising process and then we recover the probability and then we recover the probability and then we recover the probability distribution which matches distribution which matches distribution which matches this underlying distribution which is this underlying distribution which is this underlying distribution which is the orange one. The orange is the true the orange one. The orange is the true the orange one. The orange is the true underlying distribution of meaningful underlying distribution of meaningful underlying distribution of meaningful sentences and the red is the predicted sentences and the red is the predicted sentences and the red is the predicted probability distribution which matches probability distribution which matches probability distribution which matches or which tries to match the true or which tries to match the true or which tries to match the true probability distribution. probability distribution. probability distribution. Now what we are going to do is that we Now what we are going to do is that we Now what we are going to do is that we are going to walk through a code and uh are going to walk through a code and uh are going to walk through a code and uh I'm I'm going to I'm I'm going to I'm I'm going to so if you zoom into here I'm going to so if you zoom into here I'm going to
- L16 @ 374s
so if you zoom into here I'm going to share this code file with all of you and share this code file with all of you and share this code file with all of you and at the end of this code file we are at the end of this code file we are at the end of this code file we are going to train a diffusion model fully going to train a diffusion model fully going to train a diffusion model fully from scratch and you'll also generate a from scratch and you'll also generate a from scratch and you'll also generate a cool GIF like this um which I'm showing cool GIF like this um which I'm showing cool GIF like this um which I'm showing over here you'll generate a GIF like over here you'll generate a GIF like over here you'll generate a GIF like this. So let's start with this uh coding this. So let's start with this uh coding this. So let's start with this uh coding module right now.
- L18 @ 3s
So here you can see the file which has So here you can see the file which has been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was getting some error but now it has run getting some error but now it has run getting some error but now it has run pretty well. And uh the same approach as pretty well. And uh the same approach as pretty well. And uh the same approach as we saw on Google collab works here. And we saw on Google collab works here. And we saw on Google collab works here. And uh you'll see the gif being printed uh you'll see the gif being printed uh you'll see the gif being printed right on the screen in this code. And right on the screen in this code. And right on the screen in this code. And this uh fancy neon style gif also can be this uh fancy neon style gif also can be this uh fancy neon style gif also can be seen over here. This does not work very seen over here. This does not work very seen over here. This does not work very well right now but you can tweak the well right now but you can tweak the well right now but you can tweak the code with the help of claude or chat GPT code with the help of claude or chat GPT code with the help of claude or chat GPT to get this neon style gif to work. to get this neon style gif to work. to get this neon style gif to work. All right. So here what I've done is All right. So here what I've done is All right. So here what I've done is that I have just mentioned the model to that I have just mentioned the model to that I have just mentioned the model to go ahead for uh go ahead for uh go ahead for uh around 200,000 iterations and uh if you around 200,000 iterations and uh if you around 200,000 iterations and uh if you go ahead for these many number of go ahead for these many number of go ahead for these many number of iterations you'll definitely see samples iterations you'll definitely see samples
- L18 @ 56s
iterations you'll definitely see samples which are as co inference samples which which are as co inference samples which which are as co inference samples which are as coherent as what you are seeing are as coherent as what you are seeing are as coherent as what you are seeing on the screen right now. on the screen right now. on the screen right now. The reason I'm showing runpod to you The reason I'm showing runpod to you The reason I'm showing runpod to you along with Google collab is because along with Google collab is because along with Google collab is because uh runpod also allows you to chain uh runpod also allows you to chain uh runpod also allows you to chain multiple GPUs together and in run you multiple GPUs together and in run you multiple GPUs together and in run you can also see the different GPUs which can also see the different GPUs which can also see the different GPUs which are available whereas on Google collab are available whereas on Google collab are available whereas on Google collab if you see you're kind of restricted if you see you're kind of restricted if you see you're kind of restricted right by the GPUs which they give. So if right by the GPUs which they give. So if right by the GPUs which they give. So if you go to run type you'll see you have you go to run type you'll see you have you go to run type you'll see you have H100 A100 you can't see the H100 A100 you can't see the H100 A100 you can't see the specifications it's really like a black specifications it's really like a black specifications it's really like a black box but on the run pod you can see the box but on the run pod you can see the box but on the run pod you can see the VRAM of each GPU you can see the number VRAM of each GPU you can see the number VRAM of each GPU you can see the number of GPUs available you can see the cost of GPUs available you can see the cost of GPUs available you can see the cost you can even chain different GPUs you can even chain different GPUs
- L18 @ 97s
you can even chain different GPUs together to do distributed computing if together to do distributed computing if together to do distributed computing if possible possible possible so let's actually pause this right now so let's actually pause this right now so let's actually pause this right now let's interrupt this after 500 let's interrupt this after 500 let's interrupt this after 500 iterations S and uh yeah then we just iterations S and uh yeah then we just iterations S and uh yeah then we just have the sampling code over here have the sampling code over here have the sampling code over here once we have the sample here you'll see once we have the sample here you'll see once we have the sample here you'll see the sampling is pretty bad because I the sampling is pretty bad because I the sampling is pretty bad because I only run for 500 iterations so that's only run for 500 iterations so that's only run for 500 iterations so that's natural natural natural um and in this next piece of code you'll and in this next piece of code you'll save the GIF and after you save the GIF save the GIF and after you save the GIF save the GIF and after you save the GIF then you'll print the GIF. So this code then you'll print the GIF. So this code then you'll print the GIF. So this code will save the GIF as inference.jif and will save the GIF as inference.jif and will save the GIF as inference.jif and this will uh essentially print it on the this will uh essentially print it on the this will uh essentially print it on the screen screen screen right now. Right now I'm only doing for right now. Right now I'm only doing for right now. Right now I'm only doing for 500 iterations but you see already the 500 iterations but you see already the 500 iterations but you see already the GIF is appearing on the screen. We don't GIF is appearing on the screen. We don't GIF is appearing on the screen. We don't get that error which I was getting get that error which I was getting get that error which I was getting before. Um yeah if you train for 200,000 before. Um yeah if you train for 200,000
- L18 @ 156s
before. Um yeah if you train for 200,000 iterations you'll see an amazing iterations you'll see an amazing iterations you'll see an amazing coherent language diffusion language coherent language diffusion language coherent language diffusion language model working from scratch as I'm model working from scratch as I'm model working from scratch as I'm showing on the screen right now. showing on the screen right now. showing on the screen right now. I'll share the entire code file with I'll share the entire code file with I'll share the entire code file with you. You'll also now be able to you. You'll also now be able to you. You'll also now be able to understand this large language diffusion understand this large language diffusion understand this large language diffusion models paper in a lot of depth because models paper in a lot of depth because models paper in a lot of depth because we have implemented this right from we have implemented this right from we have implemented this right from scratch. They also have a section of scratch. They also have a section of scratch. They also have a section of fine-tuning towards the end which you fine-tuning towards the end which you fine-tuning towards the end which you can easily do can easily do can easily do after you have finished the after you have finished the after you have finished the pre-training. You just have to assemble pre-training. You just have to assemble pre-training. You just have to assemble a fine fine tuning data set and then run a fine fine tuning data set and then run a fine fine tuning data set and then run fine tuning on top of it. So let me just fine tuning on top of it. So let me just fine tuning on top of it. So let me just Yeah. Yeah. Yeah. So as I mentioned this paper also So as I mentioned this paper also So as I mentioned this paper also includes fine-tuning. I have not includes fine-tuning. I have not includes fine-tuning. I have not included fine-tuning in this journey included fine-tuning in this journey included fine-tuning in this journey because I believe pre-training involves because I believe pre-training involves because I believe pre-training involves all the necessary things which you will all the necessary things which you will
- L18 @ 203s
all the necessary things which you will need to learn about language diffusion need to learn about language diffusion need to learn about language diffusion models. All right. So yeah we have this All right. So yeah we have this traditional ARM language model code traditional ARM language model code traditional ARM language model code which runs and predicts sequential which runs and predicts sequential which runs and predicts sequential output which is coherent. And we have output which is coherent. And we have output which is coherent. And we have this uh this uh this uh diffusion model based language model diffusion model based language model diffusion model based language model code which predicts output such as this code which predicts output such as this code which predicts output such as this which is much faster in inference but which is much faster in inference but which is much faster in inference but pre-training might take more time. So as pre-training might take more time. So as pre-training might take more time. So as Andre Karpati mentioned in this tweet Andre Karpati mentioned in this tweet Andre Karpati mentioned in this tweet diffusion language models actually can diffusion language models actually can diffusion language models actually can be a frontier milestone in the future be a frontier milestone in the future be a frontier milestone in the future because why do we have to stick with because why do we have to stick with because why do we have to stick with auto reggressive models right of course. auto reggressive models right of course. auto reggressive models right of course. So uh I believe this is one of the first So uh I believe this is one of the first So uh I believe this is one of the first tutorials or one of the first journey tutorials or one of the first journey tutorials or one of the first journey videos which shows how to build a small videos which shows how to build a small videos which shows how to build a small scale diffusion model fully from scratch scale diffusion model fully from scratch scale diffusion model fully from scratch and I hope this codebase which I've and I hope this codebase which I've and I hope this codebase which I've shown on the screen right now serves as
- L18 @ 259s
shown on the screen right now serves as shown on the screen right now serves as a reference for you to build multiple a reference for you to build multiple a reference for you to build multiple such small language models from scratch. such small language models from scratch. such small language models from scratch. So the cool thing about diffusion models So the cool thing about diffusion models So the cool thing about diffusion models is that the space is wide open, right? is that the space is wide open, right? is that the space is wide open, right? You can see small language models um let's say small language models um let's say small language models GitHub or Quel right all of these small GitHub or Quel right all of these small GitHub or Quel right all of these small language models which have been built language models which have been built language models which have been built they are auto reggressive models but the they are auto reggressive models but the they are auto reggressive models but the space for building diffusion models is space for building diffusion models is space for building diffusion models is wide open. I know [clears throat] wide open. I know [clears throat] wide open. I know [clears throat] Inception Labs which is one company Inception Labs which is one company Inception Labs which is one company which builds diffusion models which are which builds diffusion models which are which builds diffusion models which are pretty cool but I think this is closed pretty cool but I think this is closed pretty cool but I think this is closed source. source. source. I'm yet to know and opensource model I'm yet to know and opensource model I'm yet to know and opensource model which is as fast and as good as this which is as fast and as good as this which is as fast and as good as this mercury model. So you see their whole uh mercury model. So you see their whole uh mercury model. So you see their whole uh value proposition is blazing fast value proposition is blazing fast value proposition is blazing fast inference with frontier quality and a inference with frontier quality and a inference with frontier quality and a fraction of the cost which might be fraction of the cost which might be
- L18 @ 308s
fraction of the cost which might be awesome for the future. But now you know awesome for the future. But now you know awesome for the future. But now you know how to build this fully yourself. Um, okay. After this, I'll make a final okay. After this, I'll make a final conclusion video on my thoughts and conclusion video on my thoughts and conclusion video on my thoughts and we'll wrap it up.
- L17 @ 2s
Okay. So now I'm going to take you Okay. So now I'm going to take you through the code of assembling this through the code of assembling this through the code of assembling this diffusion language model from scratch diffusion language model from scratch diffusion language model from scratch and u this code is actually and u this code is actually and u this code is actually exactly following these steps which we exactly following these steps which we exactly following these steps which we have seen on the whiteboard. So I think have seen on the whiteboard. So I think have seen on the whiteboard. So I think all of you will be able to follow along all of you will be able to follow along all of you will be able to follow along if you have understood the theory which if you have understood the theory which if you have understood the theory which we have covered in the earlier modules we have covered in the earlier modules we have covered in the earlier modules of this course. Okay. of this course. Okay. of this course. Okay. All right. So All right. So All right. So the first thing which you should note is the first thing which you should note is the first thing which you should note is that here I have a subscription of that here I have a subscription of that here I have a subscription of Google Collab Pro. So I have access to Google Collab Pro. So I have access to Google Collab Pro. So I have access to an A1 H100 GPU. an A1 H100 GPU. an A1 H100 GPU. Diffusion language models take time to Diffusion language models take time to Diffusion language models take time to train compared to train compared to train compared to the auto reggressive models. So you the auto reggressive models. So you the auto reggressive models. So you should have access to Google Collab Pro. should have access to Google Collab Pro. should have access to Google Collab Pro. You can get A100 GPU or an H100 GPU You can get A100 GPU or an H100 GPU You can get A100 GPU or an H100 GPU here. The H100 GPU is much faster. here. The H100 GPU is much faster.
- L17 @ 60s
here. The H100 GPU is much faster. If you don't have access to the Google If you don't have access to the Google If you don't have access to the Google Collab Pro, you can start with the T4 Collab Pro, you can start with the T4 Collab Pro, you can start with the T4 GPU which is free, but it will just take GPU which is free, but it will just take GPU which is free, but it will just take a large amount of time for the model to a large amount of time for the model to a large amount of time for the model to run. run. run. Okay. Okay. Okay. So remember that So remember that So remember that first thing which we have to do is that first thing which we have to do is that first thing which we have to do is that we have to select the mode which we have we have to select the mode which we have we have to select the mode which we have to run. If you have a if you are on a to run. If you have a if you are on a to run. If you have a if you are on a free tire of Google Collab, I suggest free tire of Google Collab, I suggest free tire of Google Collab, I suggest that you that you that you select the run mode to be quick. select the run mode to be quick. select the run mode to be quick. So what you can do is if you are on a So what you can do is if you are on a So what you can do is if you are on a free trial, you can do the run mode free trial, you can do the run mode free trial, you can do the run mode equal to quick equal to quick equal to quick and then run this notebook. You'll see and then run this notebook. You'll see and then run this notebook. You'll see that in run mode equal to quick the that in run mode equal to quick the that in run mode equal to quick the number of training steps is only 2,000. number of training steps is only 2,000.
- L17 @ 103s
number of training steps is only 2,000. Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 which is if you have compute budget or which is if you have compute budget or which is if you have compute budget or if you have Google collab pro you can if you have Google collab pro you can if you have Google collab pro you can even run for 60,000 steps. Some even run for 60,000 steps. Some even run for 60,000 steps. Some variables which I've defined here are variables which I've defined here are variables which I've defined here are the training examples which is the the training examples which is the the training examples which is the number of rows which I want from my tiny number of rows which I want from my tiny number of rows which I want from my tiny stories data set. So let me show that stories data set. So let me show that stories data set. So let me show that data set once more. data set once more. data set once more. Yeah. So essentially So essentially train examples is number of training train examples is number of training train examples is number of training examples which I have here. There are examples which I have here. There are examples which I have here. There are 2.2 million. So we can choose how many 2.2 million. So we can choose how many 2.2 million. So we can choose how many we want. Here I've chosen 1 million in we want. Here I've chosen 1 million in we want. Here I've chosen 1 million in this budget 100 and here I've only this budget 100 and here I've only this budget 100 and here I've only chosen 50,000 in the quick version. Then chosen 50,000 in the quick version. Then chosen 50,000 in the quick version. Then we have to select the sequence length. we have to select the sequence length. we have to select the sequence length. Sequence length is essentially the Sequence length is essentially the Sequence length is essentially the embedding dimension. Right? So if you embedding dimension. Right? So if you embedding dimension. Right? So if you take a look at the take a look at the
- L17 @ 154s
take a look at the whiteboard you'll see that whiteboard you'll see that whiteboard you'll see that this dimension which I keep talking this dimension which I keep talking this dimension which I keep talking about right this this dimension the about right this this dimension the about right this this dimension the number of columns of the token embedding number of columns of the token embedding number of columns of the token embedding matrix and the number of columns of the matrix and the number of columns of the matrix and the number of columns of the position embedding matrix and in the position embedding matrix and in the position embedding matrix and in the case of diffusion model the time case of diffusion model the time case of diffusion model the time embedding matrix also that's my embedding matrix also that's my embedding matrix also that's my embedding dimension and I've chosen that embedding dimension and I've chosen that embedding dimension and I've chosen that to be equal to 256. to be equal to 256. to be equal to 256. Now actually that's the D model. So the Now actually that's the D model. So the Now actually that's the D model. So the embedding dimension is the D model which embedding dimension is the D model which embedding dimension is the D model which has been chosen to be 512. So I'm going has been chosen to be 512. So I'm going has been chosen to be 512. So I'm going to look at budget 100 for now. Embedding to look at budget 100 for now. Embedding to look at budget 100 for now. Embedding dimension is 512. Sequence length that's dimension is 512. Sequence length that's dimension is 512. Sequence length that's the context length basically the number the context length basically the number the context length basically the number of tokens model can look at at one time. of tokens model can look at at one time. of tokens model can look at at one time. That's the number of rows of position That's the number of rows of position That's the number of rows of position embedding matrix. So sequence length embedding matrix. So sequence length embedding matrix. So sequence length essentially here in these examples we essentially here in these examples we
- L17 @ 200s
essentially here in these examples we have a sequence length of four tokens, have a sequence length of four tokens, have a sequence length of four tokens, right? Whereas in actual code we have right? Whereas in actual code we have right? Whereas in actual code we have taken a sequence length to be 256. taken a sequence length to be 256. taken a sequence length to be 256. vocabulary size is 26,000 vocabulary size is 26,000 vocabulary size is 26,000 whereas uh here I just here when I whereas uh here I just here when I whereas uh here I just here when I mentioned the size of the token mentioned the size of the token mentioned the size of the token embedding matrix I had shown a embedding matrix I had shown a embedding matrix I had shown a vocabulary size of 100,000 you can play vocabulary size of 100,000 you can play vocabulary size of 100,000 you can play around with these variables these are around with these variables these are around with these variables these are not set in stone not set in stone not set in stone number of layers these are actually the number of layers these are actually the number of layers these are actually the number of uh number of uh number of uh transformer blocks so remember this transformer blocks so remember this transformer blocks so remember this mirror notebook which I showed all of mirror notebook which I showed all of mirror notebook which I showed all of you you you uh which was essentially called journey uh which was essentially called journey uh which was essentially called journey of a token. So if you take a look at of a token. So if you take a look at of a token. So if you take a look at this miro notebook this miro notebook this miro notebook you'll see that there are when I show you'll see that there are when I show you'll see that there are when I show the transformer block there are actually the transformer block there are actually the transformer block there are actually multiple transformer blocks which are multiple transformer blocks which are multiple transformer blocks which are chained to each other. So these are the chained to each other. So these are the
- L17 @ 253s
chained to each other. So these are the uh this variable is the number of uh this variable is the number of uh this variable is the number of layers. The number of heads is layers. The number of heads is layers. The number of heads is essentially for multi head attention. In essentially for multi head attention. In essentially for multi head attention. In this journey we have so far seen only this journey we have so far seen only this journey we have so far seen only single head attention. In multi single head attention. In multi single head attention. In multi attention you have different heads. Each attention you have different heads. Each attention you have different heads. Each head has its own attention score matrix head has its own attention score matrix head has its own attention score matrix or attention weight matrix. So we or attention weight matrix. So we or attention weight matrix. So we capture a different perspective. This capture a different perspective. This capture a different perspective. This DFF is for the DFF is for the DFF is for the field forward field forward field forward neural network. So if you take a look at neural network. So if you take a look at neural network. So if you take a look at this this this uh if you take a look at our schematic uh if you take a look at our schematic uh if you take a look at our schematic over here, you'll see that the feed over here, you'll see that the feed over here, you'll see that the feed forward in the feed forward neural forward in the feed forward neural forward in the feed forward neural network. Yeah, we mentioned that it's an network. Yeah, we mentioned that it's an network. Yeah, we mentioned that it's an expansion expansion expansion um and a contraction, right? So when we um and a contraction, right? So when we um and a contraction, right? So when we expand the number of dimensions in the expand the number of dimensions in the expand the number of dimensions in the hidden layer are four times the hidden layer are four times the hidden layer are four times the embedding dimension. So this four here embedding dimension. So this four here embedding dimension. So this four here that's this uh four times the D model.
- L17 @ 311s
that's this uh four times the D model. that's this uh four times the D model. So DFF is the feed forward neural So DFF is the feed forward neural So DFF is the feed forward neural network hidden layer dimension. diffusion steps is 128. So this is diffusion steps is 128. So this is basically here in this animation I have basically here in this animation I have basically here in this animation I have shown the diffusion steps to be equal to shown the diffusion steps to be equal to shown the diffusion steps to be equal to six. Right? Time goes from 1 to 6. So six. Right? Time goes from 1 to 6. So six. Right? Time goes from 1 to 6. So 128 is just we have taken 128 time 128 is just we have taken 128 time 128 is just we have taken 128 time steps. steps. steps. That's it. And the rest of the things That's it. And the rest of the things That's it. And the rest of the things are for the optimizer. We have gradient are for the optimizer. We have gradient are for the optimizer. We have gradient accumulation so that a huge batch does accumulation so that a huge batch does accumulation so that a huge batch does not live on a GPU. We reduce the batch not live on a GPU. We reduce the batch not live on a GPU. We reduce the batch size by half so that half of the batch size by half so that half of the batch size by half so that half of the batch lives on the GPU at one time. We have a lives on the GPU at one time. We have a lives on the GPU at one time. We have a learning rate of 2 into 10 -4 batch size learning rate of 2 into 10 -4 batch size learning rate of 2 into 10 -4 batch size of 32 weight TK of.1 and warm-up steps of 32 weight TK of.1 and warm-up steps of 32 weight TK of.1 and warm-up steps of,000. These will be needed in the ADM of,000. These will be needed in the ADM of,000. These will be needed in the ADM optimizer. Okay. So you run this initial optimizer. Okay. So you run this initial
- L17 @ 361s
optimizer. Okay. So you run this initial configuration and then you move forward. configuration and then you move forward. configuration and then you move forward. You install a bunch of dependencies You install a bunch of dependencies You install a bunch of dependencies and make sure that you have GPU support and make sure that you have GPU support and make sure that you have GPU support here. So once we run this fully uh here. So once we run this fully uh here. So once we run this fully uh sometimes H100 GPU is not available in sometimes H100 GPU is not available in sometimes H100 GPU is not available in the Google Collab Pro version. So then the Google Collab Pro version. So then the Google Collab Pro version. So then you will be given an A100 version or an you will be given an A100 version or an you will be given an A100 version or an A100 GPU. That's also fine but it takes A100 GPU. That's also fine but it takes A100 GPU. That's also fine but it takes a very long time to run. I'll give you a very long time to run. I'll give you a very long time to run. I'll give you other options after showing this Google other options after showing this Google other options after showing this Google collab notebook which are much much collab notebook which are much much collab notebook which are much much faster such as going to run pod for faster such as going to run pod for faster such as going to run pod for example. example. example. Okay. So it's still connecting. So I Okay. So it's still connecting. So I Okay. So it's still connecting. So I think H100 is not available. So let's think H100 is not available. So let's think H100 is not available. So let's see see see once this is done the next step for us once this is done the next step for us once this is done the next step for us is to load the tiny stories data. So I is to load the tiny stories data. So I is to load the tiny stories data. So I go to this uh hugging face repository go to this uh hugging face repository
- L17 @ 409s
go to this uh hugging face repository and I load my data. I load my training and I load my data. I load my training and I load my data. I load my training data set. I load my validation data set data set. I load my validation data set data set. I load my validation data set and uh my data is loaded. The next thing and uh my data is loaded. The next thing and uh my data is loaded. The next thing which we have to do is break down my which we have to do is break down my which we have to do is break down my sentences into tokens. Right? Now we sentences into tokens. Right? Now we sentences into tokens. Right? Now we could have used a bite pair encoder could have used a bite pair encoder could have used a bite pair encoder tokenizer which is there are number of tokenizer which is there are number of tokenizer which is there are number of tokenizers which are available from tokenizers which are available from tokenizers which are available from scratch. So if you go to the tick scratch. So if you go to the tick scratch. So if you go to the tick tokenizer, tokenizer, tokenizer, I think if you go to tick token library I think if you go to tick token library I think if you go to tick token library here, it's the tokenizer which openai here, it's the tokenizer which openai here, it's the tokenizer which openai uses. So you can directly use a uses. So you can directly use a uses. So you can directly use a tokenizer from here. But just so that we tokenizer from here. But just so that we tokenizer from here. But just so that we all learn something, I have shown how to all learn something, I have shown how to all learn something, I have shown how to train a tokenizer from scratch. So I got train a tokenizer from scratch. So I got train a tokenizer from scratch. So I got a message that GPU was not available. So a message that GPU was not available. So a message that GPU was not available. So we I'm connected to an L4 GPU at the we I'm connected to an L4 GPU at the
- L17 @ 456s
we I'm connected to an L4 GPU at the moment. See that's fine. So let's see. Um Um yeah so here you'll see that I'm yeah so here you'll see that I'm yeah so here you'll see that I'm connected to an L4 Nvidia GPU connected to an L4 Nvidia GPU connected to an L4 Nvidia GPU and uh my data set is also loaded right and uh my data set is also loaded right and uh my data set is also loaded right now to train a tokenizer from scratch. now to train a tokenizer from scratch. now to train a tokenizer from scratch. Basically we are just going to use this Basically we are just going to use this Basically we are just going to use this BP trainer module. I'm not going to go BP trainer module. I'm not going to go BP trainer module. I'm not going to go into too many details of this but bite into too many details of this but bite into too many details of this but bite pair encoding essentially merges pair encoding essentially merges pair encoding essentially merges commonly occurring tokens or characters commonly occurring tokens or characters commonly occurring tokens or characters and forms tokens which needed to be and forms tokens which needed to be and forms tokens which needed to be which need to be added in the which need to be added in the which need to be added in the vocabulary. The variable which we need vocabulary. The variable which we need vocabulary. The variable which we need to create this tokenizer is my to create this tokenizer is my to create this tokenizer is my vocabulary size. So we keep on adding vocabulary size. So we keep on adding vocabulary size. So we keep on adding tokens to the vocabulary until we reach tokens to the vocabulary until we reach tokens to the vocabulary until we reach a tokenizer size. So here you can see a tokenizer size. So here you can see a tokenizer size. So here you can see that that that uh I have also created special tokens uh I have also created special tokens uh I have also created special tokens here such as a padding token, unknown here such as a padding token, unknown here such as a padding token, unknown token, beginning of sequence, end of
- L17 @ 512s
token, beginning of sequence, end of token, beginning of sequence, end of sequence and a mask token. This is very sequence and a mask token. This is very sequence and a mask token. This is very important. This mask token is the one important. This mask token is the one important. This mask token is the one which uh we'll be using during the which uh we'll be using during the which uh we'll be using during the noising process as I've already noising process as I've already noising process as I've already explained to you. and the tokenizer explained to you. and the tokenizer explained to you. and the tokenizer training training training all the training text is wrapped in this all the training text is wrapped in this all the training text is wrapped in this write a short story then the assistant write a short story then the assistant write a short story then the assistant and then end and then end and then end uh so basically I'll take this story uh so basically I'll take this story uh so basically I'll take this story from my data set and then the actual from my data set and then the actual from my data set and then the actual data set will be write a story then the data set will be write a story then the data set will be write a story then the actual story and then end and this this actual story and then end and this this actual story and then end and this this whole thing will be tokenized whole thing will be tokenized whole thing will be tokenized this is just a format which I'm using this is just a format which I'm using this is just a format which I'm using for the tokenization right now I don't for the tokenization right now I don't for the tokenization right now I don't think you think you think you worry too much about tokenization at the worry too much about tokenization at the worry too much about tokenization at the moment. In fact, you can even replace moment. In fact, you can even replace moment. In fact, you can even replace this whole tokenizer section with your this whole tokenizer section with your this whole tokenizer section with your with a tokenizer directly borrowed from
- L17 @ 560s
with a tokenizer directly borrowed from with a tokenizer directly borrowed from tick token. I I I think the code should tick token. I I I think the code should tick token. I I I think the code should still work. still work. still work. Uh okay, what's important is that if you Uh okay, what's important is that if you Uh okay, what's important is that if you have a sentence such as hello world, have a sentence such as hello world, have a sentence such as hello world, right, based on the tokenizer which we right, based on the tokenizer which we right, based on the tokenizer which we have created, we have 2 1 3 0 0 1 1 5 6 have created, we have 2 1 3 0 0 1 1 5 6 have created, we have 2 1 3 0 0 1 1 5 6 9 and 3. These are the tokens. If I have 9 and 3. These are the tokens. If I have 9 and 3. These are the tokens. If I have something like uh these are the tokens associated with it. these are the tokens associated with it. So I think this two and three is So I think this two and three is So I think this two and three is basically the beginning of sequence and basically the beginning of sequence and basically the beginning of sequence and the end of sequence I think. Um and the the end of sequence I think. Um and the the end of sequence I think. Um and the token ID corresponding to the mask is token ID corresponding to the mask is token ID corresponding to the mask is four. Yeah, two is the beginning of four. Yeah, two is the beginning of four. Yeah, two is the beginning of sequence ID. Three is the end of sequence ID. Three is the end of sequence ID. Three is the end of sequence ID and the token ID sequence ID and the token ID sequence ID and the token ID corresponding to mask is four. This corresponding to mask is four. This corresponding to mask is four. This token ID corresponding to mask is token ID corresponding to mask is token ID corresponding to mask is very important very important very important because in the noising process we are
- L17 @ 621s
because in the noising process we are because in the noising process we are going to replace the tokens with masks going to replace the tokens with masks going to replace the tokens with masks and then we are going to use this token and then we are going to use this token and then we are going to use this token ID to represent those masks. Why have we ID to represent those masks. Why have we ID to represent those masks. Why have we added this write a short story here? added this write a short story here? added this write a short story here? Because when we are going to give the Because when we are going to give the Because when we are going to give the inference prompt, we are going to tell inference prompt, we are going to tell inference prompt, we are going to tell to write a short story. So if it's to write a short story. So if it's to write a short story. So if it's included in the tokenizer training, it included in the tokenizer training, it included in the tokenizer training, it generally helps although it's strictly generally helps although it's strictly generally helps although it's strictly not needed at all. [snorts] not needed at all. [snorts] not needed at all. [snorts] Okay. Now we come to the process of Okay. Now we come to the process of Okay. Now we come to the process of actually assembling the uh diffusion actually assembling the uh diffusion actually assembling the uh diffusion language model architecture. Right? So language model architecture. Right? So language model architecture. Right? So the diffusion language model the diffusion language model the diffusion language model architecture again to show everyone it architecture again to show everyone it architecture again to show everyone it looks something like this. Yeah, this is how the diffusion language Yeah, this is how the diffusion language model architecture actually looks like. model architecture actually looks like. model architecture actually looks like. And let me rub all of this so that And let me rub all of this so that And let me rub all of this so that becomes clearer to all of you. Yeah. becomes clearer to all of you. Yeah.
- L17 @ 673s
becomes clearer to all of you. Yeah. And if you see that you first define a And if you see that you first define a And if you see that you first define a token embedding, position embedding and token embedding, position embedding and token embedding, position embedding and time embedding. time embedding. time embedding. And the place to get started to actually And the place to get started to actually And the place to get started to actually understand this code is that [snorts] understand this code is that [snorts] understand this code is that [snorts] you first have the token embeddings. you you first have the token embeddings. you you first have the token embeddings. you add it to the position embedding and add it to the position embedding and add it to the position embedding and then you add the time embedding after then you add the time embedding after then you add the time embedding after it. Right? And then this this input it. Right? And then this this input it. Right? And then this this input sequence you pass through the encoder sequence you pass through the encoder sequence you pass through the encoder block. Now what's the encoder block? block. Now what's the encoder block? block. Now what's the encoder block? That's essentially just a transformer That's essentially just a transformer That's essentially just a transformer encoder. So transformer encoder if you encoder. So transformer encoder if you encoder. So transformer encoder if you see yeah, we are just borrowing this yeah, we are just borrowing this transformer encoder from Pytor. So we transformer encoder from Pytor. So we transformer encoder from Pytor. So we don't have to write this or code this don't have to write this or code this don't have to write this or code this from scratch. That just simplifies the from scratch. That just simplifies the from scratch. That just simplifies the code a lot. It just creates this entire code a lot. It just creates this entire code a lot. It just creates this entire transformer block like this. Okay. transformer block like this. Okay. transformer block like this. Okay. So here we are just going to do So here we are just going to do So here we are just going to do a transformer a transformer
- L17 @ 740s
a transformer um we are going to do a transformer um we are going to do a transformer um we are going to do a transformer encoder layer over here and uh let's see and attention mask equal to none. This and attention mask equal to none. This is extremely important attention mask is extremely important attention mask is extremely important attention mask equal to none essentially means that we equal to none essentially means that we equal to none essentially means that we are not having that causal attention are not having that causal attention are not having that causal attention layer which we saw in the whiteboard. So layer which we saw in the whiteboard. So layer which we saw in the whiteboard. So on the whiteboard what we saw is that on the whiteboard what we saw is that on the whiteboard what we saw is that for auto reggressive models we put this for auto reggressive models we put this for auto reggressive models we put this elements above the diagonal to be zero elements above the diagonal to be zero elements above the diagonal to be zero right in the attention scores but we are right in the attention scores but we are right in the attention scores but we are not going to do this in the diffusion not going to do this in the diffusion not going to do this in the diffusion model. So we don't have any attention model. So we don't have any attention model. So we don't have any attention mask over here. So it's very important mask over here. So it's very important mask over here. So it's very important to mention that attention mask equal to to mention that attention mask equal to to mention that attention mask equal to none. [snorts] none. [snorts] none. [snorts] Okay. So we say that attention mask Okay. So we say that attention mask Okay. So we say that attention mask equal to none. Uh we add the again we equal to none. Uh we add the again we equal to none. Uh we add the again we add the token embedding to the position add the token embedding to the position add the token embedding to the position embedding. We add the time embedding to embedding. We add the time embedding to
- L17 @ 791s
embedding. We add the time embedding to it. The resultant input embedding we it. The resultant input embedding we it. The resultant input embedding we pass through the transformer block. Then pass through the transformer block. Then pass through the transformer block. Then we come to the output. We do a layer we come to the output. We do a layer we come to the output. We do a layer normalization and we do the logits and normalization and we do the logits and normalization and we do the logits and we return the logits. That's all. This we return the logits. That's all. This we return the logits. That's all. This is the entire diffusion model is the entire diffusion model is the entire diffusion model architecture. architecture. architecture. Okay. Okay. Um and once you have understood this on Um and once you have understood this on Um and once you have understood this on the whiteboard, if you understood this the whiteboard, if you understood this the whiteboard, if you understood this architecture on the whiteboard, [snorts] architecture on the whiteboard, [snorts] architecture on the whiteboard, [snorts] what we are doing in code is exactly a what we are doing in code is exactly a what we are doing in code is exactly a representation of this architecture. representation of this architecture. representation of this architecture. There is nothing different which is There is nothing different which is There is nothing different which is being done over here. Note one more being done over here. Note one more being done over here. Note one more thing that uh we have in the forward thing that uh we have in the forward thing that uh we have in the forward pass we also have the time steps. Why do pass we also have the time steps. Why do pass we also have the time steps. Why do we have the time steps? Because for the we have the time steps? Because for the we have the time steps? Because for the time embedding we need to know which time embedding we need to know which time embedding we need to know which time time time um we are looking at because if you look um we are looking at because if you look um we are looking at because if you look at the time embedding matrix
- L17 @ 839s
at the time embedding matrix at the time embedding matrix if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need to know at what time step we are in to to know at what time step we are in to to know at what time step we are in to choose the vector corresponding to that choose the vector corresponding to that choose the vector corresponding to that time step during the forward pass. time step during the forward pass. time step during the forward pass. Right. So I run this and I get the Right. So I run this and I get the Right. So I run this and I get the uh diffusion language model uh diffusion language model uh diffusion language model architecture. architecture. Then I'm just uh creating a training Then I'm just uh creating a training Then I'm just uh creating a training loader and I'm creating a validation loader and I'm creating a validation loader and I'm creating a validation loader so that we can create training loader so that we can create training loader so that we can create training and validation data quite easily. The and validation data quite easily. The and validation data quite easily. The next step is very important. Take a look next step is very important. Take a look next step is very important. Take a look at this corrupt with mask. Essentially at this corrupt with mask. Essentially at this corrupt with mask. Essentially what we are doing is before we pass the what we are doing is before we pass the what we are doing is before we pass the input ids to the uh transformer block or input ids to the uh transformer block or input ids to the uh transformer block or through the architecture itself we have through the architecture itself we have through the architecture itself we have to corrupt the input ids right. What to corrupt the input ids right. What to corrupt the input ids right. What does it mean? It means that we we have does it mean? It means that we we have does it mean? It means that we we have to based on the time step we have to add
- L17 @ 887s
to based on the time step we have to add to based on the time step we have to add the mask tokens. This is what is done the mask tokens. This is what is done the mask tokens. This is what is done here. So we have t which is basically at here. So we have t which is basically at here. So we have t which is basically at what time step we are in and here what's what time step we are in and here what's what time step we are in and here what's simply done is that we have this mask simply done is that we have this mask simply done is that we have this mask positions where the mask token ID is positions where the mask token ID is positions where the mask token ID is added and the rest of the token ids are added and the rest of the token ids are added and the rest of the token ids are replaced with minus 100. The reason they replaced with minus 100. The reason they replaced with minus 100. The reason they are replaced with minus 100 is because are replaced with minus 100 is because are replaced with minus 100 is because they are not going to come in the uh they are not going to come in the uh they are not going to come in the uh loss function calculation. So if you loss function calculation. So if you loss function calculation. So if you look at look at look at cross entropy loss pytor torch cross entropy loss pytor torch cross entropy loss pytor torch you'll see that there is this ignore you'll see that there is this ignore you'll see that there is this ignore index of minus 100 which means that index of minus 100 which means that index of minus 100 which means that wherever we have minus 100 that's not wherever we have minus 100 that's not wherever we have minus 100 that's not involved in the cross entropy loss involved in the cross entropy loss involved in the cross entropy loss calculation. So remember why is this calculation. So remember why is this calculation. So remember why is this done because we have already seen this done because we have already seen this
- L17 @ 929s
done because we have already seen this on the whiteboard right to calculate the on the whiteboard right to calculate the on the whiteboard right to calculate the loss I only need to look at the mask loss I only need to look at the mask loss I only need to look at the mask tokens. I don't need to look at the tokens. I don't need to look at the tokens. I don't need to look at the other tokens at all. So that's why these other tokens at all. So that's why these other tokens at all. So that's why these other tokens are replaced with minus other tokens are replaced with minus other tokens are replaced with minus 100. So they don't come in my loss 100. So they don't come in my loss 100. So they don't come in my loss function calculation. So these minute function calculation. So these minute function calculation. So these minute differences between the diffusion differences between the diffusion differences between the diffusion language models and auto reggressive language models and auto reggressive language models and auto reggressive models are carefully scattered across models are carefully scattered across models are carefully scattered across the code. And unless you understand the the code. And unless you understand the the code. And unless you understand the theory, it will be very difficult for theory, it will be very difficult for theory, it will be very difficult for you to understand okay where is the you to understand okay where is the you to understand okay where is the where is it implemented that only the where is it implemented that only the where is it implemented that only the masked positions will be used in the masked positions will be used in the masked positions will be used in the loss function calculations because all loss function calculations because all loss function calculations because all the other input ids are replaced with the other input ids are replaced with the other input ids are replaced with minus 100. Where is it included that we minus 100. Where is it included that we minus 100. Where is it included that we won't have a causal attention here? won't have a causal attention here? won't have a causal attention here? Well, it's included over here where we
- L17 @ 973s
Well, it's included over here where we Well, it's included over here where we pass an attention mask equal to none. pass an attention mask equal to none. pass an attention mask equal to none. These are the small details which all of These are the small details which all of These are the small details which all of you need to be paying attention to. you need to be paying attention to. you need to be paying attention to. Okay, Okay, Okay, so this is how we define the input so this is how we define the input so this is how we define the input sequence and we corrupt it with noise sequence and we corrupt it with noise sequence and we corrupt it with noise which means we ask we just add masks at which means we ask we just add masks at which means we ask we just add masks at the positions where the positions where the positions where um we have chosen to add the mask based um we have chosen to add the mask based um we have chosen to add the mask based on the time step. Essentially the deeper on the time step. Essentially the deeper on the time step. Essentially the deeper we are into the noising schedule the we are into the noising schedule the we are into the noising schedule the more probability of adding the mask more probability of adding the mask more probability of adding the mask right and then we define the loss. right and then we define the loss. right and then we define the loss. To define the loss we just have to get To define the loss we just have to get To define the loss we just have to get the noisy ids and then we have to the noisy ids and then we have to the noisy ids and then we have to calculate the cross entropy loss between calculate the cross entropy loss between calculate the cross entropy loss between the noisy ids and the actual values at the noisy ids and the actual values at the noisy ids and the actual values at those mask positions. That's it. those mask positions. That's it. those mask positions. That's it. So this is the cross entropy loss. So So this is the cross entropy loss. So
- L17 @ 1016s
So this is the cross entropy loss. So you can run this right now. Once we have you can run this right now. Once we have you can run this right now. Once we have this defined, the last step is just to this defined, the last step is just to this defined, the last step is just to define the training function and the define the training function and the define the training function and the loss function. So this is my training loss function. So this is my training loss function. So this is my training function at the moment where what I'm function at the moment where what I'm function at the moment where what I'm doing first is that this evaluation loss doing first is that this evaluation loss doing first is that this evaluation loss is after every 20 batches I'm going to is after every 20 batches I'm going to is after every 20 batches I'm going to print the loss. But here is where the print the loss. But here is where the print the loss. But here is where the real uh training happens. So what I'm real uh training happens. So what I'm real uh training happens. So what I'm going to do is that I'm going to take going to do is that I'm going to take going to do is that I'm going to take every batch from my training iteration. every batch from my training iteration. every batch from my training iteration. I'm going to uh I'm going to uh I'm going to uh find the diffusion loss on this batch find the diffusion loss on this batch find the diffusion loss on this batch and then what I'm going to do is that uh and then what I'm going to do is that uh and then what I'm going to do is that uh one more thing along with the batch as I one more thing along with the batch as I one more thing along with the batch as I mentioned we also need to define the mentioned we also need to define the mentioned we also need to define the time step right so when you go to the time step right so when you go to the
- L17 @ 1055s
time step right so when you go to the diffusion loss function you'll see that diffusion loss function you'll see that diffusion loss function you'll see that uh where is the diffusion loss yeah when uh where is the diffusion loss yeah when uh where is the diffusion loss yeah when you see the diffusion loss function we you see the diffusion loss function we you see the diffusion loss function we also get this time step which is chosen also get this time step which is chosen also get this time step which is chosen uh randomly we need this time step uh randomly we need this time step uh randomly we need this time step because the corruption level depends on because the corruption level depends on because the corruption level depends on that time step Right. So if you go here, that time step Right. So if you go here, that time step Right. So if you go here, what's actually done is that we first what's actually done is that we first what's actually done is that we first choose a batch of the data. choose a batch of the data. choose a batch of the data. We pass we choose a batch. Then we pass We pass we choose a batch. Then we pass We pass we choose a batch. Then we pass the model and the batch to the diffusion the model and the batch to the diffusion the model and the batch to the diffusion loss function. In the diffusion loss loss function. In the diffusion loss loss function. In the diffusion loss function, we have by default chosen a function, we have by default chosen a function, we have by default chosen a time step. And then here we pass the time step. And then here we pass the time step. And then here we pass the input batch and the time step to our input batch and the time step to our input batch and the time step to our model. This model has been defined over model. This model has been defined over model. This model has been defined over here. here. here. This model has been defined over here. This model has been defined over here. This model has been defined over here. Now this thing which I'm telling right Now this thing which I'm telling right
- L17 @ 1099s
Now this thing which I'm telling right now the corrupted input sequence is now the corrupted input sequence is now the corrupted input sequence is passed through the model is exactly passed through the model is exactly passed through the model is exactly shown over here. The corrupted sequence shown over here. The corrupted sequence shown over here. The corrupted sequence is passed through the model. It goes is passed through the model. It goes is passed through the model. It goes through this entire model architecture through this entire model architecture through this entire model architecture and we get the loss. and we get the loss. and we get the loss. All of this is implemented over here. We All of this is implemented over here. We All of this is implemented over here. We get the loss and then we do this is the get the loss and then we do this is the get the loss and then we do this is the optimizing step optimizer step. Once we optimizing step optimizer step. Once we optimizing step optimizer step. Once we get the loss function then we go into get the loss function then we go into get the loss function then we go into this loop of uh this loop of uh this loop of uh uh yeah we go into this loop of finding uh yeah we go into this loop of finding uh yeah we go into this loop of finding the partial derivative of the loss then the partial derivative of the loss then the partial derivative of the loss then updating the parameters through the updating the parameters through the updating the parameters through the optimizer that's done over here we do optimizer that's done over here we do optimizer that's done over here we do optimizer step and we uh update the optimizer step and we uh update the optimizer step and we uh update the learning rate in the adm optimizer etc. learning rate in the adm optimizer etc. learning rate in the adm optimizer etc. So you run this right now and then you So you run this right now and then you So you run this right now and then you will see that the training process will see that the training process will see that the training process starts automatically. Yeah, see the training process has
- L17 @ 1151s
Yeah, see the training process has started. Now I'm on the L I think let's started. Now I'm on the L I think let's started. Now I'm on the L I think let's see which GPU I'm on. I'm on the L4 GPU see which GPU I'm on. I'm on the L4 GPU see which GPU I'm on. I'm on the L4 GPU which is actually not very fast. So the which is actually not very fast. So the which is actually not very fast. So the iterations are proceeding slowly. So iterations are proceeding slowly. So iterations are proceeding slowly. So here it's showing that it will take here it's showing that it will take here it's showing that it will take around 3 hours for me. Still it's quite around 3 hours for me. Still it's quite around 3 hours for me. Still it's quite good right for 60,000 iterations. If you good right for 60,000 iterations. If you good right for 60,000 iterations. If you are on T4 GPU it may take 8 to 9 hours. are on T4 GPU it may take 8 to 9 hours. are on T4 GPU it may take 8 to 9 hours. One thing which I want to point out to One thing which I want to point out to One thing which I want to point out to all of you is that although the all of you is that although the all of you is that although the inference of a diffusion model is inference of a diffusion model is inference of a diffusion model is faster, training actually is a bit faster, training actually is a bit faster, training actually is a bit slower. So slower. So slower. So if you scroll to the top, I've actually if you scroll to the top, I've actually if you scroll to the top, I've actually compared the training speeds of a compared the training speeds of a compared the training speeds of a diffusion language model and the auto diffusion language model and the auto diffusion language model and the auto reggressive model. So here if you see um reggressive model. So here if you see um reggressive model. So here if you see um in u the convergence speed so if you see in u the convergence speed so if you see
- L17 @ 1197s
in u the convergence speed so if you see the auto reggressive model in 15,000 the auto reggressive model in 15,000 the auto reggressive model in 15,000 epochs I go from a loss of 8 to a loss epochs I go from a loss of 8 to a loss epochs I go from a loss of 8 to a loss of 2.6. So almost the whole model has of 2.6. So almost the whole model has of 2.6. So almost the whole model has converged in 15,000 epochs itself. converged in 15,000 epochs itself. converged in 15,000 epochs itself. Whereas in diffusion LLM in 15,000 steps Whereas in diffusion LLM in 15,000 steps Whereas in diffusion LLM in 15,000 steps I go from a loss of 10 to just a loss of I go from a loss of 10 to just a loss of I go from a loss of 10 to just a loss of 5.6. 5.6. 5.6. So the model actually converges slowly. So the model actually converges slowly. So the model actually converges slowly. Diffusion language models take time to Diffusion language models take time to converge and that's a trait trait of converge and that's a trait trait of converge and that's a trait trait of diffusion models always they are a bit diffusion models always they are a bit diffusion models always they are a bit slower slower slower because of the mask predictions being because of the mask predictions being because of the mask predictions being involved at and we are looking at it involved at and we are looking at it involved at and we are looking at it from both sides so the attention matrix from both sides so the attention matrix from both sides so the attention matrix is a bit larger etc. is a bit larger etc. is a bit larger etc. Um so diffusion language models in my Um so diffusion language models in my Um so diffusion language models in my experience train a bit slowly compared experience train a bit slowly compared experience train a bit slowly compared to auto reggressive models but as you to auto reggressive models but as you to auto reggressive models but as you can see the training has started over can see the training has started over
- L17 @ 1244s
can see the training has started over here and we can print the loss we can here and we can print the loss we can here and we can print the loss we can print the learning rate etc. While the print the learning rate etc. While the print the learning rate etc. While the training is going on, I'll tell you the training is going on, I'll tell you the training is going on, I'll tell you the next steps. What we'll do after this next steps. What we'll do after this next steps. What we'll do after this training is that we then just have to training is that we then just have to training is that we then just have to proceed with generation. Right? So in proceed with generation. Right? So in proceed with generation. Right? So in generation, as I've mentioned already, generation, as I've mentioned already, generation, as I've mentioned already, we just carry out sequential unmasking. we just carry out sequential unmasking. we just carry out sequential unmasking. So here is a animation of generation or So here is a animation of generation or So here is a animation of generation or dnoising which we have seen. We start dnoising which we have seen. We start dnoising which we have seen. We start with masks and then sequentially unmask with masks and then sequentially unmask with masks and then sequentially unmask and uh we unmask tokens with the highest and uh we unmask tokens with the highest and uh we unmask tokens with the highest confidence. That's the general rule, confidence. That's the general rule, confidence. That's the general rule, right? So this is the diffusion generate right? So this is the diffusion generate right? So this is the diffusion generate code where we start with all the masks code where we start with all the masks code where we start with all the masks and then and then and then the unmasking is actually done for the the unmasking is actually done for the the unmasking is actually done for the maximum confidence. That's the entire maximum confidence. That's the entire maximum confidence. That's the entire logic which has been implemented over logic which has been implemented over logic which has been implemented over here. I am not going to go through this
- L17 @ 1294s
here. I am not going to go through this here. I am not going to go through this code in detail because it just code in detail because it just code in detail because it just implements the same thing, right? We we implements the same thing, right? We we implements the same thing, right? We we get the confidence scores and then we get the confidence scores and then we get the confidence scores and then we unmask the token ids with the maximum unmask the token ids with the maximum unmask the token ids with the maximum confidence. The reason I spent so much confidence. The reason I spent so much confidence. The reason I spent so much time on the whiteboard is because going time on the whiteboard is because going time on the whiteboard is because going through code is I found that if I go through code is I found that if I go through code is I found that if I go through this code generally a bit through this code generally a bit through this code generally a bit difficult for folks to understand right difficult for folks to understand right difficult for folks to understand right but if I take an actual example and show but if I take an actual example and show but if I take an actual example and show you how unmasking is done step by step you how unmasking is done step by step you how unmasking is done step by step you can relate this very easily to the you can relate this very easily to the you can relate this very easily to the code code code right so here's here actually the same right so here's here actually the same right so here's here actually the same unmasking uh procedure is happening unmasking uh procedure is happening unmasking uh procedure is happening right now let me do this okay let me right now let me do this okay let me right now let me do this okay let me stop this for a moment and Let's run stop this for a moment and Let's run stop this for a moment and Let's run with whatever I have. with whatever I have. with whatever I have. Uh so this is run right now and uh here
- L17 @ 1339s
Uh so this is run right now and uh here Uh so this is run right now and uh here I'm just seeing a sample inference. Once I'm just seeing a sample inference. Once I'm just seeing a sample inference. Once upon a time is the input. I'll see that upon a time is the input. I'll see that upon a time is the input. I'll see that the inference is very bad right now the inference is very bad right now the inference is very bad right now because I I just stopped at uh because I I just stopped at uh because I I just stopped at uh how many steps just thousand steps but I how many steps just thousand steps but I how many steps just thousand steps but I just want to show you the workflow. just want to show you the workflow. just want to show you the workflow. Right? So technically we have finished Right? So technically we have finished Right? So technically we have finished coding the diffusion language model from coding the diffusion language model from coding the diffusion language model from scratch here. But I have just added a scratch here. But I have just added a scratch here. But I have just added a simple code to render a terminal style simple code to render a terminal style simple code to render a terminal style diff terminal style GIF. Right. Uh I diff terminal style GIF. Right. Uh I diff terminal style GIF. Right. Uh I think there is an error over here think there is an error over here think there is an error over here probably because we have not run till uh probably because we have not run till uh probably because we have not run till uh till completion and that's I think till completion and that's I think till completion and that's I think that's fine that's fine that's fine but no such file or directory as no such file or directory as inference.jif. here also there is an error which says here also there is an error which says cannot import. cannot import. cannot import. I think this is probably because the I think this is probably because the I think this is probably because the iterations did not finish. But if you iterations did not finish. But if you
- L17 @ 1407s
iterations did not finish. But if you actually run um actually run um actually run um 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iterations over here, you will generally see that over here, you will generally see that over here, you will generally see that the code finishes till completion. the code finishes till completion. the code finishes till completion. Um and then you just have to export this Um and then you just have to export this Um and then you just have to export this GIF and you have to save this GIF. So GIF and you have to save this GIF. So GIF and you have to save this GIF. So I've just written I also have another I've just written I also have another I've just written I also have another font here if you want to export the GIF font here if you want to export the GIF font here if you want to export the GIF in a slightly cooler format. But if you in a slightly cooler format. But if you in a slightly cooler format. But if you export the normal GIF, it will look export the normal GIF, it will look export the normal GIF, it will look something like this. So this is trained something like this. So this is trained something like this. So this is trained for 100,000 iterations, right? And after for 100,000 iterations, right? And after for 100,000 iterations, right? And after 100,000 iterations, you will start to 100,000 iterations, you will start to 100,000 iterations, you will start to see that the output starts becoming see that the output starts becoming see that the output starts becoming coherent. So the output is that once coherent. So the output is that once coherent. So the output is that once upon a time, there was a little girl upon a time, there was a little girl upon a time, there was a little girl named Lily. She loved to play with her named Lily. She loved to play with her named Lily. She loved to play with her toys and put them in a room. One day, toys and put them in a room. One day, toys and put them in a room. One day, Lily's mom asked her to help clean up
- L17 @ 1449s
Lily's mom asked her to help clean up Lily's mom asked her to help clean up her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom put them in a box. Isn't this amazing? put them in a box. Isn't this amazing? put them in a box. Isn't this amazing? This entirely coherent text generated This entirely coherent text generated This entirely coherent text generated through a pure diffusion model which we through a pure diffusion model which we through a pure diffusion model which we have built from scratch for 100,000 have built from scratch for 100,000 have built from scratch for 100,000 iterations through the exact same code iterations through the exact same code iterations through the exact same code which I have shown to you right now. And which I have shown to you right now. And which I have shown to you right now. And once you run this code, once you once you run this code, once you once you run this code, once you replicate this code, you'll also see replicate this code, you'll also see replicate this code, you'll also see this for yourself. The only requirement this for yourself. The only requirement this for yourself. The only requirement which you'll need is that you'll need to which you'll need is that you'll need to which you'll need is that you'll need to run at least for 50,000 or 60,000 run at least for 50,000 or 60,000 run at least for 50,000 or 60,000 iterations and you'll need a A100 GPU, iterations and you'll need a A100 GPU, iterations and you'll need a A100 GPU, H100 or if you have time, you can even H100 or if you have time, you can even H100 or if you have time, you can even do it on a T4 GPU or an L4 GPU. What I do it on a T4 GPU or an L4 GPU. What I do it on a T4 GPU or an L4 GPU. What I want to show you is that uh want to show you is that uh want to show you is that uh we can also do this on runpod. So if you we can also do this on runpod. So if you
- L17 @ 1490s
we can also do this on runpod. So if you have credentials or if you have some have credentials or if you have some have credentials or if you have some budget even $6 to $7 what I'll suggest budget even $6 to $7 what I'll suggest budget even $6 to $7 what I'll suggest is go to runpod click on sign in over is go to runpod click on sign in over is go to runpod click on sign in over here here here and uh if you sign in through Google uh what you can do is that so I have uh what you can do is that so I have around $100 here you can actually select around $100 here you can actually select around $100 here you can actually select a GPU which is very fast. So there's a GPU which is very fast. So there's a GPU which is very fast. So there's something running here right now. If I something running here right now. If I something running here right now. If I click on deploy, let's say I choose this click on deploy, let's say I choose this click on deploy, let's say I choose this B200 GPU which is $519 per hour, but it B200 GPU which is $519 per hour, but it B200 GPU which is $519 per hour, but it is very it has 180 GB of VRAM. H200 has is very it has 180 GB of VRAM. H200 has is very it has 180 GB of VRAM. H200 has 1441 GB of VRAM. Right now I was on L40, 1441 GB of VRAM. Right now I was on L40, 1441 GB of VRAM. Right now I was on L40, I think, which is 48 GB. Not too bad. I think, which is 48 GB. Not too bad. I think, which is 48 GB. Not too bad. But let's choose B200 But let's choose B200 But let's choose B200 and let me do let's say container disk and let me do let's say container disk and let me do let's say container disk of 200 GB and uh volume disk of 200 GB. of 200 GB and uh volume disk of 200 GB. of 200 GB and uh volume disk of 200 GB. I'll set overrides. Oops. I think it was
- L17 @ 1545s
I'll set overrides. Oops. I think it was I'll set overrides. Oops. I think it was 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set overrides over here. And then I'll click overrides over here. And then I'll click overrides over here. And then I'll click on deploy on demand. Um when you click on deploy on demand, Um when you click on deploy on demand, you'll see that uh you'll see that uh you'll see that uh it will start to spin up. Okay. So now it's still waiting for Okay. So now it's still waiting for connection. So probably I should just uh connection. So probably I should just uh connection. So probably I should just uh uh uh uh Yeah. So Jupyter Lab is ready. Ready. So Yeah. So Jupyter Lab is ready. Ready. So Yeah. So Jupyter Lab is ready. Ready. So just click on this Jupyter lab just click on this Jupyter lab just click on this Jupyter lab and then it will open this Jupyter lab and then it will open this Jupyter lab and then it will open this Jupyter lab notebook environment for you and uh once this Jupyter lab environment and uh once this Jupyter lab environment is opened for you what you can do is is opened for you what you can do is is opened for you what you can do is that you can copy paste the code into that you can copy paste the code into that you can copy paste the code into this environment. [snorts] So I click on this Python 3 kernel right So I click on this Python 3 kernel right now and open this. So you can see on the now and open this. So you can see on the now and open this. So you can see on the left hand side left hand side left hand side um let me get my code and I'll upload it um let me get my code and I'll upload it um let me get my code and I'll upload it on the left hand side over here.
- L19 @ 2s
All right, everyone. So, this brings us All right, everyone. So, this brings us to the end of this diffusion language to the end of this diffusion language to the end of this diffusion language model learning journey model learning journey model learning journey which I went on. which I went on. which I went on. And ultimately, we built a diffusion And ultimately, we built a diffusion And ultimately, we built a diffusion language model fully from scratch. We language model fully from scratch. We language model fully from scratch. We understood the nuts and bolts of how the understood the nuts and bolts of how the understood the nuts and bolts of how the architecture is assembled. architecture is assembled. architecture is assembled. The idea was not to make a 5-minute or The idea was not to make a 5-minute or The idea was not to make a 5-minute or 10-minute tutorial, but to show you 10-minute tutorial, but to show you 10-minute tutorial, but to show you everything from scratch to open the everything from scratch to open the everything from scratch to open the black box so that you can do research on black box so that you can do research on black box so that you can do research on top of this, which means that you can top of this, which means that you can top of this, which means that you can vary different hyper parameters, run vary different hyper parameters, run vary different hyper parameters, run ablation studies, maybe build your first ablation studies, maybe build your first ablation studies, maybe build your first open-source diffusion language model, open-source diffusion language model, open-source diffusion language model, which is as good as Gwen. which is as good as Gwen. which is as good as Gwen. All of these things are possible now All of these things are possible now All of these things are possible now once this black box is opened and you once this black box is opened and you once this black box is opened and you realize realize realize exactly how diffusion models work from exactly how diffusion models work from exactly how diffusion models work from scratch. Okay. I know this tutorial is scratch. Okay. I know this tutorial is scratch. Okay. I know this tutorial is not completed. not completed. not completed. I've not shown supervised fine-tuning,
- L19 @ 51s
I've not shown supervised fine-tuning, I've not shown supervised fine-tuning, which you can do on your own since at which you can do on your own since at which you can do on your own since at different sections of this video, I've different sections of this video, I've different sections of this video, I've also shown you how to learn things on also shown you how to learn things on also shown you how to learn things on your own. your own. your own. So, SFT is a part which you can do on So, SFT is a part which you can do on So, SFT is a part which you can do on your own. your own. And I hope hope that I will cover some And I hope hope that I will cover some And I hope hope that I will cover some more things which I did not get more things which I did not get more things which I did not get time to cover. I hope that I will cover time to cover. I hope that I will cover time to cover. I hope that I will cover some more mathematical aspects, some more mathematical aspects, some more mathematical aspects, but I could not fit it in this tutorial. but I could not fit it in this tutorial. but I could not fit it in this tutorial. So, that is also something which someone So, that is also something which someone So, that is also something which someone wants to dive deeper mathematically, you wants to dive deeper mathematically, you wants to dive deeper mathematically, you can feel free to do that, of course. can feel free to do that, of course. can feel free to do that, of course. I also wanted to uh take I also wanted to uh take I also wanted to uh take for example, this fine-web-edu fine-web-edu dataset and train a fine-web-edu dataset and train a diffusion model on top of it. I don't diffusion model on top of it. I don't diffusion model on top of it. I don't think I've not seen this being done in think I've not seen this being done in think I've not seen this being done in an open-source way. So, you can be the an open-source way. So, you can be the
- L19 @ 100s
an open-source way. So, you can be the first one to do this. The same pipeline first one to do this. The same pipeline first one to do this. The same pipeline which we have built for small language which we have built for small language which we have built for small language models for the tiny stories dataset, you models for the tiny stories dataset, you models for the tiny stories dataset, you can do it for the fine-web-edu dataset can do it for the fine-web-edu dataset can do it for the fine-web-edu dataset and that will be an amazing and an and that will be an amazing and an and that will be an amazing and an awesome project. awesome project. awesome project. So, the goal at Visual AI is always to So, the goal at Visual AI is always to So, the goal at Visual AI is always to show you things from scratch, um make show you things from scratch, um make show you things from scratch, um make sure foundations are built and at the sure foundations are built and at the sure foundations are built and at the same time inculcate a research mindset. same time inculcate a research mindset. same time inculcate a research mindset. Thanks a lot, everyone. I hope you found Thanks a lot, everyone. I hope you found Thanks a lot, everyone. I hope you found this tutorial useful. this tutorial useful. this tutorial useful. And I hope many of you get into And I hope many of you get into And I hope many of you get into diffusion language models as a research diffusion language models as a research diffusion language models as a research topic because this can be the next big topic because this can be the next big topic because this can be the next big thing thing thing apart from autoregressive language apart from autoregressive language apart from autoregressive language models. Thanks.
- L2 @ 3s
Hello everyone. This is Dr. Raj Dandeek Hello everyone. This is Dr. Raj Dandeek and uh today I'm very excited to present and uh today I'm very excited to present and uh today I'm very excited to present this learning material to all of you. this learning material to all of you. this learning material to all of you. I've been thinking about this for quite I've been thinking about this for quite I've been thinking about this for quite some time now and uh some time now and uh some time now and uh just like many other things. Um when I just like many other things. Um when I just like many other things. Um when I started thinking about diffusion started thinking about diffusion started thinking about diffusion language models, I thought it was very language models, I thought it was very language models, I thought it was very difficult and uh the initial phase took difficult and uh the initial phase took difficult and uh the initial phase took a very long amount of time or a large a very long amount of time or a large a very long amount of time or a large amount of time for me to really amount of time for me to really amount of time for me to really understand what's going on. But uh the understand what's going on. But uh the understand what's going on. But uh the final implementation final implementation final implementation and uh building it fully from scratch, and uh building it fully from scratch, and uh building it fully from scratch, comparing it with auto reggressive comparing it with auto reggressive comparing it with auto reggressive models was fairly straightforward and models was fairly straightforward and models was fairly straightforward and simple. simple. simple. So I could just show you that fast thing So I could just show you that fast thing So I could just show you that fast thing which I did towards the end where when which I did towards the end where when which I did towards the end where when things became clearer to me how I things became clearer to me how I things became clearer to me how I implemented language based diffusion implemented language based diffusion implemented language based diffusion models from scratch. But that's not how
- L2 @ 61s
models from scratch. But that's not how models from scratch. But that's not how I want this course or this learning I want this course or this learning I want this course or this learning material to be. I want you to take I material to be. I want you to take I material to be. I want you to take I want to take you all through the journey want to take you all through the journey want to take you all through the journey which I went through because the initial which I went through because the initial which I went through because the initial uh uh uh learning was difficult for me. Diffusion learning was difficult for me. Diffusion learning was difficult for me. Diffusion is not that simple. It's not that is not that simple. It's not that is not that simple. It's not that straightforward. So to understand straightforward. So to understand straightforward. So to understand diffusion and to get to that stage where diffusion and to get to that stage where diffusion and to get to that stage where I could build a diffusion based language I could build a diffusion based language I could build a diffusion based language model from scratch was not easy. So this model from scratch was not easy. So this model from scratch was not easy. So this is not going to be like those is not going to be like those is not going to be like those traditional courses or learning traditional courses or learning traditional courses or learning materials where we directly start from materials where we directly start from materials where we directly start from the the the uh fundamentals directly start from the uh fundamentals directly start from the uh fundamentals directly start from the content basically. But I'm I'll take you content basically. But I'm I'll take you content basically. But I'm I'll take you through a journey. I'll take you through through a journey. I'll take you through through a journey. I'll take you through my story of how I stumbled upon my story of how I stumbled upon my story of how I stumbled upon diffusion models, how I was initially diffusion models, how I was initially diffusion models, how I was initially scared of them, then how I learned the
- L2 @ 109s
scared of them, then how I learned the scared of them, then how I learned the subject, how I mastered the subject now subject, how I mastered the subject now subject, how I mastered the subject now so that you can follow along this so that you can follow along this so that you can follow along this journey with me. That's my intention journey with me. That's my intention journey with me. That's my intention with this uh um course or learning with this uh um course or learning with this uh um course or learning material. I want to I want this to be a material. I want to I want this to be a material. I want to I want this to be a journey for all of you so that you feel journey for all of you so that you feel journey for all of you so that you feel relatable, right? Because whenever we relatable, right? Because whenever we relatable, right? Because whenever we start learning a new subject, it's not start learning a new subject, it's not start learning a new subject, it's not easy. So I felt that if I documented my easy. So I felt that if I documented my easy. So I felt that if I documented my learning process or journey all of you learning process or journey all of you learning process or journey all of you will feel that okay you are also in a will feel that okay you are also in a will feel that okay you are also in a similar boat and this will give you more similar boat and this will give you more similar boat and this will give you more confidence to learn new subjects on your confidence to learn new subjects on your confidence to learn new subjects on your own. That being said at the end of this own. That being said at the end of this own. That being said at the end of this journey you will build a language model journey you will build a language model journey you will build a language model uh a diffusion based small language uh a diffusion based small language uh a diffusion based small language model fully from scratch. You'll compare model fully from scratch. You'll compare
- L2 @ 147s
model fully from scratch. You'll compare it with the auto reggressive language it with the auto reggressive language it with the auto reggressive language models. So it's an amazing exercise or models. So it's an amazing exercise or models. So it's an amazing exercise or it'll be a beautiful conclusion for all it'll be a beautiful conclusion for all it'll be a beautiful conclusion for all of you. As with all of the videos which of you. As with all of the videos which of you. As with all of the videos which we record at Vijara, this is going to be we record at Vijara, this is going to be we record at Vijara, this is going to be long form content. Uh these are not this long form content. Uh these are not this long form content. Uh these are not this is not a short content. I'll explain is not a short content. I'll explain is not a short content. I'll explain everything in depth. Everything from everything in depth. Everything from everything in depth. Everything from scratch and from first principles. If scratch and from first principles. If scratch and from first principles. If you already know some aspect of what I'm you already know some aspect of what I'm you already know some aspect of what I'm about to explain. Uh you can skip that about to explain. Uh you can skip that about to explain. Uh you can skip that particular portion and move to the next particular portion and move to the next particular portion and move to the next part. But for a complete understanding, part. But for a complete understanding, part. But for a complete understanding, I recommend you to watch it all. Thanks I recommend you to watch it all. Thanks I recommend you to watch it all. Thanks again everyone and let's get started.
- L3 @ 2s
Let's get started. Let's get started. Initially, as I mentioned, I'll be Initially, as I mentioned, I'll be Initially, as I mentioned, I'll be taking you through my journey, right? taking you through my journey, right? taking you through my journey, right? So, I'll be taking you through the exact So, I'll be taking you through the exact So, I'll be taking you through the exact timeline which I followed timeline which I followed timeline which I followed um and how I made this series on um and how I made this series on um and how I made this series on diffusion based language models. So diffusion based language models. So diffusion based language models. So honestly I had seen the this tweet by honestly I had seen the this tweet by honestly I had seen the this tweet by Andre Karpati first where he mentioned Andre Karpati first where he mentioned Andre Karpati first where he mentioned that this is interesting as first as a that this is interesting as first as a that this is interesting as first as a first large diffusion based LLM. Most of first large diffusion based LLM. Most of first large diffusion based LLM. Most of the LLMs you have been seeing are clones the LLMs you have been seeing are clones the LLMs you have been seeing are clones as far as the core modeling approach as far as the core modeling approach as far as the core modeling approach goes. They are all trained auto goes. They are all trained auto goes. They are all trained auto reggressively that is predicting tokens reggressively that is predicting tokens reggressively that is predicting tokens from left to right. Diffusion is from left to right. Diffusion is from left to right. Diffusion is different. It doesn't go left to right different. It doesn't go left to right different. It doesn't go left to right but it goes all at once. You start with but it goes all at once. You start with but it goes all at once. You start with noise and gradually den noiseise into a noise and gradually den noiseise into a noise and gradually den noiseise into a token stream. So when I read this I token stream. So when I read this I token stream. So when I read this I thought that okay this is something
- L3 @ 51s
thought that okay this is something thought that okay this is something cool. I had thought about diffusion only cool. I had thought about diffusion only cool. I had thought about diffusion only for images and videos but someone seems for images and videos but someone seems for images and videos but someone seems to have made it for language. to have made it for language. to have made it for language. But honestly there is one GIF or one But honestly there is one GIF or one But honestly there is one GIF or one video which when I saw it really blew my video which when I saw it really blew my video which when I saw it really blew my mind and then I knew I had to learn this mind and then I knew I had to learn this mind and then I knew I had to learn this material and make a course on it. Here's material and make a course on it. Here's material and make a course on it. Here's this GIF or here's the video. If you see this GIF or here's the video. If you see this GIF or here's the video. If you see on the left hand side we have an auto on the left hand side we have an auto on the left hand side we have an auto reggressive LLM over here and on the reggressive LLM over here and on the reggressive LLM over here and on the right hand side we have diffusion. So right hand side we have diffusion. So right hand side we have diffusion. So let's play it from the start. Yeah, take let's play it from the start. Yeah, take let's play it from the start. Yeah, take a look on the left hand side first how a look on the left hand side first how a look on the left hand side first how tokens appear sequentially one after the tokens appear sequentially one after the tokens appear sequentially one after the other. That's how LLMs have been other. That's how LLMs have been other. That's how LLMs have been operating naturally here. I'll zoom zoom operating naturally here. I'll zoom zoom operating naturally here. I'll zoom zoom over here. So take a look here first over here. So take a look here first
- L3 @ 93s
over here. So take a look here first only on the left hand side. Okay. And only on the left hand side. Okay. And only on the left hand side. Okay. And now when we restart the video then take now when we restart the video then take now when we restart the video then take a look at on the right hand side. Right? a look at on the right hand side. Right? a look at on the right hand side. Right? Take a look at the right hand side and Take a look at the right hand side and Take a look at the right hand side and how tokens appear here. tokens don't how tokens appear here. tokens don't how tokens appear here. tokens don't appear one after the other but they just appear one after the other but they just appear one after the other but they just fill the screen right just how images fill the screen right just how images fill the screen right just how images are created from noise right that's are created from noise right that's are created from noise right that's called dn noising in images so here's called dn noising in images so here's called dn noising in images so here's how the dnoising process actually looks how the dnoising process actually looks how the dnoising process actually looks like like like dnoising images and if I say diffusion dnoising images and if I say diffusion dnoising images and if I say diffusion right many of you might have seen this right many of you might have seen this right many of you might have seen this right when we want to make or we want to right when we want to make or we want to right when we want to make or we want to build images one way to do this is build images one way to do this is build images one way to do this is diffusion right so you take an image you diffusion right so you take an image you diffusion right so you take an image you keep keep on adding noise to it and then keep keep on adding noise to it and then keep keep on adding noise to it and then you learn how much noise is added. So
- L3 @ 138s
you learn how much noise is added. So you learn how much noise is added. So later you are able to take noise and later you are able to take noise and later you are able to take noise and then build images from it. People are then build images from it. People are then build images from it. People are familiar with this and this now comes familiar with this and this now comes familiar with this and this now comes naturally to people. But to think about naturally to people. But to think about naturally to people. But to think about diffusion for language it's awesome diffusion for language it's awesome diffusion for language it's awesome right? We start from let's say noise and right? We start from let's say noise and right? We start from let's say noise and then suddenly amazing text or meaningful then suddenly amazing text or meaningful then suddenly amazing text or meaningful text starts appearing on the screen. Now text starts appearing on the screen. Now text starts appearing on the screen. Now compare the left hand side with the compare the left hand side with the compare the left hand side with the right hand side. On the left, one token right hand side. On the left, one token right hand side. On the left, one token appears at a time. On the right, appears at a time. On the right, appears at a time. On the right, everything fills the screen at once. everything fills the screen at once. everything fills the screen at once. This GIF or this video should stay in This GIF or this video should stay in This GIF or this video should stay in your mind because we are going to build your mind because we are going to build your mind because we are going to build this. This will serve as the baseline this. This will serve as the baseline this. This will serve as the baseline material for us as the end goal which we material for us as the end goal which we material for us as the end goal which we want to reach. We want to make a want to reach. We want to make a want to reach. We want to make a diffusion model which fills the screen diffusion model which fills the screen
- L3 @ 187s
diffusion model which fills the screen like this and it's not one token at a like this and it's not one token at a like this and it's not one token at a time. So when I saw this GIF, I time. So when I saw this GIF, I time. So when I saw this GIF, I immediately thought that the left hand immediately thought that the left hand immediately thought that the left hand side where one token is predicted at a side where one token is predicted at a side where one token is predicted at a time is pretty inefficient, right? Why time is pretty inefficient, right? Why time is pretty inefficient, right? Why should only one token be predicted at a should only one token be predicted at a should only one token be predicted at a time? Why can't we just fill the whole time? Why can't we just fill the whole time? Why can't we just fill the whole screen? screen? screen? Uh and tokens come in from left, right, Uh and tokens come in from left, right, Uh and tokens come in from left, right, anywhere. anywhere. anywhere. When I saw this GIF, that's when I When I saw this GIF, that's when I When I saw this GIF, that's when I started to get an interest in the started to get an interest in the started to get an interest in the concept of diffusion. basically concept of diffusion. basically concept of diffusion. basically and u I knew that I had to build this and u I knew that I had to build this and u I knew that I had to build this from scratch. from scratch. from scratch. So to give you a bit of context I So to give you a bit of context I So to give you a bit of context I already had built this series of already had built this series of already had built this series of building a large language model from building a large language model from building a large language model from scratch which was fully auto reggressive scratch which was fully auto reggressive scratch which was fully auto reggressive which means that it predicts one token which means that it predicts one token which means that it predicts one token at a time. Um this has now become very
- L3 @ 239s
at a time. Um this has now become very at a time. Um this has now become very popular on YouTube and it's one of the popular on YouTube and it's one of the popular on YouTube and it's one of the go-to materials to learn how to build a go-to materials to learn how to build a go-to materials to learn how to build a language model from scratch. But this is language model from scratch. But this is language model from scratch. But this is auto reggressive which means it's like auto reggressive which means it's like auto reggressive which means it's like something on the left hand side which we something on the left hand side which we something on the left hand side which we saw over here. saw over here. saw over here. Uh it's an auto reggressive model right? Uh it's an auto reggressive model right? Uh it's an auto reggressive model right? So I wanted to build something similar So I wanted to build something similar So I wanted to build something similar now for diffusion models. now for diffusion models. now for diffusion models. And uh last year I had conducted a live And uh last year I had conducted a live And uh last year I had conducted a live lecture on when I was teaching this lecture on when I was teaching this lecture on when I was teaching this generative AI fundamentals course. Uh I generative AI fundamentals course. Uh I generative AI fundamentals course. Uh I had conducted a lecture on diffusion had conducted a lecture on diffusion had conducted a lecture on diffusion models for image generation. And these models for image generation. And these models for image generation. And these are the notes which I have prepared for are the notes which I have prepared for are the notes which I have prepared for the lecture. the lecture. the lecture. So before we all start building So before we all start building So before we all start building diffusion based LLMs, we first need to diffusion based LLMs, we first need to diffusion based LLMs, we first need to understand what exactly is diffusion. understand what exactly is diffusion. understand what exactly is diffusion. And uh I'm going to first walk you And uh I'm going to first walk you
- L3 @ 287s
And uh I'm going to first walk you through diffusion in an intuitive way through diffusion in an intuitive way through diffusion in an intuitive way because I I strongly think that all of because I I strongly think that all of because I I strongly think that all of you really need to understand the you really need to understand the you really need to understand the uh intuition behind this diffusion uh intuition behind this diffusion uh intuition behind this diffusion process. Even if you don't understand process. Even if you don't understand process. Even if you don't understand the mathematics, it's fine. I don't want the mathematics, it's fine. I don't want the mathematics, it's fine. I don't want you to be intimidated by the mathematics you to be intimidated by the mathematics you to be intimidated by the mathematics of diffusion but at least for now we of diffusion but at least for now we of diffusion but at least for now we need to understand the intuition behind need to understand the intuition behind need to understand the intuition behind diffusion. So basically when I say diffusion. So basically when I say diffusion. So basically when I say intuition we all need to understand how intuition we all need to understand how intuition we all need to understand how what is diffusion why it how it works what is diffusion why it how it works what is diffusion why it how it works for image generation and then we'll for image generation and then we'll for image generation and then we'll apply the similar concepts to basically apply the similar concepts to basically apply the similar concepts to basically text right so first I want to remove text right so first I want to remove text right so first I want to remove this fear of diffusion which I had I this fear of diffusion which I had I this fear of diffusion which I had I thought whenever I thought of diffusion thought whenever I thought of diffusion thought whenever I thought of diffusion I thought it's it involves complex I thought it's it involves complex I thought it's it involves complex probability and mathematical probability and mathematical probability and mathematical formulations will which I'll not so I'll formulations will which I'll not so I'll
- L3 @ 336s
formulations will which I'll not so I'll not take that route when I explain this not take that route when I explain this not take that route when I explain this to you I'll explain this to you in an to you I'll explain this to you in an to you I'll explain this to you in an intuitive So that all of you have very intuitive So that all of you have very intuitive So that all of you have very nice mental model nice mental model nice mental model at the end of this uh journey. So when at the end of this uh journey. So when at the end of this uh journey. So when we conclude and when we actually build we conclude and when we actually build we conclude and when we actually build a small diffusion model, a small a small diffusion model, a small a small diffusion model, a small language based diffusion model from language based diffusion model from language based diffusion model from scratch, it will look something like scratch, it will look something like scratch, it will look something like this. So you are going to build this. So you are going to build this. So you are going to build something like this at the end of this something like this at the end of this something like this at the end of this learning journey or at the end of this learning journey or at the end of this learning journey or at the end of this course. course. course. This has been built purely locally by me This has been built purely locally by me This has been built purely locally by me u fully from scratch. I have not relied u fully from scratch. I have not relied u fully from scratch. I have not relied on any library nothing whatsoever. This on any library nothing whatsoever. This on any library nothing whatsoever. This GIF which is a diffusion based small GIF which is a diffusion based small GIF which is a diffusion based small language model is is producing coherent language model is is producing coherent language model is is producing coherent sentences as you can see and it's built sentences as you can see and it's built sentences as you can see and it's built fully from scratch. You'll be doing this
- L3 @ 383s
fully from scratch. You'll be doing this fully from scratch. You'll be doing this if you follow you follow along with me if you follow you follow along with me if you follow you follow along with me throughout this journey. throughout this journey. throughout this journey. Right? So let's actually get uh get Right? So let's actually get uh get Right? So let's actually get uh get started. started. started. and uh and uh and uh let me connect my lecture notes. All right. So, okay. So, first we need All right. So, okay. So, first we need to understand to understand to understand image generation and uh what exactly are image generation and uh what exactly are image generation and uh what exactly are diffusion models and then we'll diffusion models and then we'll diffusion models and then we'll understand how and why diffusion models understand how and why diffusion models understand how and why diffusion models exactly work. exactly work. exactly work. So, let's take a look at this video So, let's take a look at this video So, let's take a look at this video first. I'm going to play this video first. I'm going to play this video first. I'm going to play this video right now. You'll see that a small dye or a small You'll see that a small dye or a small chemical is inserted into this liquid chemical is inserted into this liquid chemical is inserted into this liquid and then we are kind of and then we are kind of and then we are kind of uh steering the liquid. We are adding uh steering the liquid. We are adding uh steering the liquid. We are adding motion. motion. motion. So you'll see that as the liquid is So you'll see that as the liquid is So you'll see that as the liquid is moved the D basically gets spread out moved the D basically gets spread out moved the D basically gets spread out the D gets diffused. the D gets diffused. the D gets diffused. See See See this is like adding noise. Let's say you this is like adding noise. Let's say you this is like adding noise. Let's say you take an image you keep on adding noise take an image you keep on adding noise
- L3 @ 463s
take an image you keep on adding noise to it. Right? You keep on adding noise to it. Right? You keep on adding noise to it. Right? You keep on adding noise to an image until it becomes fully to an image until it becomes fully to an image until it becomes fully blurred. blurred. blurred. That's the forward noising process or That's the forward noising process or That's the forward noising process or just noising. just noising. just noising. And uh in diffusion what we actually do And uh in diffusion what we actually do And uh in diffusion what we actually do is that is that is that we have to approximate the reverse we have to approximate the reverse we have to approximate the reverse process which means that if we start process which means that if we start process which means that if we start from pure noise can we reconstruct the from pure noise can we reconstruct the from pure noise can we reconstruct the original image back from the pure noise. original image back from the pure noise. original image back from the pure noise. So the whole idea is that if the forward So the whole idea is that if the forward So the whole idea is that if the forward process is done slowly process is done slowly process is done slowly and uh in that forward process and uh in that forward process and uh in that forward process if we learn how much noise is added at if we learn how much noise is added at if we learn how much noise is added at each step that same information can be each step that same information can be each step that same information can be used to go back and to generate a clean used to go back and to generate a clean used to go back and to generate a clean image from noise. So see now here we are image from noise. So see now here we are image from noise. So see now here we are going in the reverse direction now right going in the reverse direction now right going in the reverse direction now right and the goal is to recover back the and the goal is to recover back the and the goal is to recover back the clean image. clean image.
- L3 @ 523s
clean image. So now you'll see that as we are going So now you'll see that as we are going So now you'll see that as we are going in the reverse direction slowly the dye in the reverse direction slowly the dye in the reverse direction slowly the dye or the color which we have added will or the color which we have added will or the color which we have added will come back to its original configuration. You'll already see that it's getting You'll already see that it's getting back. See here back. See here back. See here the D has kind of come back to its the D has kind of come back to its the D has kind of come back to its original configuration over here. original configuration over here. original configuration over here. See See this is beautiful right? We started from this is beautiful right? We started from this is beautiful right? We started from this similar configuration. this similar configuration. this similar configuration. Um like this Um like this Um like this we added small amount of noise and then we added small amount of noise and then we added small amount of noise and then we reversed the process and got back we reversed the process and got back we reversed the process and got back this exact same configuration. this exact same configuration. this exact same configuration. The reason this recovery was possible is The reason this recovery was possible is The reason this recovery was possible is because the noise was injected in a because the noise was injected in a because the noise was injected in a sequential manner. The noise was sequential manner. The noise was sequential manner. The noise was injected very slowly injected very slowly injected very slowly and that is the exact same process which and that is the exact same process which and that is the exact same process which is followed for images. is followed for images. is followed for images. So let's now take a look at images and So let's now take a look at images and So let's now take a look at images and see how this analogy applies to images.
- L6 @ 2s
So I hope all of us are on the same page So I hope all of us are on the same page now with respect to the motivation now with respect to the motivation now with respect to the motivation behind this learning journey. behind this learning journey. behind this learning journey. It seems like a natural question that It seems like a natural question that It seems like a natural question that why can't diffusion models only be why why can't diffusion models only be why why can't diffusion models only be why should they only be used for images? Is should they only be used for images? Is should they only be used for images? Is there some bias in our mind? This is there some bias in our mind? This is there some bias in our mind? This is exactly what Andre Karpati also talks exactly what Andre Karpati also talks exactly what Andre Karpati also talks about, right? He says that most of the about, right? He says that most of the about, right? He says that most of the image generation tools actually use image generation tools actually use image generation tools actually use diffusion and not auto reggression. diffusion and not auto reggression. diffusion and not auto reggression. We'll see what auto reggression is in a We'll see what auto reggression is in a We'll see what auto reggression is in a moment. Don't worry about it. It's only moment. Don't worry about it. It's only moment. Don't worry about it. It's only text which has been left behind. So it text which has been left behind. So it text which has been left behind. So it is a bit of a mystery to me and many is a bit of a mystery to me and many is a bit of a mystery to me and many others that why for some reason text others that why for some reason text others that why for some reason text prefers autogeneration but image videos prefers autogeneration but image videos prefers autogeneration but image videos prefer diffusion. Why should this be? prefer diffusion. Why should this be? prefer diffusion. Why should this be? These are the kind of counterintuitive These are the kind of counterintuitive These are the kind of counterintuitive questions which researchers should ask questions which researchers should ask
- L6 @ 50s
questions which researchers should ask that leads to the best research. that leads to the best research. that leads to the best research. If you ask this question, you'll slowly If you ask this question, you'll slowly If you ask this question, you'll slowly start get going into a rabbit hole. You start get going into a rabbit hole. You start get going into a rabbit hole. You may end up with a new research problem may end up with a new research problem may end up with a new research problem and an impact on discovery. But the main and an impact on discovery. But the main and an impact on discovery. But the main counterintuitive question which we want counterintuitive question which we want counterintuitive question which we want to ask is why should only text prefer to ask is why should only text prefer to ask is why should only text prefer auto reggression. Right? And for images, auto reggression. Right? And for images, auto reggression. Right? And for images, videos we use diffusion. Why? videos we use diffusion. Why? videos we use diffusion. Why? A stream of papers came out which tried A stream of papers came out which tried A stream of papers came out which tried to answer this. So you'll see that from to answer this. So you'll see that from to answer this. So you'll see that from 2023 2022 onwards itself we have papers 2023 2022 onwards itself we have papers 2023 2022 onwards itself we have papers which talk about structured denoising which talk about structured denoising which talk about structured denoising diffusion models in discrete spaces then diffusion models in discrete spaces then diffusion models in discrete spaces then discrete diffusion modeling by discrete diffusion modeling by discrete diffusion modeling by estimating the ratios of data estimating the ratios of data estimating the ratios of data distribution then simplified and distribution then simplified and distribution then simplified and generalized mask diffusion. So the generalized mask diffusion. So the generalized mask diffusion. So the concept of mask diffusion was one of the concept of mask diffusion was one of the concept of mask diffusion was one of the major breakthroughs in language modeling major breakthroughs in language modeling major breakthroughs in language modeling diffusion diffusion diffusion models. This so this paper I believe models. This so this paper I believe
- L6 @ 109s
models. This so this paper I believe is a very important one mask diffusion is a very important one mask diffusion is a very important one mask diffusion language models this is where many language models this is where many language models this is where many things actually started to become big things actually started to become big things actually started to become big for small language diffusion models for small language diffusion models for small language diffusion models you'll see that the paper mostly came you'll see that the paper mostly came you'll see that the paper mostly came from Cornell University and uh they from Cornell University and uh they from Cornell University and uh they actually actually actually motivated the concept of motivated the concept of motivated the concept of why can't we bridge the gap between the why can't we bridge the gap between the why can't we bridge the gap between the diffusion models and auto reggressive diffusion models and auto reggressive diffusion models and auto reggressive methods in language modeling This was in November 2024. The most This was in November 2024. The most recent paper which I am aware of which recent paper which I am aware of which recent paper which I am aware of which kind of summarizes everything and its kind of summarizes everything and its kind of summarizes everything and its state-of-the-art at this moment while state-of-the-art at this moment while state-of-the-art at this moment while recording this video or this journey recording this video or this journey recording this video or this journey videos is large language diffusion videos is large language diffusion videos is large language diffusion models. This paper came out in 18th on models. This paper came out in 18th on models. This paper came out in 18th on 18th October 2025 and we are going to 18th October 2025 and we are going to 18th October 2025 and we are going to replicate this paper fully from scratch replicate this paper fully from scratch replicate this paper fully from scratch in this journey. Right. So this is a in this journey. Right. So this is a in this journey. Right. So this is a paper which paper which paper which actually introduces in their uh actually introduces in their uh actually introduces in their uh introduction section itself introduction section itself
- L6 @ 171s
introduction section itself uh that why should images and video uh that why should images and video uh that why should images and video generation models have all the fun why generation models have all the fun why generation models have all the fun why can't languages be used for diffusion can't languages be used for diffusion can't languages be used for diffusion and they actually uh talk about one more and they actually uh talk about one more and they actually uh talk about one more thing right what they talk about is that thing right what they talk about is that thing right what they talk about is that if we take a look at the probabilistic if we take a look at the probabilistic if we take a look at the probabilistic lens of looking at generative AI which lens of looking at generative AI which lens of looking at generative AI which we saw in one of the prior videos, we saw in one of the prior videos, we saw in one of the prior videos, right? If we take a look at this right? If we take a look at this right? If we take a look at this probabilistic lens, what does this lens probabilistic lens, what does this lens probabilistic lens, what does this lens say? This lens says that say? This lens says that say? This lens says that what is generative AI? Essentially, if what is generative AI? Essentially, if what is generative AI? Essentially, if we want to generate something, right? we want to generate something, right? we want to generate something, right? And one way to look at generation is And one way to look at generation is And one way to look at generation is finding the underlying probability finding the underlying probability finding the underlying probability distribution and sampling from it, distribution and sampling from it, distribution and sampling from it, right? What does it mean generating an right? What does it mean generating an right? What does it mean generating an image? It means finding the underlying image? It means finding the underlying image? It means finding the underlying probability distribution of where the probability distribution of where the probability distribution of where the images might lie and then sampling from images might lie and then sampling from
- L6 @ 219s
images might lie and then sampling from it. So why can't this same thing be it. So why can't this same thing be it. So why can't this same thing be applied to text as well? applied to text as well? applied to text as well? So for that let us consider that what So for that let us consider that what So for that let us consider that what does it mean to have a probability does it mean to have a probability does it mean to have a probability distribution of text distribution of text distribution of text and again let's use Google Gemini and again let's use Google Gemini and again let's use Google Gemini because as I told you I want you to show because as I told you I want you to show because as I told you I want you to show I want to show you how I learned. So I want to show you how I learned. So I want to show you how I learned. So I'll ask here that I understood I'll ask here that I understood I'll ask here that I understood the concept of probability distribution for images and how images distribution for images and how images might live in high dimensional might live in high dimensional might live in high dimensional pixel spaces. pixel spaces. pixel spaces. What about text? How can they live in high dimensional How can they live in high dimensional spaces? spaces? spaces? What does it mean? Probability Probability distribution of meaningful text. So I distribution of meaningful text. So I distribution of meaningful text. So I want all of you to pause here and think want all of you to pause here and think want all of you to pause here and think about this for a moment. If I ask you about this for a moment. If I ask you about this for a moment. If I ask you this question in an interview, how would this question in an interview, how would this question in an interview, how would you answer this? you answer this? you answer this? Well, the way I would think about it is Well, the way I would think about it is Well, the way I would think about it is when we talked about images, first we when we talked about images, first we
- L6 @ 298s
when we talked about images, first we broke down the images into pixels, broke down the images into pixels, broke down the images into pixels, right? Similarly, text can be broken right? Similarly, text can be broken right? Similarly, text can be broken down into tokens, broken down into token down into tokens, broken down into token down into tokens, broken down into token ids and they can also be represented in ids and they can also be represented in ids and they can also be represented in higher dimensional pixel spaces. In this higher dimensional pixel spaces. In this higher dimensional pixel spaces. In this they can be represented in high they can be represented in high they can be represented in high dimensional token space. In this huge dimensional token space. In this huge dimensional token space. In this huge highdimensional token space, there might highdimensional token space, there might highdimensional token space, there might be only a very small area which consists be only a very small area which consists be only a very small area which consists of meaningful text. So let's say this is of meaningful text. So let's say this is of meaningful text. So let's say this is a higher dimensional token space of a higher dimensional token space of a higher dimensional token space of sentences. All possible sentences in sentences. All possible sentences in sentences. All possible sentences in human history. There might be only a human history. There might be only a human history. There might be only a small area of meaningful text and this small area of meaningful text and this small area of meaningful text and this might have certain probability might have certain probability might have certain probability distribution. This is what generating distribution. This is what generating distribution. This is what generating meaningful sentences might mean. So one meaningful sentences might mean. So one meaningful sentences might mean. So one way of looking at generating sentences way of looking at generating sentences way of looking at generating sentences is just this, right? create generate is just this, right? create generate is just this, right? create generate create a poem on let's say create a poem on let's say create a poem on let's say uh diffusion models. uh diffusion models. uh diffusion models. So one way to look at generating text is
- L6 @ 354s
So one way to look at generating text is So one way to look at generating text is just one word at a time. Another way of just one word at a time. Another way of just one word at a time. Another way of looking at generating text is finding looking at generating text is finding looking at generating text is finding this probability distribution where this probability distribution where this probability distribution where meaningful text lives related to my meaningful text lives related to my meaningful text lives related to my given sentence. So conditional given sentence. So conditional given sentence. So conditional probability distribution related to my probability distribution related to my probability distribution related to my input prompt and then sampling from it. input prompt and then sampling from it. input prompt and then sampling from it. This part which I'm speaking is the key This part which I'm speaking is the key This part which I'm speaking is the key intuition behind language diffusion intuition behind language diffusion intuition behind language diffusion models. models. models. One another way of looking at generating One another way of looking at generating One another way of looking at generating text is finding this probability text is finding this probability text is finding this probability distribution where meaningful text lives distribution where meaningful text lives distribution where meaningful text lives conditioned to the input prompt and then conditioned to the input prompt and then conditioned to the input prompt and then sampling from it. So yeah, maybe this poem is sampled from So yeah, maybe this poem is sampled from the meaningful space where the text the meaningful space where the text the meaningful space where the text actually lives. Let's see. So this would be my answer if Let's see. So this would be my answer if this was asked in an interview. Let's this was asked in an interview. Let's this was asked in an interview. Let's see what Google Gemini has to say about see what Google Gemini has to say about see what Google Gemini has to say about this. What about text? How can they live this. What about text? How can they live this. What about text? How can they live in high dimensional spaces? What does it in high dimensional spaces? What does it
- L6 @ 417s
in high dimensional spaces? What does it mean probability distribution of mean probability distribution of mean probability distribution of meaningful text? Let's see. So text is trickier than images. So text is trickier than images. With images you had 128x 128 pixels. In With images you had 128x 128 pixels. In With images you had 128x 128 pixels. In text words can be converted into text words can be converted into text words can be converted into embeddings. That's even awesome, right? embeddings. That's even awesome, right? embeddings. That's even awesome, right? Words can be converted into higher Words can be converted into higher Words can be converted into higher dimensional vector spaces. uh and each dimensional vector spaces. uh and each dimensional vector spaces. uh and each embedding vector can be in a higher embedding vector can be in a higher embedding vector can be in a higher dimensional space. Awesome. dimensional space. Awesome. dimensional space. Awesome. To understand the probability To understand the probability To understand the probability distribution, imagine a book space. So distribution, imagine a book space. So distribution, imagine a book space. So suppose we define a book as any sequence suppose we define a book as any sequence suppose we define a book as any sequence of 100,000 characters. This is a huge of 100,000 characters. This is a huge of 100,000 characters. This is a huge number of characters and a huge >> okay I found this on the web. So this is >> okay I found this on the web. So this is >> Siri got activated for some reason. The >> Siri got activated for some reason. The >> Siri got activated for some reason. The what it means to generate meaningful what it means to generate meaningful what it means to generate meaningful text is in this huge space to find that text is in this huge space to find that text is in this huge space to find that set of characters which are meaningful. Uh Uh if you feed the sentence, so the if you feed the sentence, so the if you feed the sentence, so the probability distribution is a probability distribution is a probability distribution is a mathematical function that scores a mathematical function that scores a mathematical function that scores a sequence of words based on two things.
- L6 @ 492s
sequence of words based on two things. sequence of words based on two things. Grammar and logic both need to be Grammar and logic both need to be Grammar and logic both need to be correct, right? And that probably lives correct, right? And that probably lives correct, right? And that probably lives in a small space in this higher in a small space in this higher in a small space in this higher dimensional area. That's the way to think about That's the way to think about probability distribution for text, probability distribution for text, probability distribution for text, right? So why can't we find this right? So why can't we find this right? So why can't we find this probability distribution where the probability distribution where the probability distribution where the meaningful text lies using diffusion? If meaningful text lies using diffusion? If meaningful text lies using diffusion? If this if generative AI can be looked at this if generative AI can be looked at this if generative AI can be looked at through a probabilistic lens, why should through a probabilistic lens, why should through a probabilistic lens, why should it be restricted to images? Why not it be restricted to images? Why not it be restricted to images? Why not text? That's what this these people text? That's what this these people text? That's what this these people actually asked, actually asked, actually asked, right? What they said is that if you look at how language models if you look at how language models traditionally operate, it's through the traditionally operate, it's through the traditionally operate, it's through the concept of auto reggression which we are concept of auto reggression which we are concept of auto reggression which we are going to look at very soon. But what going to look at very soon. But what going to look at very soon. But what these people proposed is that the major these people proposed is that the major these people proposed is that the major reason why reason why reason why uh uh uh major reason why diffusion models work major reason why diffusion models work major reason why diffusion models work for images can be the same reason which for images can be the same reason which for images can be the same reason which can be carried forward to languages.
- L6 @ 552s
can be carried forward to languages. can be carried forward to languages. What these people suggested is auto What these people suggested is auto What these people suggested is auto reggressive models or auto reggressive reggressive models or auto reggressive reggressive models or auto reggressive methods are not the reason for LLMs to methods are not the reason for LLMs to methods are not the reason for LLMs to work. So why are we sticking to it? work. So why are we sticking to it? work. So why are we sticking to it? Let's look at auto reggression actually Let's look at auto reggression actually Let's look at auto reggression actually right now so that all of you have some right now so that all of you have some right now so that all of you have some baseline to compare to.
diffusion-principles-vizuara
- L11 @ 3s
Hello everyone. I'm very excited to be Hello everyone. I'm very excited to be recording this video today because uh recording this video today because uh recording this video today because uh I'm going to make an announcement for I'm going to make an announcement for I'm going to make an announcement for the launch of a new course. Before we the launch of a new course. Before we the launch of a new course. Before we discuss about the course, first let us discuss about the course, first let us discuss about the course, first let us rewind and go back to the year 2022. rewind and go back to the year 2022. rewind and go back to the year 2022. I remember this was the year where I I remember this was the year where I I remember this was the year where I truly truly truly understood the power and scope of understood the power and scope of understood the power and scope of generative AI and the main reason was generative AI and the main reason was generative AI and the main reason was that applications like midjourney and that applications like midjourney and that applications like midjourney and stable diffusion came out which could stable diffusion came out which could stable diffusion came out which could generate images which were very very generate images which were very very generate images which were very very realistic realistic realistic and I remember being completely blown and I remember being completely blown and I remember being completely blown away by the realism of the images which away by the realism of the images which away by the realism of the images which these AI applications could generate and these AI applications could generate and these AI applications could generate and that was the time where I really started that was the time where I really started that was the time where I really started taking AI generation and generative AI taking AI generation and generative AI taking AI generation and generative AI overall as a field very seriously. overall as a field very seriously. overall as a field very seriously. I I remember initially I I I did not I I remember initially I I I did not I I remember initially I I I did not focus on the technology behind focus on the technology behind
- L11 @ 60s
focus on the technology behind uh midjourney or stable diffusion but I uh midjourney or stable diffusion but I uh midjourney or stable diffusion but I was so excited to try this out that I was so excited to try this out that I was so excited to try this out that I took this to a lot of schools in India took this to a lot of schools in India took this to a lot of schools in India and uh I started showing teachers what and uh I started showing teachers what and uh I started showing teachers what you can do with applications like you can do with applications like you can do with applications like midjourney. We had a fun exercise where midjourney. We had a fun exercise where midjourney. We had a fun exercise where I used to tell the teachers that tell me I used to tell the teachers that tell me I used to tell the teachers that tell me a name of an animal, a place and an a name of an animal, a place and an a name of an animal, a place and an activity and uh let's generate an image activity and uh let's generate an image activity and uh let's generate an image using midjourney. So we used to come up using midjourney. So we used to come up using midjourney. So we used to come up with crazy prompts like monkeys playing with crazy prompts like monkeys playing with crazy prompts like monkeys playing cricket on Mars etc. And as you remember cricket on Mars etc. And as you remember cricket on Mars etc. And as you remember the smile it had on teachers faces it the smile it had on teachers faces it the smile it had on teachers faces it was really great to see. was really great to see. was really great to see. uh when I started using these models the uh when I started using these models the uh when I started using these models the word diffusion really caught my word diffusion really caught my word diffusion really caught my attention I used to wonder is this attention I used to wonder is this attention I used to wonder is this related to the diffusion which I know related to the diffusion which I know
- L11 @ 109s
related to the diffusion which I know about I know about diffusion where let's about I know about diffusion where let's about I know about diffusion where let's say I spray a perfume in one corner of say I spray a perfume in one corner of say I spray a perfume in one corner of the room the smell slowly percolates in the room the smell slowly percolates in the room the smell slowly percolates in the room and reaches to the other corner the room and reaches to the other corner the room and reaches to the other corner of the room so this process of of the room so this process of of the room so this process of percolation percolation percolation and uh slow movement of particles from and uh slow movement of particles from and uh slow movement of particles from one area to another That's diffusion. one area to another That's diffusion. one area to another That's diffusion. And I always wondered at the back of my And I always wondered at the back of my And I always wondered at the back of my mind where does the word diffusion in mind where does the word diffusion in mind where does the word diffusion in stable diffusion come from. So recently stable diffusion come from. So recently stable diffusion come from. So recently in this year I started studying more in this year I started studying more in this year I started studying more about it and I realized that there is a about it and I realized that there is a about it and I realized that there is a beautiful story behind the development beautiful story behind the development beautiful story behind the development of midjourney and stable diffusion. of midjourney and stable diffusion. of midjourney and stable diffusion. And the word diffusion actually is used And the word diffusion actually is used And the word diffusion actually is used in the same sense like we use in in in the same sense like we use in in in the same sense like we use in in physics. physics. physics. This development and journey and the This development and journey and the This development and journey and the research which has happened to develop research which has happened to develop research which has happened to develop these diffusion models have
- L11 @ 165s
these diffusion models have these diffusion models have lasted for 10 to 15 years and every lasted for 10 to 15 years and every lasted for 10 to 15 years and every development has used breakthroughs made development has used breakthroughs made development has used breakthroughs made by a previous researcher. So it has been by a previous researcher. So it has been by a previous researcher. So it has been truly an incremental journey. truly an incremental journey. truly an incremental journey. But why I was hooked onto this field But why I was hooked onto this field But why I was hooked onto this field more was that more was that more was that firstly it was a field where I could see firstly it was a field where I could see firstly it was a field where I could see mathematics being influenced by physics. mathematics being influenced by physics. mathematics being influenced by physics. So we are actually looking at the So we are actually looking at the So we are actually looking at the physical diffusion process and then physical diffusion process and then physical diffusion process and then uh using the intuition from that process uh using the intuition from that process uh using the intuition from that process to develop an AI technique which I found to develop an AI technique which I found to develop an AI technique which I found to be amazing and the literature reveals to be amazing and the literature reveals to be amazing and the literature reveals that there are three different tracks that there are three different tracks that there are three different tracks which kind of merge together in the year which kind of merge together in the year which kind of merge together in the year 2022 and that's when we get these 2022 and that's when we get these 2022 and that's when we get these brilliant models. So I wanted to brilliant models. So I wanted to brilliant models. So I wanted to understand what these three different understand what these three different understand what these three different tracks are and can we understand them tracks are and can we understand them tracks are and can we understand them from the fundamentals. from the fundamentals. from the fundamentals. So this series has been made from that
- L11 @ 223s
So this series has been made from that So this series has been made from that point of view. I want to explain point of view. I want to explain point of view. I want to explain diffusion models to you completely from diffusion models to you completely from diffusion models to you completely from scratch. And I'm giving the title to scratch. And I'm giving the title to scratch. And I'm giving the title to this series as the principles of this series as the principles of this series as the principles of diffusion models from origins to diffusion models from origins to diffusion models from origins to advances. advances. advances. And we are going to go from complete And we are going to go from complete And we are going to go from complete basics and we are going to understand basics and we are going to understand basics and we are going to understand how it has evolved over 10 15 years and how it has evolved over 10 15 years and how it has evolved over 10 15 years and why we are able to create such realistic why we are able to create such realistic why we are able to create such realistic images. Now how are mathematics and images. Now how are mathematics and images. Now how are mathematics and physics so beautifully combined and how physics so beautifully combined and how physics so beautifully combined and how they work well with each other. they work well with each other. they work well with each other. Diffusion models are not just limited to Diffusion models are not just limited to Diffusion models are not just limited to images. You can create beautiful videos, images. You can create beautiful videos, images. You can create beautiful videos, audios, protein synthesis. There are so audios, protein synthesis. There are so audios, protein synthesis. There are so many application of diffusion models many application of diffusion models many application of diffusion models that this whole field is very that this whole field is very that this whole field is very fascinating. fascinating. fascinating. Now let me quickly show you how exactly Now let me quickly show you how exactly Now let me quickly show you how exactly I have structured this course and how I I have structured this course and how I
- L11 @ 276s
I have structured this course and how I plan to teach it. It'll be very similar plan to teach it. It'll be very similar plan to teach it. It'll be very similar to Vijuara's other teaching material and to Vijuara's other teaching material and to Vijuara's other teaching material and I'll be teaching these series of I'll be teaching these series of I'll be teaching these series of lectures and it'll be uploaded on lectures and it'll be uploaded on lectures and it'll be uploaded on YouTube completely for free. Now let's YouTube completely for free. Now let's YouTube completely for free. Now let's dive in and understand the curriculum in dive in and understand the curriculum in dive in and understand the curriculum in detail. Thank you. Let us try to understand how this course Let us try to understand how this course is going to be structured. The title of is going to be structured. The title of is going to be structured. The title of this course is the principles of this course is the principles of this course is the principles of diffusion models from origins to advances. As I said before in the introduction, As I said before in the introduction, As I said before in the introduction, diffusion models has really diffusion models has really diffusion models has really revolutionized the text to image revolutionized the text to image revolutionized the text to image synthesis as they produce images which synthesis as they produce images which synthesis as they produce images which are very realistic and some examples we are very realistic and some examples we are very realistic and some examples we see are Delhi 3 and stable diffusion. see are Delhi 3 and stable diffusion. see are Delhi 3 and stable diffusion. And apart from these applications uh And apart from these applications uh And apart from these applications uh which also include midjourney, there are which also include midjourney, there are which also include midjourney, there are also other domains where diffusion also other domains where diffusion also other domains where diffusion models find models find models find a lot of applications. a lot of applications. a lot of applications. The next domain that we are going to The next domain that we are going to The next domain that we are going to discuss is that of protein
- L11 @ 343s
discuss is that of protein discuss is that of protein structure prediction. structure prediction. structure prediction. This has been a problem which has This has been a problem which has This has been a problem which has fascinated biologists for a very long fascinated biologists for a very long fascinated biologists for a very long time. And it turns out that now you can time. And it turns out that now you can time. And it turns out that now you can use diffusion models to generate 3D use diffusion models to generate 3D use diffusion models to generate 3D molecular structures and protein molecular structures and protein molecular structures and protein confirmations. confirmations. confirmations. And there are some excellent models And there are some excellent models And there are some excellent models which have been released in the last 2 which have been released in the last 2 which have been released in the last 2 three years which have diffusion models three years which have diffusion models three years which have diffusion models at the heart of um the methodology for at the heart of um the methodology for at the heart of um the methodology for predicting these protein structures and predicting these protein structures and predicting these protein structures and I find this application very I find this application very I find this application very fascinating. fascinating. Another application which is kind of a Another application which is kind of a Another application which is kind of a direct synthesis or extension of image direct synthesis or extension of image direct synthesis or extension of image generation capabilities is the ability generation capabilities is the ability generation capabilities is the ability to generate videos using AI. to generate videos using AI. to generate videos using AI. Video generation models are still Video generation models are still Video generation models are still improving but we are now seeing improving but we are now seeing improving but we are now seeing increasingly models becoming more increasingly models becoming more increasingly models becoming more effective at generating realistic effective at generating realistic effective at generating realistic videos. It started from generating 3 videos. It started from generating 3 videos. It started from generating 3 seconds, 5 seconds, 10 seconds and now seconds, 5 seconds, 10 seconds and now
- L11 @ 402s
seconds, 5 seconds, 10 seconds and now you can generate up to 5 minute videos you can generate up to 5 minute videos you can generate up to 5 minute videos using u generative AI and a lot of these using u generative AI and a lot of these using u generative AI and a lot of these models like Lumiere by Google research models like Lumiere by Google research models like Lumiere by Google research have diffusion process at the heart of have diffusion process at the heart of have diffusion process at the heart of how these models are prepared in the how these models are prepared in the how these models are prepared in the first place. first place. first place. Another very key application which I Another very key application which I Another very key application which I really want to highlight is that of really want to highlight is that of really want to highlight is that of modern robot learning. modern robot learning. modern robot learning. All of us have, you know, always been All of us have, you know, always been All of us have, you know, always been mesmerized by the possibilities of mesmerized by the possibilities of mesmerized by the possibilities of robotics. You see all these Boston robotics. You see all these Boston robotics. You see all these Boston Dynamics robots flying around doing Dynamics robots flying around doing Dynamics robots flying around doing these acrobatic stunts. But the question these acrobatic stunts. But the question these acrobatic stunts. But the question is that why haven't we seen a lot of is that why haven't we seen a lot of is that why haven't we seen a lot of these robots in our houses doing these robots in our houses doing these robots in our houses doing household chores? household chores? household chores? And there has been a lot of focus in the And there has been a lot of focus in the And there has been a lot of focus in the AI community to advance the field of AI community to advance the field of AI community to advance the field of robotics robotics robotics and diffusion has given a huge push to and diffusion has given a huge push to and diffusion has given a huge push to this advancement. Recently we are seeing
- L11 @ 462s
this advancement. Recently we are seeing this advancement. Recently we are seeing a lot of foundational robotics models a lot of foundational robotics models a lot of foundational robotics models being released which are trained on being released which are trained on being released which are trained on massive amounts of robotics trajectories massive amounts of robotics trajectories massive amounts of robotics trajectories similar to how a LLM is trained on similar to how a LLM is trained on similar to how a LLM is trained on massive amounts of web data. Similarly, massive amounts of web data. Similarly, massive amounts of web data. Similarly, a robotics foundation model is trained a robotics foundation model is trained a robotics foundation model is trained on massive amounts of robotics on massive amounts of robotics on massive amounts of robotics trajectories. And then trajectories. And then trajectories. And then you have these foundational models which you have these foundational models which you have these foundational models which can do a lot of tasks like folding can do a lot of tasks like folding can do a lot of tasks like folding clothes, pushing a block, pick and place clothes, pushing a block, pick and place clothes, pushing a block, pick and place operations etc. operations etc. operations etc. A fascinating example of this is the A fascinating example of this is the A fascinating example of this is the model pi0 which is developed by physical model pi0 which is developed by physical model pi0 which is developed by physical intelligence and they use diffusion intelligence and they use diffusion intelligence and they use diffusion models at the heart of their policy. models at the heart of their policy. models at the heart of their policy. And this is fascinating right because And this is fascinating right because And this is fascinating right because the process of diffusion which appears the process of diffusion which appears the process of diffusion which appears to be limited to physics somehow seems to be limited to physics somehow seems to be limited to physics somehow seems to have applications in a lot of these to have applications in a lot of these to have applications in a lot of these different domains. That leads us to different domains. That leads us to different domains. That leads us to wonder is there something intrinsic wonder is there something intrinsic
- L11 @ 523s
wonder is there something intrinsic which is happening in diffusion that is which is happening in diffusion that is which is happening in diffusion that is common in all these applications. But common in all these applications. But common in all these applications. But robotics is a field which has fascinated robotics is a field which has fascinated robotics is a field which has fascinated me for quite some time and diffusion me for quite some time and diffusion me for quite some time and diffusion models have you know advanced the field models have you know advanced the field models have you know advanced the field of robot policy learning significantly. of robot policy learning significantly. of robot policy learning significantly. Uh I I I I will be following the Uh I I I I will be following the Uh I I I I will be following the progress of physical intelligence very progress of physical intelligence very progress of physical intelligence very closely. I'm very impressed by the work closely. I'm very impressed by the work closely. I'm very impressed by the work uh done by their team and uh this model uh done by their team and uh this model uh done by their team and uh this model which appears to be a frontier model in which appears to be a frontier model in which appears to be a frontier model in uh in in the class of robotics uh in in the class of robotics uh in in the class of robotics foundation model is very interesting and foundation model is very interesting and foundation model is very interesting and the architecture the development process the architecture the development process the architecture the development process is also very fascinating. is also very fascinating. is also very fascinating. Another application of diffusion models Another application of diffusion models Another application of diffusion models is to generate realistic music and is to generate realistic music and is to generate realistic music and speech maintaining the temporal speech maintaining the temporal speech maintaining the temporal coherence and style. So you can imagine coherence and style. So you can imagine coherence and style. So you can imagine diffusion models taking care of not just diffusion models taking care of not just diffusion models taking care of not just images and videos but also audio
- L11 @ 582s
images and videos but also audio images and videos but also audio synthesis. Now that we have established that Now that we have established that diffusion models have a huge number of diffusion models have a huge number of diffusion models have a huge number of applications and they have a lot of applications and they have a lot of applications and they have a lot of potential. They have already potential. They have already potential. They have already revolutionized revolutionized revolutionized few industries and they might have a lot few industries and they might have a lot few industries and they might have a lot of other applications waiting to be of other applications waiting to be of other applications waiting to be uncovered. This is a field which has a uncovered. This is a field which has a uncovered. This is a field which has a lot of potential not just from the lot of potential not just from the lot of potential not just from the research side but also application side. research side but also application side. research side but also application side. So anyone who is looking to understand So anyone who is looking to understand So anyone who is looking to understand AI from a more deeper lens and develop a AI from a more deeper lens and develop a AI from a more deeper lens and develop a skill set to build AI applications skill set to build AI applications skill set to build AI applications should definitely have a very clearcut should definitely have a very clearcut should definitely have a very clearcut fundamental understanding of diffusion fundamental understanding of diffusion fundamental understanding of diffusion models in their arsenal. models in their arsenal. models in their arsenal. However, very little is known about how However, very little is known about how However, very little is known about how they work. Let's do a small they work. Let's do a small they work. Let's do a small study. What we'll do is we will go to study. What we'll do is we will go to study. What we'll do is we will go to YouTube and we'll try to understand what kind of videos are available on what kind of videos are available on diffusion models. So let's see diffusion diffusion models. So let's see diffusion diffusion models. So let's see diffusion models explained.
- L11 @ 655s
So you see there are these small videos So you see there are these small videos 15 minutes, 30 minutes, 12 minutes and 15 minutes, 30 minutes, 12 minutes and 15 minutes, 30 minutes, 12 minutes and some of them have got really good views some of them have got really good views some of them have got really good views but all of these videos are very surface but all of these videos are very surface but all of these videos are very surface level. They don't really help you level. They don't really help you level. They don't really help you understand what is happening underneath. understand what is happening underneath. understand what is happening underneath. You get a sense that okay fine I have You get a sense that okay fine I have You get a sense that okay fine I have understood diffusion but really you understood diffusion but really you understood diffusion but really you don't and my level of understanding was don't and my level of understanding was don't and my level of understanding was this two years back. I broadly unders this two years back. I broadly unders this two years back. I broadly unders understood what was happening but then I understood what was happening but then I understood what was happening but then I realized that there is a whole realized that there is a whole realized that there is a whole mystery waiting to be discovered. It's mystery waiting to be discovered. It's mystery waiting to be discovered. It's like this is just covering the tip of like this is just covering the tip of like this is just covering the tip of the iceberg whereas the iceberg is very the iceberg whereas the iceberg is very the iceberg whereas the iceberg is very huge huge huge and the rich mystery and the rich and the rich mystery and the rich and the rich mystery and the rich fundamentals are are are not presented fundamentals are are are not presented fundamentals are are are not presented anywhere. anywhere. anywhere. All of these videos are 10 minutes 15 All of these videos are 10 minutes 15 All of these videos are 10 minutes 15 minutes maximum 30 minutes long. So very minutes maximum 30 minutes long. So very
- L11 @ 708s
minutes maximum 30 minutes long. So very little is known about how they work and little is known about how they work and little is known about how they work and I think that diffusion models should be I think that diffusion models should be I think that diffusion models should be very fundamental in every AI course. very fundamental in every AI course. very fundamental in every AI course. There should be a separate course just There should be a separate course just There should be a separate course just reserved for diffusion models. So this reserved for diffusion models. So this reserved for diffusion models. So this course is designed for people who seek a course is designed for people who seek a course is designed for people who seek a very clear conceptual and mathematically very clear conceptual and mathematically very clear conceptual and mathematically grounded understanding of diffusion grounded understanding of diffusion grounded understanding of diffusion models. models. models. It clarifies the theoretical foundations It clarifies the theoretical foundations It clarifies the theoretical foundations explaining the reasoning behind their explaining the reasoning behind their explaining the reasoning behind their diverse formulations and provides a diverse formulations and provides a diverse formulations and provides a stable footing for further study and stable footing for further study and stable footing for further study and research in this rapidly evolving field. research in this rapidly evolving field. research in this rapidly evolving field. And this is a very great or accessible And this is a very great or accessible And this is a very great or accessible entry point for learners. We'll also entry point for learners. We'll also entry point for learners. We'll also make this course very practical. We will make this course very practical. We will make this course very practical. We will include a lot of practical applications include a lot of practical applications include a lot of practical applications and practical examples uh which will and practical examples uh which will and practical examples uh which will make every single concept very clear. So make every single concept very clear. So make every single concept very clear. So we'll have a very nice balance of theory we'll have a very nice balance of theory we'll have a very nice balance of theory and practicals. and practicals. and practicals. Now the reason I'm coming out with this
- L11 @ 767s
Now the reason I'm coming out with this Now the reason I'm coming out with this course right now is because I received a course right now is because I received a course right now is because I received a major boost when this book was released. major boost when this book was released. major boost when this book was released. The principles of diffusion models from The principles of diffusion models from The principles of diffusion models from origins to advances and the writers of origins to advances and the writers of origins to advances and the writers of these books are some of the authors these books are some of the authors these books are some of the authors which have themselves contributed to which have themselves contributed to which have themselves contributed to some key advances in the development of some key advances in the development of some key advances in the development of diffusion models and it's a fabulously diffusion models and it's a fabulously diffusion models and it's a fabulously written book. It's it's it's kind of written book. It's it's it's kind of written book. It's it's it's kind of long. It's around 500 pages. My aim is long. It's around 500 pages. My aim is long. It's around 500 pages. My aim is to take this book as the primary source to take this book as the primary source to take this book as the primary source of reference and uh add my own thinking of reference and uh add my own thinking of reference and uh add my own thinking practical applications and practical applications and practical applications and uh you know relate it to some real life uh you know relate it to some real life uh you know relate it to some real life examples wherever necessary. for example examples wherever necessary. for example examples wherever necessary. for example uh related to the field of robotics then uh related to the field of robotics then uh related to the field of robotics then image generation and take real life image generation and take real life image generation and take real life examples so that the theory doesn't examples so that the theory doesn't examples so that the theory doesn't become very complicated become very complicated become very complicated and one thing uh I realized after
- L11 @ 822s
and one thing uh I realized after and one thing uh I realized after reading this book and these thoughts reading this book and these thoughts reading this book and these thoughts were also circling in my mind is that were also circling in my mind is that were also circling in my mind is that uh anyone who is learning artificial uh anyone who is learning artificial uh anyone who is learning artificial intelligence and wants to go a little intelligence and wants to go a little intelligence and wants to go a little bit deep you need to start looking at bit deep you need to start looking at bit deep you need to start looking at the subject from multiple different the subject from multiple different the subject from multiple different lenses. lenses. lenses. You know looking at diffusion models You know looking at diffusion models You know looking at diffusion models from a surface level understanding is from a surface level understanding is from a surface level understanding is okay. But if you go one level deeper you okay. But if you go one level deeper you okay. But if you go one level deeper you start to understand that there is start to understand that there is start to understand that there is something very deep which is happening something very deep which is happening something very deep which is happening which is really connecting everything which is really connecting everything which is really connecting everything together. And that is what we are going together. And that is what we are going together. And that is what we are going to understand. So at the heart of this to understand. So at the heart of this to understand. So at the heart of this is something very simple but to uncover is something very simple but to uncover is something very simple but to uncover that simple fact we are going to go that simple fact we are going to go that simple fact we are going to go through a lot of material. through a lot of material. through a lot of material. So let's try to briefly understand how I So let's try to briefly understand how I So let's try to briefly understand how I have divided the curriculum according to have divided the curriculum according to
- L11 @ 871s
have divided the curriculum according to this book. Uh and I I will not go very this book. Uh and I I will not go very this book. Uh and I I will not go very theoretical in this section. So it might theoretical in this section. So it might theoretical in this section. So it might appear that I'm skimming some of the appear that I'm skimming some of the appear that I'm skimming some of the details but it's it's very deliberate. details but it's it's very deliberate. details but it's it's very deliberate. It's to not burden you with a lot of It's to not burden you with a lot of It's to not burden you with a lot of keywords right now. But at the same time keywords right now. But at the same time keywords right now. But at the same time for people looking to know whether this for people looking to know whether this for people looking to know whether this course is relevant for them. I want to course is relevant for them. I want to course is relevant for them. I want to give a very broad level understanding of give a very broad level understanding of give a very broad level understanding of what all concepts I am planning to cover what all concepts I am planning to cover what all concepts I am planning to cover as a part of this course and how is the as a part of this course and how is the as a part of this course and how is the course really structured. course really structured. course really structured. So part A of uh the course is going to So part A of uh the course is going to So part A of uh the course is going to be introduction to deep generative be introduction to deep generative be introduction to deep generative modeling. What are the fundamentals? modeling. What are the fundamentals? modeling. What are the fundamentals? What is the meaning of deep generative What is the meaning of deep generative What is the meaning of deep generative models? models? models? What is generative models itself in in What is generative models itself in in What is generative models itself in in the first place? And what is the broader
- L11 @ 920s
the first place? And what is the broader the first place? And what is the broader objective we are trying to achieve here? objective we are trying to achieve here? objective we are trying to achieve here? This is going to be very fundamental. This is going to be very fundamental. This is going to be very fundamental. And uh it it turns out that what we are And uh it it turns out that what we are And uh it it turns out that what we are trying to do is we are trying to trying to do is we are trying to trying to do is we are trying to generate an image. But generate an image. But generate an image. But the way we do that is we try to the way we do that is we try to the way we do that is we try to look at what would be the distribution look at what would be the distribution look at what would be the distribution which is which is which is used to generate all these images. used to generate all these images. used to generate all these images. we predict the distribution and then we we predict the distribution and then we we predict the distribution and then we sample from the distribution to predict sample from the distribution to predict sample from the distribution to predict a new image. So I will explain all of a new image. So I will explain all of a new image. So I will explain all of this once we start the course but that's this once we start the course but that's this once we start the course but that's in a in a broad sense this is what we in a in a broad sense this is what we in a in a broad sense this is what we are trying to do and the field of deep are trying to do and the field of deep are trying to do and the field of deep generative modeling has evolved quite a generative modeling has evolved quite a generative modeling has evolved quite a lot in the last 10 years. As you can see lot in the last 10 years. As you can see
- L11 @ 968s
lot in the last 10 years. As you can see one of the first methods actually came one of the first methods actually came one of the first methods actually came 40 years back which was energy based 40 years back which was energy based 40 years back which was energy based methods. Since then we have made a lot methods. Since then we have made a lot methods. Since then we have made a lot of progress. Variation autoenccoders, of progress. Variation autoenccoders, of progress. Variation autoenccoders, normalizing flows, diffusion normalizing flows, diffusion normalizing flows, diffusion probabilistic models, neural oDs, probabilistic models, neural oDs, probabilistic models, neural oDs, noise condition score network, dnoising noise condition score network, dnoising noise condition score network, dnoising diffusion probabilistic models. This is diffusion probabilistic models. This is diffusion probabilistic models. This is very interesting. And then flow very interesting. And then flow very interesting. And then flow matching. matching. matching. All of these are different techniques All of these are different techniques All of these are different techniques which have given rise or expanded the which have given rise or expanded the which have given rise or expanded the breadth the the breadth of this field of breadth the the breadth of this field of breadth the the breadth of this field of deep generative modeling. deep generative modeling. deep generative modeling. Now how are we going to understand all Now how are we going to understand all Now how are we going to understand all these concepts? For example, let's take these concepts? For example, let's take these concepts? For example, let's take pick variation autoenccoders. pick variation autoenccoders. pick variation autoenccoders. What will be our strategy to understand What will be our strategy to understand What will be our strategy to understand these concepts? Let's try to understand these concepts? Let's try to understand these concepts? Let's try to understand that. that. that. So in part B, we will actually touch So in part B, we will actually touch So in part B, we will actually touch upon the core perspectives on diffusion upon the core perspectives on diffusion upon the core perspectives on diffusion models and this is why in in this part models and this is why in in this part models and this is why in in this part we will cover all of these different
- L11 @ 1025s
we will cover all of these different we will cover all of these different techniques individually. techniques individually. techniques individually. So broadly speaking there are three So broadly speaking there are three So broadly speaking there are three views which I mentioned at the start of views which I mentioned at the start of views which I mentioned at the start of the video which converge very nicely. the video which converge very nicely. the video which converge very nicely. The variational view, scorebased view The variational view, scorebased view The variational view, scorebased view and flow-based view. I I don't want you and flow-based view. I I don't want you and flow-based view. I I don't want you to focus on these terminologies a lot to focus on these terminologies a lot to focus on these terminologies a lot but but but for example let's take an example of for example let's take an example of for example let's take an example of variational autoenccoder. It falls variational autoenccoder. It falls variational autoenccoder. It falls within the variational view and we will within the variational view and we will within the variational view and we will understand the variational autoenccoder understand the variational autoenccoder understand the variational autoenccoder from the complete basics. How exactly is from the complete basics. How exactly is from the complete basics. How exactly is it formulated? The mathematics behind it it formulated? The mathematics behind it it formulated? The mathematics behind it and then finally we will see a practical and then finally we will see a practical and then finally we will see a practical example of where it is implemented in example of where it is implemented in example of where it is implemented in practice. Then we'll move on to the next practice. Then we'll move on to the next practice. Then we'll move on to the next view etc. So every single subsection of view etc. So every single subsection of view etc. So every single subsection of this course is going to be devoted to this course is going to be devoted to this course is going to be devoted to one view. First we'll start with one view. First we'll start with one view. First we'll start with variational view then we will cover variational view then we will cover
- L11 @ 1079s
variational view then we will cover score based view. Then we will cover score based view. Then we will cover score based view. Then we will cover flow-based view and then after that we flow-based view and then after that we flow-based view and then after that we will see how these views are coming will see how these views are coming will see how these views are coming together. For example, if you see this together. For example, if you see this together. For example, if you see this robotics application which I mentioned robotics application which I mentioned robotics application which I mentioned pi0 pi 0 is built on a policy which uses pi0 pi 0 is built on a policy which uses pi0 pi 0 is built on a policy which uses this flow-based view this flow-based view this flow-based view and if you look at midjourney and if you look at midjourney and if you look at midjourney midjourney is built on midjourney is built on midjourney is built on a policy which uses variational view. So a policy which uses variational view. So a policy which uses variational view. So variational view and flowbased view are variational view and flowbased view are variational view and flowbased view are very common but everything started from very common but everything started from very common but everything started from score based view. So we need to look at score based view. So we need to look at score based view. So we need to look at that as well so that we get a very that as well so that we get a very that as well so that we get a very complete picture of complete picture of complete picture of how the different different perspectives how the different different perspectives how the different different perspectives on diffusion models are and how they on diffusion models are and how they on diffusion models are and how they work together with each other. work together with each other. work together with each other. So this is how we are going to So this is how we are going to So this is how we are going to understand each lecture. And then understand each lecture. And then understand each lecture. And then finally in part three we are going to finally in part three we are going to
- L11 @ 1139s
finally in part three we are going to understand how to sample from these understand how to sample from these understand how to sample from these diffusion models. How to steer the diffusion models. How to steer the diffusion models. How to steer the generation using classifier guidance, generation using classifier guidance, generation using classifier guidance, classifier free guidance etc. We are classifier free guidance etc. We are classifier free guidance etc. We are going to look at a lot of practical going to look at a lot of practical going to look at a lot of practical examples as as you can see in this examples as as you can see in this examples as as you can see in this figure. figure. figure. This is an example where you know This is an example where you know This is an example where you know Chinese letters are generated using the Chinese letters are generated using the Chinese letters are generated using the process of diffusion and it's it's very process of diffusion and it's it's very process of diffusion and it's it's very exciting right you can see that exciting right you can see that exciting right you can see that when the green bar is almost zero it's when the green bar is almost zero it's when the green bar is almost zero it's complete noise but we are generating complete noise but we are generating complete noise but we are generating something from noise. So initially you something from noise. So initially you something from noise. So initially you start with noise and then slowly you start with noise and then slowly you start with noise and then slowly you start the process of denoising and then start the process of denoising and then start the process of denoising and then finally you get what you want. [snorts] finally you get what you want. [snorts] finally you get what you want. [snorts] So there is there is a feeling of some So there is there is a feeling of some So there is there is a feeling of some magic happening here and that is what magic happening here and that is what magic happening here and that is what actually attracted me to this field. I actually attracted me to this field. I
- L11 @ 1191s
actually attracted me to this field. I want I I I wanted to study it in in in want I I I wanted to study it in in in want I I I wanted to study it in in in more depth. more depth. more depth. People who have taken Vijuara's courses People who have taken Vijuara's courses People who have taken Vijuara's courses before uh know our style of teaching is before uh know our style of teaching is before uh know our style of teaching is completely from the basics. We do not completely from the basics. We do not completely from the basics. We do not assume any prior knowledge and we try to assume any prior knowledge and we try to assume any prior knowledge and we try to impart the intuition to you which is impart the intuition to you which is impart the intuition to you which is going to be crucial for you to develop a going to be crucial for you to develop a going to be crucial for you to develop a very deep understanding and uh enjoyment very deep understanding and uh enjoyment very deep understanding and uh enjoyment of the subject because we believe that of the subject because we believe that of the subject because we believe that if you start enjoying the subject of AI if you start enjoying the subject of AI if you start enjoying the subject of AI uh the the sky is the limit for you then uh the the sky is the limit for you then uh the the sky is the limit for you then you can take it up ahead in a lot of you can take it up ahead in a lot of you can take it up ahead in a lot of different ways. different ways. different ways. a bit about your course instructor. U a bit about your course instructor. U a bit about your course instructor. U I have taught multiple courses before I have taught multiple courses before I have taught multiple courses before including reasoning LLM from scratch and including reasoning LLM from scratch and including reasoning LLM from scratch and hands-on reinforcement learning. The hands-on reinforcement learning. The hands-on reinforcement learning. The reinforcement learning course just reinforcement learning course just
- L11 @ 1240s
reinforcement learning course just finished a while back and uh after that finished a while back and uh after that finished a while back and uh after that I have come up with this diffusion I have come up with this diffusion I have come up with this diffusion series. Both these courses have received series. Both these courses have received series. Both these courses have received very good traction and every single day very good traction and every single day very good traction and every single day I see people either commenting or I see people either commenting or I see people either commenting or reaching out to us personally regarding reaching out to us personally regarding reaching out to us personally regarding how this course has helped them. how this course has helped them. how this course has helped them. So this is how we are going to uh teach So this is how we are going to uh teach So this is how we are going to uh teach you this course also in the same manner you this course also in the same manner you this course also in the same manner and my teaching style is very simple. I and my teaching style is very simple. I and my teaching style is very simple. I focus on three things. First is the focus on three things. First is the focus on three things. First is the fundamentals and then assume no prior fundamentals and then assume no prior fundamentals and then assume no prior knowledge and mix theory and practical. knowledge and mix theory and practical. knowledge and mix theory and practical. Why the fundamentals are important is Why the fundamentals are important is Why the fundamentals are important is because you can look at these YouTube because you can look at these YouTube because you can look at these YouTube videos and you can understand broad videos and you can understand broad videos and you can understand broad level what's happening but level what's happening but level what's happening but you know when you have understood you know when you have understood you know when you have understood something from the very basics and that something from the very basics and that something from the very basics and that feeling you will not get by looking at feeling you will not get by looking at
- L11 @ 1292s
feeling you will not get by looking at these concepts from a very surface level these concepts from a very surface level these concepts from a very surface level to get that you need to understand the to get that you need to understand the to get that you need to understand the mathematics and the physics behind these mathematics and the physics behind these mathematics and the physics behind these concepts as I explained in the concepts as I explained in the concepts as I explained in the introduction the many of the concepts introduction the many of the concepts introduction the many of the concepts are actually inspired from physics which are actually inspired from physics which are actually inspired from physics which is also very interesting. is also very interesting. is also very interesting. So this is how we are planning to So this is how we are planning to So this is how we are planning to conduct this course and uh I'm very conduct this course and uh I'm very conduct this course and uh I'm very excited to teach this course to all of excited to teach this course to all of excited to teach this course to all of you. I'm planning to release every you. I'm planning to release every you. I'm planning to release every lecture either once in a week or once in lecture either once in a week or once in lecture either once in a week or once in two weeks. two weeks. two weeks. Thank you very much everyone and I look Thank you very much everyone and I look Thank you very much everyone and I look forward to seeing you in this course.