← back to concept

Masked Diffusion — across courses

cmu-10799-diffusion-flow

  1. L11 @ 2s

    All right. All right. Right. So, so far what we have learn we Right. So, so far what we have learn we Right. So, so far what we have learn we have learned about like pretty much have learned about like pretty much have learned about like pretty much everything about diffusion models or everything about diffusion models or everything about diffusion models or like the overall family of the diffusion like the overall family of the diffusion like the overall family of the diffusion models right for image generation models right for image generation models right for image generation specifically. Uh so we have covered the specifically. Uh so we have covered the specifically. Uh so we have covered the fundamentals the denoising the diffusion fundamentals the denoising the diffusion fundamentals the denoising the diffusion models the scorebased models and models the scorebased models and models the scorebased models and the full matching all of them are pretty the full matching all of them are pretty the full matching all of them are pretty much the same thing right and in order much the same thing right and in order much the same thing right and in order to improve upon that we have covered a to improve upon that we have covered a to improve upon that we have covered a bunch of advanced topics that includes bunch of advanced topics that includes bunch of advanced topics that includes the design space of diffusion model uh the design space of diffusion model uh the design space of diffusion model uh you know how can we do faster sampling you know how can we do faster sampling you know how can we do faster sampling by various inference time method uh how by various inference time method uh how by various inference time method uh how can we do controllable generations and can we do controllable generations and can we do controllable generations and specifically how can we do test the specifically how can we do test the specifically how can we do test the image generation and we last week we image generation and we last week we image generation and we last week we also talked about many different you also talked about many different you also talked about many different you know method know method know method to do distillation or uh you know self

  2. L11 @ 56s

    to do distillation or uh you know self to do distillation or uh you know self dissolations or flow maps or consistency dissolations or flow maps or consistency dissolations or flow maps or consistency models those things um but on the other models those things um but on the other models those things um but on the other hand I know a lot of you guys are from hand I know a lot of you guys are from hand I know a lot of you guys are from LTI LTI LTI so far everything we have learned about so far everything we have learned about so far everything we have learned about are about image or uh in image that's are about image or uh in image that's are about image or uh in image that's get represented in a continuous space get represented in a continuous space get represented in a continuous space right because you know in your homework right because you know in your homework right because you know in your homework one and two you need to like you know one and two you need to like you know one and two you need to like you know rescale it to negative one and one and rescale it to negative one and one and rescale it to negative one and one and stuff, right? So the the problem stuff, right? So the the problem stuff, right? So the the problem is that then how about discrete data is that then how about discrete data is that then how about discrete data then? Like then? Like then? Like is it possible for us to extend what we is it possible for us to extend what we is it possible for us to extend what we have learned to discrete data? Now why have learned to discrete data? Now why have learned to discrete data? Now why do we care about discrete data is do we care about discrete data is do we care about discrete data is because it's actually quite important. because it's actually quite important. because it's actually quite important. Matter of fact, some people, not to name Matter of fact, some people, not to name Matter of fact, some people, not to name who, actually, I'll just name who, like who, actually, I'll just name who, like

  3. L11 @ 105s

    who, actually, I'll just name who, like the people from Anthropic have claimed the people from Anthropic have claimed the people from Anthropic have claimed to me that uh I I believe it's ICML 2024 to me that uh I I believe it's ICML 2024 to me that uh I I believe it's ICML 2024 that like uh they said, well, anything that like uh they said, well, anything that like uh they said, well, anything that cannot be expressed or compressed that cannot be expressed or compressed that cannot be expressed or compressed into text is uninteresting information into text is uninteresting information into text is uninteresting information and uh you know, image generation is and uh you know, image generation is and uh you know, image generation is just not on the path to AGI. This is just not on the path to AGI. This is just not on the path to AGI. This is what they said. Okay, rumor has that what they said. Okay, rumor has that what they said. Okay, rumor has that they're going to release a image uh they're going to release a image uh they're going to release a image uh generation model. So, we shall see if generation model. So, we shall see if generation model. So, we shall see if it's actually on the path to AGI. But it's actually on the path to AGI. But it's actually on the path to AGI. But regardless, you know, all of those AGI regardless, you know, all of those AGI regardless, you know, all of those AGI bros probably uh they really believe bros probably uh they really believe bros probably uh they really believe that text is like uh like the the path that text is like uh like the the path that text is like uh like the the path to, you know, super intelligence and to, you know, super intelligence and to, you know, super intelligence and whatever. So, it's like super important whatever. So, it's like super important whatever. So, it's like super important I guess right now uh for investment I guess right now uh for investment I guess right now uh for investment reasons and whatever. Um reasons and whatever. Um reasons and whatever. Um uh I mean obviously you know we have uh I mean obviously you know we have

  4. L11 @ 157s

    uh I mean obviously you know we have other important modalities right for other important modalities right for other important modalities right for example like molecules or like DNA example like molecules or like DNA example like molecules or like DNA sequences and stuff right and those sequences and stuff right and those sequences and stuff right and those things are oftent times at least things are oftent times at least things are oftent times at least partially um represented by uh discrete partially um represented by uh discrete partially um represented by uh discrete data and moreover even image right so data and moreover even image right so data and moreover even image right so like right now like right now like right now do we store image is by not actually in do we store image is by not actually in do we store image is by not actually in a continuous space but actually in like a continuous space but actually in like a continuous space but actually in like a quantiz a quantiz a quantiz discretized pixel space, right? So you discretized pixel space, right? So you discretized pixel space, right? So you actually what you're seeing is a bunch actually what you're seeing is a bunch actually what you're seeing is a bunch of like different pixels and each pixel of like different pixels and each pixel of like different pixels and each pixel actually has three channels and each actually has three channels and each actually has three channels and each channel actually has a value like a channel actually has a value like a channel actually has a value like a discrete value from 0 to 255, right? So discrete value from 0 to 255, right? So discrete value from 0 to 255, right? So this is like technically image should this is like technically image should this is like technically image should also be represented as discrete data. So also be represented as discrete data. So also be represented as discrete data. So uh uh uh how do we make diffusion models work on how do we make diffusion models work on how do we make diffusion models work on discrete data? discrete data? discrete data? What? Yes. What? Yes. What? Yes. >> Near space and do like a nearest >> Near space and do like a nearest

  5. L11 @ 215s

    >> Near space and do like a nearest neighbors type thing to match it. neighbors type thing to match it. neighbors type thing to match it. >> Uh so diffusion model on a continuous >> Uh so diffusion model on a continuous >> Uh so diffusion model on a continuous space and then do nearest neighbor. This space and then do nearest neighbor. This space and then do nearest neighbor. This is a good good uh good uh good approach. is a good good uh good uh good approach. is a good good uh good uh good approach. Uh are do you have any allergies? Great. Uh are do you have any allergies? Great. Uh are do you have any allergies? Great. All right. All right. It has nuts in it. So that's why I need It has nuts in it. So that's why I need It has nuts in it. So that's why I need to ask. All right. Uh, do you prefer to to ask. All right. Uh, do you prefer to to ask. All right. Uh, do you prefer to have blessing on getting rich or getting have blessing on getting rich or getting have blessing on getting rich or getting good academic achievement? good academic achievement? good academic achievement? >> I think they're related. >> I think they're related. >> I think they're related. >> Well, which one you choose one? >> Well, which one you choose one? >> Well, which one you choose one? >> Maybe rich. The other one I can do. >> Maybe rich. The other one I can do. >> Maybe rich. The other one I can do. >> Okay. >> Okay. >> Okay. All right. All right. All right. Anyone else want to have a All right. Anyone else want to have a All right. Anyone else want to have a take a guess take a guess take a guess or like some you know attempt I hear something but I don't see okay I hear something but I don't see okay so even right now text is first gone it so even right now text is first gone it so even right now text is first gone it goes through an embedding process and goes through an embedding process and goes through an embedding process and then embedding is what goes through then embedding is what goes through

  6. L11 @ 284s

    then embedding is what goes through >> so you first take the text and go to >> so you first take the text and go to >> so you first take the text and go to embedding embedding existing embedding embedding existing embedding embedding existing of the embeddings of the embeddings of the embeddings >> and then when you have to unimpify it >> and then when you have to unimpify it >> and then when you have to unimpify it that time then you need some other model that time then you need some other model that time then you need some other model to do the to do the to do the >> decoding or something right okay so you >> decoding or something right okay so you >> decoding or something right okay so you guys kind of have a similar uh like idea guys kind of have a similar uh like idea guys kind of have a similar uh like idea and this idea actually was explored by and this idea actually was explored by and this idea actually was explored by people from Stanford and it's actually people from Stanford and it's actually people from Stanford and it's actually also it's like kind of like the first also it's like kind of like the first also it's like kind of like the first like diffusion language model if you like diffusion language model if you like diffusion language model if you will uh but this is like pretty much will uh but this is like pretty much will uh but this is like pretty much exactly what they did uh unfortunately exactly what they did uh unfortunately exactly what they did uh unfortunately it it didn't work as well as what we're it it didn't work as well as what we're it it didn't work as well as what we're going to talk about today. But that was going to talk about today. But that was going to talk about today. But that was the first attempt. Do I have allergies? the first attempt. Do I have allergies? the first attempt. Do I have allergies? All right. Get get uh I'm kind of All right. Get get uh I'm kind of All right. Get get uh I'm kind of running out of the allergyfree options. running out of the allergyfree options.

  7. L11 @ 326s

    running out of the allergyfree options. Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy Hold on. Uh sorry. The the allergy sensitive option. Okay. Anyway, you're sensitive option. Okay. Anyway, you're sensitive option. Okay. Anyway, you're getting something that is an allergy getting something that is an allergy getting something that is an allergy invariant. Uh invariant. Uh invariant. Uh uh uh uh. Do you want to get rich or do uh uh uh. Do you want to get rich or do uh uh uh. Do you want to get rich or do you want to get you want to get you want to get >> Okay. Dang. I don't want to get rich. That's Dang. I don't want to get rich. That's crazy. All right. Anyway, crazy. All right. Anyway, crazy. All right. Anyway, okay. Um anyway, but um if you if we okay. Um anyway, but um if you if we okay. Um anyway, but um if you if we think about it, right, diffusion models think about it, right, diffusion models think about it, right, diffusion models can be viewed in three ways, right? As can be viewed in three ways, right? As can be viewed in three ways, right? As we have mentioned before, at least we have mentioned before, at least we have mentioned before, at least continuous diffusion. We have denoising continuous diffusion. We have denoising continuous diffusion. We have denoising diffusion models, which is basically diffusion models, which is basically diffusion models, which is basically we're trying to learn well add noise in we're trying to learn well add noise in we're trying to learn well add noise in the forward process and learn to dn the forward process and learn to dn the forward process and learn to dn noiseise in the in the reverse process. noiseise in the in the reverse process. noiseise in the in the reverse process. And in the scorebased models, we're And in the scorebased models, we're And in the scorebased models, we're trying to learn a score function and trying to learn a score function and trying to learn a score function and then somehow sample through a OD or SDE. then somehow sample through a OD or SDE. then somehow sample through a OD or SDE. And then uh for flow matching is pretty

  8. L11 @ 379s

    And then uh for flow matching is pretty And then uh for flow matching is pretty much the same thing, but instead of much the same thing, but instead of much the same thing, but instead of learning the score function directly, learning the score function directly, learning the score function directly, we're learning some velocity, right? So we're learning some velocity, right? So we're learning some velocity, right? So basically can do we have like the like a basically can do we have like the like a basically can do we have like the like a like a mapping from the formulations in like a mapping from the formulations in like a mapping from the formulations in continuous diffusion to discrete continuous diffusion to discrete continuous diffusion to discrete diffusion. This is the question that diffusion. This is the question that diffusion. This is the question that we're mainly going to be as uh be we're mainly going to be as uh be we're mainly going to be as uh be answering today. Okay, so let's look at answering today. Okay, so let's look at answering today. Okay, so let's look at the first or maybe the most like naive the first or maybe the most like naive the first or maybe the most like naive way to think about this, right? Um, so way to think about this, right? Um, so way to think about this, right? Um, so say like how do we add noise uh and say like how do we add noise uh and say like how do we add noise uh and learn to d noiseise in a discrete space learn to d noiseise in a discrete space learn to d noiseise in a discrete space first. Okay, so let's get a reminder of first. Okay, so let's get a reminder of first. Okay, so let's get a reminder of how diffusion works. I'm sure you guys how diffusion works. I'm sure you guys how diffusion works. I'm sure you guys know, but like you know you just add know, but like you know you just add know, but like you know you just add noise and then gradually d noiseise. The noise and then gradually d noiseise. The noise and then gradually d noiseise. The adding noise process is the forward adding noise process is the forward

  9. L11 @ 423s

    adding noise process is the forward process and the gradually d noiseis process and the gradually d noiseis process and the gradually d noiseis process is the reverse process. Right. process is the reverse process. Right. process is the reverse process. Right. So I guess the first question is how do So I guess the first question is how do So I guess the first question is how do we define noise quote unquote in say we define noise quote unquote in say we define noise quote unquote in say like a text space right? Okay like a text space right? Okay like a text space right? Okay >> tokens. >> tokens. >> tokens. >> Masking tokens. Okay. Good idea. Anyone >> Masking tokens. Okay. Good idea. Anyone >> Masking tokens. Okay. Good idea. Anyone else? Yeah. >> Incorrect like just like rand random >> Incorrect like just like rand random things, right? Basically things, right? Basically things, right? Basically >> Yeah. Yeah. Yeah. like wrong tokens. >> Yeah. Yeah. Yeah. like wrong tokens. >> Yeah. Yeah. Yeah. like wrong tokens. Okay, great. Uh well, allergy. Okay, great. Uh well, allergy. Okay, great. Uh well, allergy. Okay, everything is a vegetarian by the Okay, everything is a vegetarian by the Okay, everything is a vegetarian by the way, so no worries. But let's hope Oh. way, so no worries. But let's hope Oh. way, so no worries. But let's hope Oh. Uh okay. Uh good academic or ah man, you Uh okay. Uh good academic or ah man, you Uh okay. Uh good academic or ah man, you guys are just uh too serious. Okay. guys are just uh too serious. Okay. guys are just uh too serious. Okay. Okay. Or okay. All right. Okay. Or okay. All right. Okay. Or okay. All right. Cool. Anyway. Cool. Anyway. Cool. Anyway. All right. So let's see All right. So let's see All right. So let's see the like how do we define noise in text the like how do we define noise in text the like how do we define noise in text right so let's this is like a noised right so let's this is like a noised right so let's this is like a noised version of a tweet from Andre Kapathi uh version of a tweet from Andre Kapathi uh

  10. L11 @ 489s

    version of a tweet from Andre Kapathi uh so let's identify some of the noise in so let's identify some of the noise in so let's identify some of the noise in this tweet what do we see this tweet what do we see this tweet what do we see >> I see random emojis in between and >> I see random emojis in between and >> I see random emojis in between and random characters random characters random characters >> random emojis >> random emojis >> random emojis all right you're getting allergy all right you're getting allergy all right you're getting allergy >> all right hold on >> all right hold on >> all right hold on poor Dvanchu is getting uh invarian poor Dvanchu is getting uh invarian poor Dvanchu is getting uh invarian version but it's okay. All right. Uh version but it's okay. All right. Uh version but it's okay. All right. Uh academic or >> get rich. >> get rich. Okay. All right. Okay. All right. Okay. All right. Okay. Uh random emojis. What else? Okay. Uh random emojis. What else? Okay. Uh random emojis. What else? >> Spellings are wrong. >> Spellings are wrong. >> Spellings are wrong. >> Spellings are wrong. >> Spellings are wrong. Hold on. Hold on. Hold on. Hold on. Hold on. Hold on. Get rich or Get rich or Get rich or >> Okay. >> Okay. Oh, this is actually a good one. way the Oh, this is actually a good one. way the Oh, this is actually a good one. way the the the envelope. Okay. Uh the the envelope. Okay. Uh the the envelope. Okay. Uh uh yeah, but basically pretty much that, uh yeah, but basically pretty much that, uh yeah, but basically pretty much that, right? So like so this is the original right? So like so this is the original right? So like so this is the original tweet and you'll get like some random tweet and you'll get like some random tweet and you'll get like some random tokens here and there. You get some like tokens here and there. You get some like tokens here and there. You get some like weird modifications to the tokens and weird modifications to the tokens and weird modifications to the tokens and then you get some weird emojis and you

  11. L11 @ 555s

    then you get some weird emojis and you then you get some weird emojis and you get some like out of place like six, get some like out of place like six, get some like out of place like six, seven like tokens just who knows what's seven like tokens just who knows what's seven like tokens just who knows what's going on. Um yeah so essentially these going on. Um yeah so essentially these going on. Um yeah so essentially these are the noise right then uh how so if we are the noise right then uh how so if we are the noise right then uh how so if we want to define noise as something like want to define noise as something like want to define noise as something like that basically just some random out that basically just some random out that basically just some random out ofplace tokens then how do we add noise ofplace tokens then how do we add noise ofplace tokens then how do we add noise then to the text then to the text then to the text to a piece of like given a piece of text to a piece of like given a piece of text to a piece of like given a piece of text how do we add noise how do we add noise how do we add noise >> well yeah >> well yeah >> well yeah that is uh well hold on who who who that is uh well hold on who who who that is uh well hold on who who who raised their hands first I actually raised their hands first I actually raised their hands first I actually don't okay yeah don't okay yeah don't okay yeah >> I don't know something irrelevant. >> I don't know something irrelevant. >> I don't know something irrelevant. >> Add something add something irrelevant. >> Add something add something irrelevant. >> Add something add something irrelevant. Sort sort of sort of but um uh maybe Sort sort of sort of but um uh maybe Sort sort of sort of but um uh maybe formalize it as like a say given a piece formalize it as like a say given a piece formalize it as like a say given a piece of text. >> Yeah. Okay.

  12. L11 @ 613s

    >> Yeah. Okay. >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh >> Masking also works. Okay. Uh hold on. Uh do you have allergy? Okay. This is also do you have allergy? Okay. This is also do you have allergy? Okay. This is also allergy inv. Do you have allergy inv. Do you have allergy inv. Do you have >> get rich or >> get rich or >> get rich or >> both? >> both? >> both? >> Choose. >> Choose. >> Choose. >> Oh, yeah. Ask him too much, man. Only >> Oh, yeah. Ask him too much, man. Only >> Oh, yeah. Ask him too much, man. Only one. Okay. You're getting rich. Okay. one. Okay. You're getting rich. Okay. one. Okay. You're getting rich. Okay. What about you? What about you? What about you? >> Hold on. Hold on. No allergy. >> Hold on. Hold on. No allergy. >> Hold on. Hold on. No allergy. Give me a sec. All right. Let's see. Let's see. Uh, no. Okay. Uh, no. Okay. Uh, no. Okay. This is the allergen version, I guess. This is the allergen version, I guess. This is the allergen version, I guess. Dang, nobody has allergy in this class. Dang, nobody has allergy in this class. Dang, nobody has allergy in this class. This is amazing. This is amazing. This is amazing. >> Okay. Uh, which one? Get rich. >> Okay. Uh, which one? Get rich. >> Okay. Uh, which one? Get rich. >> Rich for sure. >> Rich for sure. >> Rich for sure. >> Okay. Oh, thank you. Um anyway, >> Okay. Oh, thank you. Um anyway, >> Okay. Oh, thank you. Um anyway, yeah, but essentially yeah, but essentially yeah, but essentially the idea is that the idea is that the idea is that the idea is that like for every token, the idea is that like for every token, the idea is that like for every token, we can just have some like random we can just have some like random we can just have some like random probability that transform this token probability that transform this token probability that transform this token from the like original uh clean data to from the like original uh clean data to

  13. L11 @ 683s

    from the like original uh clean data to some random tokens right in the some random tokens right in the some random tokens right in the vocabulary. So that basically say like vocabulary. So that basically say like vocabulary. So that basically say like you just randomly select a token and you just randomly select a token and you just randomly select a token and then this token get randomly transformed then this token get randomly transformed then this token get randomly transformed into 67 right okay so mathematically into 67 right okay so mathematically into 67 right okay so mathematically uh or like I guess like not so formally uh or like I guess like not so formally uh or like I guess like not so formally mathematically how we say just let's say mathematically how we say just let's say mathematically how we say just let's say we have a vocabulary uh say like the we have a vocabulary uh say like the we have a vocabulary uh say like the vocabulary is really short just I love vocabulary is really short just I love vocabulary is really short just I love cat three possible tokens and uh then cat three possible tokens and uh then cat three possible tokens and uh then the sentence I love cat can be the sentence I love cat can be the sentence I love cat can be represented by three one whole vectors represented by three one whole vectors represented by three one whole vectors vectors. Uh vectors. Uh vectors. Uh the one half vectors is incorrect the one half vectors is incorrect the one half vectors is incorrect because I you I wanted to have five because I you I wanted to have five because I you I wanted to have five tokens but then it's too much math to tokens but then it's too much math to tokens but then it's too much math to do. So uh but yeah the one half vector do. So uh but yeah the one half vector do. So uh but yeah the one half vector should be give me a second. All right. Anyway you can you can All right. Anyway you can you can represent everything in a one half represent everything in a one half represent everything in a one half vector. Uh then basically say we have a

  14. L11 @ 743s

    vector. Uh then basically say we have a vector. Uh then basically say we have a beta chance. This is just a some chance beta chance. This is just a some chance beta chance. This is just a some chance to turn that an existing token into a to turn that an existing token into a to turn that an existing token into a random one in the vocabulary. Uh that random one in the vocabulary. Uh that random one in the vocabulary. Uh that this transformation can be represented this transformation can be represented this transformation can be represented by this transition matrix essentially uh by this transition matrix essentially uh by this transition matrix essentially uh which we represent as Q. Uh where yeah which we represent as Q. Uh where yeah which we represent as Q. Uh where yeah like it's literally just like um sorry like it's literally just like um sorry like it's literally just like um sorry not beta. Yeah. So it's literally not beta. Yeah. So it's literally not beta. Yeah. So it's literally because we have a beta chance to uh you because we have a beta chance to uh you because we have a beta chance to uh you know sample it to be any um like know sample it to be any um like know sample it to be any um like vocabulary in uh any token in the vocabulary in uh any token in the vocabulary in uh any token in the vocabulary. You can also resample the vocabulary. You can also resample the vocabulary. You can also resample the same one. So that's why it's basically same one. So that's why it's basically same one. So that's why it's basically just like every everyone has like just like every everyone has like just like every everyone has like onethird of beta of a chance to get onethird of beta of a chance to get onethird of beta of a chance to get sampled when we try to you know resample sampled when we try to you know resample sampled when we try to you know resample but otherwise we stay the same. So this but otherwise we stay the same. So this but otherwise we stay the same. So this is why this is like one minus you know

  15. L11 @ 789s

    is why this is like one minus you know is why this is like one minus you know every everything else has one beta over every everything else has one beta over every everything else has one beta over three chance and then staying the same three chance and then staying the same three chance and then staying the same has a 1 minus 2 beta over 3. has a 1 minus 2 beta over 3. has a 1 minus 2 beta over 3. Okay, make sense? Okay, make sense? Okay, make sense? All right. Now, uh let's apply this All right. Now, uh let's apply this All right. Now, uh let's apply this transition matrix to the third token say transition matrix to the third token say transition matrix to the third token say cat, right? Uh then we literally just do cat, right? Uh then we literally just do cat, right? Uh then we literally just do you know one hot vector times transition you know one hot vector times transition you know one hot vector times transition matrix equals to the new uh probability matrix equals to the new uh probability matrix equals to the new uh probability like literally equal to the new like like literally equal to the new like like literally equal to the new like probability distribution. So essentially probability distribution. So essentially probability distribution. So essentially right now this is like you have like right now this is like you have like right now this is like you have like beta over three chance to get transition beta over three chance to get transition beta over three chance to get transition into I uh beta over three chance to get into I uh beta over three chance to get into I uh beta over three chance to get transition love and then one minus 2 transition love and then one minus 2 transition love and then one minus 2 beta over three chance into staying at beta over three chance into staying at beta over three chance into staying at cat. Okay cat. Okay cat. Okay any question? Yeah, any question? Yeah, any question? Yeah, >> when we actually do this for real >> when we actually do this for real >> when we actually do this for real sentences with real vocabulary, it's not

  16. L11 @ 848s

    sentences with real vocabulary, it's not sentences with real vocabulary, it's not necessary that the noise sentence is necessary that the noise sentence is necessary that the noise sentence is actually like it might like there is a actually like it might like there is a actually like it might like there is a chance that say we replace some word chance that say we replace some word chance that say we replace some word which is not conse. >> Yes, that's right. But like just think >> Yes, that's right. But like just think about how many like tokens available out about how many like tokens available out about how many like tokens available out there, right? the chances of that is non there, right? the chances of that is non there, right? the chances of that is non zero but like also not high right okay zero but like also not high right okay zero but like also not high right okay uh but so basically uh we can describe uh but so basically uh we can describe uh but so basically uh we can describe this whole thing into a categorical this whole thing into a categorical this whole thing into a categorical distribution right so essentially it distribution right so essentially it distribution right so essentially it just like the transform token given the just like the transform token given the just like the transform token given the previous token which is cat uh follows a previous token which is cat uh follows a previous token which is cat uh follows a categorical distribution that can be you categorical distribution that can be you categorical distribution that can be you know uh represented by this uh know uh represented by this uh know uh represented by this uh probability probability probability distribution or like this vector distribution or like this vector distribution or like this vector essentially Okay. Uh and if we stack all essentially Okay. Uh and if we stack all essentially Okay. Uh and if we stack all three uh then we can apply three uh then we can apply three uh then we can apply hold on if we stack all three tokens and hold on if we stack all three tokens and hold on if we stack all three tokens and then you know so you just so the first

  17. L11 @ 910s

    then you know so you just so the first then you know so you just so the first row is uh I second row is love uh third row is uh I second row is love uh third row is uh I second row is love uh third row is cat uh then and then we can row is cat uh then and then we can row is cat uh then and then we can independently analyze to each token independently analyze to each token independently analyze to each token right uh then we can get basically just right uh then we can get basically just right uh then we can get basically just matrix multiplication and then you get matrix multiplication and then you get matrix multiplication and then you get the thing that you want. Uh and then the the thing that you want. Uh and then the the thing that you want. Uh and then the categorical probability distribution can categorical probability distribution can categorical probability distribution can also be represented as this matrix also be represented as this matrix also be represented as this matrix multiplication version of the multiplication version of the multiplication version of the representation. representation. representation. Okay. Any question? No question. Great. All right. So now No question. Great. All right. So now let's build the diffusion for process let's build the diffusion for process let's build the diffusion for process using this definition of noise. Right. using this definition of noise. Right. using this definition of noise. Right. Uh so literally you just add a subscript Uh so literally you just add a subscript Uh so literally you just add a subscript t to everything and that's diffusion a t to everything and that's diffusion a t to everything and that's diffusion a so easy. so easy. so easy. Um well uh the well it is pretty easy Um well uh the well it is pretty easy Um well uh the well it is pretty easy and also uh the the good thing about and also uh the the good thing about and also uh the the good thing about having this kind of transformation is having this kind of transformation is having this kind of transformation is that by induction we can not only get that by induction we can not only get

  18. L11 @ 981s

    that by induction we can not only get like basically it's the same thing as like basically it's the same thing as like basically it's the same thing as like you know two gausians added like you know two gausians added like you know two gausians added together equal to the same equal to together equal to the same equal to together equal to the same equal to another gausian. Uh this one is also another gausian. Uh this one is also another gausian. Uh this one is also like basically uh you know transforming like basically uh you know transforming like basically uh you know transforming from uh t minus 2 to t is equivalent to from uh t minus 2 to t is equivalent to from uh t minus 2 to t is equivalent to first transition from t minus2 to t first transition from t minus2 to t first transition from t minus2 to t minus one and then transition from t minus one and then transition from t minus one and then transition from t minus one to t right uh so basically you minus one to t right uh so basically you minus one to t right uh so basically you just like literally have two matrix just like literally have two matrix just like literally have two matrix multiplications right so by induction we multiplications right so by induction we multiplications right so by induction we can literally get the onestep uh you can literally get the onestep uh you can literally get the onestep uh you know transformation from time zero to know transformation from time zero to know transformation from time zero to time t to be something like this Right. time t to be something like this Right. time t to be something like this Right. So essentially just the the the the one So essentially just the the the the one So essentially just the the the the one step before jumping is like the same step before jumping is like the same step before jumping is like the same like anal like analogy to continuous like anal like analogy to continuous like anal like analogy to continuous diffusion DDPN one step four right. So diffusion DDPN one step four right. So diffusion DDPN one step four right. So this is like literally how to jump

  19. L11 @ 1031s

    this is like literally how to jump this is like literally how to jump directly from x0 to xt is you just like directly from x0 to xt is you just like directly from x0 to xt is you just like kind of like apply this like apply this kind of like apply this like apply this kind of like apply this like apply this giant uh matrix multiplication term giant uh matrix multiplication term giant uh matrix multiplication term which you can calculate you know either which you can calculate you know either which you can calculate you know either ahead of time or in some smart ways or ahead of time or in some smart ways or ahead of time or in some smart ways or yeah stuff like that basically. yeah stuff like that basically. yeah stuff like that basically. Any questions? if you go back. if you go back. >> H you should uh did I not add tea? I may >> H you should uh did I not add tea? I may >> H you should uh did I not add tea? I may not add tea. Let's see. >> So like it's you also have like a >> So like it's you also have like a noising schedule if you will. Yeah. >> All right. Any question? >> All right. Any question? Yeah. Yeah. Yeah. How do you really exactly like calculate How do you really exactly like calculate How do you really exactly like calculate beta at each time step? beta at each time step? beta at each time step? >> How do you exactly calculate beta is >> How do you exactly calculate beta is >> How do you exactly calculate beta is set? It's like the same as the beta in set? It's like the same as the beta in set? It's like the same as the beta in DDPM. DDPM. DDPM. You choose it. Yeah. You choose it. Yeah. You choose it. Yeah. >> Why is beta? I thought it depend on >> Why is beta? I thought it depend on >> Why is beta? I thought it depend on >> Yeah. Beta t is a set schedule. The same >> Yeah. Beta t is a set schedule. The same >> Yeah. Beta t is a set schedule. The same as the beta t in uh DDPM.

  20. L11 @ 1104s

    as the beta t in uh DDPM. as the beta t in uh DDPM. >> Yeah. Don't worry. Everything's gonna be Don't worry. Everything's gonna be simplified later on in this class. right simplified later on in this class. right simplified later on in this class. right now is the the more difficult one now is the the more difficult one now is the the more difficult one because this is from like 2021 or because this is from like 2021 or because this is from like 2021 or something. So people are still confused. something. So people are still confused. something. So people are still confused. All right. Anyway, uh any questions? >> Is this engram? What do you think? This >> Is this engram? What do you think? This is actually a good question. What is is actually a good question. What is is actually a good question. What is this? Is this What's the difference this? Is this What's the difference this? Is this What's the difference between engram and this actually? depends on the tokens that come before depends on the tokens that come before it. But this doesn't have that. it. But this doesn't have that. it. But this doesn't have that. >> Yeah. So, Ang needs to be neighboring uh >> Yeah. So, Ang needs to be neighboring uh >> Yeah. So, Ang needs to be neighboring uh tokens, right? And this one it can be tokens, right? And this one it can be tokens, right? And this one it can be any. any. any. >> Yeah. All right. Uh well, I think I run out. I don't know. well, I think I run out. I don't know. I'm not sure. Uh I'm not sure. Uh I'm not sure. Uh >> okay. Okay. >> okay. Okay. >> okay. Okay. All right. All right. All right. Okay. All right. All right. All right. Okay. All right. All right. All right. Okay. Um anyway, cool. Now, how do we learn Um anyway, cool. Now, how do we learn Um anyway, cool. Now, how do we learn this reverse process? Anyone have any this reverse process? Anyone have any this reverse process? Anyone have any ideas? ideas? ideas? What do we think? What do we think? What do we think? >> Yeah. >> Yeah. >> Yeah. >> Estimate these points and then Q

  21. L11 @ 1180s

    >> Estimate these points and then Q >> Estimate these points and then Q transpose. transpose. transpose. >> Yes, pretty much. And then specifically, >> Yes, pretty much. And then specifically, >> Yes, pretty much. And then specifically, which what kind of loss fun do we want which what kind of loss fun do we want which what kind of loss fun do we want to learn? Well, yeah, kind of. Well, actually we we we kind of just Well, actually we we we kind of just need to use the same elbow from DDPM need to use the same elbow from DDPM need to use the same elbow from DDPM because it's kind of the same thing. The because it's kind of the same thing. The because it's kind of the same thing. The only difference is that we have only difference is that we have only difference is that we have different like different like different like uh I guess uh expressions for the Q's uh I guess uh expressions for the Q's uh I guess uh expressions for the Q's and the P's. And so we have the Q and and the P's. And so we have the Q and and the P's. And so we have the Q and that just like a you know reconstruction that just like a you know reconstruction that just like a you know reconstruction cross entropy. So we can just ignore we cross entropy. So we can just ignore we cross entropy. So we can just ignore we know how to do those. So the only thing know how to do those. So the only thing know how to do those. So the only thing we pretty much need to deal with is this we pretty much need to deal with is this we pretty much need to deal with is this this new Q, right? So how do we derive this new Q, right? So how do we derive this new Q, right? So how do we derive this new Q here? Well, it's actually this new Q here? Well, it's actually this new Q here? Well, it's actually very simple. So you based it and then very simple. So you based it and then very simple. So you based it and then because you're getting you're marovian because you're getting you're marovian

  22. L11 @ 1237s

    because you're getting you're marovian like everything is marovian here. So you like everything is marovian here. So you like everything is marovian here. So you can ignore the x0 here. So then this can ignore the x0 here. So then this can ignore the x0 here. So then this becomes uh xt given xt minus one. Uh and becomes uh xt given xt minus one. Uh and becomes uh xt given xt minus one. Uh and then um this is so qxt given x minus one then um this is so qxt given x minus one then um this is so qxt given x minus one times q which we know which we define times q which we know which we define times q which we know which we define and then q of xtus one given x which we and then q of xtus one given x which we and then q of xtus one given x which we also know from previous slides and also know from previous slides and also know from previous slides and similarly for the denominator um similarly for the denominator um similarly for the denominator um denominator. Yeah. Um so basically you denominator. Yeah. Um so basically you denominator. Yeah. Um so basically you just plug everything in and then you get just plug everything in and then you get just plug everything in and then you get something like this. And because this something like this. And because this something like this. And because this needs to be a function of uh xt minus needs to be a function of uh xt minus needs to be a function of uh xt minus one uh so you basically rearrange some one uh so you basically rearrange some one uh so you basically rearrange some terms and you get something that is terms and you get something that is terms and you get something that is equivalent. Um so now equivalent. Um so now equivalent. Um so now essentially um q of x um t minus one is essentially um q of x um t minus one is essentially um q of x um t minus one is literally just equal to um yeah the literally just equal to um yeah the literally just equal to um yeah the categorical distribution that is defined

  23. L11 @ 1295s

    categorical distribution that is defined categorical distribution that is defined by this black font term here. >> Oh dot is like elementwise >> Oh dot is like elementwise multiplication. Anyway, honestly, doesn't matter. The Anyway, honestly, doesn't matter. The only thing that you need to know is that only thing that you need to know is that only thing that you need to know is that we can have closed form solution for we can have closed form solution for we can have closed form solution for this term. All right, so now we have this term. All right, so now we have this term. All right, so now we have that as well. So basically uh we can that as well. So basically uh we can that as well. So basically uh we can just directly parameterize it this way. just directly parameterize it this way. just directly parameterize it this way. But uh you know by DDPM we also know But uh you know by DDPM we also know But uh you know by DDPM we also know that you know a better parameterization that you know a better parameterization that you know a better parameterization you know homework one uh question six you know homework one uh question six you know homework one uh question six right better parameterization can have a right better parameterization can have a right better parameterization can have a lot of effect on your training lot of effect on your training lot of effect on your training performance. Uh so what is the best performance. Uh so what is the best performance. Uh so what is the best parameterization in this case? Like what parameterization in this case? Like what parameterization in this case? Like what what should be the training target like what should be the training target like what should be the training target like what should be the the thing that the what should be the the thing that the what should be the the thing that the neuronet network produce neuronet network produce neuronet network produce then then then >> uh >> uh >> uh the noise may be a little bit more the noise may be a little bit more the noise may be a little bit more difficult to define in this case. Right? difficult to define in this case. Right?

  24. L11 @ 1362s

    difficult to define in this case. Right? What is the other thing? What is the other thing? What is the other thing? >> Tokens. >> Tokens. >> Tokens. >> The what >> The what >> The what >> the probability is of the tokens. the >> the probability is of the tokens. the >> the probability is of the tokens. the categorical distribution at each categorical distribution at each categorical distribution at each position. position. position. >> Well, that is just that, right? So, is >> Well, that is just that, right? So, is >> Well, that is just that, right? So, is there like any like easier target that there like any like easier target that there like any like easier target that we can that we can um you know, we can that we can um you know, we can that we can um you know, we have the noise prediction and we have we have the noise prediction and we have we have the noise prediction and we have >> clean prediction, right? Literally just >> clean prediction, right? Literally just >> clean prediction, right? Literally just that. Uh but basically what you can do that. Uh but basically what you can do that. Uh but basically what you can do is you can just channel your inner bay is you can just channel your inner bay is you can just channel your inner bay again. Uh and then you can basically again. Uh and then you can basically again. Uh and then you can basically just like decompose uh you know your just like decompose uh you know your just like decompose uh you know your target into or like your objective or target into or like your objective or target into or like your objective or not objective your your uh you know the not objective your your uh you know the not objective your your uh you know the thing that you're supposed to predict thing that you're supposed to predict thing that you're supposed to predict into like the four process times the into like the four process times the into like the four process times the clean data estimation or the clean data clean data estimation or the clean data clean data estimation or the clean data prediction right so literally you need prediction right so literally you need

  25. L11 @ 1410s

    prediction right so literally you need to only you only need to predict the to only you only need to predict the to only you only need to predict the logist of the final clean output given logist of the final clean output given logist of the final clean output given your current erh like noisy input and your current erh like noisy input and your current erh like noisy input and then everything just be calculated ated then everything just be calculated ated then everything just be calculated ated by you know like like the KL divergence by you know like like the KL divergence by you know like like the KL divergence between the two and uh also uh in the between the two and uh also uh in the between the two and uh also uh in the paper they also said oh actually uh you paper they also said oh actually uh you paper they also said oh actually uh you can also add another cross entropy loss can also add another cross entropy loss can also add another cross entropy loss just like just just like the BERT just like just just like the BERT just like just just like the BERT essentially and then it's going to essentially and then it's going to essentially and then it's going to improve the performance a little bit. So improve the performance a little bit. So improve the performance a little bit. So if you put everything together, we got if you put everything together, we got if you put everything together, we got the first uh like proper discrete the first uh like proper discrete the first uh like proper discrete diffusion like so it's actually the the diffusion like so it's actually the the diffusion like so it's actually the the noising process and everything happens noising process and everything happens noising process and everything happens in the discrete space and this is what in the discrete space and this is what in the discrete space and this is what we call a discree uh dn noising we call a discree uh dn noising we call a discree uh dn noising diffusion model because we're 3D so it's diffusion model because we're 3D so it's diffusion model because we're 3D so it's a D3PM. a D3PM.

  26. L11 @ 1458s

    a D3PM. All right, any questions about D3PM? Yes, Yes, >> this won't have like an equivalence to >> this won't have like an equivalence to >> this won't have like an equivalence to like score matching, right? Because of like score matching, right? Because of like score matching, right? Because of the discrete space, the discrete space, the discrete space, >> we have not talked about score matching >> we have not talked about score matching >> we have not talked about score matching yet. Uh but we're going to talk about it yet. Uh but we're going to talk about it yet. Uh but we're going to talk about it next. Any other questions before we move next. Any other questions before we move next. Any other questions before we move on to score matching? on to score matching? on to score matching? >> Yeah. >> Oh, you cannot do variable length here. >> Oh, you cannot do variable length here. Yeah. Yeah. Oh, yeah. We're g we were Yeah. Yeah. Oh, yeah. We're g we were Yeah. Yeah. Oh, yeah. We're g we were gonna talk about it in next class but gonna talk about it in next class but gonna talk about it in next class but they actually fix this kind of fix it I they actually fix this kind of fix it I they actually fix this kind of fix it I guess u but but but but but like using guess u but but but but but like using guess u but but but but but like using some like hacks I guess u but yeah one some like hacks I guess u but yeah one some like hacks I guess u but yeah one downside about diffusion language model downside about diffusion language model downside about diffusion language model is that everything needs to be trained is that everything needs to be trained is that everything needs to be trained in a fixed length uh scenario in a fixed length uh scenario in a fixed length uh scenario unfortunately yes it's not variable yeah unfortunately yes it's not variable yeah unfortunately yes it's not variable yeah >> autogressive models as some kind of uh >> autogressive models as some kind of uh >> autogressive models as some kind of uh like discrete diffusion models like discrete diffusion models

  27. L11 @ 1517s

    like discrete diffusion models >> great question and the answer is Yes. >> great question and the answer is Yes. >> great question and the answer is Yes. And we're going to talk about it next And we're going to talk about it next And we're going to talk about it next class actually. Yeah. But uh allergy. class actually. Yeah. But uh allergy. class actually. Yeah. But uh allergy. All right. Actually, I don't even know All right. Actually, I don't even know All right. Actually, I don't even know if I have non-allergy things anymore, if I have non-allergy things anymore, if I have non-allergy things anymore, but let's see. All right. I give up. All right. I give up. Academic. Dang. Why you guys Nobody's graduating Dang. Why you guys Nobody's graduating soon or something. soon or something. soon or something. All right. Cool. Cool. Cool. All right. So, right. Cool. Cool. Cool. All right. So, you know, as people have been asking, you know, as people have been asking, you know, as people have been asking, uh, so we covered like one aspect of so we covered like one aspect of discrete diffusion which is like kind of discrete diffusion which is like kind of discrete diffusion which is like kind of corresponding to the process of adding corresponding to the process of adding corresponding to the process of adding noise and learning to d noise, right? noise and learning to d noise, right? noise and learning to d noise, right? And then we basically just um, you know And then we basically just um, you know And then we basically just um, you know define our noise in the categorical define our noise in the categorical define our noise in the categorical distribution. So we get that covered. Uh distribution. So we get that covered. Uh distribution. So we get that covered. Uh now what about score function right like now what about score function right like now what about score function right like the score functions the whole concept the score functions the whole concept the score functions the whole concept seems really wrong here but we can do seems really wrong here but we can do seems really wrong here but we can do that. Yeah that. Yeah that. Yeah >> previous >> previous >> previous fixed length. fixed length.

  28. L11 @ 1593s

    fixed length. >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is >> Yeah it's fixed length. Everything is fixed length. Yeah. fixed length. Yeah. fixed length. Yeah. >> It's the same for everything. >> It's the same for everything. >> It's the same for everything. Yeah. So maybe it's fine but like it Yeah. So maybe it's fine but like it Yeah. So maybe it's fine but like it just like the well like formulation wise just like the well like formulation wise just like the well like formulation wise it so like mathematically it has to be it so like mathematically it has to be it so like mathematically it has to be uh like fixed length whereas if you do uh like fixed length whereas if you do uh like fixed length whereas if you do auto reggressive mathematically it auto reggressive mathematically it auto reggressive mathematically it doesn't have to right it's just that doesn't have to right it's just that doesn't have to right it's just that when we're training a transformer the when we're training a transformer the when we're training a transformer the transformer is fixed length. Yeah. So transformer is fixed length. Yeah. So transformer is fixed length. Yeah. So like in transformer it doesn't really like in transformer it doesn't really like in transformer it doesn't really matter that much I guess but yeah matter that much I guess but yeah matter that much I guess but yeah but it's still we're going to talk about but it's still we're going to talk about but it's still we're going to talk about how like it's more in it's actually not how like it's more in it's actually not how like it's more in it's actually not as efficient uh and how people are as efficient uh and how people are as efficient uh and how people are trying to solve it um next class though. trying to solve it um next class though. trying to solve it um next class though. Yeah. Any other questions before me Yeah. Any other questions before me Yeah. Any other questions before me before we move on? before we move on? before we move on? Alrighty. Okay. So h just a reminder Alrighty. Okay. So h just a reminder

  29. L11 @ 1644s

    Alrighty. Okay. So h just a reminder what is a uh what is a continuous score what is a uh what is a continuous score what is a uh what is a continuous score and why is it weird to think about score and why is it weird to think about score and why is it weird to think about score functions in the discrete space. So functions in the discrete space. So functions in the discrete space. So continuously if if your status space is continuously if if your status space is continuously if if your status space is a continuous space like a 2D plane right a continuous space like a 2D plane right a continuous space like a 2D plane right uh then the score function is like the uh then the score function is like the uh then the score function is like the gradient uh that goes towards the higher gradient uh that goes towards the higher gradient uh that goes towards the higher likelihood region right so you just like likelihood region right so you just like likelihood region right so you just like usually and then you just like follow usually and then you just like follow usually and then you just like follow the flow uh so to speak whereas if the flow uh so to speak whereas if the flow uh so to speak whereas if you're talking about a discreet um like you're talking about a discreet um like you're talking about a discreet um like score I guess what you do is you do not score I guess what you do is you do not score I guess what you do is you do not you cannot really flow continuously you cannot really flow continuously you cannot really flow continuously anymore right what you can do is you anymore right what you can do is you anymore right what you can do is you have like sort like three disconnected have like sort like three disconnected have like sort like three disconnected points and you're low key just like points and you're low key just like points and you're low key just like jumping between the points. So there's jumping between the points. So there's jumping between the points. So there's no continuous flow anymore. There's no no continuous flow anymore. There's no

  30. L11 @ 1690s

    no continuous flow anymore. There's no gradient like you cannot like really do gradient like you cannot like really do gradient like you cannot like really do any gradient stuff anymore here, right? any gradient stuff anymore here, right? any gradient stuff anymore here, right? So that's like really weird. Uh so how So that's like really weird. Uh so how So that's like really weird. Uh so how do we do that, right? So essentially but do we do that, right? So essentially but do we do that, right? So essentially but but like basically if you think about it but like basically if you think about it but like basically if you think about it just just if you uh if you vibe with it just just if you uh if you vibe with it just just if you uh if you vibe with it uh you know the continuous score is uh you know the continuous score is uh you know the continuous score is basically saying that like compare my basically saying that like compare my basically saying that like compare my likelihood with my neighbors with like likelihood with my neighbors with like likelihood with my neighbors with like my surrounding and uh if they are higher my surrounding and uh if they are higher my surrounding and uh if they are higher if if my neighbors has higher likelihood if if my neighbors has higher likelihood if if my neighbors has higher likelihood than me then I will flow to them than me then I will flow to them than me then I will flow to them otherwise they flow to me right so this otherwise they flow to me right so this otherwise they flow to me right so this is like the same kind of like so we if is like the same kind of like so we if is like the same kind of like so we if We follow the same vibe as that like We follow the same vibe as that like We follow the same vibe as that like just like comparing with my neighbors just like comparing with my neighbors just like comparing with my neighbors type of thing. Uh which is not a healthy type of thing. Uh which is not a healthy

  31. L11 @ 1736s

    type of thing. Uh which is not a healthy thing to do but like you know just for thing to do but like you know just for thing to do but like you know just for the sake of the class anyway um you know the sake of the class anyway um you know the sake of the class anyway um you know it if uh like so in discrete case what it if uh like so in discrete case what it if uh like so in discrete case what we do is we don't flow to them we just we do is we don't flow to them we just we do is we don't flow to them we just directly jump. So like if your neighbors directly jump. So like if your neighbors directly jump. So like if your neighbors uh has higher likelihood than you then uh has higher likelihood than you then uh has higher likelihood than you then you you you jump to them instead of float. But like jump to them instead of float. But like jump to them instead of float. But like how to describe this jumping rate, how to describe this jumping rate, how to describe this jumping rate, right? This like this jumping frequency right? This like this jumping frequency right? This like this jumping frequency I guess. Uh then what you can do is I guess. Uh then what you can do is I guess. Uh then what you can do is literally basically have something so literally basically have something so literally basically have something so that the discrete score quote unquote that the discrete score quote unquote that the discrete score quote unquote the score is sort of just a comparison the score is sort of just a comparison the score is sort of just a comparison between the likelihood at your current between the likelihood at your current between the likelihood at your current point and the likelihood at the point point and the likelihood at the point point and the likelihood at the point that you're going to jump to. So uh that you're going to jump to. So uh that you're going to jump to. So uh basically what people does is that they basically what people does is that they

  32. L11 @ 1781s

    basically what people does is that they define something called uh the concrete define something called uh the concrete define something called uh the concrete score where it's literally just the score where it's literally just the score where it's literally just the probability or the probability ratio probability or the probability ratio probability or the probability ratio between the point that you're jumping to between the point that you're jumping to between the point that you're jumping to and the point that you're currently at. and the point that you're currently at. and the point that you're currently at. Does it make sense people? >> Yeah. wouldn't be able to just jump to any wouldn't be able to just jump to any wouldn't be able to just jump to any neighbors because there's no like neighbors because there's no like neighbors because there's no like inherent ordering between the tokens, inherent ordering between the tokens, inherent ordering between the tokens, right? right? right? >> Um that is still fine. >> Um that is still fine. >> Um that is still fine. >> And we're going to see why. Yeah. Okay. >> And we're going to see why. Yeah. Okay. >> And we're going to see why. Yeah. Okay. Uh let yeah >> you could be definitely which is why you >> you could be definitely which is why you need to define your trans transition need to define your trans transition need to define your trans transition like clearly essentially. But anyway, like clearly essentially. But anyway, like clearly essentially. But anyway, let let me let me go through a little let let me let me go through a little let let me let me go through a little bit more of the map. Yeah. bit more of the map. Yeah. bit more of the map. Yeah. >> To look at the subgradient when compared >> To look at the subgradient when compared >> To look at the subgradient when compared to this to this to this >> uh you the the the gradient >> uh you the the the gradient >> uh you the the the gradient >> the subgradient >> the network generalization >> the network generalization >> I see I have no idea actually but >> I see I have no idea actually but >> I see I have no idea actually but apparently apparently

  33. L11 @ 1849s

    apparently uh if you read this particular paper uh if you read this particular paper uh if you read this particular paper they have some connections to something they have some connections to something they have some connections to something but I'm not entirely sure unfortunately. but I'm not entirely sure unfortunately. but I'm not entirely sure unfortunately. Sorry. Okay. Anyway, Sorry. Okay. Anyway, Sorry. Okay. Anyway, uh so uh so uh so uh another piece of the thing that we're uh another piece of the thing that we're uh another piece of the thing that we're missing here is that in continuous missing here is that in continuous missing here is that in continuous score-based models, not only that we um score-based models, not only that we um score-based models, not only that we um like model everything in score function, like model everything in score function, like model everything in score function, we also represent everything in a we also represent everything in a we also represent everything in a continuous time SD or OD, right? Uh so continuous time SD or OD, right? Uh so continuous time SD or OD, right? Uh so what what does even mean to have a what what does even mean to have a what what does even mean to have a continuous time discrete jump? Uh well, continuous time discrete jump? Uh well, continuous time discrete jump? Uh well, essentially there is this other notion. essentially there is this other notion. essentially there is this other notion. It's called uh continuous time markoff It's called uh continuous time markoff It's called uh continuous time markoff chain or CTMC. And what it does is chain or CTMC. And what it does is chain or CTMC. And what it does is basically you still have some sort of basically you still have some sort of basically you still have some sort of like uh transition matrix. Uh but like like uh transition matrix. Uh but like like uh transition matrix. Uh but like basically but but now this transition basically but but now this transition basically but but now this transition matrix so sort of like represents some matrix so sort of like represents some matrix so sort of like represents some like instantaneous rate essentially like instantaneous rate essentially like instantaneous rate essentially people actually call it a rate matrix.

  34. L11 @ 1908s

    people actually call it a rate matrix. people actually call it a rate matrix. So it's a instantaneous rate to jump So it's a instantaneous rate to jump So it's a instantaneous rate to jump like just just just think about if you like just just just think about if you like just just just think about if you just discretize your the the trend the just discretize your the the trend the just discretize your the the trend the evolution like infiniteesimally small evolution like infiniteesimally small evolution like infiniteesimally small and uh the probability to jump from one and uh the probability to jump from one and uh the probability to jump from one state uh to the next infiniteimally state uh to the next infiniteimally state uh to the next infiniteimally small steps can be written as this thing small steps can be written as this thing small steps can be written as this thing which is basically which is basically which is basically your rate times your infinite decimally your rate times your infinite decimally your rate times your infinite decimally small time. small time. small time. uh uh if so you're jumping to another uh uh if so you're jumping to another uh uh if so you're jumping to another point point point uh you know with the rate times infinite uh you know with the rate times infinite uh you know with the rate times infinite decimal small time and then the rate or decimal small time and then the rate or decimal small time and then the rate or like the probability that you're staying like the probability that you're staying like the probability that you're staying at your current point is literally just at your current point is literally just at your current point is literally just one minus whatever that is left whatever one minus whatever that is left whatever one minus whatever that is left whatever that is sum over all the other states. that is sum over all the other states. that is sum over all the other states. Okay. Um, so essentially if you if you Okay. Um, so essentially if you if you Okay. Um, so essentially if you if you want something to satisfy this whole want something to satisfy this whole want something to satisfy this whole thing, then you're actually basically if

  35. L11 @ 1963s

    thing, then you're actually basically if thing, then you're actually basically if you do the math then then then you're you do the math then then then you're you do the math then then then you're actually going to get like the sort of actually going to get like the sort of actually going to get like the sort of like the the rate to stay at the current like the the rate to stay at the current like the the rate to stay at the current state is the same as the negative of all state is the same as the negative of all state is the same as the negative of all the other rate combined. So basically the other rate combined. So basically the other rate combined. So basically just like just imagine how like remember just like just imagine how like remember just like just imagine how like remember how we have like a like a conservation how we have like a like a conservation how we have like a like a conservation of mass, right? like basically this the of mass, right? like basically this the of mass, right? like basically this the the mass of water or like the volume of the mass of water or like the volume of the mass of water or like the volume of the water or the or the sum of the water or the or the sum of the water or the or the sum of probability stays at one right so probability stays at one right so probability stays at one right so basically essentially all the rate you basically essentially all the rate you basically essentially all the rate you should cancel out each other at the end should cancel out each other at the end should cancel out each other at the end so that's why everything should sum up so that's why everything should sum up so that's why everything should sum up to zero in the rate space to zero in the rate space to zero in the rate space to be like a fully connected everyone to be like a fully connected everyone to be like a fully connected everyone what is what is what is >> uh yeah like so like basically >> uh yeah like so like basically >> uh yeah like so like basically everything everything

  36. L11 @ 2007s

    everything well you can represent everything in a well you can represent everything in a well you can represent everything in a fully connected graph and then assign fully connected graph and then assign fully connected graph and then assign zero rate right zero rate right zero rate right yeah yeah yeah so it's going to be the same thing so it's going to be the same thing so it's going to be the same thing essentially essentially essentially okay but basically we can also write okay but basically we can also write okay but basically we can also write this turns out we can also write this this turns out we can also write this this turns out we can also write this thing into OD and this OD can be thing into OD and this OD can be thing into OD and this OD can be represented as this thing uh and represented as this thing uh and represented as this thing uh and basically uh going in reverse uh basically uh going in reverse uh basically uh going in reverse uh okay so basically now I'm so sorry for okay so basically now I'm so sorry for okay so basically now I'm so sorry for the confusion of the notation but now Q the confusion of the notation but now Q the confusion of the notation but now Q R means the reverse time in in R means the reverse time in in R means the reverse time in in instantaneous instantaneous instantaneous rate. Okay, so it's not the the the the rate. Okay, so it's not the the the the rate. Okay, so it's not the the the the product anymore. It's the reversed rate. product anymore. It's the reversed rate. product anymore. It's the reversed rate. Uh and then like basically if you do the Uh and then like basically if you do the Uh and then like basically if you do the math correctly bas why why are you doing math correctly bas why why are you doing math correctly bas why why are you doing this way is because of detail balance. this way is because of detail balance. this way is because of detail balance. So if you want your chain to satisfy So if you want your chain to satisfy

  37. L11 @ 2062s

    So if you want your chain to satisfy detail balance you're going to get this detail balance you're going to get this detail balance you're going to get this thing so that your chain is reversible, thing so that your chain is reversible, thing so that your chain is reversible, right? Uh and then uh this is just some right? Uh and then uh this is just some right? Uh and then uh this is just some marco chain things but basically if you marco chain things but basically if you marco chain things but basically if you want your chain to be reversible you want your chain to be reversible you want your chain to be reversible you need to satisfy detail balance. Detail need to satisfy detail balance. Detail need to satisfy detail balance. Detail balance is going to give you this balance is going to give you this balance is going to give you this equation and uh equation and uh equation and uh we have our concrete score in this in we have our concrete score in this in we have our concrete score in this in this equation. Uh so long story short this equation. Uh so long story short this equation. Uh so long story short you need to learn the concrete score. you need to learn the concrete score. you need to learn the concrete score. Yeah this is like the only thing that Yeah this is like the only thing that Yeah this is like the only thing that you need to remember from all of these you need to remember from all of these you need to remember from all of these math. It's like basically we need to math. It's like basically we need to math. It's like basically we need to learn the concrete score in order to get learn the concrete score in order to get learn the concrete score in order to get the score-based model equivalence of the score-based model equivalence of the score-based model equivalence of discrete diffusion. discrete diffusion. discrete diffusion. Now how to learn a concrete score? What Now how to learn a concrete score? What Now how to learn a concrete score? What do we think? There should be a very easy answer, There should be a very easy answer, right? I see everyone's confused by how this

  38. L11 @ 2119s

    I see everyone's confused by how this whole thing but we need basically we whole thing but we need basically we whole thing but we need basically we need a neuronet network to predict this need a neuronet network to predict this need a neuronet network to predict this thing. thing. thing. >> Yeah. >> Yeah. >> Distribution over all the >> Distribution over all the >> Distribution over all the >> yes learn the distribution will be >> yes learn the distribution will be >> yes learn the distribution will be better. We're going to talk about it better. We're going to talk about it better. We're going to talk about it later. But uh there's a there's a even later. But uh there's a there's a even later. But uh there's a there's a even easier way. You haven't got my easier way. You haven't got my easier way. You haven't got my blessings, have you? Have you got got my blessings, have you? Have you got got my blessings, have you? Have you got got my blessings yet? >> Get get richer. Get get good grade, I >> Get get richer. Get get good grade, I guess. guess. guess. >> Good grade. Oh man. Okay. >> Good grade. Oh man. Okay. >> Good grade. Oh man. Okay. Um, this is no allergy version. Anyway, Um, this is no allergy version. Anyway, Um, this is no allergy version. Anyway, okay. What else? There should be a super okay. What else? There should be a super okay. What else? There should be a super easy easy easy version. that consists of a loss function that we that consists of a loss function that we use for this entire class. What What >> did you say? >> did you say? >> did you say? >> Elbow. Yeah. But also L2, right? You can just Yeah. But also L2, right? You can just L2 it. Yeah. So you can do concrete score Yeah. So you can do concrete score matching instead of score matching, matching instead of score matching, matching instead of score matching, right? And the score matching is right? And the score matching is right? And the score matching is literally L2 loss between your predicted literally L2 loss between your predicted literally L2 loss between your predicted score versus your uh you know your your

  39. L11 @ 2208s

    score versus your uh you know your your score versus your uh you know your your your your your real score. And you your your your real score. And you your your your real score. And you literally just do this, right? L2 score literally just do this, right? L2 score literally just do this, right? L2 score L2 between your predicted concrete score L2 between your predicted concrete score L2 between your predicted concrete score versus your real score. versus your real score. versus your real score. Is there any problem with this? What do Is there any problem with this? What do Is there any problem with this? What do we think? >> well we do know we we we we could >> well we do know we we we we could construct this. construct this. construct this. >> Yeah. Well there there there is a way to >> Yeah. Well there there there is a way to >> Yeah. Well there there there is a way to construct this I guess if if if you construct this I guess if if if you construct this I guess if if if you want. Yeah want. Yeah want. Yeah >> infeasible because of the number of >> infeasible because of the number of >> infeasible because of the number of combinations of tokens. combinations of tokens. combinations of tokens. Um not quite not quite uh but but but Um not quite not quite uh but but but Um not quite not quite uh but but but there's some actually I just tell you there's some actually I just tell you there's some actually I just tell you guys uh basically the biggest problem is guys uh basically the biggest problem is guys uh basically the biggest problem is that you actually um so like the that you actually um so like the that you actually um so like the probability ratio here should always be probability ratio here should always be probability ratio here should always be positive because it's a positive number positive because it's a positive number positive because it's a positive number divided by a positive number or a divided by a positive number or a divided by a positive number or a non-zero number divided by a non-zero non-zero number divided by a non-zero

  40. L11 @ 2267s

    non-zero number divided by a non-zero number or like or sorry non- negative number or like or sorry non- negative number or like or sorry non- negative number divided by a non- negative number number divided by a non- negative number number divided by a non- negative number right but uh as if we assume like full right but uh as if we assume like full right but uh as if we assume like full support then it's a not then it's support then it's a not then it's support then it's a not then it's positive number divided by positive positive number divided by positive positive number divided by positive number but the problem is that there's number but the problem is that there's number but the problem is that there's no way to enforce that right in this no way to enforce that right in this no way to enforce that right in this loss function. So negative if you loss function. So negative if you loss function. So negative if you predict the likelihood ratio to be predict the likelihood ratio to be predict the likelihood ratio to be negative.5 negative.5 negative.5 on a loss function it may seems fine on a loss function it may seems fine on a loss function it may seems fine right because it's like maybe it's like right because it's like maybe it's like right because it's like maybe it's like negative.5 versus ne 0.2 two. So the negative.5 versus ne 0.2 two. So the negative.5 versus ne 0.2 two. So the loss function is actually quite small. loss function is actually quite small. loss function is actually quite small. Like the loss is actually quite small, Like the loss is actually quite small, Like the loss is actually quite small, but you are actually predicting but you are actually predicting but you are actually predicting something that's like super super wrong. something that's like super super wrong. something that's like super super wrong. You're actually predicting something You're actually predicting something You're actually predicting something that's invalid. So this is one thing. that's invalid. So this is one thing. that's invalid. So this is one thing. >> Yeah. Yeah. Yeah. Yeah. Very good. Very >> Yeah. Yeah. Yeah. Yeah. Very good. Very >> Yeah. Yeah. Yeah. Yeah. Very good. Very good. Exactly. That's exactly correct.

  41. L11 @ 2315s

    good. Exactly. That's exactly correct. good. Exactly. That's exactly correct. Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you Um uh the other thing is that yeah, you you may notice that the like the you may notice that the like the you may notice that the like the denominator gets small then the thing denominator gets small then the thing denominator gets small then the thing explode, right? And the other thing is explode, right? And the other thing is explode, right? And the other thing is that a lot of you guys have said that in that a lot of you guys have said that in that a lot of you guys have said that in your um homework. MSE is just like not a your um homework. MSE is just like not a your um homework. MSE is just like not a great metric in general for distribution great metric in general for distribution great metric in general for distribution matching. Um so what do we do? Okay, matching. Um so what do we do? Okay, matching. Um so what do we do? Okay, let's derive a better loss function let's derive a better loss function let's derive a better loss function here. Okay, so basically if you think here. Okay, so basically if you think here. Okay, so basically if you think about it, right? Uh the reverse Q about it, right? Uh the reverse Q about it, right? Uh the reverse Q because we need to satisfy detail because we need to satisfy detail because we need to satisfy detail balance. This is how how this is what we balance. This is how how this is what we balance. This is how how this is what we derived, right? the reverse the uh rate derived, right? the reverse the uh rate derived, right? the reverse the uh rate equal to some base rate times the ratio equal to some base rate times the ratio equal to some base rate times the ratio right so the base rate is set so just right so the base rate is set so just right so the base rate is set so just like a like a like a like a like a what like a like a like a like a like a what

  42. L11 @ 2360s

    like a like a like a like a like a what we call uh constant so let's just ignore we call uh constant so let's just ignore we call uh constant so let's just ignore that for now so we basically we can that for now so we basically we can that for now so we basically we can represent this like frequency of jumping represent this like frequency of jumping represent this like frequency of jumping like if you want to describe a frequency like if you want to describe a frequency like if you want to describe a frequency of something happening in a discrete of something happening in a discrete of something happening in a discrete space right then the the first uh space right then the the first uh space right then the the first uh distribution that you should think of is distribution that you should think of is distribution that you should think of is the pson distribution. By the way, I the pson distribution. By the way, I the pson distribution. By the way, I don't actually know if they derive it don't actually know if they derive it don't actually know if they derive it this way. This is sort of like my this way. This is sort of like my this way. This is sort of like my derivation and didn't really check their derivation and didn't really check their derivation and didn't really check their derivation, but I think it's it's okay. derivation, but I think it's it's okay. derivation, but I think it's it's okay. It's going to work. Trust me. Okay. It's going to work. Trust me. Okay. It's going to work. Trust me. Okay. Anyway, point being so let's just like Anyway, point being so let's just like Anyway, point being so let's just like define uh the you know the the the the define uh the you know the the the the define uh the you know the the the the the the the data distribution and the predicted data distribution and the predicted data distribution and the predicted distribution as some person. Okay. And distribution as some person. Okay. And distribution as some person. Okay. And then the person is going to be this then the person is going to be this

  43. L11 @ 2406s

    then the person is going to be this thing, right? So what you do is you thing, right? So what you do is you thing, right? So what you do is you essentially so K is like you you jump to essentially so K is like you you jump to essentially so K is like you you jump to you know you the the jumping event you know you the the jumping event you know you the the jumping event essentially and the basically the KL essentially and the basically the KL essentially and the basically the KL divergence between two person is plug in divergence between two person is plug in divergence between two person is plug in the formula equal to this and then this the formula equal to this and then this the formula equal to this and then this thing is basically uh if you cancel out thing is basically uh if you cancel out thing is basically uh if you cancel out everything that you can cancel uh it's everything that you can cancel uh it's everything that you can cancel uh it's going to become something like this and going to become something like this and going to become something like this and then because this is a summation of two then because this is a summation of two then because this is a summation of two parts uh so you can break it break it up parts uh so you can break it break it up parts uh so you can break it break it up and uh this summation here. So basically and uh this summation here. So basically and uh this summation here. So basically and then you you you you move everything and then you you you you move everything and then you you you you move everything that is not relevant to the to to the to that is not relevant to the to to the to that is not relevant to the to to the to the to K outside of the summation and the to K outside of the summation and the to K outside of the summation and then this part becomes one uh s minus r then this part becomes one uh s minus r

  44. L11 @ 2454s

    then this part becomes one uh s minus r uh times the sum of all the possible all uh times the sum of all the possible all uh times the sum of all the possible all the probabilities and then this one is the probabilities and then this one is the probabilities and then this one is the log minus the other log times the the log minus the other log times the the log minus the other log times the expectation essentially right so the sum expectation essentially right so the sum expectation essentially right so the sum of probability equals one the the of probability equals one the the of probability equals one the the expectation of a hor distribution equals expectation of a hor distribution equals expectation of a hor distribution equals the rate. So it's R right and then the rate. So it's R right and then the rate. So it's R right and then basically and then you can ignore all basically and then you can ignore all basically and then you can ignore all the things that is not relevant to your the things that is not relevant to your the things that is not relevant to your theta to your learning to your parameter theta to your learning to your parameter theta to your learning to your parameter then you get something like this right then you get something like this right then you get something like this right does it make sense yeah does it make sense yeah does it make sense yeah >> use is there any reason apart from the >> use is there any reason apart from the >> use is there any reason apart from the fact that you want to enforce the fact that you want to enforce the fact that you want to enforce the exponential is that the reason exponential is that the reason exponential is that the reason >> that is part of the that is literally >> that is part of the that is literally >> that is part of the that is literally yeah that is part of the reason it gives yeah that is part of the reason it gives yeah that is part of the reason it gives us something that is that has that has us something that is that has that has

  45. L11 @ 2497s

    us something that is that has that has exponential in it and also literally exponential in it and also literally exponential in it and also literally right the the the easiest right the the the easiest right the the the easiest uh distribution that you can think of to uh distribution that you can think of to uh distribution that you can think of to represent the rate of something represent the rate of something represent the rate of something happening is in a discrete space is happening is in a discrete space is happening is in a discrete space is person right so okay but like I said I person right so okay but like I said I person right so okay but like I said I don't know if this is like the official don't know if this is like the official don't know if this is like the official derivation that they did this is derivation that they did this is derivation that they did this is something you know that I did but I something you know that I did but I something you know that I did but I think it works anyway we get to the same think it works anyway we get to the same think it works anyway we get to the same thing this is a screenshot from the thing this is a screenshot from the thing this is a screenshot from the paper don't worry uh but yeah but paper don't worry uh but yeah but paper don't worry uh but yeah but basically this is a paper uh that that basically this is a paper uh that that basically this is a paper uh that that the paper some paper defined uh this the paper some paper defined uh this the paper some paper defined uh this loss called the the score entropy loss loss called the the score entropy loss loss called the the score entropy loss because it resembles an entropy, you because it resembles an entropy, you because it resembles an entropy, you know, uh and then so basically what's know, uh and then so basically what's know, uh and then so basically what's happening is literally the same thing, happening is literally the same thing, happening is literally the same thing, right? So you the the S minus the

  46. L11 @ 2540s

    right? So you the the S minus the right? So you the the S minus the likelihood ratio times the log of the likelihood ratio times the log of the likelihood ratio times the log of the your S. Um so because you have a log your S. Um so because you have a log your S. Um so because you have a log here, right? So everything needs to be here, right? So everything needs to be here, right? So everything needs to be positive now, right? And because you're positive now, right? And because you're positive now, right? And because you're in a log form, so you also kind of deal in a log form, so you also kind of deal in a log form, so you also kind of deal with the crazy crazy ratio better, a with the crazy crazy ratio better, a with the crazy crazy ratio better, a little bit better. Uh and then also like little bit better. Uh and then also like little bit better. Uh and then also like now this is a KL divergence right rather now this is a KL divergence right rather now this is a KL divergence right rather than just a L2 loss right so it's like than just a L2 loss right so it's like than just a L2 loss right so it's like distributionally it's better distributionally it's better distributionally it's better >> what does the notation Y sim X mean in >> what does the notation Y sim X mean in >> what does the notation Y sim X mean in the the >> oh this is like jumping from X to Y yeah >> oh this is like jumping from X to Y yeah >> oh this is like jumping from X to Y yeah okay okay okay question okay anyway point being we can also okay anyway point being we can also apply the same reparameterization trick apply the same reparameterization trick apply the same reparameterization trick here right it's the same thing um So here right it's the same thing um So here right it's the same thing um So this is how you solve the how to this is how you solve the how to this is how you solve the how to calculate the ground truth by literally calculate the ground truth by literally

  47. L11 @ 2592s

    calculate the ground truth by literally the same trick the den noising score the same trick the den noising score the same trick the den noising score matching. This is just the noising score matching. This is just the noising score matching. This is just the noising score entropy I guess right. Um so essentially entropy I guess right. Um so essentially entropy I guess right. Um so essentially what you can do is you can uh yeah what you can do is you can uh yeah what you can do is you can uh yeah basically just uh just sample a data and basically just uh just sample a data and basically just uh just sample a data and then sample a perturb the data and and then sample a perturb the data and and then sample a perturb the data and and and then calculate the likelihood and then calculate the likelihood and then calculate the likelihood based on the the perturb uh rate that we based on the the perturb uh rate that we based on the the perturb uh rate that we have. set in in our model assumption have. set in in our model assumption have. set in in our model assumption essentially. essentially. essentially. All right, any questions? But anyway, if All right, any questions? But anyway, if All right, any questions? But anyway, if you don't have questions, now you have you don't have questions, now you have you don't have questions, now you have the score entropy discrete diffusion the score entropy discrete diffusion the score entropy discrete diffusion sde. sde. sde. Okay, Okay, Okay, these things are confusing. Um, any these things are confusing. Um, any these things are confusing. Um, any questions about SE DD? Yeah. >> How do you generate from this >> How do you generate from this >> How do you generate from this train? uh basically you the uh basically you the hold on hold on. hold on hold on. hold on hold on. Okay. Oh yeah. Okay. Oh yeah. Okay. Oh yeah. Give me a second. All right. So, this is Give me a second. All right. So, this is Give me a second. All right. So, this is the reverse rate, right? So, you just the reverse rate, right? So, you just

  48. L11 @ 2680s

    the reverse rate, right? So, you just replace this concrete score with the replace this concrete score with the replace this concrete score with the concrete score that you trained. And concrete score that you trained. And concrete score that you trained. And then this is the then you apply the then this is the then you apply the then this is the then you apply the rate. rate. rate. Yeah. Make sense? Make sense? Okay. Cool. Okay. Cool. Okay. Cool. All right. All right. Cool. All right. So we get to the score Cool. All right. So we get to the score Cool. All right. So we get to the score matching the score matching uh version matching the score matching uh version matching the score matching uh version of the discrete diffusion. Very nice. of the discrete diffusion. Very nice. of the discrete diffusion. Very nice. Very nice. H Very nice. H Very nice. H uh we're going to talk about the the the uh we're going to talk about the the the uh we're going to talk about the the the flow matching version next class. The flow matching version next class. The flow matching version next class. The class is not over. Uh notice how now all class is not over. Uh notice how now all class is not over. Uh notice how now all our models are pretty generic, right? In our models are pretty generic, right? In our models are pretty generic, right? In terms of like the Q's and stuff. This Q terms of like the Q's and stuff. This Q terms of like the Q's and stuff. This Q can be any Q essentially, right? But can be any Q essentially, right? But can be any Q essentially, right? But like many people have uh mentioned like many people have uh mentioned like many people have uh mentioned that you know like you can choose that you know like you can choose that you know like you can choose different cues uh and in fact many of different cues uh and in fact many of different cues uh and in fact many of you have chosen different cues you have chosen different cues you have chosen different cues >> earlier how are we choosing why do we >> earlier how are we choosing why do we

  49. L11 @ 2739s

    >> earlier how are we choosing why do we just randomly sample any token choices just randomly sample any token choices just randomly sample any token choices >> yeah based on >> yeah based on >> yeah based on >> okay so but notice how this kind of >> okay so but notice how this kind of >> okay so but notice how this kind of related I guess um so what is the related I guess um so what is the related I guess um so what is the easiest uh you know transition that we easiest uh you know transition that we easiest uh you know transition that we can have here what is the easiest can have here what is the easiest can have here what is the easiest easiest transition matrix that we can easiest transition matrix that we can easiest transition matrix that we can define or what is the easiest way to add define or what is the easiest way to add define or what is the easiest way to add noise besides you know randomly picking noise besides you know randomly picking noise besides you know randomly picking one that is called uniform or something. one that is called uniform or something. one that is called uniform or something. Yeah. Yeah. And what is or maybe like what is the And what is or maybe like what is the And what is or maybe like what is the one that is closest to LM one that is closest to LM one that is closest to LM and some of you have already mentioned and some of you have already mentioned and some of you have already mentioned it. No way. like it's literally in the Yeah. No way. like it's literally in the Yeah. >> Yeah. Right. It's just random asking >> Yeah. Right. It's just random asking >> Yeah. Right. It's just random asking literally it's just so easy. Um so in a literally it's just so easy. Um so in a literally it's just so easy. Um so in a in a graph in a graphical way. So in a graph in a graphical way. So in a graph in a graphical way. So basically like how about we can only do basically like how about we can only do

  50. L11 @ 2798s

    basically like how about we can only do masking and unmasking, right? So in the masking and unmasking, right? So in the masking and unmasking, right? So in the forward process all you can do is you forward process all you can do is you forward process all you can do is you can you can turn an existing tokens into can you can turn an existing tokens into can you can turn an existing tokens into a mask. You cannot unmask it. Like the a mask. You cannot unmask it. Like the a mask. You cannot unmask it. Like the all you can do like if you if you all you can do like if you if you all you can do like if you if you already mask it, you cannot unmask it in already mask it, you cannot unmask it in already mask it, you cannot unmask it in in in the forward process. And then the in in the forward process. And then the in in the forward process. And then the reverse process is the opposite, right? reverse process is the opposite, right? reverse process is the opposite, right? So like you try to unmask something and So like you try to unmask something and So like you try to unmask something and then you you will not like you know then you you will not like you know then you you will not like you know remask it if you if you have already remask it if you if you have already remask it if you if you have already unmasked it. So literally this is how unmasked it. So literally this is how unmasked it. So literally this is how you do uh you know just just add random you do uh you know just just add random you do uh you know just just add random masking to each um token. And the masking to each um token. And the masking to each um token. And the transition matrix is also super simple, transition matrix is also super simple, transition matrix is also super simple, right? So essentially what you do is right? So essentially what you do is right? So essentially what you do is like basically the only two columns that like basically the only two columns that

  51. L11 @ 2840s

    like basically the only two columns that you're going to have non- negative value you're going to have non- negative value you're going to have non- negative value is the columns on the diagonal and the is the columns on the diagonal and the is the columns on the diagonal and the last column which uh like kind of last column which uh like kind of last column which uh like kind of represents the the the the transition represents the the the the transition represents the the the the transition rate to the to the to the masks. Right? rate to the to the to the masks. Right? rate to the to the to the masks. Right? So in the diagonal it represents the So in the diagonal it represents the So in the diagonal it represents the rate to stay unmasked and this is this rate to stay unmasked and this is this rate to stay unmasked and this is this is the forward transition by the way. is the forward transition by the way. is the forward transition by the way. And then and in the last column it And then and in the last column it And then and in the last column it represents the transition rate to mask. represents the transition rate to mask. represents the transition rate to mask. So just like the rate to mass the token. So just like the rate to mass the token. So just like the rate to mass the token. Okay. Okay. Okay. Yeah. Yeah. >> So in normal diffusion >> So in normal diffusion >> So in normal diffusion >> if we let it go for a long while and we >> if we let it go for a long while and we >> if we let it go for a long while and we have absolute uninformative blockchain have absolute uninformative blockchain have absolute uninformative blockchain noise but that's still samples from a noise but that's still samples from a noise but that's still samples from a distribution. So we have different distribution. So we have different distribution. So we have different starting points to go reverse. starting points to go reverse. starting points to go reverse. >> But if we go all the way end of the day >> But if we go all the way end of the day

  52. L11 @ 2886s

    >> But if we go all the way end of the day everything will be masked. everything will be masked. everything will be masked. >> Yeah. And it's still a distribution >> Yeah. And it's still a distribution >> Yeah. And it's still a distribution right? It's a point mask. So we always right? It's a point mask. So we always right? It's a point mask. So we always go if you start reverse from that we go if you start reverse from that we go if you start reverse from that we >> always go to a point mass distribution >> always go to a point mass distribution >> always go to a point mass distribution where nothing has support but the all where nothing has support but the all where nothing has support but the all mask tokens. Yeah. Yeah. Good question. mask tokens. Yeah. Yeah. Good question. mask tokens. Yeah. Yeah. Good question. >> Yeah. >> Yeah. >> With this matrix mass tokens would just >> With this matrix mass tokens would just >> With this matrix mass tokens would just stay mass tokens with the probability of stay mass tokens with the probability of stay mass tokens with the probability of one. one. one. >> Uh yes >> Uh yes >> Uh yes >> because it's a because like I said right >> because it's a because like I said right >> because it's a because like I said right if you already masked it you should not if you already masked it you should not if you already masked it you should not unmask it. Yeah. In the forward process unmask it. Yeah. In the forward process unmask it. Yeah. In the forward process only. This is a forward process. only. This is a forward process. only. This is a forward process. All right, any other questions? All right, any other questions? All right, any other questions? Okay, cool. But basically what's Okay, cool. But basically what's Okay, cool. But basically what's happening is that now we actually don't happening is that now we actually don't happening is that now we actually don't even need to consider all the matrix even need to consider all the matrix even need to consider all the matrix multiplication and stuff anymore. Uh multiplication and stuff anymore. Uh

  53. L11 @ 2935s

    multiplication and stuff anymore. Uh because we simplified everything, right? because we simplified everything, right? because we simplified everything, right? So basically what what we're essentially So basically what what we're essentially So basically what what we're essentially doing is literally right, we're actually doing is literally right, we're actually doing is literally right, we're actually just interpolating right between a clean just interpolating right between a clean just interpolating right between a clean data and all mass data, right? and all data and all mass data, right? and all data and all mass data, right? and all mass vector. Um, so what you can do is mass vector. Um, so what you can do is mass vector. Um, so what you can do is we can literally just formulate it in we can literally just formulate it in we can literally just formulate it in this way, right? So your it is is this way, right? So your it is is this way, right? So your it is is literally just a linear interpolation or literally just a linear interpolation or literally just a linear interpolation or like basically a scheduled interpolation like basically a scheduled interpolation like basically a scheduled interpolation I guess uh between your data and or I guess uh between your data and or I guess uh between your data and or mask. So you can literally simply mask. So you can literally simply mask. So you can literally simply describe your forward function like this describe your forward function like this describe your forward function like this or you can write it as you know in a or you can write it as you know in a or you can write it as you know in a clearer form. Basically, it's like with clearer form. Basically, it's like with clearer form. Basically, it's like with uh alpha t you know with alpha t um uh alpha t you know with alpha t um uh alpha t you know with alpha t um like basically one forward jump is like like basically one forward jump is like like basically one forward jump is like with alpha t probability you're going to with alpha t probability you're going to with alpha t probability you're going to stay unmasked and then with one minus stay unmasked and then with one minus

  54. L11 @ 2991s

    stay unmasked and then with one minus alpha t uh probability you're going to alpha t uh probability you're going to alpha t uh probability you're going to stay masked and uh basically if we if stay masked and uh basically if we if stay masked and uh basically if we if you do it this way um then like the one you do it this way um then like the one you do it this way um then like the one step like the one small step um four can step like the one small step um four can step like the one small step um four can also be you know represented in a very also be you know represented in a very also be you know represented in a very very nice and simple form. And specifically, if you were already And specifically, if you were already get masked in the previous time step, get masked in the previous time step, get masked in the previous time step, you always stay masked. So that's why you always stay masked. So that's why you always stay masked. So that's why this is one No question. All right, let's move on. No question. All right, let's move on. Um, Um, Um, so, uh, why why is this a nice thing? so, uh, why why is this a nice thing? so, uh, why why is this a nice thing? Because remember how previously when we Because remember how previously when we Because remember how previously when we were trying to do the whole elbow thing were trying to do the whole elbow thing were trying to do the whole elbow thing and then we had this like full like and then we had this like full like and then we had this like full like multip matrix multiplication like multip matrix multiplication like multip matrix multiplication like different things happening in the D3 PM different things happening in the D3 PM different things happening in the D3 PM thing. Now this very complicated thing thing. Now this very complicated thing thing. Now this very complicated thing becomes super simple. uh literally right becomes super simple. uh literally right becomes super simple. uh literally right when uh xt equals x0 and x t minus one when uh xt equals x0 and x t minus one

  55. L11 @ 3068s

    when uh xt equals x0 and x t minus one also equals x zero um like basically if also equals x zero um like basically if also equals x zero um like basically if if you have not been masked uh at time t if you have not been masked uh at time t if you have not been masked uh at time t then you also should not have been then you also should not have been then you also should not have been masked in xt minus one. So this thing masked in xt minus one. So this thing masked in xt minus one. So this thing has probability one and like everything has probability one and like everything has probability one and like everything else have probability zero, right? But else have probability zero, right? But else have probability zero, right? But if you're if you're already masked at if you're if you're already masked at if you're if you're already masked at time t, then there are two options for time t, then there are two options for time t, then there are two options for you to go, right? One is that you got you to go, right? One is that you got you to go, right? One is that you got masked at this particular time step or masked at this particular time step or masked at this particular time step or you got masked before like in in the you got masked before like in in the you got masked before like in in the history, right? So basically here is history, right? So basically here is history, right? So basically here is what we're saying that okay we're only what we're saying that okay we're only what we're saying that okay we're only getting masked at time t. So what is the getting masked at time t. So what is the getting masked at time t. So what is the probability of that? Then that probability of that? Then that probability of that? Then that probability if you just like plug probability if you just like plug probability if you just like plug everything in then it actually gets everything in then it actually gets everything in then it actually gets simplified into this really really simplified into this really really

  56. L11 @ 3122s

    simplified into this really really simple uh ratio. And similarly simple uh ratio. And similarly simple uh ratio. And similarly because the only other option for you is because the only other option for you is because the only other option for you is that the this basically the only opt that the this basically the only opt that the this basically the only opt other option for uh the the only other other option for uh the the only other other option for uh the the only other valid choice for the values for xt minus valid choice for the values for xt minus valid choice for the values for xt minus one will be a mask token. So just one will be a mask token. So just one will be a mask token. So just literally one minus the ratio that you literally one minus the ratio that you literally one minus the ratio that you just calculated which can be simplified just calculated which can be simplified just calculated which can be simplified into another nice ratio. All right. But anyway, um so if we sum All right. But anyway, um so if we sum everything up, then you can actually everything up, then you can actually everything up, then you can actually represent this supposedly super represent this supposedly super represent this supposedly super complicated, you know, um distribution complicated, you know, um distribution complicated, you know, um distribution where it it has a lot of m matrix where it it has a lot of m matrix where it it has a lot of m matrix multiplication into this like super multiplication into this like super multiplication into this like super super super simple form, right? So you super super simple form, right? So you super super simple form, right? So you really really only have three cases. One really really only have three cases. One really really only have three cases. One is if XT hasn't been masked, XT minus is if XT hasn't been masked, XT minus is if XT hasn't been masked, XT minus one also you should just like take it one also you should just like take it one also you should just like take it over. You should you should have the over. You should you should have the

  57. L11 @ 3191s

    over. You should you should have the same value because it hasn't been same value because it hasn't been same value because it hasn't been masked. And if XT is masked, then you masked. And if XT is masked, then you masked. And if XT is masked, then you have two cases. One is that you at at have two cases. One is that you at at have two cases. One is that you at at time Xt minus one you already masked. time Xt minus one you already masked. time Xt minus one you already masked. The other case is that uh at time t The other case is that uh at time t The other case is that uh at time t minus one you haven't been masked and minus one you haven't been masked and minus one you haven't been masked and you only get masked recently. So each you only get masked recently. So each you only get masked recently. So each will has this nice ratio as your will has this nice ratio as your will has this nice ratio as your probability. probability. probability. Okay. >> great question because we're gonna like >> great question because we're gonna like basically uh you know because we can basically uh you know because we can basically uh you know because we can again uh like obviously you you this is again uh like obviously you you this is again uh like obviously you you this is still your your um your objective, still your your um your objective, still your your um your objective, right? We're dealing with the same right? We're dealing with the same right? We're dealing with the same elbow. So but this thing can again be elbow. So but this thing can again be elbow. So but this thing can again be reparameterized into this thing times uh reparameterized into this thing times uh reparameterized into this thing times uh you know uh the the clean uh h you know uh the the clean uh h you know uh the the clean uh h prediction. So literally your prediction prediction. So literally your prediction prediction. So literally your prediction is literally just going to be a ratio is literally just going to be a ratio is literally just going to be a ratio times the clean data estimation.

  58. L11 @ 3263s

    times the clean data estimation. times the clean data estimation. That's it. >> We have that problem because we'll start >> We have that problem because we'll start with full mask. So if you start with with full mask. So if you start with with full mask. So if you start with full mass, we can only ever end up in full mass, we can only ever end up in full mass, we can only ever end up in one possible x0. one possible x0. one possible x0. >> Uh but this is for every token position. >> Uh but this is for every token position. >> Uh but this is for every token position. So for every token position, you you So for every token position, you you So for every token position, you you sort of independently mask sort of independently mask sort of independently mask >> it will end up being full mask, right? >> it will end up being full mask, right? >> it will end up being full mask, right? Because every token Because every token Because every token >> yeah eventually end up being full mask >> yeah eventually end up being full mask >> yeah eventually end up being full mask and then in reverse time you just apply and then in reverse time you just apply and then in reverse time you just apply this transition essentially every single this transition essentially every single this transition essentially every single step at each token. So like when you are step at each token. So like when you are step at each token. So like when you are unmasked already you don't change it. If unmasked already you don't change it. If unmasked already you don't change it. If you if you haven't been unmasked then you if you haven't been unmasked then you if you haven't been unmasked then you decide whether to unmask it in the you decide whether to unmask it in the you decide whether to unmask it in the next time step with this probability or next time step with this probability or next time step with this probability or not. >> The starting reverse is full mask. >> The starting reverse is full mask. >> Yeah. >> Yeah. >> So you're always starting with the full >> So you're always starting with the full

  59. L11 @ 3326s

    >> So you're always starting with the full mask. mask. mask. >> That's right. >> That's right. >> That's right. >> So if that one reverse gets you to I >> So if that one reverse gets you to I >> So if that one reverse gets you to I love cat. What possibility is for you to love cat. What possibility is for you to love cat. What possibility is for you to read some other thing? But this okay th read some other thing? But this okay th read some other thing? But this okay th this this thing is so stochastic right this this thing is so stochastic right this this thing is so stochastic right this whole thing is so stocastic this whole thing is so stocastic this whole thing is so stocastic like basically this is a categorical like basically this is a categorical like basically this is a categorical distribution and then you sample from distribution and then you sample from distribution and then you sample from the categorical distribution. the categorical distribution. the categorical distribution. >> Yeah. Okay. Yeah. >> Yeah. Okay. Yeah. >> Yeah. Okay. Yeah. >> Uh is there also something do we also >> Uh is there also something do we also >> Uh is there also something do we also like remask stuff? like remask stuff? like remask stuff? >> No you don't you never remask >> No you don't you never remask >> No you don't you never remask related to this question. What if the related to this question. What if the related to this question. What if the model wants to change some word after it model wants to change some word after it model wants to change some word after it is generated when it generates is generated when it generates is generated when it generates something? in this case. Yeah. But in something? in this case. Yeah. But in something? in this case. Yeah. But in the other case, in the in in the uniform the other case, in the in in the uniform the other case, in the in in the uniform case, it does, right? Yeah. Which is why case, it does, right? Yeah. Which is why case, it does, right? Yeah. Which is why actually a good point. Basically, uh the actually a good point. Basically, uh the

  60. L11 @ 3372s

    actually a good point. Basically, uh the the the mercury um founder, Inception the the mercury um founder, Inception the the mercury um founder, Inception Lab founder, Stephano, who is the one of Lab founder, Stephano, who is the one of Lab founder, Stephano, who is the one of the inventors of diffusion models. He the inventors of diffusion models. He the inventors of diffusion models. He said that in in on his website, I guess, said that in in on his website, I guess, said that in in on his website, I guess, that like a diffusion language model can that like a diffusion language model can that like a diffusion language model can correct its own uh hallucination correct its own uh hallucination correct its own uh hallucination is only true because they were using the is only true because they were using the is only true because they were using the uniform thing. uniform thing. uniform thing. >> Yeah, >> Yeah, >> Yeah, they're probably not using the mass they're probably not using the mass they're probably not using the mass thing. Well, I don't know now, but at thing. Well, I don't know now, but at thing. Well, I don't know now, but at the time that he claimed this, they were the time that he claimed this, they were the time that he claimed this, they were using the uniform thing. Oh, by the way, using the uniform thing. Oh, by the way, using the uniform thing. Oh, by the way, the SD thing is also from Stephano. So, the SD thing is also from Stephano. So, the SD thing is also from Stephano. So, yeah. Well, like this this categorical you're Well, like this this categorical you're sampling from a categorical distribution sampling from a categorical distribution sampling from a categorical distribution stoastically. stoastically. stoastically. Yeah. For each token. Yeah. For each token. Yeah. For each token. Okay. Uh but anyway, uh the objective Okay. Uh but anyway, uh the objective Okay. Uh but anyway, uh the objective function is actually surprisingly easy function is actually surprisingly easy function is actually surprisingly easy because basically what you do is you you because basically what you do is you you because basically what you do is you you you can also do like continuous time you can also do like continuous time

  61. L11 @ 3429s

    you can also do like continuous time training with this. You literally just training with this. You literally just training with this. You literally just take the continuous time limit of the of take the continuous time limit of the of take the continuous time limit of the of this ratio here and then this thing this ratio here and then this thing this ratio here and then this thing happened to just be this right. It's happened to just be this right. It's happened to just be this right. It's super easy. And then this thing is just super easy. And then this thing is just super easy. And then this thing is just cross entropy between the clean data and cross entropy between the clean data and cross entropy between the clean data and the and your estimation is like bird the and your estimation is like bird the and your estimation is like bird basically. So is low key just a uh way basically. So is low key just a uh way basically. So is low key just a uh way to burn to burn to burn zooming out a little bit but I thought zooming out a little bit but I thought zooming out a little bit but I thought the whole point so in this objective the whole point so in this objective the whole point so in this objective we're still sampling one token at a time we're still sampling one token at a time we're still sampling one token at a time but I thought that but I thought that but I thought that >> well you can parallely sample all tokens >> well you can parallely sample all tokens >> well you can parallely sample all tokens right so like all of these all of the right so like all of these all of the right so like all of these all of the categorical sampling happens categorical sampling happens categorical sampling happens independently at each token location independently at each token location independently at each token location >> the number of tokens we sample >> the number of tokens we sample >> the number of tokens we sample >> uh you yes kind of so it's the same >> uh you yes kind of so it's the same

  62. L11 @ 3478s

    >> uh you yes kind of so it's the same thing it's uh like analogy to you choose thing it's uh like analogy to you choose thing it's uh like analogy to you choose the number of sampling steps right in the number of sampling steps right in the number of sampling steps right in continuous diffusion. So here you also continuous diffusion. So here you also continuous diffusion. So here you also kind of like you you you kind you also kind of like you you you kind you also kind of like you you you kind you also choose the number of time steps kind of choose the number of time steps kind of choose the number of time steps kind of and then basically depending on the and then basically depending on the and then basically depending on the number of time steps um like you just number of time steps um like you just number of time steps um like you just like you don't really choose the like like you don't really choose the like like you don't really choose the like the number of unmasking to tokens that the number of unmasking to tokens that the number of unmasking to tokens that you have at each step but what you do is you have at each step but what you do is you have at each step but what you do is you basically adjust your scheduleuler you basically adjust your scheduleuler you basically adjust your scheduleuler based on the current step that you add based on the current step that you add based on the current step that you add and then you sample via the categorical and then you sample via the categorical and then you sample via the categorical distribution and if the categorical distribution and if the categorical distribution and if the categorical distribution give you I'm mass token distribution give you I'm mass token distribution give you I'm mass token there you are. Uh but if they decided to there you are. Uh but if they decided to there you are. Uh but if they decided to stay mass then they will stay mass but stay mass then they will stay mass but stay mass then they will stay mass but this is purely depending on like the the

  63. L11 @ 3522s

    this is purely depending on like the the this is purely depending on like the the model that you got trained but usually model that you got trained but usually model that you got trained but usually if you are trying to sample with smaller if you are trying to sample with smaller if you are trying to sample with smaller time steps then it will try to basically time steps then it will try to basically time steps then it will try to basically for each individual tokens it'll be more for each individual tokens it'll be more for each individual tokens it'll be more aggressive and so it's more likely for aggressive and so it's more likely for aggressive and so it's more likely for you to see like parallel multiple tokens you to see like parallel multiple tokens you to see like parallel multiple tokens get unmasked at the same time. All get unmasked at the same time. All get unmasked at the same time. All right, we I don't think I gave you a a right, we I don't think I gave you a a right, we I don't think I gave you a a blessing yet. Uh, Rich, get rich. Uh, Rich, get rich. >> Sure. >> Sure. >> Sure. >> All right. Oh, this is a good good one. >> All right. Oh, this is a good good one. >> All right. Oh, this is a good good one. Okay. Anyway. All right. Any other Okay. Anyway. All right. Any other Okay. Anyway. All right. Any other questions? questions? questions? We don't have any heavy math after this, We don't have any heavy math after this, We don't have any heavy math after this, so we can stay for a little bit longer. so we can stay for a little bit longer. so we can stay for a little bit longer. Yeah. Does the ratio change over time Yeah. Does the ratio change over time Yeah. Does the ratio change over time like or is it the same? like or is it the same? like or is it the same? >> Uh it change over time, right? Like >> Uh it change over time, right? Like >> Uh it change over time, right? Like basically the mask ratio is depending on basically the mask ratio is depending on

  64. L11 @ 3574s

    basically the mask ratio is depending on this alpha t thing which you set. So you this alpha t thing which you set. So you this alpha t thing which you set. So you you can you can choose uh but like you you can you can choose uh but like you you can you can choose uh but like you can you can have a linear it's the same can you can have a linear it's the same can you can have a linear it's the same thing as DDPM right so you can you can thing as DDPM right so you can you can thing as DDPM right so you can you can choose a linear schedule you can choose choose a linear schedule you can choose choose a linear schedule you can choose a cosine schedule you can choose a cosine schedule you can choose a cosine schedule you can choose whatever schedule you want. Yeah. Is whatever schedule you want. Yeah. Is whatever schedule you want. Yeah. Is this tied to the >> uh it like statistically? Yes. But like >> uh it like statistically? Yes. But like you don't decide how many tokens you uh you don't decide how many tokens you uh you don't decide how many tokens you uh amass basically. >> Yeah. Yeah. Yeah. Yeah. Yeah. So that's >> Yeah. Yeah. Yeah. Yeah. Yeah. So that's why so so that's why you have this why so so that's why you have this why so so that's why you have this ratio, right? Like basically um yeah ratio, right? Like basically um yeah ratio, right? Like basically um yeah here is t the number of is t the number of the token that we've generated like say the token that we've generated like say the token that we've generated like say third token in subsequ third token in subsequ third token in subsequ steps. Yeah. T is the diffusion time steps. Yeah. T is the diffusion time steps. Yeah. T is the diffusion time step here. Yeah. And then the the the step here. Yeah. And then the the the step here. Yeah. And then the the the the tokens are the L L's here. L L are the tokens are the L L's here. L L are the tokens are the L L's here. L L are the tokens.

  65. L11 @ 3634s

    the tokens. the tokens. >> Yeah. >> Yeah. >> But then in the last you have something >> But then in the last you have something >> But then in the last you have something like XT equal to M. So like XT equal to M. So like XT equal to M. So >> uh yeah. So that just means that Oh, >> uh yeah. So that just means that Oh, >> uh yeah. So that just means that Oh, sorry. Okay. Yeah. So everything before sorry. Okay. Yeah. So everything before sorry. Okay. Yeah. So everything before this slide XT is one token but all the this slide XT is one token but all the this slide XT is one token but all the tokens are independent. So this is why I tokens are independent. So this is why I tokens are independent. So this is why I just didn't this why they can be added just didn't this why they can be added just didn't this why they can be added together which is sounds wrong right it together which is sounds wrong right it together which is sounds wrong right it just doesn't sound right but anyway but just doesn't sound right but anyway but just doesn't sound right but anyway but but but yeah but we're going to talk but but yeah but we're going to talk but but yeah but we're going to talk about uh about uh about uh how may or may not people try to solve how may or may not people try to solve how may or may not people try to solve it actually but yeah they don't really it actually but yeah they don't really it actually but yeah they don't really but you know but you know but you know >> this output is in the token space does >> this output is in the token space does >> this output is in the token space does this mean we cannot do like a latent this mean we cannot do like a latent this mean we cannot do like a latent text decoding of diffusion models text decoding of diffusion models text decoding of diffusion models >> uh what do you mean by latent text well >> uh what do you mean by latent text well

  66. L11 @ 3676s

    >> uh what do you mean by latent text well like so you mean like like so you mean like like so you mean like a region space for images and then use a region space for images and then use a region space for images and then use like a V come back. like a V come back. like a V come back. >> Ah, I see. I see. >> Ah, I see. I see. >> Ah, I see. I see. >> That's >> That's >> That's >> Yeah. Yeah. Right. Yeah. So basically >> Yeah. Yeah. Right. Yeah. So basically >> Yeah. Yeah. Right. Yeah. So basically you don't you don't really have a you don't you don't really have a you don't you don't really have a discrete. So there are too many discrete discrete. So there are too many discrete discrete. So there are too many discrete steps here I guess to have a like a VA steps here I guess to have a like a VA steps here I guess to have a like a VA style uh thing here. Yeah. Yeah. So style uh thing here. Yeah. Yeah. So style uh thing here. Yeah. Yeah. So right right now nobody is doing that. right right now nobody is doing that. right right now nobody is doing that. But may maybe it's visible. Who knows? But may maybe it's visible. Who knows? But may maybe it's visible. Who knows? >> Like you have to decide what length you >> Like you have to decide what length you >> Like you have to decide what length you want to I don't know the want to I don't know the want to I don't know the >> Yeah. Yeah. Yeah. Yeah. Yeah. The the >> Yeah. Yeah. Yeah. Yeah. Yeah. The the >> Yeah. Yeah. Yeah. Yeah. Yeah. The the the variable length here, the L thing is the variable length here, the L thing is the variable length here, the L thing is a big thing. They Yeah. This is why it's a big thing. They Yeah. This is why it's a big thing. They Yeah. This is why it's kind of not very efficient the the kind of not very efficient the the kind of not very efficient the the model. model. model. >> Does it produce something coherent if

  67. L11 @ 3722s

    >> Does it produce something coherent if >> Does it produce something coherent if all the things are independently getting all the things are independently getting all the things are independently getting masked? masked? masked? Well, the model itself is trained in Well, the model itself is trained in Well, the model itself is trained in with like self attention from from with like self attention from from with like self attention from from everywhere, right? So, basically, it's everywhere, right? So, basically, it's everywhere, right? So, basically, it's sort of like just imagine like the edge sort of like just imagine like the edge sort of like just imagine like the edge case where you only have one token case where you only have one token case where you only have one token masked, right? Then it's very easy to to masked, right? Then it's very easy to to masked, right? Then it's very easy to to to to determine which token it is, to to determine which token it is, to to determine which token it is, right, based on your data distribution right, based on your data distribution right, based on your data distribution and then you can just induction. and then you can just induction. and then you can just induction. >> Yeah. >> Yeah. >> Assume that because you're summing over >> Assume that because you're summing over >> Assume that because you're summing over all tokens, it will learn some structure all tokens, it will learn some structure all tokens, it will learn some structure between the tokens. between the tokens. between the tokens. >> Yeah, basically. So because you're >> Yeah, basically. So because you're >> Yeah, basically. So because you're you're dealing like all the tokens share you're dealing like all the tokens share you're dealing like all the tokens share the same model, they see the same thing. the same model, they see the same thing. the same model, they see the same thing. So yeah, you kind of depend on the the So yeah, you kind of depend on the the So yeah, you kind of depend on the the model. model. >> It's like recursive birth basically. >> It's like recursive birth basically. >> It's like recursive birth basically. >> It is pretty much just birth literally >> It is pretty much just birth literally

  68. L11 @ 3769s

    >> It is pretty much just birth literally if you think about it. Yeah. if you think about it. Yeah. if you think about it. Yeah. >> Well, between like >> Well, between like >> Well, between like >> you call arbitrary length. I'm going to >> you call arbitrary length. I'm going to >> you call arbitrary length. I'm going to talk about it like next class, but it's talk about it like next class, but it's talk about it like next class, but it's like very very hacky. Well, sorry for like very very hacky. Well, sorry for like very very hacky. Well, sorry for the people who developed the method, but the people who developed the method, but the people who developed the method, but I think it's kind of hacky. Okay. I think it's kind of hacky. Okay. I think it's kind of hacky. Okay. Anyway, Anyway, Anyway, >> you didn't specify any sort of >> you didn't specify any sort of >> you didn't specify any sort of location position. location position. location position. >> Oh, yeah. Yeah, they do have all the >> Oh, yeah. Yeah, they do have all the >> Oh, yeah. Yeah, they do have all the positional embeddings and everything. positional embeddings and everything. positional embeddings and everything. Yeah. Yeah. >> Yeah. >> Yeah. >> Yeah. >> Oh, no. So, this thing cannot, right? >> Oh, no. So, this thing cannot, right? But the thing that we saw before the But the thing that we saw before the But the thing that we saw before the uniform version, it can, right? Because uniform version, it can, right? Because uniform version, it can, right? Because you can sample any tokens. Even if you you can sample any tokens. Even if you you can sample any tokens. Even if you has already sampled the token, you can has already sampled the token, you can has already sampled the token, you can still sample again at this location. still sample again at this location. still sample again at this location. Right? So this is what Stephano meant Right? So this is what Stephano meant Right? So this is what Stephano meant when they when he said like oh we can when they when he said like oh we can

  69. L11 @ 3820s

    when they when he said like oh we can correct hallucination but like here we correct hallucination but like here we correct hallucination but like here we cannot really it's the same as cannot really it's the same as cannot really it's the same as >> LM how do we know when to stop? We have how do we know when to stop? We have like a fixed time. like a fixed time. like a fixed time. >> You have a fixed time step, right? So >> You have a fixed time step, right? So >> You have a fixed time step, right? So you just run it till the end. you just run it till the end. you just run it till the end. >> Okay. Uh we have like one more minute. >> Okay. Uh we have like one more minute. >> Okay. Uh we have like one more minute. So but yeah, now you have mass diffusion So but yeah, now you have mass diffusion So but yeah, now you have mass diffusion model. Hey. All right. So we uh let's go model. Hey. All right. So we uh let's go model. Hey. All right. So we uh let's go through the last thing. Uh so basically through the last thing. Uh so basically through the last thing. Uh so basically it works super well and also like it works super well and also like it works super well and also like uh very coincidentally I feel like this uh very coincidentally I feel like this uh very coincidentally I feel like this is happening more and more in the age of is happening more and more in the age of is happening more and more in the age of gender modeling unfortunately. Oh I gender modeling unfortunately. Oh I gender modeling unfortunately. Oh I don't know if it's unfortunate fortunate don't know if it's unfortunate fortunate don't know if it's unfortunate fortunate like if you're outside of the drama like if you're outside of the drama like if you're outside of the drama you're you you feel really nice. This is you're you you feel really nice. This is you're you you feel really nice. This is like a sort of like a like a like a like like a sort of like a like a like a like

  70. L11 @ 3864s

    like a sort of like a like a like a like a interesting moment in science. But if a interesting moment in science. But if a interesting moment in science. But if you're one of the authors, you may get, you're one of the authors, you may get, you're one of the authors, you may get, you know, you may be like what what what you know, you may be like what what what you know, you may be like what what what am I what is going on? But anyway, uh am I what is going on? But anyway, uh am I what is going on? But anyway, uh but they kind of like the math diffusion but they kind of like the math diffusion but they kind of like the math diffusion formulation like literally pretty much formulation like literally pretty much formulation like literally pretty much the same uh like loss function was the same uh like loss function was the same uh like loss function was discovered by two concurrent work at new discovered by two concurrent work at new discovered by two concurrent work at new rips uh 2024. Literally they're even rips uh 2024. Literally they're even rips uh 2024. Literally they're even even their titles are kind of similar even their titles are kind of similar even their titles are kind of similar honestly. So it just uh but but yeah but honestly. So it just uh but but yeah but honestly. So it just uh but but yeah but they are concurrent. So they they they they are concurrent. So they they they they are concurrent. So they they they both discovered it um you know both discovered it um you know both discovered it um you know independently. So this is just shows how independently. So this is just shows how independently. So this is just shows how this is a super effective um like method this is a super effective um like method this is a super effective um like method essentially. Yeah. essentially. Yeah. essentially. Yeah. >> Both papers were written by AI >> Both papers were written by AI >> Both papers were written by AI >> very low because it's 2024. Okay. Anyway >> very low because it's 2024. Okay. Anyway

  71. L11 @ 3908s

    >> very low because it's 2024. Okay. Anyway um but since like you know people show um but since like you know people show um but since like you know people show saw the potential of this they have saw the potential of this they have saw the potential of this they have actually tried to scale it up. Uh so actually tried to scale it up. Uh so actually tried to scale it up. Uh so there this thing called lada uh which is there this thing called lada uh which is there this thing called lada uh which is basically try to scale up in an LM scale basically try to scale up in an LM scale basically try to scale up in an LM scale and try to do like instruction and try to do like instruction and try to do like instruction fine-tunings and everything like that fine-tunings and everything like that fine-tunings and everything like that and the loss function they use basically and the loss function they use basically and the loss function they use basically they just like have like a very specific they just like have like a very specific they just like have like a very specific uh you know uh noise scheduling and then uh you know uh noise scheduling and then uh you know uh noise scheduling and then basically the the the the ratio that we basically the the the the ratio that we basically the the the the ratio that we saw at the beginning uh in the last saw at the beginning uh in the last saw at the beginning uh in the last slides reduce to one over t. So it's slides reduce to one over t. So it's slides reduce to one over t. So it's literally just 1 / t times cross entropy literally just 1 / t times cross entropy literally just 1 / t times cross entropy at places where you have masks. at places where you have masks. at places where you have masks. That's it. Like super super super easy. That's it. Like super super super easy. That's it. Like super super super easy. Um but it works pretty well. Let's just Um but it works pretty well. Let's just Um but it works pretty well. Let's just say and seems like super compatible uh

  72. L11 @ 3955s

    say and seems like super compatible uh say and seems like super compatible uh to comparable to like large language to comparable to like large language to comparable to like large language models like llama. So it's like super models like llama. So it's like super models like llama. So it's like super nice. Um yeah but anyway uh so we have nice. Um yeah but anyway uh so we have nice. Um yeah but anyway uh so we have talked about two formulations for talked about two formulations for talked about two formulations for discrete diffusion. uh the next class discrete diffusion. uh the next class discrete diffusion. uh the next class we're going to talk about essentially we're going to talk about essentially we're going to talk about essentially the flow matching equivalence of uh the flow matching equivalence of uh the flow matching equivalence of uh discrete diffusion and it's actually a discrete diffusion and it's actually a discrete diffusion and it's actually a very very interesting recent work about very very interesting recent work about very very interesting recent work about that and uh lastly this is like a very that and uh lastly this is like a very that and uh lastly this is like a very very big trend right now essentially very big trend right now essentially very big trend right now essentially like people are trying to connect LMS like people are trying to connect LMS like people are trying to connect LMS with uh with uh with uh um discrete diffusion and uh yeah let's um discrete diffusion and uh yeah let's um discrete diffusion and uh yeah let's just like kind of like look at some of just like kind of like look at some of just like kind of like look at some of the efforts that people have put in it the efforts that people have put in it the efforts that people have put in it yeah but that's it yeah thanks for yeah but that's it yeah thanks for yeah but that's it yeah thanks for coming uh yeah if you guys want snacks coming uh yeah if you guys want snacks coming uh yeah if you guys want snacks Please come up and uh can get rich as Please come up and uh can get rich as

  73. L11 @ 3999s

    Please come up and uh can get rich as well. All right, see you guys next well. All right, see you guys next well. All right, see you guys next class.

diffusion-lm-vizuara

  1. L13 @ 3s

    So what we aim to do is as follows. We So what we aim to do is as follows. We have a true probability distribution, have a true probability distribution, have a true probability distribution, right? So let's say the meaningful right? So let's say the meaningful right? So let's say the meaningful sentences look like this in probability sentences look like this in probability sentences look like this in probability distribution space. We want to get a distribution space. We want to get a distribution space. We want to get a probability distribution probability distribution probability distribution or we want to predict a probability or we want to predict a probability or we want to predict a probability distribution which is as close to this distribution which is as close to this distribution which is as close to this true probability distribution. true probability distribution. true probability distribution. But we want to predict this not with an But we want to predict this not with an But we want to predict this not with an auto reggressive model but we want to auto reggressive model but we want to auto reggressive model but we want to use a diffusion model use a diffusion model use a diffusion model for getting this probability for getting this probability for getting this probability distribution of the predicted sentences. distribution of the predicted sentences. distribution of the predicted sentences. Right? Right? Right? So we will integrate the three main So we will integrate the three main So we will integrate the three main characteristics of diffusion models. characteristics of diffusion models. characteristics of diffusion models. First we need a noising process. Second First we need a noising process. Second First we need a noising process. Second we need to predict the noise using some we need to predict the noise using some we need to predict the noise using some sort of a model. And third we need a sort of a model. And third we need a sort of a model. And third we need a dnoising process. The noising process in dnoising process. The noising process in dnoising process. The noising process in the case of images was adding noise to the case of images was adding noise to the case of images was adding noise to an image using a noising schedule maybe

  2. L13 @ 57s

    an image using a noising schedule maybe an image using a noising schedule maybe a goshian schedule. The predicting noise a goshian schedule. The predicting noise a goshian schedule. The predicting noise was the ML model called unit in the case was the ML model called unit in the case was the ML model called unit in the case of images and the dnoising process which of images and the dnoising process which of images and the dnoising process which they had was just uh reversing uh the they had was just uh reversing uh the they had was just uh reversing uh the noising process backwards so that we can noising process backwards so that we can noising process backwards so that we can generate images from noise. Again I want generate images from noise. Again I want generate images from noise. Again I want to bring your attention to this visual to bring your attention to this visual to bring your attention to this visual which we saw in the dnoising process. which we saw in the dnoising process. which we saw in the dnoising process. What happens is we start with complete What happens is we start with complete What happens is we start with complete noise and then we bring it back to the noise and then we bring it back to the noise and then we bring it back to the original image. Now let's start original image. Now let's start original image. Now let's start implementing these characteristics step implementing these characteristics step implementing these characteristics step by step. So how do we first of all let's by step. So how do we first of all let's by step. So how do we first of all let's start with noising process. How do we start with noising process. How do we start with noising process. How do we integrate noising process in the text? integrate noising process in the text? integrate noising process in the text? Okay, this is where our understanding of Okay, this is where our understanding of Okay, this is where our understanding of the ARM model is going to play a very the ARM model is going to play a very the ARM model is going to play a very very crucial role. Right? So let's see very crucial role. Right? So let's see

  3. L13 @ 101s

    very crucial role. Right? So let's see how noising is included. how noising is included. how noising is included. What is the noising process in the text? What is the noising process in the text? What is the noising process in the text? Right? Right? And for this the first thing which is And for this the first thing which is And for this the first thing which is going to be extremely crucial for us is going to be extremely crucial for us is going to be extremely crucial for us is that in the case of images we had an that in the case of images we had an that in the case of images we had an image right which we want to noise or image right which we want to noise or image right which we want to noise or which we want to subsequently add noise which we want to subsequently add noise which we want to subsequently add noise to. Um let's say the image was that of a to. Um let's say the image was that of a to. Um let's say the image was that of a Chinese character. That was an image. We Chinese character. That was an image. We Chinese character. That was an image. We can successfully go on adding noise to can successfully go on adding noise to can successfully go on adding noise to this image. this image. this image. We add noise. So this is for images. So this is for images. What about text? How do we go about What about text? How do we go about What about text? How do we go about adding noise to a piece of text? So how adding noise to a piece of text? So how adding noise to a piece of text? So how do we corrupt a piece of text? So the do we corrupt a piece of text? So the do we corrupt a piece of text? So the whole idea would be to take a bunch of whole idea would be to take a bunch of whole idea would be to take a bunch of text to corrupt it with noise to predict text to corrupt it with noise to predict text to corrupt it with noise to predict that noise and then when we start with

  4. L13 @ 152s

    that noise and then when we start with that noise and then when we start with noise, we can predict the original text noise, we can predict the original text noise, we can predict the original text just like how we did in images. So first just like how we did in images. So first just like how we did in images. So first is how do we add noise to text? is how do we add noise to text? is how do we add noise to text? So let's think about ARMs, right? What So let's think about ARMs, right? What So let's think about ARMs, right? What happens in ARMs? ARMs have this uh ARM have this architecture, right? And ARM have this architecture, right? And here I'm again it's the exact same here I'm again it's the exact same here I'm again it's the exact same architecture but just a different uh architecture but just a different uh architecture but just a different uh visual here. So what ARMs do is that you visual here. So what ARMs do is that you visual here. So what ARMs do is that you have this piece of text which is passed have this piece of text which is passed have this piece of text which is passed into this architecture. Right? Let's say into this architecture. Right? Let's say into this architecture. Right? Let's say I have this piece of text called the I have this piece of text called the I have this piece of text called the next day is bright. next day is bright. next day is bright. Okay, I have tokenized it. So let's say Okay, I have tokenized it. So let's say Okay, I have tokenized it. So let's say the token ids are 1 11 20,000 the token ids are 1 11 20,000 the token ids are 1 11 20,000 55 and 3,000. 55 and 3,000. 55 and 3,000. And then what I have done is that I have And then what I have done is that I have And then what I have done is that I have converted it into embedding vectors. converted it into embedding vectors. converted it into embedding vectors. That's the next step. So token That's the next step. So token That's the next step. So token embedding. So this is a 384 dimensional

  5. L13 @ 212s

    embedding. So this is a 384 dimensional embedding. So this is a 384 dimensional vector. vector. vector. This is a 384 dimensional vector. This is a 384 dimensional vector. This is a 384 dimensional vector. I'll just mute my discord. This is a I'll just mute my discord. This is a I'll just mute my discord. This is a 384dimensional vector. 384dimensional vector. 384dimensional vector. This is a 384dimensional vector. This is a 384dimensional vector. This is a 384dimensional vector. And this is again a 384dimensional And this is again a 384dimensional And this is again a 384dimensional vector. Right? This is how it works for vector. Right? This is how it works for vector. Right? This is how it works for auto reggressive models. Now the auto reggressive models. Now the auto reggressive models. Now the question which I'm asking to all of you question which I'm asking to all of you question which I'm asking to all of you is that is that is that I want to start with the first step of I want to start with the first step of I want to start with the first step of these three these three these three three characteristics which is the three characteristics which is the three characteristics which is the noising process. How do I corrupt the noising process. How do I corrupt the noising process. How do I corrupt the given piece of text by adding noise? given piece of text by adding noise? given piece of text by adding noise? What can I do over here? So that noise What can I do over here? So that noise What can I do over here? So that noise is added. Now what does noise mean? is added. Now what does noise mean? is added. Now what does noise mean? Noise essentially means that this piece Noise essentially means that this piece Noise essentially means that this piece of text right now let's say I have the of text right now let's say I have the of text right now let's say I have the actual distribution of true sentences. actual distribution of true sentences. actual distribution of true sentences. Right? This is the true true sentences Right? This is the true true sentences Right? This is the true true sentences probability distribution. Let me Let me this is the true this is the true

  6. L13 @ 284s

    this is the true sentences probability distribution. sentences probability distribution. sentences probability distribution. This and this sentence the next day is This and this sentence the next day is This and this sentence the next day is bright belongs somewhere here. The next bright belongs somewhere here. The next bright belongs somewhere here. The next day is bright lies somewhere here. So day is bright lies somewhere here. So day is bright lies somewhere here. So the probability associated with this. So the probability associated with this. So the probability associated with this. So if this sentence is X, probability if this sentence is X, probability if this sentence is X, probability associated with X is very high because associated with X is very high because associated with X is very high because this is a good sentence. Now I want to this is a good sentence. Now I want to this is a good sentence. Now I want to corrupt this text which means I want to corrupt this text which means I want to corrupt this text which means I want to take it to spaces. So if this is my take it to spaces. So if this is my take it to spaces. So if this is my probability distribution space, I want probability distribution space, I want probability distribution space, I want to take it to spaces here where the to take it to spaces here where the to take it to spaces here where the probability is very low of it being a probability is very low of it being a probability is very low of it being a true sentence. How do I do that? How do true sentence. How do I do that? How do true sentence. How do I do that? How do I corrupt a sentence? The best way to I corrupt a sentence? The best way to I corrupt a sentence? The best way to corrupt a sentence is to just remove corrupt a sentence is to just remove corrupt a sentence is to just remove portions of it. Right? portions of it. Right? portions of it. Right? If I remove portion of this, if I remove If I remove portion of this, if I remove If I remove portion of this, if I remove a portion of this, this is no longer a

  7. L13 @ 325s

    a portion of this, this is no longer a a portion of this, this is no longer a true sentence. It becomes a cor, it true sentence. It becomes a cor, it true sentence. It becomes a cor, it becomes a corrupt text or it becomes a becomes a corrupt text or it becomes a becomes a corrupt text or it becomes a noisy text. Um, this is exactly how noisy text. Um, this is exactly how noisy text. Um, this is exactly how noising is done at the sentence level. noising is done at the sentence level. noising is done at the sentence level. So we have input sequences or we have So we have input sequences or we have So we have input sequences or we have sequences of tokens, right? sequences of tokens, right? sequences of tokens, right? In diffusion language models, we corrupt In diffusion language models, we corrupt In diffusion language models, we corrupt this text by introducing masks. this text by introducing masks. this text by introducing masks. What this means is that if I have the What this means is that if I have the What this means is that if I have the next day is bright, next day is bright, next day is bright, I will mask this input sequence. Which I will mask this input sequence. Which I will mask this input sequence. Which means I will let I will say that the means I will let I will say that the means I will let I will say that the input sequence is this. The mask input sequence is this. The mask input sequence is this. The mask day mask mask bright. bright. bright. This is basically corrupting the input This is basically corrupting the input This is basically corrupting the input sequence by adding masks. sequence by adding masks. sequence by adding masks. Right? Uh now how do I add these masks? Right? Uh now how do I add these masks? Right? Uh now how do I add these masks? The the way I add these masks is by The the way I add these masks is by The the way I add these masks is by defining something like a noising defining something like a noising defining something like a noising schedule. Right? So first I define a schedule. Right? So first I define a

  8. L13 @ 393s

    schedule. Right? So first I define a time step. Let's say I go from time time step. Let's say I go from time time step. Let's say I go from time equal to 1 till time equal to 6. equal to 1 till time equal to 6. equal to 1 till time equal to 6. And I define the number of tokens uh proportional to uh proportional to let's say my time let's say t divided by let's say my time let's say t divided by let's say my time let's say t divided by so if 6 is capital t divided by capital so if 6 is capital t divided by capital so if 6 is capital t divided by capital t so if my t is equal to 1 t so if my t is equal to 1 t so if my t is equal to 1 um I will approximately mask um I will approximately mask um I will approximately mask 16th of the tokens If my t is equal to 2, I will If my t is equal to 2, I will approximately mask 26 of the tokens. If approximately mask 26 of the tokens. If approximately mask 26 of the tokens. If my t is equal to 6, I will mask all of my t is equal to 6, I will mask all of my t is equal to 6, I will mask all of the tokens. So you see at t equal to0, I have my So you see at t equal to0, I have my entire entire entire clean sequence which is not masked. So clean sequence which is not masked. So clean sequence which is not masked. So that's the correct sequence. Think of it that's the correct sequence. Think of it that's the correct sequence. Think of it like lying in the correct probability like lying in the correct probability like lying in the correct probability distribution space. As more and more distribution space. As more and more distribution space. As more and more noise gets added, I move out of this noise gets added, I move out of this noise gets added, I move out of this probability distribution space and start probability distribution space and start probability distribution space and start going haywire. So maybe initially when going haywire. So maybe initially when

  9. L13 @ 462s

    going haywire. So maybe initially when only one mask is added, I go to a lower only one mask is added, I go to a lower only one mask is added, I go to a lower probability. When two masks are added, I probability. When two masks are added, I probability. When two masks are added, I go to even lower probability. When all go to even lower probability. When all go to even lower probability. When all the masks are added, it's complete the masks are added, it's complete the masks are added, it's complete noise. So I go to a probability noise. So I go to a probability noise. So I go to a probability distribution or I go to a space in my distribution or I go to a space in my distribution or I go to a space in my probability distribution space which probability distribution space which probability distribution space which contains no meaningful text at all. I go contains no meaningful text at all. I go contains no meaningful text at all. I go to a random location basically. to a random location basically. to a random location basically. Uh so if you see Uh so if you see Uh so if you see diffusion noise schedule images diffusion noise schedule images diffusion noise schedule images and maybe let's see a gif. Noising schedule. Noising schedule Noising schedule. Noising schedule diffusion. Yeah, take a look at this. This is a Yeah, take a look at this. This is a noising schedule, right? I have a dog noising schedule, right? I have a dog noising schedule, right? I have a dog image on the left. I keep on adding image on the left. I keep on adding image on the left. I keep on adding noise and then it becomes complete noise and then it becomes complete noise and then it becomes complete noise. The way it happens in text is can noise. The way it happens in text is can noise. The way it happens in text is can be represented by be represented by be represented by uh this kind of a thing. Yeah. So the uh this kind of a thing. Yeah. So the uh this kind of a thing. Yeah. So the way it happens in text is that uh at way it happens in text is that uh at

  10. L13 @ 536s

    way it happens in text is that uh at time equal to 1, you almost have an time equal to 1, you almost have an time equal to 1, you almost have an entire clean text with maybe some mask entire clean text with maybe some mask entire clean text with maybe some mask over here. But as time increases, over here. But as time increases, over here. But as time increases, as time becomes two, I have two masks. as time becomes two, I have two masks. as time becomes two, I have two masks. At time as time becomes three, I have At time as time becomes three, I have At time as time becomes three, I have three masks. As time increases almost three masks. As time increases almost three masks. As time increases almost everything becomes masked. So if I watch everything becomes masked. So if I watch everything becomes masked. So if I watch the full noising you'll see that you the full noising you'll see that you the full noising you'll see that you start with lower masks and then you start with lower masks and then you start with lower masks and then you increase the number of masks. That's increase the number of masks. That's increase the number of masks. That's basically what is meant by a noising basically what is meant by a noising basically what is meant by a noising schedule. In the case of diffusion LLMs schedule. In the case of diffusion LLMs schedule. In the case of diffusion LLMs it's also called as a masking schedule it's also called as a masking schedule it's also called as a masking schedule and the masking schedule is basically and the masking schedule is basically and the masking schedule is basically proportional to the time. The further proportional to the time. The further proportional to the time. The further along we are in the noising schedule the along we are in the noising schedule the along we are in the noising schedule the more masks there will be. Usually there more masks there will be. Usually there more masks there will be. Usually there is a probability associated with it. So is a probability associated with it. So is a probability associated with it. So it's a Bernoli distribution. So the it's a Bernoli distribution. So the

  11. L13 @ 580s

    it's a Bernoli distribution. So the probability of masking is way higher. probability of masking is way higher. probability of masking is way higher. Probability of masking all tokens is way Probability of masking all tokens is way Probability of masking all tokens is way higher as time is increased. But this higher as time is increased. But this higher as time is increased. But this animation suffices to understand how animation suffices to understand how animation suffices to understand how masking works for text. Okay. masking works for text. Okay. masking works for text. Okay. Um Um Um yeah. So yeah. So yeah. So what does it mean by masking essentially what does it mean by masking essentially what does it mean by masking essentially right? What it essentially means is that right? What it essentially means is that right? What it essentially means is that we just in our vocabulary. So if we have we just in our vocabulary. So if we have we just in our vocabulary. So if we have a vocabulary of 100,000 a vocabulary of 100,000 a vocabulary of 100,000 100,000 100,000 100,000 and the vocabulary is uh the boy etc. We and the vocabulary is uh the boy etc. We and the vocabulary is uh the boy etc. We add one more token to this vocabulary add one more token to this vocabulary add one more token to this vocabulary that's the mask token right. So now that's the mask token right. So now that's the mask token right. So now wherever there is a mask so here we have wherever there is a mask so here we have wherever there is a mask so here we have a mask right or here there is a mask and a mask right or here there is a mask and a mask right or here there is a mask and here there is a mask here there is a mask here there is a mask right or here there is a mask. So the right or here there is a mask. So the right or here there is a mask. So the original tokens were 1 11 20,000 55 and original tokens were 1 11 20,000 55 and original tokens were 1 11 20,000 55 and 3,000. Right? Now these are masked. So 3,000. Right? Now these are masked. So

  12. L13 @ 644s

    3,000. Right? Now these are masked. So this is masked and this is masked. So this is masked and this is masked. So this is masked and this is masked. So this will be replaced with 1 0 0 0 1 this will be replaced with 1 0 0 0 1 this will be replaced with 1 0 0 0 1 which is the token ID for the mask. And which is the token ID for the mask. And which is the token ID for the mask. And this will also be replaced with 10 0 01 this will also be replaced with 10 0 01 this will also be replaced with 10 0 01 which is the token ID for the mask. And which is the token ID for the mask. And then this input sequence will go into then this input sequence will go into then this input sequence will go into the transformer architecture. Right? the transformer architecture. Right? the transformer architecture. Right? This input sequence goes into the This input sequence goes into the This input sequence goes into the transformer architecture transformer architecture transformer architecture or when I say transformer architecture or when I say transformer architecture or when I say transformer architecture actually this is the input sequence actually this is the input sequence actually this is the input sequence which goes over here which goes through which goes over here which goes through which goes over here which goes through the input the processor and the output. the input the processor and the output. the input the processor and the output. So we start with a noisy input with mask So we start with a noisy input with mask So we start with a noisy input with mask tokens whereas on the left hand side is tokens whereas on the left hand side is tokens whereas on the left hand side is the ARM. In the ARM we start with the the ARM. In the ARM we start with the the ARM. In the ARM we start with the entire sequence whereas in the case of entire sequence whereas in the case of entire sequence whereas in the case of diffusion language models we start with diffusion language models we start with diffusion language models we start with a noisy input with mask tokens. Okay.

  13. L13 @ 695s

    a noisy input with mask tokens. Okay. a noisy input with mask tokens. Okay. Everything else remains exactly the Everything else remains exactly the Everything else remains exactly the same. Now we'll see what are the changes same. Now we'll see what are the changes same. Now we'll see what are the changes when we go through this IPO block. What when we go through this IPO block. What when we go through this IPO block. What are the changes when we go through the I are the changes when we go through the I are the changes when we go through the I um um um I P and the O block in the next step. I P and the O block in the next step. I P and the O block in the next step. But the first thing which I wanted all But the first thing which I wanted all But the first thing which I wanted all of you to understand is how the noising of you to understand is how the noising of you to understand is how the noising process actually works. So to give a process actually works. So to give a process actually works. So to give a more concrete feel to it, let's say if more concrete feel to it, let's say if more concrete feel to it, let's say if we have four time steps, t= 1, t=2, we have four time steps, t= 1, t=2, we have four time steps, t= 1, t=2, uh here we have t= 3 and here we have t uh here we have t= 3 and here we have t uh here we have t= 3 and here we have t equal to 4. Right? Initially nothing is equal to 4. Right? Initially nothing is equal to 4. Right? Initially nothing is masked but as the time increases you'll masked but as the time increases you'll masked but as the time increases you'll see more and more masks to the input see more and more masks to the input see more and more masks to the input tokens. Um yeah so the first difference between Um yeah so the first difference between the ARM architecture and the diffusion the ARM architecture and the diffusion the ARM architecture and the diffusion architecture if you compare them side by

  14. L13 @ 743s

    architecture if you compare them side by architecture if you compare them side by side is the input which goes into this side is the input which goes into this side is the input which goes into this input block the processor block and the input block the processor block and the input block the processor block and the output block this input itself is output block this input itself is output block this input itself is different here we take a bunch of token different here we take a bunch of token different here we take a bunch of token ids and we add masks right so when the ids and we add masks right so when the ids and we add masks right so when the forward pass is done first a random time forward pass is done first a random time forward pass is done first a random time step is chosen based on that time step step is chosen based on that time step step is chosen based on that time step we decide how much noise is added how do we decide how much noise is added how do we decide how much noise is added how do we know how much noise to add because we we know how much noise to add because we we know how much noise to add because we a noise schedule which is defined. So a noise schedule which is defined. So a noise schedule which is defined. So let's say if a time step equal to three let's say if a time step equal to three let's say if a time step equal to three is chosen. The time step is chosen is chosen. The time step is chosen is chosen. The time step is chosen randomly in each forward pass. So here randomly in each forward pass. So here randomly in each forward pass. So here what happens is that in ARM architecture what happens is that in ARM architecture what happens is that in ARM architecture I told you that a batch is selected I told you that a batch is selected I told you that a batch is selected right? batch is selected then we get the right? batch is selected then we get the right? batch is selected then we get the targets targets

  15. L13 @ 783s

    targets then we sorry we get the predictions then we sorry we get the predictions then we sorry we get the predictions then we get the loss then we get the loss then we get the loss then we update the parameters then we update the parameters then we update the parameters and then we choose a new batch this and then we choose a new batch this and then we choose a new batch this process stays the same for the diffusion process stays the same for the diffusion process stays the same for the diffusion LLM but it's just that whenever a new LLM but it's just that whenever a new LLM but it's just that whenever a new batch is selected a new time step is batch is selected a new time step is batch is selected a new time step is also randomly selected also randomly selected also randomly selected so if you have sim six time steps a new so if you have sim six time steps a new so if you have sim six time steps a new time step will be selected from 1 to time step will be selected from 1 to time step will be selected from 1 to six. So if time step number three is six. So if time step number three is six. So if time step number three is selected, selected, selected, if time step number three is se if time step number three is se if time step number three is se selected, roughly 50% of the input token selected, roughly 50% of the input token selected, roughly 50% of the input token sequence will be masked and they'll be sequence will be masked and they'll be sequence will be masked and they'll be fed into the architecture. The input, fed into the architecture. The input, fed into the architecture. The input, the processor and the output, the input, the processor and the output, the input, the processor and the output, the input, the processor and the output. Okay. the processor and the output. Okay. the processor and the output. Okay. So until now what we have seen is that So until now what we have seen is that So until now what we have seen is that what is the noising process in the case what is the noising process in the case

  16. L13 @ 831s

    what is the noising process in the case of diffusion language models. The next of diffusion language models. The next of diffusion language models. The next step which we are going to see is that step which we are going to see is that step which we are going to see is that how do we predict the noise. So in the how do we predict the noise. So in the how do we predict the noise. So in the case of uh images the noise is predicted case of uh images the noise is predicted case of uh images the noise is predicted by unit. But let's see in the case of by unit. But let's see in the case of by unit. But let's see in the case of diffusion language models how do we diffusion language models how do we diffusion language models how do we predict the noise? What kind of a model predict the noise? What kind of a model predict the noise? What kind of a model do we need to predict the noise? Um and do we need to predict the noise? Um and do we need to predict the noise? Um and the answer is much simpler than you the answer is much simpler than you the answer is much simpler than you think. So, let's get into that right

  17. L14 @ 4s

    Okay. So the next step in these key Okay. So the next step in these key characteristics which we mentioned and characteristics which we mentioned and characteristics which we mentioned and probably that's the most important step probably that's the most important step probably that's the most important step is this noise prediction. Right. is this noise prediction. Right. is this noise prediction. Right. Basically we need a model Basically we need a model Basically we need a model which can help us predict the noise in which can help us predict the noise in which can help us predict the noise in the noising process the noising process the noising process because the whole idea is as follows. because the whole idea is as follows. because the whole idea is as follows. Right? If you take a look at this video, if you take a look at this video again, if you take a look at this video again, what we want to do is that when we start what we want to do is that when we start what we want to do is that when we start rotating the fluid slowly, we want to rotating the fluid slowly, we want to rotating the fluid slowly, we want to predict how much noise is added at each predict how much noise is added at each predict how much noise is added at each step. So that during dinoising, it will step. So that during dinoising, it will step. So that during dinoising, it will it will help us recover the original it will help us recover the original it will help us recover the original state. state. state. So at each step of the noising process, So at each step of the noising process, So at each step of the noising process, how can we predict the amount of noise how can we predict the amount of noise how can we predict the amount of noise which is added? The way this is done in which is added? The way this is done in which is added? The way this is done in in the case of images is that in the case of images is that in the case of images is that you take a noisy image. you take a noisy image. you take a noisy image. You take a noisy image.

  18. L14 @ 61s

    You take a noisy image. You take a noisy image. Let's say this is a noisy image of a Let's say this is a noisy image of a Let's say this is a noisy image of a panda which is the example which we have panda which is the example which we have panda which is the example which we have been seeing. You take a noisy image and been seeing. You take a noisy image and been seeing. You take a noisy image and then you use a noise prediction model then you use a noise prediction model then you use a noise prediction model which is a unit model to predict the which is a unit model to predict the which is a unit model to predict the noise. How does the unit model work? noise. How does the unit model work? noise. How does the unit model work? Well, it's a combination of Well, it's a combination of Well, it's a combination of CNN which is convolutional neural CNN which is convolutional neural CNN which is convolutional neural network upsampling and down sampling network upsampling and down sampling network upsampling and down sampling layer. So it can extract features from layer. So it can extract features from layer. So it can extract features from the image. So it can identify what's the image. So it can identify what's the image. So it can identify what's noise and what's not. We'll also we also noise and what's not. We'll also we also noise and what's not. We'll also we also know the actual noise right during know the actual noise right during know the actual noise right during training. So we can predict whether the training. So we can predict whether the training. So we can predict whether the predicted noise is actually close to the predicted noise is actually close to the predicted noise is actually close to the actual noise. So loss function is actual noise. So loss function is actual noise. So loss function is defined based on the predicted noise and defined based on the predicted noise and defined based on the predicted noise and the actual noise. So we need a loss the actual noise. So we need a loss the actual noise. So we need a loss function, right? We need to predict the

  19. L14 @ 104s

    function, right? We need to predict the function, right? We need to predict the noise and then we need to compare it noise and then we need to compare it noise and then we need to compare it with the actual noise. Essentially we we with the actual noise. Essentially we we with the actual noise. Essentially we we need a model which can predict the noise need a model which can predict the noise need a model which can predict the noise added during the noising process and added during the noising process and added during the noising process and compare it with some true value. What is compare it with some true value. What is compare it with some true value. What is such model in the case of text? I want such model in the case of text? I want such model in the case of text? I want you all to take a pause at this moment you all to take a pause at this moment you all to take a pause at this moment and think about it yourself. Let's say and think about it yourself. Let's say and think about it yourself. Let's say you have this noisy input text, right? you have this noisy input text, right? you have this noisy input text, right? which is uh which is uh which is uh uh uh uh the the the so the next day is bright and we are so the next day is bright and we are so the next day is bright and we are masking next and is so it's the masking next and is so it's the masking next and is so it's the mask mask mask day day day mask and bright mask and bright mask and bright now that's my input sequence right now now that's my input sequence right now now that's my input sequence right now okay okay okay and uh so this is my input sequence this and uh so this is my input sequence this and uh so this is my input sequence this is mask and uh this is mask. Okay. is mask and uh this is mask. Okay. is mask and uh this is mask. Okay. How do I predict the true values here? How do I predict the true values here?

  20. L14 @ 164s

    How do I predict the true values here? So what I want to do what does it mean So what I want to do what does it mean So what I want to do what does it mean prediction of noise? I somehow want to prediction of noise? I somehow want to prediction of noise? I somehow want to predict predict predict the true tokens the true tokens the true tokens at the places This is exactly what is done in images This is exactly what is done in images also. Right? What what does it mean to also. Right? What what does it mean to also. Right? What what does it mean to predict the noise from a noisy image? It predict the noise from a noisy image? It predict the noise from a noisy image? It just means that we need to predict how just means that we need to predict how just means that we need to predict how we go from the noisy image to the clean we go from the noisy image to the clean we go from the noisy image to the clean image. This is exactly what we are doing over This is exactly what we are doing over here. We need to predict how to go from here. We need to predict how to go from here. We need to predict how to go from this noisy text to the clean text. So we this noisy text to the clean text. So we this noisy text to the clean text. So we need to predict what's the text which need to predict what's the text which need to predict what's the text which was here in the first place. was here in the first place. was here in the first place. Again, I invite you to pause this video Again, I invite you to pause this video Again, I invite you to pause this video at this moment and think about it. at this moment and think about it. at this moment and think about it. How do you know what the true token was How do you know what the true token was How do you know what the true token was at this position? And how do you get the at this position? And how do you get the

  21. L14 @ 221s

    at this position? And how do you get the loss function? loss function? loss function? Well, the answer is extremely simple, Well, the answer is extremely simple, Well, the answer is extremely simple, right? Because we we already know what right? Because we we already know what right? Because we we already know what the true tokens are. We know that here the true tokens are. We know that here the true tokens are. We know that here the true token is next and we know that the true token is next and we know that the true token is next and we know that here the true token is is, here the true token is is, here the true token is is, right? Then how do we get the loss right? Then how do we get the loss right? Then how do we get the loss function? We have exactly seen how to function? We have exactly seen how to function? We have exactly seen how to propagate this input sequence through my propagate this input sequence through my propagate this input sequence through my model model model and I'll get my predictions right. I'll and I'll get my predictions right. I'll and I'll get my predictions right. I'll get my predictions get my predictions get my predictions for the next I'll get my predictions for for the next I'll get my predictions for for the next I'll get my predictions for each of these. each of these. each of these. So I'll get the next token prediction So I'll get the next token prediction So I'll get the next token prediction for the I'll get the next token for the I'll get the next token for the I'll get the next token prediction for this mask. I'll get the prediction for this mask. I'll get the prediction for this mask. I'll get the next token prediction for day. I'll get next token prediction for day. I'll get next token prediction for day. I'll get the next token for mask and I'll get the the next token for mask and I'll get the the next token for mask and I'll get the next token prediction for bride. next token prediction for bride. next token prediction for bride. Right. So the next token which is Right. So the next token which is

  22. L14 @ 272s

    Right. So the next token which is predicted at this position I'll compare it with the true next token I'll compare it with the true next token which should be next. which should be next. which should be next. The next token which is predicted at The next token which is predicted at The next token which is predicted at this position. I'll compare it with the this position. I'll compare it with the this position. I'll compare it with the true next token which is bright. true next token which is bright. true next token which is bright. And I'll use this to compute my loss And I'll use this to compute my loss And I'll use this to compute my loss function. So this is L1 and this is L2. function. So this is L1 and this is L2. function. So this is L1 and this is L2. So my loss function will then be L1 + L2 So my loss function will then be L1 + L2 So my loss function will then be L1 + L2 / 2. Think about this for a moment, right? We Think about this for a moment, right? We are corrupting the text. We are masking are corrupting the text. We are masking are corrupting the text. We are masking it. I'm just passing the whole input it. I'm just passing the whole input it. I'm just passing the whole input sequence with the masks through the sequence with the masks through the sequence with the masks through the architecture and I'm getting the architecture and I'm getting the architecture and I'm getting the predictions. Why? How do I know I get predictions. Why? How do I know I get predictions. Why? How do I know I get the predictions? Because this is exactly the predictions? Because this is exactly the predictions? Because this is exactly what we saw in ARMs. If you see, if you what we saw in ARMs. If you see, if you what we saw in ARMs. If you see, if you remember what we saw in ARMs, remember what we saw in ARMs, remember what we saw in ARMs, in ARMs, for every input, we are getting in ARMs, for every input, we are getting in ARMs, for every input, we are getting predictions, right? For the next, these

  23. L14 @ 325s

    predictions, right? For the next, these predictions, right? For the next, these are my predictions. are my predictions. are my predictions. These are my predictions. I'm just going These are my predictions. I'm just going These are my predictions. I'm just going to compare the predictions for my mask to compare the predictions for my mask to compare the predictions for my mask with the true tokens with the true tokens with the true tokens and I'm going to get the loss function. and I'm going to get the loss function. and I'm going to get the loss function. It's as simple as that. So I'll compare It's as simple as that. So I'll compare It's as simple as that. So I'll compare the predicted token at this position. the predicted token at this position. the predicted token at this position. I'll compare it with the true token at I'll compare it with the true token at I'll compare it with the true token at this position. I'll get the predicted this position. I'll get the predicted this position. I'll get the predicted token at this position. I'll compare it token at this position. I'll compare it token at this position. I'll compare it with the true token at this position. with the true token at this position. with the true token at this position. And then I'll get the loss function And then I'll get the loss function And then I'll get the loss function based on the predicted token and the based on the predicted token and the based on the predicted token and the true token. And then I'll take an true token. And then I'll take an true token. And then I'll take an average of these losses. That's the average of these losses. That's the average of these losses. That's the model which I'm going to use. So in model which I'm going to use. So in model which I'm going to use. So in simple way the model which predicts the simple way the model which predicts the simple way the model which predicts the this model which predicts the noise this model which predicts the noise this model which predicts the noise right which is the second right which is the second right which is the second um second characteristics the model um second characteristics the model

  24. L14 @ 371s

    um second characteristics the model which predicts the noise which was a which predicts the noise which was a which predicts the noise which was a unit in the case of diffusion for unit in the case of diffusion for unit in the case of diffusion for images. This model is nothing but a images. This model is nothing but a images. This model is nothing but a transformer. is the or I should say the language is the or I should say the language model architecture itself. We are going to see this architecture We are going to see this architecture now just in a moment. But this now just in a moment. But this now just in a moment. But this architecture which I'm showing on on architecture which I'm showing on on architecture which I'm showing on on this side that is the model which this side that is the model which this side that is the model which predicts the noise. Okay. So now I'm predicts the noise. Okay. So now I'm predicts the noise. Okay. So now I'm just going to just going to just going to uh take you through this entire model uh take you through this entire model uh take you through this entire model because you'll see that most of it because you'll see that most of it because you'll see that most of it actually remains exactly the same. Here actually remains exactly the same. Here actually remains exactly the same. Here we have the input in the ARM we have the input in the ARM we have the input in the ARM architecture. We saw that there is a architecture. We saw that there is a architecture. We saw that there is a input, processor and output. Here also input, processor and output. Here also input, processor and output. Here also there will be an input, processor and there will be an input, processor and there will be an input, processor and output. Here also there is a transformer output. Here also there is a transformer output. Here also there is a transformer block. Many things remain exactly the block. Many things remain exactly the block. Many things remain exactly the same. Let's see what the differences same. Let's see what the differences same. Let's see what the differences are. Right? In the ARM architecture, we

  25. L14 @ 434s

    are. Right? In the ARM architecture, we are. Right? In the ARM architecture, we remember we had the uh token embedding remember we had the uh token embedding remember we had the uh token embedding and position embedding. We added them and position embedding. We added them and position embedding. We added them for the input sequences. Right? In the for the input sequences. Right? In the for the input sequences. Right? In the case of uh case of uh case of uh uh diffusion models, the first change uh diffusion models, the first change uh diffusion models, the first change which happens is that in these token which happens is that in these token which happens is that in these token embeddings, one or more of these tokens embeddings, one or more of these tokens embeddings, one or more of these tokens are masked. So this is masked. are masked. So this is masked. are masked. So this is masked. That's what we have already seen. So That's what we have already seen. So That's what we have already seen. So token embeddings, it's masked. Position token embeddings, it's masked. Position token embeddings, it's masked. Position embeddings, these remain the same. Here embeddings, these remain the same. Here embeddings, these remain the same. Here we don't have anything masked but token we don't have anything masked but token we don't have anything masked but token embedding we have a mask token here and embedding we have a mask token here and embedding we have a mask token here and which tokens to mask are is decided which tokens to mask are is decided which tokens to mask are is decided probabilistically but how many tokens to probabilistically but how many tokens to probabilistically but how many tokens to mask is decided by that noise schedule mask is decided by that noise schedule mask is decided by that noise schedule which we have. So this this noise should which we have. So this this noise should which we have. So this this noise should so let's say we get the token embedding so let's say we get the token embedding so let's say we get the token embedding we get the position embedding these we get the position embedding these we get the position embedding these tokens are now masked we add the token embedding plus position

  26. L14 @ 486s

    we add the token embedding plus position embedding the one major difference in embedding the one major difference in embedding the one major difference in diffusion language models is that along diffusion language models is that along diffusion language models is that along with adding token embedding plus with adding token embedding plus with adding token embedding plus position embedding we also add time position embedding we also add time position embedding we also add time embeddings right so we have time steps embeddings right so we have time steps embeddings right so we have time steps for the noise schedule if those are from for the noise schedule if those are from for the noise schedule if those are from 1 to 4 we have a 768 embed 768 create 1 to 4 we have a 768 embed 768 create 1 to 4 we have a 768 embed 768 create dimensional vector for each of these dimensional vector for each of these dimensional vector for each of these time steps. time steps. time steps. And remember what I mentioned for And remember what I mentioned for And remember what I mentioned for diffusion models. In diffusion models, diffusion models. In diffusion models, diffusion models. In diffusion models, whenever we are doing a forward pass, we whenever we are doing a forward pass, we whenever we are doing a forward pass, we not only choose a batch, we also choose not only choose a batch, we also choose not only choose a batch, we also choose the time step as well. the time step as well. the time step as well. So if you have chosen let's say time So if you have chosen let's say time So if you have chosen let's say time step equal to two, the time embedding, step equal to two, the time embedding, step equal to two, the time embedding, we'll go to the time embedding matrix we'll go to the time embedding matrix we'll go to the time embedding matrix and we'll see the vector corresponding and we'll see the vector corresponding and we'll see the vector corresponding to time step equal to two and that same to time step equal to two and that same to time step equal to two and that same time embedding vector is added to all my time embedding vector is added to all my

  27. L14 @ 531s

    time embedding vector is added to all my tokens. So now the input embedding is tokens. So now the input embedding is tokens. So now the input embedding is not just the token embedding plus not just the token embedding plus not just the token embedding plus position embedding. My input embedding position embedding. My input embedding position embedding. My input embedding is token embedding plus position is token embedding plus position is token embedding plus position embedding plus time embedding. There are embedding plus time embedding. There are embedding plus time embedding. There are three embedding vectors which are added three embedding vectors which are added three embedding vectors which are added together. Why do we consider time together. Why do we consider time together. Why do we consider time embedding here? Because the time at embedding here? Because the time at embedding here? Because the time at which we are predicting the noise, it's which we are predicting the noise, it's which we are predicting the noise, it's also important. also important. also important. While dnoising, we need to know the time While dnoising, we need to know the time While dnoising, we need to know the time step at which we are denoising. Right? step at which we are denoising. Right? step at which we are denoising. Right? If we are way if it's extremely if it's If we are way if it's extremely if it's If we are way if it's extremely if it's extremely noisy image, it's further extremely noisy image, it's further extremely noisy image, it's further along in time. If the image is not that along in time. If the image is not that along in time. If the image is not that noisy, it's earlier in time. This is one noisy, it's earlier in time. This is one noisy, it's earlier in time. This is one major change. We add the time embedding major change. We add the time embedding major change. We add the time embedding also along with the token embedding plus also along with the token embedding plus also along with the token embedding plus the position embedding. That's the first the position embedding. That's the first the position embedding. That's the first change. So now if you see if you compare change. So now if you see if you compare

  28. L14 @ 575s

    change. So now if you see if you compare these block diagrams these block diagrams these block diagrams in the ARM architecture, we only had the in the ARM architecture, we only had the in the ARM architecture, we only had the token embedding and position embedding. token embedding and position embedding. token embedding and position embedding. In the uh diffusion model architecture, In the uh diffusion model architecture, In the uh diffusion model architecture, we have token embedding, we have we have token embedding, we have we have token embedding, we have position embedding, and we have time position embedding, and we have time position embedding, and we have time embedding. And all these are added to embedding. And all these are added to embedding. And all these are added to each other. This resulting input each other. This resulting input each other. This resulting input embedding is then passed through through embedding is then passed through through embedding is then passed through through the transformer block. Okay. Now the the transformer block. Okay. Now the the transformer block. Okay. Now the transformer block remains exactly the transformer block remains exactly the transformer block remains exactly the same. same. same. The transformer block has no change. We The transformer block has no change. We The transformer block has no change. We have the layer norm, we have the multi have the layer norm, we have the multi have the layer norm, we have the multi head attention, we have dropout, head attention, we have dropout, head attention, we have dropout, shortcut, layer norm, feed forward shortcut, layer norm, feed forward shortcut, layer norm, feed forward network and dropout. So the whole flow network and dropout. So the whole flow network and dropout. So the whole flow which we have seen layer norm, multi which we have seen layer norm, multi which we have seen layer norm, multi head attention, dropout, shortcut head attention, dropout, shortcut head attention, dropout, shortcut connection here. then another layer norm connection here. then another layer norm connection here. then another layer norm feed forward neural network dropout and feed forward neural network dropout and feed forward neural network dropout and another shortcut connection. This stays another shortcut connection. This stays another shortcut connection. This stays exactly the same. The difference is in exactly the same. The difference is in

  29. L14 @ 625s

    exactly the same. The difference is in the attention mechanism. the attention mechanism. the attention mechanism. Now think about it right here. We no Now think about it right here. We no Now think about it right here. We no longer have a next token prediction longer have a next token prediction longer have a next token prediction task. What we have to do is we just have task. What we have to do is we just have task. What we have to do is we just have to predict the masks right? We just have to predict the masks right? We just have to predict the masks right? We just have to predict the masks. So if we are to predict the masks. So if we are to predict the masks. So if we are looking at this mask, I can look at looking at this mask, I can look at looking at this mask, I can look at tokens before this mask and I can look tokens before this mask and I can look tokens before this mask and I can look at tokens after this mask also. I get at tokens after this mask also. I get at tokens after this mask also. I get information of tokens behind me and I information of tokens behind me and I information of tokens behind me and I get information of tokens in front of me get information of tokens in front of me get information of tokens in front of me also. This is very different than ARMs. also. This is very different than ARMs. also. This is very different than ARMs. In ARM, if I'm looking at this position, In ARM, if I'm looking at this position, In ARM, if I'm looking at this position, I only have access to tokens which come I only have access to tokens which come I only have access to tokens which come before this position. But in diffusion before this position. But in diffusion before this position. But in diffusion models, for one token, I have access to models, for one token, I have access to models, for one token, I have access to tokens before it and after it. Which tokens before it and after it. Which tokens before it and after it. Which means that when I'm calculating the means that when I'm calculating the

  30. L14 @ 668s

    means that when I'm calculating the attention so the mask the mask day the mask day mask bright right which means that if mask bright right which means that if mask bright right which means that if I'm calculating the attention score for I'm calculating the attention score for I'm calculating the attention score for any token I can calculate the attention any token I can calculate the attention any token I can calculate the attention score for tokens before that token and score for tokens before that token and score for tokens before that token and also after that token right so if this also after that token right so if this also after that token right so if this is my attention scores matrix this will is my attention scores matrix this will is my attention scores matrix this will be a 5x5 matrix 1 2 3 4 5 1 2 3 4 5 1 2 be a 5x5 matrix 1 2 3 4 5 1 2 3 4 5 1 2 be a 5x5 matrix 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 In the case of auto reggressive models I In the case of auto reggressive models I will not I will not have so these will not I will not have so these will not I will not have so these attention scores right which I'm marking attention scores right which I'm marking attention scores right which I'm marking right now these attention scores will right now these attention scores will right now these attention scores will not be there I will only have these not be there I will only have these not be there I will only have these attention scores for an auto reggressive attention scores for an auto reggressive attention scores for an auto reggressive model model whereas whereas whereas for a diffusion model I can look at a for a diffusion model I can look at a for a diffusion model I can look at a token from all sides. So if I just rub token from all sides. So if I just rub token from all sides. So if I just rub this right now this right now this right now if I look at I essentially have this entire causal

  31. L14 @ 748s

    I essentially have this entire causal attention matrix which is available to attention matrix which is available to attention matrix which is available to me. I don't have to set these values to me. I don't have to set these values to me. I don't have to set these values to zero like in the ARM block. So one major zero like in the ARM block. So one major zero like in the ARM block. So one major difference takes place in this attention difference takes place in this attention difference takes place in this attention module. If you look here in the auto module. If you look here in the auto module. If you look here in the auto reggressive modeling the causal reggressive modeling the causal reggressive modeling the causal attention exists which means that all attention exists which means that all attention exists which means that all the attention scores above this diagonal the attention scores above this diagonal the attention scores above this diagonal are set to zero. Whereas this does not are set to zero. Whereas this does not are set to zero. Whereas this does not happen in the case of diffusion models. happen in the case of diffusion models. happen in the case of diffusion models. We don't have such a causal attention We don't have such a causal attention We don't have such a causal attention mask. We have a biirectional attention mask. We have a biirectional attention mask. We have a biirectional attention mechanism. So all of the attention mechanism. So all of the attention mechanism. So all of the attention scores remain valid. scores remain valid. scores remain valid. That's one major difference and I want That's one major difference and I want That's one major difference and I want all of you to keep this in mind. That's all of you to keep this in mind. That's all of you to keep this in mind. That's why we went into detail in the attention why we went into detail in the attention why we went into detail in the attention mechanism. That is the second major mechanism. That is the second major mechanism. That is the second major difference, right? Uh what's the third difference, right? Uh what's the third difference, right? Uh what's the third major difference? The third major major difference? The third major

  32. L14 @ 795s

    major difference? The third major difference comes in the output token difference comes in the output token difference comes in the output token prediction, right? Uh let's say here we prediction, right? Uh let's say here we prediction, right? Uh let's say here we have every effort moves you, right? and have every effort moves you, right? and have every effort moves you, right? and we have predictions and we have targets we have predictions and we have targets we have predictions and we have targets and we compute the loss for all of these and we compute the loss for all of these and we compute the loss for all of these and we add all of these losses together and we add all of these losses together and we add all of these losses together for diffusion based modeling it's very for diffusion based modeling it's very for diffusion based modeling it's very different for diffusion based models different for diffusion based models different for diffusion based models let's say this is my input sequence I let's say this is my input sequence I let's say this is my input sequence I have to only consider the prediction at have to only consider the prediction at have to only consider the prediction at this position and I have to consider the this position and I have to consider the this position and I have to consider the prediction at this position and I have prediction at this position and I have prediction at this position and I have to compare it with the actual value to compare it with the actual value to compare it with the actual value right so I just have to find the cross right so I just have to find the cross right so I just have to find the cross entropy loss for my mask one I have to entropy loss for my mask one I have to entropy loss for my mask one I have to find the cross entropy loss for my mask find the cross entropy loss for my mask find the cross entropy loss for my mask two and I have to take the average of two and I have to take the average of two and I have to take the average of these losses. That's my loss. these losses. That's my loss.

  33. L14 @ 836s

    these losses. That's my loss. I don't have to find the loss for the I don't have to find the loss for the I don't have to find the loss for the other positions at all. other positions at all. other positions at all. Even in the example which we have chosen Even in the example which we have chosen Even in the example which we have chosen over here um the mask day. So this over here um the mask day. So this over here um the mask day. So this example right the example right the example right the mask day mask and bright right. So here I only have to care. So let's So here I only have to care. So let's say I have my prediction here and the say I have my prediction here and the say I have my prediction here and the prediction here prediction here prediction here the true value is next and that comes at the true value is next and that comes at the true value is next and that comes at ID equal to 3. I just have to take ID equal to 3. I just have to take ID equal to 3. I just have to take negative log of probability at this negative log of probability at this negative log of probability at this position position position and here the true value is let's say is and here the true value is let's say is and here the true value is let's say is comes at ID equal to 500. So then I just comes at ID equal to 500. So then I just comes at ID equal to 500. So then I just have to take negative log of probability have to take negative log of probability have to take negative log of probability of 500 and the loss of 500 and the loss of 500 and the loss and the loss is just then equal to and the loss is just then equal to and the loss is just then equal to negative of 1 by 2 log of P3 plus log of negative of 1 by 2 log of P3 plus log of negative of 1 by 2 log of P3 plus log of P500. P500. P500. So if this is let's say 0.1 this So if this is let's say 0.1 this

  34. L14 @ 904s

    So if this is let's say 0.1 this probability and this is2 my loss in this probability and this is2 my loss in this probability and this is2 my loss in this case will be - 1 by2 log of.1 log of.1 plus log of 2. plus log of 2. plus log of 2. So let's see how much that is. -1 by 2 So let's see how much that is. -1 by 2 So let's see how much that is. -1 by 2 into log of.1 into log of.1 into log of.1 plus log of plus log of plus log of that's 84. So the loss which I've that's 84. So the loss which I've that's 84. So the loss which I've obtained in this case is obtained in this case is obtained in this case is um um um so the loss which I've obtained in this so the loss which I've obtained in this so the loss which I've obtained in this case is 84 case is 84 case is 84 and then I do back propagation. So the and then I do back propagation. So the and then I do back propagation. So the way this entire way this entire way this entire um um noising process happens now I've tried noising process happens now I've tried noising process happens now I've tried to explain it not the entire noising to explain it not the entire noising to explain it not the entire noising process but the entire flow we first process but the entire flow we first process but the entire flow we first decide a time point let's say time equal decide a time point let's say time equal decide a time point let's say time equal to three right I asked 50% of the input to three right I asked 50% of the input to three right I asked 50% of the input sequences the masked input then goes sequences the masked input then goes sequences the masked input then goes through this whole architecture what are through this whole architecture what are through this whole architecture what are the changes in this architecture the changes in this architecture the changes in this architecture compared to the auto reggressive model compared to the auto reggressive model compared to the auto reggressive model we have a time embedding now which was

  35. L14 @ 975s

    we have a time embedding now which was we have a time embedding now which was not there auto reggressive model in not there auto reggressive model in not there auto reggressive model in multi attention block uh we don't have multi attention block uh we don't have multi attention block uh we don't have the causal attention now we can take the the causal attention now we can take the the causal attention now we can take the whole whole attention attention scores whole whole attention attention scores whole whole attention attention scores matrix because for each token we can matrix because for each token we can matrix because for each token we can look behind and we can look ahead also look behind and we can look ahead also look behind and we can look ahead also okay that's why generally b models are okay that's why generally b models are okay that's why generally b models are good at sentiment analysis right because good at sentiment analysis right because good at sentiment analysis right because bird is also biirectional in bird models bird is also biirectional in bird models bird is also biirectional in bird models we look at a token from behind and ahead we look at a token from behind and ahead we look at a token from behind and ahead so you can look at an entire sentence so you can look at an entire sentence so you can look at an entire sentence and predict the sentiment of the and predict the sentiment of the and predict the sentiment of the sentence right It's kind of similar sentence right It's kind of similar sentence right It's kind of similar here. The only difference from the BERT here. The only difference from the BERT here. The only difference from the BERT architecture is that in BERT the masking architecture is that in BERT the masking architecture is that in BERT the masking ratio is fixed. But here the masking ratio is fixed. But here the masking ratio is fixed. But here the masking ratio is different. Right? The masking ratio is different. Right? The masking ratio is different. Right? The masking ratio uh in this case actually depends ratio uh in this case actually depends ratio uh in this case actually depends on u the time step. More the time step,

  36. L14 @ 1022s

    on u the time step. More the time step, on u the time step. More the time step, more is the masking. Less the time step, more is the masking. Less the time step, more is the masking. Less the time step, less the masking. That's the difference less the masking. That's the difference less the masking. That's the difference between BERT and uh language diffusion. between BERT and uh language diffusion. between BERT and uh language diffusion. So we have chosen time equal to three. So we have chosen time equal to three. So we have chosen time equal to three. 50% is masked that input sequence. we 50% is masked that input sequence. we 50% is masked that input sequence. we choose a batch. So if the batch size is choose a batch. So if the batch size is choose a batch. So if the batch size is 128, we choose a batch of 128 sequences. 128, we choose a batch of 128 sequences. 128, we choose a batch of 128 sequences. We mask 50% of those and we pass it We mask 50% of those and we pass it We mask 50% of those and we pass it through this entire architecture. So through this entire architecture. So through this entire architecture. So this is let's say the input sequence, it this is let's say the input sequence, it this is let's say the input sequence, it first goes through the token embedding. first goes through the token embedding. first goes through the token embedding. So every token is converted into an So every token is converted into an So every token is converted into an embedding vector. We add the positional embedding vector. We add the positional embedding vector. We add the positional embedding, we add the time embedding and embedding, we add the time embedding and embedding, we add the time embedding and uh so token embedding plus position uh so token embedding plus position uh so token embedding plus position embedding plus time embedding is my embedding plus time embedding is my embedding plus time embedding is my final input vector that goes through my final input vector that goes through my final input vector that goes through my transformer block. So I have my layer transformer block. So I have my layer

  37. L14 @ 1063s

    transformer block. So I have my layer normalization multi head attention I normalization multi head attention I normalization multi head attention I have my dropout I have my skip have my dropout I have my skip have my dropout I have my skip connection I have second layer connection I have second layer connection I have second layer normalization I have my feed forward normalization I have my feed forward normalization I have my feed forward neural network I have dropout and neural network I have dropout and neural network I have dropout and another shortcut connection on the another shortcut connection on the another shortcut connection on the outset which means at the end I have my outset which means at the end I have my outset which means at the end I have my final normalization I have this logits final normalization I have this logits final normalization I have this logits matrix and I have the cross entropy loss matrix and I have the cross entropy loss matrix and I have the cross entropy loss which is only computed for the mask which is only computed for the mask which is only computed for the mask tokens that's the second major last tokens that's the second major last tokens that's the second major last major difference between language major difference between language major difference between language diffusion and auto reggressive model diffusion and auto reggressive model diffusion and auto reggressive model diffusion here the Cross entropy loss is diffusion here the Cross entropy loss is diffusion here the Cross entropy loss is only calculated for the mask tokens and only calculated for the mask tokens and only calculated for the mask tokens and for nothing else. for nothing else. for nothing else. So to summarize the entire noising So to summarize the entire noising So to summarize the entire noising process looks like this. We have the process looks like this. We have the process looks like this. We have the let's say we choose the batch size. We let's say we choose the batch size. We let's say we choose the batch size. We choose two things, right? We choose the choose two things, right? We choose the choose two things, right? We choose the batch size. We choose the time step, batch size. We choose the time step,

  38. L14 @ 1111s

    batch size. We choose the time step, right? Let's say batch size is 128. We right? Let's say batch size is 128. We right? Let's say batch size is 128. We assemble an input sequence of assemble an input sequence of assemble an input sequence of 128 tokens uh and we choose a batch size. So we uh and we choose a batch size. So we randomly mask let's say 50. We choose a randomly mask let's say 50. We choose a randomly mask let's say 50. We choose a time step. If the time step is half of time step. If the time step is half of time step. If the time step is half of the total time, we randomly mask 50% of the total time, we randomly mask 50% of the total time, we randomly mask 50% of these. So these will be replaced with these. So these will be replaced with these. So these will be replaced with mask. Okay. Now before now these this input Okay. Now before now these this input sequence before going through the sequence before going through the sequence before going through the transformer architecture we add the transformer architecture we add the transformer architecture we add the token embeddings plus the position token embeddings plus the position token embeddings plus the position embeddings plus the time embeddings. embeddings plus the time embeddings. embeddings plus the time embeddings. So this vector which I have shown is So this vector which I have shown is So this vector which I have shown is token embedding plus position plus time. token embedding plus position plus time. token embedding plus position plus time. And this this input vectors then go into And this this input vectors then go into And this this input vectors then go into the transformer architecture. When they the transformer architecture. When they the transformer architecture. When they come out of the transformer come out of the transformer come out of the transformer architecture, we get the categorical architecture, we get the categorical architecture, we get the categorical cross entropy loss only on the mask cross entropy loss only on the mask cross entropy loss only on the mask tokens. Based on this loss, we find the tokens. Based on this loss, we find the

  39. L14 @ 1170s

    tokens. Based on this loss, we find the partial derivative of the loss with partial derivative of the loss with partial derivative of the loss with respect to all the parameters. And respect to all the parameters. And respect to all the parameters. And remember we have seen what what do I remember we have seen what what do I remember we have seen what what do I mean by all the parameters? Uh we have mean by all the parameters? Uh we have mean by all the parameters? Uh we have seen everything marked P over here are seen everything marked P over here are seen everything marked P over here are all the parameters. We have seen that in all the parameters. We have seen that in all the parameters. We have seen that in one of the previous lectures. This one of the previous lectures. This one of the previous lectures. This remains the same. All trainable remains the same. All trainable remains the same. All trainable parameters remain the same in ARM and in parameters remain the same in ARM and in parameters remain the same in ARM and in the language diffusion. the language diffusion. the language diffusion. Right? So we get the partial derivative Right? So we get the partial derivative Right? So we get the partial derivative of loss with respect to all the of loss with respect to all the of loss with respect to all the parameters. Then we update parameters. Then we update parameters. Then we update then we update the parameters then we update the parameters then we update the parameters according to ADAM optimizer or any other according to ADAM optimizer or any other according to ADAM optimizer or any other optimizer which you want. And then this optimizer which you want. And then this optimizer which you want. And then this entire thing goes into a loop. entire thing goes into a loop. entire thing goes into a loop. This is how I pre-train. This is how I pre-train. This is how I pre-train. This is how I pre-train a language This is how I pre-train a language This is how I pre-train a language diffusion model. What is implicitly happening in this What is implicitly happening in this case is the better we pre-train a case is the better we pre-train a

  40. L14 @ 1227s

    case is the better we pre-train a language diffusion model the better we language diffusion model the better we language diffusion model the better we are able to discover what noise is added are able to discover what noise is added are able to discover what noise is added and during the generation process or and during the generation process or and during the generation process or during the dnoising the better we'll be during the dnoising the better we'll be during the dnoising the better we'll be able to find the underlying probability able to find the underlying probability able to find the underlying probability distribution right so if you look at distribution right so if you look at distribution right so if you look at this analogy over here. Yeah. So what we are doing right now is Yeah. So what we are doing right now is the noising process, right? And we are the noising process, right? And we are the noising process, right? And we are training the model to recover the noise. training the model to recover the noise. training the model to recover the noise. And the better we train this model to And the better we train this model to And the better we train this model to recover the noise, the better we'll be recover the noise, the better we'll be recover the noise, the better we'll be at denoising. at denoising. at denoising. So the better we so in dnoising what we So the better we so in dnoising what we So the better we so in dnoising what we are going to do is that we are going to are going to do is that we are going to are going to do is that we are going to start with all masks and then we we are start with all masks and then we we are start with all masks and then we we are slowly going to uncover the masks and slowly going to uncover the masks and slowly going to uncover the masks and recover the actual tokens in that recover the actual tokens in that recover the actual tokens in that position. position. position. That's why we get gifs such as this. That's why we get gifs such as this. That's why we get gifs such as this. Start with all masks and then we uncover

  41. L14 @ 1278s

    Start with all masks and then we uncover Start with all masks and then we uncover tokens correct tokens at that position. tokens correct tokens at that position. tokens correct tokens at that position. But dnoising will only work if the loss But dnoising will only work if the loss But dnoising will only work if the loss function during the pre-training or function during the pre-training or function during the pre-training or understanding the noise is good. Which understanding the noise is good. Which understanding the noise is good. Which means the loss function becomes lower means the loss function becomes lower means the loss function becomes lower and lower and lower. and lower and lower. and lower and lower. In this large language diffusion models In this large language diffusion models In this large language diffusion models paper, they have mentioned a theoretical paper, they have mentioned a theoretical paper, they have mentioned a theoretical result here which I just want to result here which I just want to result here which I just want to mention. So this is the loss function mention. So this is the loss function mention. So this is the loss function during pre-training. All which I have during pre-training. All which I have during pre-training. All which I have explained right now. Everything which I explained right now. Everything which I explained right now. Everything which I have explained on the screen this this have explained on the screen this this have explained on the screen this this loss function which I have explained on loss function which I have explained on loss function which I have explained on the screen until now is explained in one the screen until now is explained in one the screen until now is explained in one simple formula over here. That's why simple formula over here. That's why simple formula over here. That's why it's very difficult to understand it's very difficult to understand it's very difficult to understand research papers until you have a firm research papers until you have a firm research papers until you have a firm background of the foundations. So this background of the foundations. So this background of the foundations. So this is the loss function. What these people is the loss function. What these people is the loss function. What these people predict is that predict is that

  42. L14 @ 1322s

    predict is that the probability of actually recovering the probability of actually recovering the probability of actually recovering the true true distribution or the the true true distribution or the the true true distribution or the chances of recovering the true chances of recovering the true chances of recovering the true distribution are directly dependent on distribution are directly dependent on distribution are directly dependent on the loss function. The lower the loss the loss function. The lower the loss the loss function. The lower the loss function, the better we can recover the function, the better we can recover the function, the better we can recover the true distribution during dinoising. This true distribution during dinoising. This true distribution during dinoising. This is what is meant by this second formula. is what is meant by this second formula. is what is meant by this second formula. So dnoising is very directly linked with So dnoising is very directly linked with So dnoising is very directly linked with this loss function over here. So there this loss function over here. So there this loss function over here. So there is a link between the noising process is a link between the noising process is a link between the noising process and the dinoising. In the noising and the dinoising. In the noising and the dinoising. In the noising process, we are adding masks and we're process, we are adding masks and we're process, we are adding masks and we're recovering the true value at those recovering the true value at those recovering the true value at those masks. The better we do this, the better masks. The better we do this, the better masks. The better we do this, the better will be the dnoising also. I'm not going will be the dnoising also. I'm not going will be the dnoising also. I'm not going into the mathematical details of this into the mathematical details of this into the mathematical details of this but they have mentioned that but they have mentioned that but they have mentioned that um this has been proved the loss um this has been proved the loss um this has been proved the loss function in equation three has been function in equation three has been function in equation three has been proved to be an upper bound on the

  43. L14 @ 1368s

    proved to be an upper bound on the proved to be an upper bound on the negative log likelihood making it a negative log likelihood making it a negative log likelihood making it a principled objective for generative principled objective for generative principled objective for generative modeling. So what they are saying is modeling. So what they are saying is modeling. So what they are saying is that the lower the loss function is the that the lower the loss function is the that the lower the loss function is the better we'll be at this uh true distri better we'll be at this uh true distri better we'll be at this uh true distri better better we'll be at approximating better better we'll be at approximating better better we'll be at approximating this true distribution. this true distribution. this true distribution. So if the green is the true distribution So if the green is the true distribution So if the green is the true distribution and the orange is the predicted, the and the orange is the predicted, the and the orange is the predicted, the lower the loss function in pre-training, lower the loss function in pre-training, lower the loss function in pre-training, the closer these two will lie. the closer these two will lie. the closer these two will lie. So the dotted red line will lie much So the dotted red line will lie much So the dotted red line will lie much closer to the orange. If the loss closer to the orange. If the loss closer to the orange. If the loss function during pre-training is lower, function during pre-training is lower, function during pre-training is lower, what is the loss function? It's exactly what is the loss function? It's exactly what is the loss function? It's exactly what we saw saw over here. what we saw saw over here. what we saw saw over here. The loss function essentially quantifies The loss function essentially quantifies The loss function essentially quantifies what is there at the masked masked what is there at the masked masked what is there at the masked masked positions versus what should be there at positions versus what should be there at positions versus what should be there at the masked positions. Okay. the masked positions. Okay.

  44. L14 @ 1414s

    the masked positions. Okay. So, until now we have seen the first two So, until now we have seen the first two So, until now we have seen the first two steps of diffusion based language steps of diffusion based language steps of diffusion based language models. First is what is the noising models. First is what is the noising models. First is what is the noising process and second is how do we predict process and second is how do we predict process and second is how do we predict the noise? We predict the noise through the noise? We predict the noise through the noise? We predict the noise through the language model architecture itself the language model architecture itself the language model architecture itself which I've depicted which I've depicted which I've depicted using this diagram. using this diagram. using this diagram. So, anytime you are confused, you can So, anytime you are confused, you can So, anytime you are confused, you can take a look at take a look at take a look at yeah this diagram. This is the diagram yeah this diagram. This is the diagram yeah this diagram. This is the diagram which is essentially used for predicting which is essentially used for predicting which is essentially used for predicting the noise in the case of diffusion the noise in the case of diffusion the noise in the case of diffusion models. And I hope all of you have seen models. And I hope all of you have seen models. And I hope all of you have seen the differences between the diffusion the differences between the diffusion the differences between the diffusion architecture and the uh ARM or the reg architecture and the uh ARM or the reg architecture and the uh ARM or the reg auto reggressive architecture. Time auto reggressive architecture. Time auto reggressive architecture. Time embedding is the first difference. Then embedding is the first difference. Then embedding is the first difference. Then in multi attention the causality is the in multi attention the causality is the in multi attention the causality is the second difference. Then the cross second difference. Then the cross second difference. Then the cross entropy only for the mask tokens that is entropy only for the mask tokens that is entropy only for the mask tokens that is the third difference. Noisy input here

  45. L14 @ 1462s

    the third difference. Noisy input here the third difference. Noisy input here that's the four difference. There are that's the four difference. There are that's the four difference. There are four major differences. Everything else four major differences. Everything else four major differences. Everything else essentially remains the same. That's why essentially remains the same. That's why essentially remains the same. That's why I mentioned earlier that understanding I mentioned earlier that understanding I mentioned earlier that understanding the auto reggressive model architecture the auto reggressive model architecture the auto reggressive model architecture is so crucial for understanding the is so crucial for understanding the is so crucial for understanding the language diffusion model architecture. language diffusion model architecture. language diffusion model architecture. Okay. Now when I'm going to share this Okay. Now when I'm going to share this Okay. Now when I'm going to share this resources with you, there is also a resources with you, there is also a resources with you, there is also a website in which I have explained all of website in which I have explained all of website in which I have explained all of these calculations from scratch. So if these calculations from scratch. So if these calculations from scratch. So if you scroll down below, I have explained you scroll down below, I have explained you scroll down below, I have explained there is a section called vector flow. there is a section called vector flow. there is a section called vector flow. So what we do over here is that we take So what we do over here is that we take So what we do over here is that we take a toy model configuration where we a toy model configuration where we a toy model configuration where we assume vocabulary size is equal to six assume vocabulary size is equal to six assume vocabulary size is equal to six sequence length is equal to 5. Uh we sequence length is equal to 5. Uh we sequence length is equal to 5. Uh we this is a we construct a token embedding this is a we construct a token embedding this is a we construct a token embedding matrix. We construct a position matrix. We construct a position matrix. We construct a position embedding matrix and we construct a time embedding matrix and we construct a time

  46. L14 @ 1509s

    embedding matrix and we construct a time embedding matrix also. And I show you embedding matrix also. And I show you embedding matrix also. And I show you how everything is done. So at each how everything is done. So at each how everything is done. So at each position we add the token embedding, position we add the token embedding, position we add the token embedding, position embedding and the time position embedding and the time position embedding and the time embedding. Then we do layer embedding. Then we do layer embedding. Then we do layer normalization vector wise. So if you normalization vector wise. So if you normalization vector wise. So if you want to go into the mathematical details want to go into the mathematical details want to go into the mathematical details of how the dimensions are retained, I'll of how the dimensions are retained, I'll of how the dimensions are retained, I'll share this link with you where even I've share this link with you where even I've share this link with you where even I've shown everything from the attention shown everything from the attention shown everything from the attention mechanism to the shortcut connections mechanism to the shortcut connections mechanism to the shortcut connections which are also called residuals to the which are also called residuals to the which are also called residuals to the second layer normalization feed forward second layer normalization feed forward second layer normalization feed forward neural network final layer normalization neural network final layer normalization neural network final layer normalization final logits matrix and the final cross final logits matrix and the final cross final logits matrix and the final cross entropy loss. I have already shown this entropy loss. I have already shown this entropy loss. I have already shown this to you on the whiteboard but just for to you on the whiteboard but just for to you on the whiteboard but just for the sake of simplicity I've added this the sake of simplicity I've added this the sake of simplicity I've added this hole over here. So the entire forward hole over here. So the entire forward hole over here. So the entire forward pass pipeline for diffusion language pass pipeline for diffusion language pass pipeline for diffusion language model can be summarized here. We have

  47. L14 @ 1554s

    model can be summarized here. We have model can be summarized here. We have token ids convert them into token token ids convert them into token token ids convert them into token embedding vectors add position embedding embedding vectors add position embedding embedding vectors add position embedding vectors add the time embedding vectors vectors add the time embedding vectors vectors add the time embedding vectors since you have to decide a time step at since you have to decide a time step at since you have to decide a time step at the start of the forward pass. Then we the start of the forward pass. Then we the start of the forward pass. Then we go into the transformer architecture. We go into the transformer architecture. We go into the transformer architecture. We pass through the first layer norm, pass through the first layer norm, pass through the first layer norm, attention mechanism, shortcut attention mechanism, shortcut attention mechanism, shortcut connection, second layer norm, feed connection, second layer norm, feed connection, second layer norm, feed forward neural network, another shortcut forward neural network, another shortcut forward neural network, another shortcut connection. Final then we come to the connection. Final then we come to the connection. Final then we come to the output header. We have the final layer output header. We have the final layer output header. We have the final layer normalization. We have the logits matrix normalization. We have the logits matrix normalization. We have the logits matrix and we have the cross entropy loss on and we have the cross entropy loss on and we have the cross entropy loss on the mask positions. the mask positions. the mask positions. All right, cool. And now we are ready to All right, cool. And now we are ready to All right, cool. And now we are ready to move to the final step which is move to the final step which is move to the final step which is essentially the dnoising process. So essentially the dnoising process. So essentially the dnoising process. So let's get to that right now.

  48. L18 @ 3s

    So here you can see the file which has So here you can see the file which has been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was been loaded on runpod. Earlier I was getting some error but now it has run getting some error but now it has run getting some error but now it has run pretty well. And uh the same approach as pretty well. And uh the same approach as pretty well. And uh the same approach as we saw on Google collab works here. And we saw on Google collab works here. And we saw on Google collab works here. And uh you'll see the gif being printed uh you'll see the gif being printed uh you'll see the gif being printed right on the screen in this code. And right on the screen in this code. And right on the screen in this code. And this uh fancy neon style gif also can be this uh fancy neon style gif also can be this uh fancy neon style gif also can be seen over here. This does not work very seen over here. This does not work very seen over here. This does not work very well right now but you can tweak the well right now but you can tweak the well right now but you can tweak the code with the help of claude or chat GPT code with the help of claude or chat GPT code with the help of claude or chat GPT to get this neon style gif to work. to get this neon style gif to work. to get this neon style gif to work. All right. So here what I've done is All right. So here what I've done is All right. So here what I've done is that I have just mentioned the model to that I have just mentioned the model to that I have just mentioned the model to go ahead for uh go ahead for uh go ahead for uh around 200,000 iterations and uh if you around 200,000 iterations and uh if you around 200,000 iterations and uh if you go ahead for these many number of go ahead for these many number of go ahead for these many number of iterations you'll definitely see samples iterations you'll definitely see samples

  49. L18 @ 56s

    iterations you'll definitely see samples which are as co inference samples which which are as co inference samples which which are as co inference samples which are as coherent as what you are seeing are as coherent as what you are seeing are as coherent as what you are seeing on the screen right now. on the screen right now. on the screen right now. The reason I'm showing runpod to you The reason I'm showing runpod to you The reason I'm showing runpod to you along with Google collab is because along with Google collab is because along with Google collab is because uh runpod also allows you to chain uh runpod also allows you to chain uh runpod also allows you to chain multiple GPUs together and in run you multiple GPUs together and in run you multiple GPUs together and in run you can also see the different GPUs which can also see the different GPUs which can also see the different GPUs which are available whereas on Google collab are available whereas on Google collab are available whereas on Google collab if you see you're kind of restricted if you see you're kind of restricted if you see you're kind of restricted right by the GPUs which they give. So if right by the GPUs which they give. So if right by the GPUs which they give. So if you go to run type you'll see you have you go to run type you'll see you have you go to run type you'll see you have H100 A100 you can't see the H100 A100 you can't see the H100 A100 you can't see the specifications it's really like a black specifications it's really like a black specifications it's really like a black box but on the run pod you can see the box but on the run pod you can see the box but on the run pod you can see the VRAM of each GPU you can see the number VRAM of each GPU you can see the number VRAM of each GPU you can see the number of GPUs available you can see the cost of GPUs available you can see the cost of GPUs available you can see the cost you can even chain different GPUs you can even chain different GPUs

  50. L18 @ 97s

    you can even chain different GPUs together to do distributed computing if together to do distributed computing if together to do distributed computing if possible possible possible so let's actually pause this right now so let's actually pause this right now so let's actually pause this right now let's interrupt this after 500 let's interrupt this after 500 let's interrupt this after 500 iterations S and uh yeah then we just iterations S and uh yeah then we just iterations S and uh yeah then we just have the sampling code over here have the sampling code over here have the sampling code over here once we have the sample here you'll see once we have the sample here you'll see once we have the sample here you'll see the sampling is pretty bad because I the sampling is pretty bad because I the sampling is pretty bad because I only run for 500 iterations so that's only run for 500 iterations so that's only run for 500 iterations so that's natural natural natural um and in this next piece of code you'll and in this next piece of code you'll save the GIF and after you save the GIF save the GIF and after you save the GIF save the GIF and after you save the GIF then you'll print the GIF. So this code then you'll print the GIF. So this code then you'll print the GIF. So this code will save the GIF as inference.jif and will save the GIF as inference.jif and will save the GIF as inference.jif and this will uh essentially print it on the this will uh essentially print it on the this will uh essentially print it on the screen screen screen right now. Right now I'm only doing for right now. Right now I'm only doing for right now. Right now I'm only doing for 500 iterations but you see already the 500 iterations but you see already the 500 iterations but you see already the GIF is appearing on the screen. We don't GIF is appearing on the screen. We don't GIF is appearing on the screen. We don't get that error which I was getting get that error which I was getting get that error which I was getting before. Um yeah if you train for 200,000 before. Um yeah if you train for 200,000

  51. L18 @ 156s

    before. Um yeah if you train for 200,000 iterations you'll see an amazing iterations you'll see an amazing iterations you'll see an amazing coherent language diffusion language coherent language diffusion language coherent language diffusion language model working from scratch as I'm model working from scratch as I'm model working from scratch as I'm showing on the screen right now. showing on the screen right now. showing on the screen right now. I'll share the entire code file with I'll share the entire code file with I'll share the entire code file with you. You'll also now be able to you. You'll also now be able to you. You'll also now be able to understand this large language diffusion understand this large language diffusion understand this large language diffusion models paper in a lot of depth because models paper in a lot of depth because models paper in a lot of depth because we have implemented this right from we have implemented this right from we have implemented this right from scratch. They also have a section of scratch. They also have a section of scratch. They also have a section of fine-tuning towards the end which you fine-tuning towards the end which you fine-tuning towards the end which you can easily do can easily do can easily do after you have finished the after you have finished the after you have finished the pre-training. You just have to assemble pre-training. You just have to assemble pre-training. You just have to assemble a fine fine tuning data set and then run a fine fine tuning data set and then run a fine fine tuning data set and then run fine tuning on top of it. So let me just fine tuning on top of it. So let me just fine tuning on top of it. So let me just Yeah. Yeah. Yeah. So as I mentioned this paper also So as I mentioned this paper also So as I mentioned this paper also includes fine-tuning. I have not includes fine-tuning. I have not includes fine-tuning. I have not included fine-tuning in this journey included fine-tuning in this journey included fine-tuning in this journey because I believe pre-training involves because I believe pre-training involves because I believe pre-training involves all the necessary things which you will all the necessary things which you will

  52. L18 @ 203s

    all the necessary things which you will need to learn about language diffusion need to learn about language diffusion need to learn about language diffusion models. All right. So yeah we have this All right. So yeah we have this traditional ARM language model code traditional ARM language model code traditional ARM language model code which runs and predicts sequential which runs and predicts sequential which runs and predicts sequential output which is coherent. And we have output which is coherent. And we have output which is coherent. And we have this uh this uh this uh diffusion model based language model diffusion model based language model diffusion model based language model code which predicts output such as this code which predicts output such as this code which predicts output such as this which is much faster in inference but which is much faster in inference but which is much faster in inference but pre-training might take more time. So as pre-training might take more time. So as pre-training might take more time. So as Andre Karpati mentioned in this tweet Andre Karpati mentioned in this tweet Andre Karpati mentioned in this tweet diffusion language models actually can diffusion language models actually can diffusion language models actually can be a frontier milestone in the future be a frontier milestone in the future be a frontier milestone in the future because why do we have to stick with because why do we have to stick with because why do we have to stick with auto reggressive models right of course. auto reggressive models right of course. auto reggressive models right of course. So uh I believe this is one of the first So uh I believe this is one of the first So uh I believe this is one of the first tutorials or one of the first journey tutorials or one of the first journey tutorials or one of the first journey videos which shows how to build a small videos which shows how to build a small videos which shows how to build a small scale diffusion model fully from scratch scale diffusion model fully from scratch scale diffusion model fully from scratch and I hope this codebase which I've and I hope this codebase which I've and I hope this codebase which I've shown on the screen right now serves as

  53. L18 @ 259s

    shown on the screen right now serves as shown on the screen right now serves as a reference for you to build multiple a reference for you to build multiple a reference for you to build multiple such small language models from scratch. such small language models from scratch. such small language models from scratch. So the cool thing about diffusion models So the cool thing about diffusion models So the cool thing about diffusion models is that the space is wide open, right? is that the space is wide open, right? is that the space is wide open, right? You can see small language models um let's say small language models um let's say small language models GitHub or Quel right all of these small GitHub or Quel right all of these small GitHub or Quel right all of these small language models which have been built language models which have been built language models which have been built they are auto reggressive models but the they are auto reggressive models but the they are auto reggressive models but the space for building diffusion models is space for building diffusion models is space for building diffusion models is wide open. I know [clears throat] wide open. I know [clears throat] wide open. I know [clears throat] Inception Labs which is one company Inception Labs which is one company Inception Labs which is one company which builds diffusion models which are which builds diffusion models which are which builds diffusion models which are pretty cool but I think this is closed pretty cool but I think this is closed pretty cool but I think this is closed source. source. source. I'm yet to know and opensource model I'm yet to know and opensource model I'm yet to know and opensource model which is as fast and as good as this which is as fast and as good as this which is as fast and as good as this mercury model. So you see their whole uh mercury model. So you see their whole uh mercury model. So you see their whole uh value proposition is blazing fast value proposition is blazing fast value proposition is blazing fast inference with frontier quality and a inference with frontier quality and a inference with frontier quality and a fraction of the cost which might be fraction of the cost which might be

  54. L18 @ 308s

    fraction of the cost which might be awesome for the future. But now you know awesome for the future. But now you know awesome for the future. But now you know how to build this fully yourself. Um, okay. After this, I'll make a final okay. After this, I'll make a final conclusion video on my thoughts and conclusion video on my thoughts and conclusion video on my thoughts and we'll wrap it up.

  55. L17 @ 2s

    Okay. So now I'm going to take you Okay. So now I'm going to take you through the code of assembling this through the code of assembling this through the code of assembling this diffusion language model from scratch diffusion language model from scratch diffusion language model from scratch and u this code is actually and u this code is actually and u this code is actually exactly following these steps which we exactly following these steps which we exactly following these steps which we have seen on the whiteboard. So I think have seen on the whiteboard. So I think have seen on the whiteboard. So I think all of you will be able to follow along all of you will be able to follow along all of you will be able to follow along if you have understood the theory which if you have understood the theory which if you have understood the theory which we have covered in the earlier modules we have covered in the earlier modules we have covered in the earlier modules of this course. Okay. of this course. Okay. of this course. Okay. All right. So All right. So All right. So the first thing which you should note is the first thing which you should note is the first thing which you should note is that here I have a subscription of that here I have a subscription of that here I have a subscription of Google Collab Pro. So I have access to Google Collab Pro. So I have access to Google Collab Pro. So I have access to an A1 H100 GPU. an A1 H100 GPU. an A1 H100 GPU. Diffusion language models take time to Diffusion language models take time to Diffusion language models take time to train compared to train compared to train compared to the auto reggressive models. So you the auto reggressive models. So you the auto reggressive models. So you should have access to Google Collab Pro. should have access to Google Collab Pro. should have access to Google Collab Pro. You can get A100 GPU or an H100 GPU You can get A100 GPU or an H100 GPU You can get A100 GPU or an H100 GPU here. The H100 GPU is much faster. here. The H100 GPU is much faster.

  56. L17 @ 60s

    here. The H100 GPU is much faster. If you don't have access to the Google If you don't have access to the Google If you don't have access to the Google Collab Pro, you can start with the T4 Collab Pro, you can start with the T4 Collab Pro, you can start with the T4 GPU which is free, but it will just take GPU which is free, but it will just take GPU which is free, but it will just take a large amount of time for the model to a large amount of time for the model to a large amount of time for the model to run. run. run. Okay. Okay. Okay. So remember that So remember that So remember that first thing which we have to do is that first thing which we have to do is that first thing which we have to do is that we have to select the mode which we have we have to select the mode which we have we have to select the mode which we have to run. If you have a if you are on a to run. If you have a if you are on a to run. If you have a if you are on a free tire of Google Collab, I suggest free tire of Google Collab, I suggest free tire of Google Collab, I suggest that you that you that you select the run mode to be quick. select the run mode to be quick. select the run mode to be quick. So what you can do is if you are on a So what you can do is if you are on a So what you can do is if you are on a free trial, you can do the run mode free trial, you can do the run mode free trial, you can do the run mode equal to quick equal to quick equal to quick and then run this notebook. You'll see and then run this notebook. You'll see and then run this notebook. You'll see that in run mode equal to quick the that in run mode equal to quick the that in run mode equal to quick the number of training steps is only 2,000. number of training steps is only 2,000.

  57. L17 @ 103s

    number of training steps is only 2,000. Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 Whereas in run mode equal to budget 100 which is if you have compute budget or which is if you have compute budget or which is if you have compute budget or if you have Google collab pro you can if you have Google collab pro you can if you have Google collab pro you can even run for 60,000 steps. Some even run for 60,000 steps. Some even run for 60,000 steps. Some variables which I've defined here are variables which I've defined here are variables which I've defined here are the training examples which is the the training examples which is the the training examples which is the number of rows which I want from my tiny number of rows which I want from my tiny number of rows which I want from my tiny stories data set. So let me show that stories data set. So let me show that stories data set. So let me show that data set once more. data set once more. data set once more. Yeah. So essentially So essentially train examples is number of training train examples is number of training train examples is number of training examples which I have here. There are examples which I have here. There are examples which I have here. There are 2.2 million. So we can choose how many 2.2 million. So we can choose how many 2.2 million. So we can choose how many we want. Here I've chosen 1 million in we want. Here I've chosen 1 million in we want. Here I've chosen 1 million in this budget 100 and here I've only this budget 100 and here I've only this budget 100 and here I've only chosen 50,000 in the quick version. Then chosen 50,000 in the quick version. Then chosen 50,000 in the quick version. Then we have to select the sequence length. we have to select the sequence length. we have to select the sequence length. Sequence length is essentially the Sequence length is essentially the Sequence length is essentially the embedding dimension. Right? So if you embedding dimension. Right? So if you embedding dimension. Right? So if you take a look at the take a look at the

  58. L17 @ 154s

    take a look at the whiteboard you'll see that whiteboard you'll see that whiteboard you'll see that this dimension which I keep talking this dimension which I keep talking this dimension which I keep talking about right this this dimension the about right this this dimension the about right this this dimension the number of columns of the token embedding number of columns of the token embedding number of columns of the token embedding matrix and the number of columns of the matrix and the number of columns of the matrix and the number of columns of the position embedding matrix and in the position embedding matrix and in the position embedding matrix and in the case of diffusion model the time case of diffusion model the time case of diffusion model the time embedding matrix also that's my embedding matrix also that's my embedding matrix also that's my embedding dimension and I've chosen that embedding dimension and I've chosen that embedding dimension and I've chosen that to be equal to 256. to be equal to 256. to be equal to 256. Now actually that's the D model. So the Now actually that's the D model. So the Now actually that's the D model. So the embedding dimension is the D model which embedding dimension is the D model which embedding dimension is the D model which has been chosen to be 512. So I'm going has been chosen to be 512. So I'm going has been chosen to be 512. So I'm going to look at budget 100 for now. Embedding to look at budget 100 for now. Embedding to look at budget 100 for now. Embedding dimension is 512. Sequence length that's dimension is 512. Sequence length that's dimension is 512. Sequence length that's the context length basically the number the context length basically the number the context length basically the number of tokens model can look at at one time. of tokens model can look at at one time. of tokens model can look at at one time. That's the number of rows of position That's the number of rows of position That's the number of rows of position embedding matrix. So sequence length embedding matrix. So sequence length embedding matrix. So sequence length essentially here in these examples we essentially here in these examples we

  59. L17 @ 200s

    essentially here in these examples we have a sequence length of four tokens, have a sequence length of four tokens, have a sequence length of four tokens, right? Whereas in actual code we have right? Whereas in actual code we have right? Whereas in actual code we have taken a sequence length to be 256. taken a sequence length to be 256. taken a sequence length to be 256. vocabulary size is 26,000 vocabulary size is 26,000 vocabulary size is 26,000 whereas uh here I just here when I whereas uh here I just here when I whereas uh here I just here when I mentioned the size of the token mentioned the size of the token mentioned the size of the token embedding matrix I had shown a embedding matrix I had shown a embedding matrix I had shown a vocabulary size of 100,000 you can play vocabulary size of 100,000 you can play vocabulary size of 100,000 you can play around with these variables these are around with these variables these are around with these variables these are not set in stone not set in stone not set in stone number of layers these are actually the number of layers these are actually the number of layers these are actually the number of uh number of uh number of uh transformer blocks so remember this transformer blocks so remember this transformer blocks so remember this mirror notebook which I showed all of mirror notebook which I showed all of mirror notebook which I showed all of you you you uh which was essentially called journey uh which was essentially called journey uh which was essentially called journey of a token. So if you take a look at of a token. So if you take a look at of a token. So if you take a look at this miro notebook this miro notebook this miro notebook you'll see that there are when I show you'll see that there are when I show you'll see that there are when I show the transformer block there are actually the transformer block there are actually the transformer block there are actually multiple transformer blocks which are multiple transformer blocks which are multiple transformer blocks which are chained to each other. So these are the chained to each other. So these are the

  60. L17 @ 253s

    chained to each other. So these are the uh this variable is the number of uh this variable is the number of uh this variable is the number of layers. The number of heads is layers. The number of heads is layers. The number of heads is essentially for multi head attention. In essentially for multi head attention. In essentially for multi head attention. In this journey we have so far seen only this journey we have so far seen only this journey we have so far seen only single head attention. In multi single head attention. In multi single head attention. In multi attention you have different heads. Each attention you have different heads. Each attention you have different heads. Each head has its own attention score matrix head has its own attention score matrix head has its own attention score matrix or attention weight matrix. So we or attention weight matrix. So we or attention weight matrix. So we capture a different perspective. This capture a different perspective. This capture a different perspective. This DFF is for the DFF is for the DFF is for the field forward field forward field forward neural network. So if you take a look at neural network. So if you take a look at neural network. So if you take a look at this this this uh if you take a look at our schematic uh if you take a look at our schematic uh if you take a look at our schematic over here, you'll see that the feed over here, you'll see that the feed over here, you'll see that the feed forward in the feed forward neural forward in the feed forward neural forward in the feed forward neural network. Yeah, we mentioned that it's an network. Yeah, we mentioned that it's an network. Yeah, we mentioned that it's an expansion expansion expansion um and a contraction, right? So when we um and a contraction, right? So when we um and a contraction, right? So when we expand the number of dimensions in the expand the number of dimensions in the expand the number of dimensions in the hidden layer are four times the hidden layer are four times the hidden layer are four times the embedding dimension. So this four here embedding dimension. So this four here embedding dimension. So this four here that's this uh four times the D model.

  61. L17 @ 311s

    that's this uh four times the D model. that's this uh four times the D model. So DFF is the feed forward neural So DFF is the feed forward neural So DFF is the feed forward neural network hidden layer dimension. diffusion steps is 128. So this is diffusion steps is 128. So this is basically here in this animation I have basically here in this animation I have basically here in this animation I have shown the diffusion steps to be equal to shown the diffusion steps to be equal to shown the diffusion steps to be equal to six. Right? Time goes from 1 to 6. So six. Right? Time goes from 1 to 6. So six. Right? Time goes from 1 to 6. So 128 is just we have taken 128 time 128 is just we have taken 128 time 128 is just we have taken 128 time steps. steps. steps. That's it. And the rest of the things That's it. And the rest of the things That's it. And the rest of the things are for the optimizer. We have gradient are for the optimizer. We have gradient are for the optimizer. We have gradient accumulation so that a huge batch does accumulation so that a huge batch does accumulation so that a huge batch does not live on a GPU. We reduce the batch not live on a GPU. We reduce the batch not live on a GPU. We reduce the batch size by half so that half of the batch size by half so that half of the batch size by half so that half of the batch lives on the GPU at one time. We have a lives on the GPU at one time. We have a lives on the GPU at one time. We have a learning rate of 2 into 10 -4 batch size learning rate of 2 into 10 -4 batch size learning rate of 2 into 10 -4 batch size of 32 weight TK of.1 and warm-up steps of 32 weight TK of.1 and warm-up steps of 32 weight TK of.1 and warm-up steps of,000. These will be needed in the ADM of,000. These will be needed in the ADM of,000. These will be needed in the ADM optimizer. Okay. So you run this initial optimizer. Okay. So you run this initial

  62. L17 @ 361s

    optimizer. Okay. So you run this initial configuration and then you move forward. configuration and then you move forward. configuration and then you move forward. You install a bunch of dependencies You install a bunch of dependencies You install a bunch of dependencies and make sure that you have GPU support and make sure that you have GPU support and make sure that you have GPU support here. So once we run this fully uh here. So once we run this fully uh here. So once we run this fully uh sometimes H100 GPU is not available in sometimes H100 GPU is not available in sometimes H100 GPU is not available in the Google Collab Pro version. So then the Google Collab Pro version. So then the Google Collab Pro version. So then you will be given an A100 version or an you will be given an A100 version or an you will be given an A100 version or an A100 GPU. That's also fine but it takes A100 GPU. That's also fine but it takes A100 GPU. That's also fine but it takes a very long time to run. I'll give you a very long time to run. I'll give you a very long time to run. I'll give you other options after showing this Google other options after showing this Google other options after showing this Google collab notebook which are much much collab notebook which are much much collab notebook which are much much faster such as going to run pod for faster such as going to run pod for faster such as going to run pod for example. example. example. Okay. So it's still connecting. So I Okay. So it's still connecting. So I Okay. So it's still connecting. So I think H100 is not available. So let's think H100 is not available. So let's think H100 is not available. So let's see see see once this is done the next step for us once this is done the next step for us once this is done the next step for us is to load the tiny stories data. So I is to load the tiny stories data. So I is to load the tiny stories data. So I go to this uh hugging face repository go to this uh hugging face repository

  63. L17 @ 409s

    go to this uh hugging face repository and I load my data. I load my training and I load my data. I load my training and I load my data. I load my training data set. I load my validation data set data set. I load my validation data set data set. I load my validation data set and uh my data is loaded. The next thing and uh my data is loaded. The next thing and uh my data is loaded. The next thing which we have to do is break down my which we have to do is break down my which we have to do is break down my sentences into tokens. Right? Now we sentences into tokens. Right? Now we sentences into tokens. Right? Now we could have used a bite pair encoder could have used a bite pair encoder could have used a bite pair encoder tokenizer which is there are number of tokenizer which is there are number of tokenizer which is there are number of tokenizers which are available from tokenizers which are available from tokenizers which are available from scratch. So if you go to the tick scratch. So if you go to the tick scratch. So if you go to the tick tokenizer, tokenizer, tokenizer, I think if you go to tick token library I think if you go to tick token library I think if you go to tick token library here, it's the tokenizer which openai here, it's the tokenizer which openai here, it's the tokenizer which openai uses. So you can directly use a uses. So you can directly use a uses. So you can directly use a tokenizer from here. But just so that we tokenizer from here. But just so that we tokenizer from here. But just so that we all learn something, I have shown how to all learn something, I have shown how to all learn something, I have shown how to train a tokenizer from scratch. So I got train a tokenizer from scratch. So I got train a tokenizer from scratch. So I got a message that GPU was not available. So a message that GPU was not available. So a message that GPU was not available. So we I'm connected to an L4 GPU at the we I'm connected to an L4 GPU at the

  64. L17 @ 456s

    we I'm connected to an L4 GPU at the moment. See that's fine. So let's see. Um Um yeah so here you'll see that I'm yeah so here you'll see that I'm yeah so here you'll see that I'm connected to an L4 Nvidia GPU connected to an L4 Nvidia GPU connected to an L4 Nvidia GPU and uh my data set is also loaded right and uh my data set is also loaded right and uh my data set is also loaded right now to train a tokenizer from scratch. now to train a tokenizer from scratch. now to train a tokenizer from scratch. Basically we are just going to use this Basically we are just going to use this Basically we are just going to use this BP trainer module. I'm not going to go BP trainer module. I'm not going to go BP trainer module. I'm not going to go into too many details of this but bite into too many details of this but bite into too many details of this but bite pair encoding essentially merges pair encoding essentially merges pair encoding essentially merges commonly occurring tokens or characters commonly occurring tokens or characters commonly occurring tokens or characters and forms tokens which needed to be and forms tokens which needed to be and forms tokens which needed to be which need to be added in the which need to be added in the which need to be added in the vocabulary. The variable which we need vocabulary. The variable which we need vocabulary. The variable which we need to create this tokenizer is my to create this tokenizer is my to create this tokenizer is my vocabulary size. So we keep on adding vocabulary size. So we keep on adding vocabulary size. So we keep on adding tokens to the vocabulary until we reach tokens to the vocabulary until we reach tokens to the vocabulary until we reach a tokenizer size. So here you can see a tokenizer size. So here you can see a tokenizer size. So here you can see that that that uh I have also created special tokens uh I have also created special tokens uh I have also created special tokens here such as a padding token, unknown here such as a padding token, unknown here such as a padding token, unknown token, beginning of sequence, end of

  65. L17 @ 512s

    token, beginning of sequence, end of token, beginning of sequence, end of sequence and a mask token. This is very sequence and a mask token. This is very sequence and a mask token. This is very important. This mask token is the one important. This mask token is the one important. This mask token is the one which uh we'll be using during the which uh we'll be using during the which uh we'll be using during the noising process as I've already noising process as I've already noising process as I've already explained to you. and the tokenizer explained to you. and the tokenizer explained to you. and the tokenizer training training training all the training text is wrapped in this all the training text is wrapped in this all the training text is wrapped in this write a short story then the assistant write a short story then the assistant write a short story then the assistant and then end and then end and then end uh so basically I'll take this story uh so basically I'll take this story uh so basically I'll take this story from my data set and then the actual from my data set and then the actual from my data set and then the actual data set will be write a story then the data set will be write a story then the data set will be write a story then the actual story and then end and this this actual story and then end and this this actual story and then end and this this whole thing will be tokenized whole thing will be tokenized whole thing will be tokenized this is just a format which I'm using this is just a format which I'm using this is just a format which I'm using for the tokenization right now I don't for the tokenization right now I don't for the tokenization right now I don't think you think you think you worry too much about tokenization at the worry too much about tokenization at the worry too much about tokenization at the moment. In fact, you can even replace moment. In fact, you can even replace moment. In fact, you can even replace this whole tokenizer section with your this whole tokenizer section with your this whole tokenizer section with your with a tokenizer directly borrowed from

  66. L17 @ 560s

    with a tokenizer directly borrowed from with a tokenizer directly borrowed from tick token. I I I think the code should tick token. I I I think the code should tick token. I I I think the code should still work. still work. still work. Uh okay, what's important is that if you Uh okay, what's important is that if you Uh okay, what's important is that if you have a sentence such as hello world, have a sentence such as hello world, have a sentence such as hello world, right, based on the tokenizer which we right, based on the tokenizer which we right, based on the tokenizer which we have created, we have 2 1 3 0 0 1 1 5 6 have created, we have 2 1 3 0 0 1 1 5 6 have created, we have 2 1 3 0 0 1 1 5 6 9 and 3. These are the tokens. If I have 9 and 3. These are the tokens. If I have 9 and 3. These are the tokens. If I have something like uh these are the tokens associated with it. these are the tokens associated with it. So I think this two and three is So I think this two and three is So I think this two and three is basically the beginning of sequence and basically the beginning of sequence and basically the beginning of sequence and the end of sequence I think. Um and the the end of sequence I think. Um and the the end of sequence I think. Um and the token ID corresponding to the mask is token ID corresponding to the mask is token ID corresponding to the mask is four. Yeah, two is the beginning of four. Yeah, two is the beginning of four. Yeah, two is the beginning of sequence ID. Three is the end of sequence ID. Three is the end of sequence ID. Three is the end of sequence ID and the token ID sequence ID and the token ID sequence ID and the token ID corresponding to mask is four. This corresponding to mask is four. This corresponding to mask is four. This token ID corresponding to mask is token ID corresponding to mask is token ID corresponding to mask is very important very important very important because in the noising process we are

  67. L17 @ 621s

    because in the noising process we are because in the noising process we are going to replace the tokens with masks going to replace the tokens with masks going to replace the tokens with masks and then we are going to use this token and then we are going to use this token and then we are going to use this token ID to represent those masks. Why have we ID to represent those masks. Why have we ID to represent those masks. Why have we added this write a short story here? added this write a short story here? added this write a short story here? Because when we are going to give the Because when we are going to give the Because when we are going to give the inference prompt, we are going to tell inference prompt, we are going to tell inference prompt, we are going to tell to write a short story. So if it's to write a short story. So if it's to write a short story. So if it's included in the tokenizer training, it included in the tokenizer training, it included in the tokenizer training, it generally helps although it's strictly generally helps although it's strictly generally helps although it's strictly not needed at all. [snorts] not needed at all. [snorts] not needed at all. [snorts] Okay. Now we come to the process of Okay. Now we come to the process of Okay. Now we come to the process of actually assembling the uh diffusion actually assembling the uh diffusion actually assembling the uh diffusion language model architecture. Right? So language model architecture. Right? So language model architecture. Right? So the diffusion language model the diffusion language model the diffusion language model architecture again to show everyone it architecture again to show everyone it architecture again to show everyone it looks something like this. Yeah, this is how the diffusion language Yeah, this is how the diffusion language model architecture actually looks like. model architecture actually looks like. model architecture actually looks like. And let me rub all of this so that And let me rub all of this so that And let me rub all of this so that becomes clearer to all of you. Yeah. becomes clearer to all of you. Yeah.

  68. L17 @ 673s

    becomes clearer to all of you. Yeah. And if you see that you first define a And if you see that you first define a And if you see that you first define a token embedding, position embedding and token embedding, position embedding and token embedding, position embedding and time embedding. time embedding. time embedding. And the place to get started to actually And the place to get started to actually And the place to get started to actually understand this code is that [snorts] understand this code is that [snorts] understand this code is that [snorts] you first have the token embeddings. you you first have the token embeddings. you you first have the token embeddings. you add it to the position embedding and add it to the position embedding and add it to the position embedding and then you add the time embedding after then you add the time embedding after then you add the time embedding after it. Right? And then this this input it. Right? And then this this input it. Right? And then this this input sequence you pass through the encoder sequence you pass through the encoder sequence you pass through the encoder block. Now what's the encoder block? block. Now what's the encoder block? block. Now what's the encoder block? That's essentially just a transformer That's essentially just a transformer That's essentially just a transformer encoder. So transformer encoder if you encoder. So transformer encoder if you encoder. So transformer encoder if you see yeah, we are just borrowing this yeah, we are just borrowing this transformer encoder from Pytor. So we transformer encoder from Pytor. So we transformer encoder from Pytor. So we don't have to write this or code this don't have to write this or code this don't have to write this or code this from scratch. That just simplifies the from scratch. That just simplifies the from scratch. That just simplifies the code a lot. It just creates this entire code a lot. It just creates this entire code a lot. It just creates this entire transformer block like this. Okay. transformer block like this. Okay. transformer block like this. Okay. So here we are just going to do So here we are just going to do So here we are just going to do a transformer a transformer

  69. L17 @ 740s

    a transformer um we are going to do a transformer um we are going to do a transformer um we are going to do a transformer encoder layer over here and uh let's see and attention mask equal to none. This and attention mask equal to none. This is extremely important attention mask is extremely important attention mask is extremely important attention mask equal to none essentially means that we equal to none essentially means that we equal to none essentially means that we are not having that causal attention are not having that causal attention are not having that causal attention layer which we saw in the whiteboard. So layer which we saw in the whiteboard. So layer which we saw in the whiteboard. So on the whiteboard what we saw is that on the whiteboard what we saw is that on the whiteboard what we saw is that for auto reggressive models we put this for auto reggressive models we put this for auto reggressive models we put this elements above the diagonal to be zero elements above the diagonal to be zero elements above the diagonal to be zero right in the attention scores but we are right in the attention scores but we are right in the attention scores but we are not going to do this in the diffusion not going to do this in the diffusion not going to do this in the diffusion model. So we don't have any attention model. So we don't have any attention model. So we don't have any attention mask over here. So it's very important mask over here. So it's very important mask over here. So it's very important to mention that attention mask equal to to mention that attention mask equal to to mention that attention mask equal to none. [snorts] none. [snorts] none. [snorts] Okay. So we say that attention mask Okay. So we say that attention mask Okay. So we say that attention mask equal to none. Uh we add the again we equal to none. Uh we add the again we equal to none. Uh we add the again we add the token embedding to the position add the token embedding to the position add the token embedding to the position embedding. We add the time embedding to embedding. We add the time embedding to

  70. L17 @ 791s

    embedding. We add the time embedding to it. The resultant input embedding we it. The resultant input embedding we it. The resultant input embedding we pass through the transformer block. Then pass through the transformer block. Then pass through the transformer block. Then we come to the output. We do a layer we come to the output. We do a layer we come to the output. We do a layer normalization and we do the logits and normalization and we do the logits and normalization and we do the logits and we return the logits. That's all. This we return the logits. That's all. This we return the logits. That's all. This is the entire diffusion model is the entire diffusion model is the entire diffusion model architecture. architecture. architecture. Okay. Okay. Um and once you have understood this on Um and once you have understood this on Um and once you have understood this on the whiteboard, if you understood this the whiteboard, if you understood this the whiteboard, if you understood this architecture on the whiteboard, [snorts] architecture on the whiteboard, [snorts] architecture on the whiteboard, [snorts] what we are doing in code is exactly a what we are doing in code is exactly a what we are doing in code is exactly a representation of this architecture. representation of this architecture. representation of this architecture. There is nothing different which is There is nothing different which is There is nothing different which is being done over here. Note one more being done over here. Note one more being done over here. Note one more thing that uh we have in the forward thing that uh we have in the forward thing that uh we have in the forward pass we also have the time steps. Why do pass we also have the time steps. Why do pass we also have the time steps. Why do we have the time steps? Because for the we have the time steps? Because for the we have the time steps? Because for the time embedding we need to know which time embedding we need to know which time embedding we need to know which time time time um we are looking at because if you look um we are looking at because if you look um we are looking at because if you look at the time embedding matrix

  71. L17 @ 839s

    at the time embedding matrix at the time embedding matrix if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need if we have t going from 1 to 128 we need to know at what time step we are in to to know at what time step we are in to to know at what time step we are in to choose the vector corresponding to that choose the vector corresponding to that choose the vector corresponding to that time step during the forward pass. time step during the forward pass. time step during the forward pass. Right. So I run this and I get the Right. So I run this and I get the Right. So I run this and I get the uh diffusion language model uh diffusion language model uh diffusion language model architecture. architecture. Then I'm just uh creating a training Then I'm just uh creating a training Then I'm just uh creating a training loader and I'm creating a validation loader and I'm creating a validation loader and I'm creating a validation loader so that we can create training loader so that we can create training loader so that we can create training and validation data quite easily. The and validation data quite easily. The and validation data quite easily. The next step is very important. Take a look next step is very important. Take a look next step is very important. Take a look at this corrupt with mask. Essentially at this corrupt with mask. Essentially at this corrupt with mask. Essentially what we are doing is before we pass the what we are doing is before we pass the what we are doing is before we pass the input ids to the uh transformer block or input ids to the uh transformer block or input ids to the uh transformer block or through the architecture itself we have through the architecture itself we have through the architecture itself we have to corrupt the input ids right. What to corrupt the input ids right. What to corrupt the input ids right. What does it mean? It means that we we have does it mean? It means that we we have does it mean? It means that we we have to based on the time step we have to add

  72. L17 @ 887s

    to based on the time step we have to add to based on the time step we have to add the mask tokens. This is what is done the mask tokens. This is what is done the mask tokens. This is what is done here. So we have t which is basically at here. So we have t which is basically at here. So we have t which is basically at what time step we are in and here what's what time step we are in and here what's what time step we are in and here what's simply done is that we have this mask simply done is that we have this mask simply done is that we have this mask positions where the mask token ID is positions where the mask token ID is positions where the mask token ID is added and the rest of the token ids are added and the rest of the token ids are added and the rest of the token ids are replaced with minus 100. The reason they replaced with minus 100. The reason they replaced with minus 100. The reason they are replaced with minus 100 is because are replaced with minus 100 is because are replaced with minus 100 is because they are not going to come in the uh they are not going to come in the uh they are not going to come in the uh loss function calculation. So if you loss function calculation. So if you loss function calculation. So if you look at look at look at cross entropy loss pytor torch cross entropy loss pytor torch cross entropy loss pytor torch you'll see that there is this ignore you'll see that there is this ignore you'll see that there is this ignore index of minus 100 which means that index of minus 100 which means that index of minus 100 which means that wherever we have minus 100 that's not wherever we have minus 100 that's not wherever we have minus 100 that's not involved in the cross entropy loss involved in the cross entropy loss involved in the cross entropy loss calculation. So remember why is this calculation. So remember why is this calculation. So remember why is this done because we have already seen this done because we have already seen this

  73. L17 @ 929s

    done because we have already seen this on the whiteboard right to calculate the on the whiteboard right to calculate the on the whiteboard right to calculate the loss I only need to look at the mask loss I only need to look at the mask loss I only need to look at the mask tokens. I don't need to look at the tokens. I don't need to look at the tokens. I don't need to look at the other tokens at all. So that's why these other tokens at all. So that's why these other tokens at all. So that's why these other tokens are replaced with minus other tokens are replaced with minus other tokens are replaced with minus 100. So they don't come in my loss 100. So they don't come in my loss 100. So they don't come in my loss function calculation. So these minute function calculation. So these minute function calculation. So these minute differences between the diffusion differences between the diffusion differences between the diffusion language models and auto reggressive language models and auto reggressive language models and auto reggressive models are carefully scattered across models are carefully scattered across models are carefully scattered across the code. And unless you understand the the code. And unless you understand the the code. And unless you understand the theory, it will be very difficult for theory, it will be very difficult for theory, it will be very difficult for you to understand okay where is the you to understand okay where is the you to understand okay where is the where is it implemented that only the where is it implemented that only the where is it implemented that only the masked positions will be used in the masked positions will be used in the masked positions will be used in the loss function calculations because all loss function calculations because all loss function calculations because all the other input ids are replaced with the other input ids are replaced with the other input ids are replaced with minus 100. Where is it included that we minus 100. Where is it included that we minus 100. Where is it included that we won't have a causal attention here? won't have a causal attention here? won't have a causal attention here? Well, it's included over here where we

  74. L17 @ 973s

    Well, it's included over here where we Well, it's included over here where we pass an attention mask equal to none. pass an attention mask equal to none. pass an attention mask equal to none. These are the small details which all of These are the small details which all of These are the small details which all of you need to be paying attention to. you need to be paying attention to. you need to be paying attention to. Okay, Okay, Okay, so this is how we define the input so this is how we define the input so this is how we define the input sequence and we corrupt it with noise sequence and we corrupt it with noise sequence and we corrupt it with noise which means we ask we just add masks at which means we ask we just add masks at which means we ask we just add masks at the positions where the positions where the positions where um we have chosen to add the mask based um we have chosen to add the mask based um we have chosen to add the mask based on the time step. Essentially the deeper on the time step. Essentially the deeper on the time step. Essentially the deeper we are into the noising schedule the we are into the noising schedule the we are into the noising schedule the more probability of adding the mask more probability of adding the mask more probability of adding the mask right and then we define the loss. right and then we define the loss. right and then we define the loss. To define the loss we just have to get To define the loss we just have to get To define the loss we just have to get the noisy ids and then we have to the noisy ids and then we have to the noisy ids and then we have to calculate the cross entropy loss between calculate the cross entropy loss between calculate the cross entropy loss between the noisy ids and the actual values at the noisy ids and the actual values at the noisy ids and the actual values at those mask positions. That's it. those mask positions. That's it. those mask positions. That's it. So this is the cross entropy loss. So So this is the cross entropy loss. So

  75. L17 @ 1016s

    So this is the cross entropy loss. So you can run this right now. Once we have you can run this right now. Once we have you can run this right now. Once we have this defined, the last step is just to this defined, the last step is just to this defined, the last step is just to define the training function and the define the training function and the define the training function and the loss function. So this is my training loss function. So this is my training loss function. So this is my training function at the moment where what I'm function at the moment where what I'm function at the moment where what I'm doing first is that this evaluation loss doing first is that this evaluation loss doing first is that this evaluation loss is after every 20 batches I'm going to is after every 20 batches I'm going to is after every 20 batches I'm going to print the loss. But here is where the print the loss. But here is where the print the loss. But here is where the real uh training happens. So what I'm real uh training happens. So what I'm real uh training happens. So what I'm going to do is that I'm going to take going to do is that I'm going to take going to do is that I'm going to take every batch from my training iteration. every batch from my training iteration. every batch from my training iteration. I'm going to uh I'm going to uh I'm going to uh find the diffusion loss on this batch find the diffusion loss on this batch find the diffusion loss on this batch and then what I'm going to do is that uh and then what I'm going to do is that uh and then what I'm going to do is that uh one more thing along with the batch as I one more thing along with the batch as I one more thing along with the batch as I mentioned we also need to define the mentioned we also need to define the mentioned we also need to define the time step right so when you go to the time step right so when you go to the

  76. L17 @ 1055s

    time step right so when you go to the diffusion loss function you'll see that diffusion loss function you'll see that diffusion loss function you'll see that uh where is the diffusion loss yeah when uh where is the diffusion loss yeah when uh where is the diffusion loss yeah when you see the diffusion loss function we you see the diffusion loss function we you see the diffusion loss function we also get this time step which is chosen also get this time step which is chosen also get this time step which is chosen uh randomly we need this time step uh randomly we need this time step uh randomly we need this time step because the corruption level depends on because the corruption level depends on because the corruption level depends on that time step Right. So if you go here, that time step Right. So if you go here, that time step Right. So if you go here, what's actually done is that we first what's actually done is that we first what's actually done is that we first choose a batch of the data. choose a batch of the data. choose a batch of the data. We pass we choose a batch. Then we pass We pass we choose a batch. Then we pass We pass we choose a batch. Then we pass the model and the batch to the diffusion the model and the batch to the diffusion the model and the batch to the diffusion loss function. In the diffusion loss loss function. In the diffusion loss loss function. In the diffusion loss function, we have by default chosen a function, we have by default chosen a function, we have by default chosen a time step. And then here we pass the time step. And then here we pass the time step. And then here we pass the input batch and the time step to our input batch and the time step to our input batch and the time step to our model. This model has been defined over model. This model has been defined over model. This model has been defined over here. here. here. This model has been defined over here. This model has been defined over here. This model has been defined over here. Now this thing which I'm telling right Now this thing which I'm telling right

  77. L17 @ 1099s

    Now this thing which I'm telling right now the corrupted input sequence is now the corrupted input sequence is now the corrupted input sequence is passed through the model is exactly passed through the model is exactly passed through the model is exactly shown over here. The corrupted sequence shown over here. The corrupted sequence shown over here. The corrupted sequence is passed through the model. It goes is passed through the model. It goes is passed through the model. It goes through this entire model architecture through this entire model architecture through this entire model architecture and we get the loss. and we get the loss. and we get the loss. All of this is implemented over here. We All of this is implemented over here. We All of this is implemented over here. We get the loss and then we do this is the get the loss and then we do this is the get the loss and then we do this is the optimizing step optimizer step. Once we optimizing step optimizer step. Once we optimizing step optimizer step. Once we get the loss function then we go into get the loss function then we go into get the loss function then we go into this loop of uh this loop of uh this loop of uh uh yeah we go into this loop of finding uh yeah we go into this loop of finding uh yeah we go into this loop of finding the partial derivative of the loss then the partial derivative of the loss then the partial derivative of the loss then updating the parameters through the updating the parameters through the updating the parameters through the optimizer that's done over here we do optimizer that's done over here we do optimizer that's done over here we do optimizer step and we uh update the optimizer step and we uh update the optimizer step and we uh update the learning rate in the adm optimizer etc. learning rate in the adm optimizer etc. learning rate in the adm optimizer etc. So you run this right now and then you So you run this right now and then you So you run this right now and then you will see that the training process will see that the training process will see that the training process starts automatically. Yeah, see the training process has

  78. L17 @ 1151s

    Yeah, see the training process has started. Now I'm on the L I think let's started. Now I'm on the L I think let's started. Now I'm on the L I think let's see which GPU I'm on. I'm on the L4 GPU see which GPU I'm on. I'm on the L4 GPU see which GPU I'm on. I'm on the L4 GPU which is actually not very fast. So the which is actually not very fast. So the which is actually not very fast. So the iterations are proceeding slowly. So iterations are proceeding slowly. So iterations are proceeding slowly. So here it's showing that it will take here it's showing that it will take here it's showing that it will take around 3 hours for me. Still it's quite around 3 hours for me. Still it's quite around 3 hours for me. Still it's quite good right for 60,000 iterations. If you good right for 60,000 iterations. If you good right for 60,000 iterations. If you are on T4 GPU it may take 8 to 9 hours. are on T4 GPU it may take 8 to 9 hours. are on T4 GPU it may take 8 to 9 hours. One thing which I want to point out to One thing which I want to point out to One thing which I want to point out to all of you is that although the all of you is that although the all of you is that although the inference of a diffusion model is inference of a diffusion model is inference of a diffusion model is faster, training actually is a bit faster, training actually is a bit faster, training actually is a bit slower. So slower. So slower. So if you scroll to the top, I've actually if you scroll to the top, I've actually if you scroll to the top, I've actually compared the training speeds of a compared the training speeds of a compared the training speeds of a diffusion language model and the auto diffusion language model and the auto diffusion language model and the auto reggressive model. So here if you see um reggressive model. So here if you see um reggressive model. So here if you see um in u the convergence speed so if you see in u the convergence speed so if you see

  79. L17 @ 1197s

    in u the convergence speed so if you see the auto reggressive model in 15,000 the auto reggressive model in 15,000 the auto reggressive model in 15,000 epochs I go from a loss of 8 to a loss epochs I go from a loss of 8 to a loss epochs I go from a loss of 8 to a loss of 2.6. So almost the whole model has of 2.6. So almost the whole model has of 2.6. So almost the whole model has converged in 15,000 epochs itself. converged in 15,000 epochs itself. converged in 15,000 epochs itself. Whereas in diffusion LLM in 15,000 steps Whereas in diffusion LLM in 15,000 steps Whereas in diffusion LLM in 15,000 steps I go from a loss of 10 to just a loss of I go from a loss of 10 to just a loss of I go from a loss of 10 to just a loss of 5.6. 5.6. 5.6. So the model actually converges slowly. So the model actually converges slowly. So the model actually converges slowly. Diffusion language models take time to Diffusion language models take time to converge and that's a trait trait of converge and that's a trait trait of converge and that's a trait trait of diffusion models always they are a bit diffusion models always they are a bit diffusion models always they are a bit slower slower slower because of the mask predictions being because of the mask predictions being because of the mask predictions being involved at and we are looking at it involved at and we are looking at it involved at and we are looking at it from both sides so the attention matrix from both sides so the attention matrix from both sides so the attention matrix is a bit larger etc. is a bit larger etc. is a bit larger etc. Um so diffusion language models in my Um so diffusion language models in my Um so diffusion language models in my experience train a bit slowly compared experience train a bit slowly compared experience train a bit slowly compared to auto reggressive models but as you to auto reggressive models but as you to auto reggressive models but as you can see the training has started over can see the training has started over

  80. L17 @ 1244s

    can see the training has started over here and we can print the loss we can here and we can print the loss we can here and we can print the loss we can print the learning rate etc. While the print the learning rate etc. While the print the learning rate etc. While the training is going on, I'll tell you the training is going on, I'll tell you the training is going on, I'll tell you the next steps. What we'll do after this next steps. What we'll do after this next steps. What we'll do after this training is that we then just have to training is that we then just have to training is that we then just have to proceed with generation. Right? So in proceed with generation. Right? So in proceed with generation. Right? So in generation, as I've mentioned already, generation, as I've mentioned already, generation, as I've mentioned already, we just carry out sequential unmasking. we just carry out sequential unmasking. we just carry out sequential unmasking. So here is a animation of generation or So here is a animation of generation or So here is a animation of generation or dnoising which we have seen. We start dnoising which we have seen. We start dnoising which we have seen. We start with masks and then sequentially unmask with masks and then sequentially unmask with masks and then sequentially unmask and uh we unmask tokens with the highest and uh we unmask tokens with the highest and uh we unmask tokens with the highest confidence. That's the general rule, confidence. That's the general rule, confidence. That's the general rule, right? So this is the diffusion generate right? So this is the diffusion generate right? So this is the diffusion generate code where we start with all the masks code where we start with all the masks code where we start with all the masks and then and then and then the unmasking is actually done for the the unmasking is actually done for the the unmasking is actually done for the maximum confidence. That's the entire maximum confidence. That's the entire maximum confidence. That's the entire logic which has been implemented over logic which has been implemented over logic which has been implemented over here. I am not going to go through this

  81. L17 @ 1294s

    here. I am not going to go through this here. I am not going to go through this code in detail because it just code in detail because it just code in detail because it just implements the same thing, right? We we implements the same thing, right? We we implements the same thing, right? We we get the confidence scores and then we get the confidence scores and then we get the confidence scores and then we unmask the token ids with the maximum unmask the token ids with the maximum unmask the token ids with the maximum confidence. The reason I spent so much confidence. The reason I spent so much confidence. The reason I spent so much time on the whiteboard is because going time on the whiteboard is because going time on the whiteboard is because going through code is I found that if I go through code is I found that if I go through code is I found that if I go through this code generally a bit through this code generally a bit through this code generally a bit difficult for folks to understand right difficult for folks to understand right difficult for folks to understand right but if I take an actual example and show but if I take an actual example and show but if I take an actual example and show you how unmasking is done step by step you how unmasking is done step by step you how unmasking is done step by step you can relate this very easily to the you can relate this very easily to the you can relate this very easily to the code code code right so here's here actually the same right so here's here actually the same right so here's here actually the same unmasking uh procedure is happening unmasking uh procedure is happening unmasking uh procedure is happening right now let me do this okay let me right now let me do this okay let me right now let me do this okay let me stop this for a moment and Let's run stop this for a moment and Let's run stop this for a moment and Let's run with whatever I have. with whatever I have. with whatever I have. Uh so this is run right now and uh here

  82. L17 @ 1339s

    Uh so this is run right now and uh here Uh so this is run right now and uh here I'm just seeing a sample inference. Once I'm just seeing a sample inference. Once I'm just seeing a sample inference. Once upon a time is the input. I'll see that upon a time is the input. I'll see that upon a time is the input. I'll see that the inference is very bad right now the inference is very bad right now the inference is very bad right now because I I just stopped at uh because I I just stopped at uh because I I just stopped at uh how many steps just thousand steps but I how many steps just thousand steps but I how many steps just thousand steps but I just want to show you the workflow. just want to show you the workflow. just want to show you the workflow. Right? So technically we have finished Right? So technically we have finished Right? So technically we have finished coding the diffusion language model from coding the diffusion language model from coding the diffusion language model from scratch here. But I have just added a scratch here. But I have just added a scratch here. But I have just added a simple code to render a terminal style simple code to render a terminal style simple code to render a terminal style diff terminal style GIF. Right. Uh I diff terminal style GIF. Right. Uh I diff terminal style GIF. Right. Uh I think there is an error over here think there is an error over here think there is an error over here probably because we have not run till uh probably because we have not run till uh probably because we have not run till uh till completion and that's I think till completion and that's I think till completion and that's I think that's fine that's fine that's fine but no such file or directory as no such file or directory as inference.jif. here also there is an error which says here also there is an error which says cannot import. cannot import. cannot import. I think this is probably because the I think this is probably because the I think this is probably because the iterations did not finish. But if you iterations did not finish. But if you

  83. L17 @ 1407s

    iterations did not finish. But if you actually run um actually run um actually run um 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iterations 60,000 or 60,000 or 100,000 iterations over here, you will generally see that over here, you will generally see that over here, you will generally see that the code finishes till completion. the code finishes till completion. the code finishes till completion. Um and then you just have to export this Um and then you just have to export this Um and then you just have to export this GIF and you have to save this GIF. So GIF and you have to save this GIF. So GIF and you have to save this GIF. So I've just written I also have another I've just written I also have another I've just written I also have another font here if you want to export the GIF font here if you want to export the GIF font here if you want to export the GIF in a slightly cooler format. But if you in a slightly cooler format. But if you in a slightly cooler format. But if you export the normal GIF, it will look export the normal GIF, it will look export the normal GIF, it will look something like this. So this is trained something like this. So this is trained something like this. So this is trained for 100,000 iterations, right? And after for 100,000 iterations, right? And after for 100,000 iterations, right? And after 100,000 iterations, you will start to 100,000 iterations, you will start to 100,000 iterations, you will start to see that the output starts becoming see that the output starts becoming see that the output starts becoming coherent. So the output is that once coherent. So the output is that once coherent. So the output is that once upon a time, there was a little girl upon a time, there was a little girl upon a time, there was a little girl named Lily. She loved to play with her named Lily. She loved to play with her named Lily. She loved to play with her toys and put them in a room. One day, toys and put them in a room. One day, toys and put them in a room. One day, Lily's mom asked her to help clean up

  84. L17 @ 1449s

    Lily's mom asked her to help clean up Lily's mom asked her to help clean up her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom her toys. Lily was happy to help her mom put them in a box. Isn't this amazing? put them in a box. Isn't this amazing? put them in a box. Isn't this amazing? This entirely coherent text generated This entirely coherent text generated This entirely coherent text generated through a pure diffusion model which we through a pure diffusion model which we through a pure diffusion model which we have built from scratch for 100,000 have built from scratch for 100,000 have built from scratch for 100,000 iterations through the exact same code iterations through the exact same code iterations through the exact same code which I have shown to you right now. And which I have shown to you right now. And which I have shown to you right now. And once you run this code, once you once you run this code, once you once you run this code, once you replicate this code, you'll also see replicate this code, you'll also see replicate this code, you'll also see this for yourself. The only requirement this for yourself. The only requirement this for yourself. The only requirement which you'll need is that you'll need to which you'll need is that you'll need to which you'll need is that you'll need to run at least for 50,000 or 60,000 run at least for 50,000 or 60,000 run at least for 50,000 or 60,000 iterations and you'll need a A100 GPU, iterations and you'll need a A100 GPU, iterations and you'll need a A100 GPU, H100 or if you have time, you can even H100 or if you have time, you can even H100 or if you have time, you can even do it on a T4 GPU or an L4 GPU. What I do it on a T4 GPU or an L4 GPU. What I do it on a T4 GPU or an L4 GPU. What I want to show you is that uh want to show you is that uh want to show you is that uh we can also do this on runpod. So if you we can also do this on runpod. So if you

  85. L17 @ 1490s

    we can also do this on runpod. So if you have credentials or if you have some have credentials or if you have some have credentials or if you have some budget even $6 to $7 what I'll suggest budget even $6 to $7 what I'll suggest budget even $6 to $7 what I'll suggest is go to runpod click on sign in over is go to runpod click on sign in over is go to runpod click on sign in over here here here and uh if you sign in through Google uh what you can do is that so I have uh what you can do is that so I have around $100 here you can actually select around $100 here you can actually select around $100 here you can actually select a GPU which is very fast. So there's a GPU which is very fast. So there's a GPU which is very fast. So there's something running here right now. If I something running here right now. If I something running here right now. If I click on deploy, let's say I choose this click on deploy, let's say I choose this click on deploy, let's say I choose this B200 GPU which is $519 per hour, but it B200 GPU which is $519 per hour, but it B200 GPU which is $519 per hour, but it is very it has 180 GB of VRAM. H200 has is very it has 180 GB of VRAM. H200 has is very it has 180 GB of VRAM. H200 has 1441 GB of VRAM. Right now I was on L40, 1441 GB of VRAM. Right now I was on L40, 1441 GB of VRAM. Right now I was on L40, I think, which is 48 GB. Not too bad. I think, which is 48 GB. Not too bad. I think, which is 48 GB. Not too bad. But let's choose B200 But let's choose B200 But let's choose B200 and let me do let's say container disk and let me do let's say container disk and let me do let's say container disk of 200 GB and uh volume disk of 200 GB. of 200 GB and uh volume disk of 200 GB. of 200 GB and uh volume disk of 200 GB. I'll set overrides. Oops. I think it was

  86. L17 @ 1545s

    I'll set overrides. Oops. I think it was I'll set overrides. Oops. I think it was 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set 187. Let's do 200. And I'll set overrides over here. And then I'll click overrides over here. And then I'll click overrides over here. And then I'll click on deploy on demand. Um when you click on deploy on demand, Um when you click on deploy on demand, you'll see that uh you'll see that uh you'll see that uh it will start to spin up. Okay. So now it's still waiting for Okay. So now it's still waiting for connection. So probably I should just uh connection. So probably I should just uh connection. So probably I should just uh uh uh uh Yeah. So Jupyter Lab is ready. Ready. So Yeah. So Jupyter Lab is ready. Ready. So Yeah. So Jupyter Lab is ready. Ready. So just click on this Jupyter lab just click on this Jupyter lab just click on this Jupyter lab and then it will open this Jupyter lab and then it will open this Jupyter lab and then it will open this Jupyter lab notebook environment for you and uh once this Jupyter lab environment and uh once this Jupyter lab environment is opened for you what you can do is is opened for you what you can do is is opened for you what you can do is that you can copy paste the code into that you can copy paste the code into that you can copy paste the code into this environment. [snorts] So I click on this Python 3 kernel right So I click on this Python 3 kernel right now and open this. So you can see on the now and open this. So you can see on the now and open this. So you can see on the left hand side left hand side left hand side um let me get my code and I'll upload it um let me get my code and I'll upload it um let me get my code and I'll upload it on the left hand side over here.