← back to concept

Flow Matching — across courses

cmu-10799-diffusion-flow

  1. L12 @ 3s

    Okay, so let's get started. uh so far uh Okay, so let's get started. uh so far uh our last class we have talked about you our last class we have talked about you our last class we have talked about you know two ways to do discrete diffusion know two ways to do discrete diffusion know two ways to do discrete diffusion like the first one is like we do the like the first one is like we do the like the first one is like we do the categorical noise and then d noiseis and categorical noise and then d noiseis and categorical noise and then d noiseis and that's corresponding to basically the that's corresponding to basically the that's corresponding to basically the ddpm formulation of continuous diffusion ddpm formulation of continuous diffusion ddpm formulation of continuous diffusion and we also talk about like the score and we also talk about like the score and we also talk about like the score matching you know formulation of matching you know formulation of matching you know formulation of or equivalence of the uh discrete or equivalence of the uh discrete or equivalence of the uh discrete diffusion and uh we haven't talked about diffusion and uh we haven't talked about diffusion and uh we haven't talked about the flow matching equivalence Uh so this the flow matching equivalence Uh so this the flow matching equivalence Uh so this is what we're going to talk about today. is what we're going to talk about today. is what we're going to talk about today. Okay. And then you may have noticed that Okay. And then you may have noticed that Okay. And then you may have noticed that I wear the meta shirt again and meta I wear the meta shirt again and meta I wear the meta shirt again and meta hat. Uh well this is because the this hat. Uh well this is because the this hat. Uh well this is because the this thing well some of it come from meta thing well some of it come from meta thing well some of it come from meta again from the same flow matching team. again from the same flow matching team. again from the same flow matching team. Okay. Uh so this is just like a quick Okay. Uh so this is just like a quick

  2. L12 @ 48s

    Okay. Uh so this is just like a quick like reminder of like like reminder of like like reminder of like what happened next last class. So what happened next last class. So what happened next last class. So basically we talked about how to add basically we talked about how to add basically we talked about how to add noise to text and how to represent it in noise to text and how to represent it in noise to text and how to represent it in like a vector slash uh matrix form and like a vector slash uh matrix form and like a vector slash uh matrix form and how we can build a diffusionesque uh how we can build a diffusionesque uh how we can build a diffusionesque uh forward process with this like you know forward process with this like you know forward process with this like you know vector and matrix representation of the vector and matrix representation of the vector and matrix representation of the text and trans transformation. Uh and text and trans transformation. Uh and text and trans transformation. Uh and then we can train it using the same uh then we can train it using the same uh then we can train it using the same uh you know elbow loss from uh DDPM you know elbow loss from uh DDPM you know elbow loss from uh DDPM essentially. And uh we also talked about essentially. And uh we also talked about essentially. And uh we also talked about how we can define the discrete score how we can define the discrete score how we can define the discrete score which is the concrete score and which is the concrete score and which is the concrete score and essentially how we can learn it uh you essentially how we can learn it uh you essentially how we can learn it uh you know using a better or like more stable know using a better or like more stable know using a better or like more stable um like loss function which is called um like loss function which is called um like loss function which is called the score entropy loss over here. And the score entropy loss over here. And the score entropy loss over here. And lastly we talked about like the easiest

  3. L12 @ 99s

    lastly we talked about like the easiest lastly we talked about like the easiest way to do discrete diffusion which is way to do discrete diffusion which is way to do discrete diffusion which is mass diffusion uh language modeling. So mass diffusion uh language modeling. So mass diffusion uh language modeling. So basically you always you try to basically you always you try to basically you always you try to interpolate between your clean data and interpolate between your clean data and interpolate between your clean data and the full mask and then you just like try the full mask and then you just like try the full mask and then you just like try to construct a discrete diffusion model to construct a discrete diffusion model to construct a discrete diffusion model like that. Okay. So uh what is the so so like that. Okay. So uh what is the so so like that. Okay. So uh what is the so so so so far we talked about everything. Um so so far we talked about everything. Um so so far we talked about everything. Um so what is the analog analogy I guess to so what is the analog analogy I guess to so what is the analog analogy I guess to the the velocity uh in flow matching in the the velocity uh in flow matching in the the velocity uh in flow matching in our discrete setting. What do we think for real for real someone? someone? someone? >> Yeah. >> Yeah. >> Yeah. >> Probability part. >> Probability part. >> Probability part. >> The what? Sorry. >> The what? Sorry. >> The what? Sorry. >> The probability part. >> The probability part. >> The probability part. >> The the the what part? >> The the the what part? >> The the the what part? >> The probability part. >> The probability part. >> Probability part. >> Probability part. >> Um yeah, like on on Yes. So basically >> Um yeah, like on on Yes. So basically >> Um yeah, like on on Yes. So basically you will try you want to somehow form a you will try you want to somehow form a you will try you want to somehow form a probability path, right? to go from like probability path, right? to go from like

  4. L12 @ 166s

    probability path, right? to go from like some noise or like some the full mass to some noise or like some the full mass to some noise or like some the full mass to clean data, right? Yeah, basically that clean data, right? Yeah, basically that clean data, right? Yeah, basically that uh and uh but yeah, let's talk about how uh and uh but yeah, let's talk about how uh and uh but yeah, let's talk about how we build it, right? So, let's first we build it, right? So, let's first we build it, right? So, let's first recall how we build a flow model first. recall how we build a flow model first. recall how we build a flow model first. Uh so, basically uh to build a flow Uh so, basically uh to build a flow Uh so, basically uh to build a flow model, we need uh essentially two things model, we need uh essentially two things model, we need uh essentially two things to happen, right? Basically this flow of to happen, right? Basically this flow of to happen, right? Basically this flow of probability need to conserve the mass probability need to conserve the mass probability need to conserve the mass which means that the probability should which means that the probability should which means that the probability should always add up to one right and we should always add up to one right and we should always add up to one right and we should also satisfy the continuity equation also satisfy the continuity equation also satisfy the continuity equation which means that the probability can which means that the probability can which means that the probability can only flow or change continuously but only flow or change continuously but only flow or change continuously but like these two things are kind of like like these two things are kind of like like these two things are kind of like weird right when you think about um weird right when you think about um weird right when you think about um discrete settings uh or maybe not but discrete settings uh or maybe not but discrete settings uh or maybe not but let let's look at them in a in the let let's look at them in a in the

  5. L12 @ 212s

    let let's look at them in a in the discrete settings. So last time we kind discrete settings. So last time we kind discrete settings. So last time we kind of skipped through this part uh or of skipped through this part uh or of skipped through this part uh or didn't really talk about it in detail didn't really talk about it in detail didn't really talk about it in detail but like the the continuous time but like the the continuous time but like the the continuous time formulation that correspond to you know formulation that correspond to you know formulation that correspond to you know this like uh this this transformation is this like uh this this transformation is this like uh this this transformation is what we call the continuous time markoff what we call the continuous time markoff what we call the continuous time markoff chain or CTMC. So CTMC is basically just chain or CTMC. So CTMC is basically just chain or CTMC. So CTMC is basically just like we we have this like uh transition like we we have this like uh transition like we we have this like uh transition matrix at time t or like parameterized matrix at time t or like parameterized matrix at time t or like parameterized by the continuous time t. uh and then we by the continuous time t. uh and then we by the continuous time t. uh and then we can have this essentially this um can have this essentially this um can have this essentially this um transformation of probability uh get transformation of probability uh get transformation of probability uh get being expressed in this way. This is being expressed in this way. This is being expressed in this way. This is what we have seen before uh from the what we have seen before uh from the what we have seen before uh from the SEDD uh paper, right? Um but basically SEDD uh paper, right? Um but basically SEDD uh paper, right? Um but basically what it means is that like because we what it means is that like because we what it means is that like because we know that it's a probability know that it's a probability know that it's a probability distribution, right? So the probability distribution, right? So the probability

  6. L12 @ 263s

    distribution, right? So the probability the sum of the probability over all the sum of the probability over all the sum of the probability over all possible states or all possible tokens possible states or all possible tokens possible states or all possible tokens uh should equal to one right. So like uh should equal to one right. So like uh should equal to one right. So like basically from any state the probability basically from any state the probability basically from any state the probability of transition like the sum of the of transition like the sum of the of transition like the sum of the probability to transition to any state probability to transition to any state probability to transition to any state any other states or like just any states any other states or like just any states any other states or like just any states I guess or all states uh should be one. I guess or all states uh should be one. I guess or all states uh should be one. So the the conservation of mass right So the the conservation of mass right So the the conservation of mass right here uh then basically if you plug in here uh then basically if you plug in here uh then basically if you plug in everything uh you should get like everything uh you should get like everything uh you should get like basically the transition rate. So the basically the transition rate. So the basically the transition rate. So the this transition matrix basically um all this transition matrix basically um all this transition matrix basically um all the columns or all the rows depending on the columns or all the rows depending on the columns or all the rows depending on which you know convention that you that which you know convention that you that which you know convention that you that you choose to to use but basically like you choose to to use but basically like you choose to to use but basically like um you like all the all the all the um you like all the all the all the um you like all the all the all the transitions should add up to zero. And transitions should add up to zero. And

  7. L12 @ 313s

    transitions should add up to zero. And this is like and this is literally just this is like and this is literally just this is like and this is literally just a con conversation uh conservation of a con conversation uh conservation of a con conversation uh conservation of mass. So basically it just means that mass. So basically it just means that mass. So basically it just means that like the amount of like the amount of like the amount of like the amount of like the amount of like the amount of flow that flowing out of the certain flow that flowing out of the certain flow that flowing out of the certain state should equal to the all the amount state should equal to the all the amount state should equal to the all the amount of flow that's flow in like basically or of flow that's flow in like basically or of flow that's flow in like basically or or like the the opposite of that or like the the opposite of that or like the the opposite of that essentially. Yeah. Yeah. Does it make essentially. Yeah. Yeah. Does it make essentially. Yeah. Yeah. Does it make sense for people? confusion. confusion. >> Yeah. >> Yeah. >> Nearly add noise kind of in discrete >> Nearly add noise kind of in discrete >> Nearly add noise kind of in discrete setting. Do you have like a fixed setting. Do you have like a fixed setting. Do you have like a fixed sequence of like jumping from word to sequence of like jumping from word to sequence of like jumping from word to word? word? word? >> Yeah. So this is not like so this this >> Yeah. So this is not like so this this >> Yeah. So this is not like so this this QT thing is like a transition mixtures QT thing is like a transition mixtures QT thing is like a transition mixtures that represent a jump essentially. So that represent a jump essentially. So that represent a jump essentially. So this is like basically how much more this is like basically how much more this is like basically how much more likely that you're going to go basically likely that you're going to go basically

  8. L12 @ 369s

    likely that you're going to go basically it's like how how how likely that you're it's like how how how likely that you're it's like how how how likely that you're going to go jump from this state to that going to go jump from this state to that going to go jump from this state to that state. Basically this is like an state. Basically this is like an state. Basically this is like an unnormalized version of that. unnormalized version of that. unnormalized version of that. Yeah. Does it make sense? Yeah. So Yeah. Does it make sense? Yeah. So Yeah. Does it make sense? Yeah. So basically basically basically this this transition matrix is a jump this this transition matrix is a jump this this transition matrix is a jump and then um like basically the the and then um like basically the the and then um like basically the the infinite testimonly like you can form infinite testimonly like you can form infinite testimonly like you can form this probability distribution like this this probability distribution like this this probability distribution like this and because we want it to be a valid and because we want it to be a valid and because we want it to be a valid probability distribution. So everything probability distribution. So everything probability distribution. So everything add up to one you the you will have this add up to one you the you will have this add up to one you the you will have this like property where literally just like like property where literally just like like property where literally just like every like just like like all the all every like just like like all the all every like just like like all the all the flows will add up to zero basically. Yeah. Yeah. >> Represent all the jumps that you >> Represent all the jumps that you >> Represent all the jumps that you contain. contain. contain. >> Yeah. >> Yeah. >> Yeah. >> Yeah. So basically just like no matter how So basically just like no matter how So basically just like no matter how like so so this this QT is is the flow. like so so this this QT is is the flow. like so so this this QT is is the flow. It's a velocity, right? Essentially. So

  9. L12 @ 423s

    It's a velocity, right? Essentially. So It's a velocity, right? Essentially. So it's basically just saying that like um it's basically just saying that like um it's basically just saying that like um all the flows will like like you will all the flows will like like you will all the flows will like like you will not create new flows like that all the not create new flows like that all the not create new flows like that all the flows will just like add up to zero. flows will just like add up to zero. flows will just like add up to zero. They'll all cancel out essentially. Yeah, Yeah, >> this is represented by a matrix like for >> this is represented by a matrix like for >> this is represented by a matrix like for state transitions. It already seems like state transitions. It already seems like state transitions. It already seems like a very big matrix. a very big matrix. a very big matrix. >> It is. It is but like I mean okay uh >> It is. It is but like I mean okay uh >> It is. It is but like I mean okay uh that's actually a good question and and that's actually a good question and and that's actually a good question and and this is also why like you will see like this is also why like you will see like this is also why like you will see like before right everyone was trying to before right everyone was trying to before right everyone was trying to represent the what they're trying to represent the what they're trying to represent the what they're trying to predict as like the logit instead of the predict as like the logit instead of the predict as like the logit instead of the actual rate matrix right because the actual rate matrix right because the actual rate matrix right because the rate matrix is like quadratics of the of rate matrix is like quadratics of the of rate matrix is like quadratics of the of the of of the logit dimension right so the of of the logit dimension right so the of of the logit dimension right so this is also why we're trying to do this is also why we're trying to do

  10. L12 @ 473s

    this is also why we're trying to do reparameterization trick yeah but that's reparameterization trick yeah but that's reparameterization trick yeah but that's a great question. All right. Anything a great question. All right. Anything a great question. All right. Anything else? Anyone else question? else? Anyone else question? else? Anyone else question? All good. All right. All right. Okay. All good. All right. All right. Okay. All good. All right. All right. Okay. So, we have uh the conservation of mass. So, we have uh the conservation of mass. So, we have uh the conservation of mass. Now, uh what is the other thing that we Now, uh what is the other thing that we Now, uh what is the other thing that we need? >> The other thing >> The other thing >> continuity equation, right? Okay. So, >> continuity equation, right? Okay. So, >> continuity equation, right? Okay. So, what is the continuity equation in in in what is the continuity equation in in in what is the continuity equation in in in this CTMC? Well, this thing is called this CTMC? Well, this thing is called this CTMC? Well, this thing is called the Clomograph the Clomograph the Clomograph equation and I just I guess this is the equation and I just I guess this is the equation and I just I guess this is the information theory people would like the information theory people would like the information theory people would like the name and stuff. uh but but essentially name and stuff. uh but but essentially name and stuff. uh but but essentially is literally uh just means that the is literally uh just means that the is literally uh just means that the infinite decimal change in probability infinite decimal change in probability infinite decimal change in probability is equal to literally the the amount of is equal to literally the the amount of is equal to literally the the amount of probability that comes into this probability that comes into this probability that comes into this particular state particular state particular state minus the probability that go out of minus the probability that go out of minus the probability that go out of this state that's it right this is like this state that's it right this is like

  11. L12 @ 538s

    this state that's it right this is like super uh very intuitive I hope is super uh very intuitive I hope is super uh very intuitive I hope is All good. What can we observe here? Can we write What can we observe here? Can we write this in like a easier form essentially this in like a easier form essentially this in like a easier form essentially using the things that we learned just using the things that we learned just using the things that we learned just now? Yeah. now? Yeah. now? Yeah. And not yet conditional, but like can we And not yet conditional, but like can we And not yet conditional, but like can we simplify this a little bit using the simplify this a little bit using the simplify this a little bit using the conservation of mass that we just conservation of mass that we just conservation of mass that we just learned? It kind of looks like it, right? If you It kind of looks like it, right? If you think about it think about it think about it like what what is this part? What is like what what is this part? What is like what what is this part? What is this part? So this part plus this part is zero, So this part plus this part is zero, right? But what is this part? Let's go back. Let's go back. So what is Let's go back. Let's go back. So what is So this is this right? So this is this right? So this is this right? So can we write it in a better form? Yeah, it will Yeah, it will well basically all right because like well basically all right because like well basically all right because like this negative sum thing is just qxx. So this negative sum thing is just qxx. So this negative sum thing is just qxx. So this is literally just like qxx times this is literally just like qxx times this is literally just like qxx times px. So you don't even need the the the px. So you don't even need the the the px. So you don't even need the the the not equal anymore. So just sum of all not equal anymore. So just sum of all

  12. L12 @ 617s

    not equal anymore. So just sum of all the z all the possible z's and then qzx the z all the possible z's and then qzx the z all the possible z's and then qzx ptz right and z could be equal to x as ptz right and z could be equal to x as ptz right and z could be equal to x as well. So this is just literally can be well. So this is just literally can be well. So this is just literally can be written as DPT uh DT equals to QP. written as DPT uh DT equals to QP. written as DPT uh DT equals to QP. Super simple, right? Make sense? Make sense? Okay, cool. Uh well, this is nice Okay, cool. Uh well, this is nice Okay, cool. Uh well, this is nice because this is a OD again, right? because this is a OD again, right? because this is a OD again, right? Amazing. Just like flow matching, we get Amazing. Just like flow matching, we get Amazing. Just like flow matching, we get a simple OD again, right? So all we need a simple OD again, right? So all we need a simple OD again, right? So all we need to do now is to simulate this OD, right? to do now is to simulate this OD, right? to do now is to simulate this OD, right? In order to in order to build a In order to in order to build a In order to in order to build a trajectory on this probability path. trajectory on this probability path. trajectory on this probability path. Does it make sense? Okay. Cool. All Does it make sense? Okay. Cool. All Does it make sense? Okay. Cool. All right. So now let's simulate a right. So now let's simulate a right. So now let's simulate a trajectory on the probability path. trajectory on the probability path. trajectory on the probability path. Right. So then we can have literally Right. So then we can have literally Right. So then we can have literally so this is like basically just like so this is like basically just like so this is like basically just like oiler, right? So you just get you you oiler, right? So you just get you you

  13. L12 @ 681s

    oiler, right? So you just get you you just literally plug everything in here just literally plug everything in here just literally plug everything in here and then you'll get this thing and the and then you'll get this thing and the and then you'll get this thing and the most important part that I want to say most important part that I want to say most important part that I want to say is that so this is the only thing that is that so this is the only thing that is that so this is the only thing that we need to learn right the transition we need to learn right the transition we need to learn right the transition like the transition rate from like your like the transition rate from like your like the transition rate from like your current time step to next time step. So current time step to next time step. So current time step to next time step. So this is like the quoteunquote velocity this is like the quoteunquote velocity this is like the quoteunquote velocity that we're that we should be learning. that we're that we should be learning. that we're that we should be learning. Make sense? So this is the the the Make sense? So this is the the the Make sense? So this is the the the correspondence correspondence correspondence of the velocity thing in the full of the velocity thing in the full of the velocity thing in the full matching. matching. matching. Okay. So now how do we learn this thing Okay. So now how do we learn this thing Okay. So now how do we learn this thing or how do we parameterize this thing? I or how do we parameterize this thing? I or how do we parameterize this thing? I guess guess guess we kind of talked about it already. All right, I'll just tell you guys the All right, I'll just tell you guys the answer. But basically you just like you answer. But basically you just like you answer. But basically you just like you just do reparameterization again. So just do reparameterization again. So just do reparameterization again. So like the the core of the the story in like the the core of the the story in

  14. L12 @ 735s

    like the the core of the the story in all the discrete diffusion papers is all the discrete diffusion papers is all the discrete diffusion papers is that like no matter what you do try to that like no matter what you do try to that like no matter what you do try to reparameterize your thing into like like reparameterize your thing into like like reparameterize your thing into like like some something that you can directly get some something that you can directly get some something that you can directly get from the forward process and the clean from the forward process and the clean from the forward process and the clean data estimation. It's actually not only data estimation. It's actually not only data estimation. It's actually not only uh the the discrete diffusion paper. uh the the discrete diffusion paper. uh the the discrete diffusion paper. It's actually all diffusion paper. It's actually all diffusion paper. It's actually all diffusion paper. Right? Basically, you just apply Right? Basically, you just apply Right? Basically, you just apply reparameterization trick and then you'll reparameterization trick and then you'll reparameterization trick and then you'll get something that is like nicer to to get something that is like nicer to to get something that is like nicer to to learn. For example, like the epsilon learn. For example, like the epsilon learn. For example, like the epsilon prediction and the the or the v prediction and the the or the v prediction and the the or the v prediction, right? And in this case, prediction, right? And in this case, prediction, right? And in this case, this is literally just the x0 prediction this is literally just the x0 prediction this is literally just the x0 prediction or like the clean data prediction. or like the clean data prediction. or like the clean data prediction. Does it make sense? And the reason why Does it make sense? And the reason why Does it make sense? And the reason why we're doing that is because a it saves we're doing that is because a it saves we're doing that is because a it saves dimensionality and b we can literally dimensionality and b we can literally dimensionality and b we can literally just use cross entropy to learn this, just use cross entropy to learn this,

  15. L12 @ 783s

    just use cross entropy to learn this, right? Because we have the ground truth right? Because we have the ground truth right? Because we have the ground truth clean data. So we just need to use cross clean data. So we just need to use cross clean data. So we just need to use cross entropy on every single token. Cool. Now you get discrete flow Cool. Now you get discrete flow matching. Yay. That's it pretty much. So matching. Yay. That's it pretty much. So matching. Yay. That's it pretty much. So this is like kind of similar to what we this is like kind of similar to what we this is like kind of similar to what we have seen before, right? Yep. have seen before, right? Yep. have seen before, right? Yep. >> What does the prior distribution look >> What does the prior distribution look >> What does the prior distribution look like? Is it just uniform overall? like? Is it just uniform overall? like? Is it just uniform overall? >> You can you can choose, right? So you >> You can you can choose, right? So you >> You can you can choose, right? So you can choose uniform, you can choose all can choose uniform, you can choose all can choose uniform, you can choose all mask, you can choose anything just the mask, you can choose anything just the mask, you can choose anything just the same as the discrete diffusion thing. Okay. Uh but basically once we have Okay. Uh but basically once we have learned this uh network then we can try learned this uh network then we can try learned this uh network then we can try to sample from it. So how do you sample to sample from it. So how do you sample to sample from it. So how do you sample from? from? from? Right. So you basically first start from Right. So you basically first start from Right. So you basically first start from t equals z because you are doing full t equals z because you are doing full t equals z because you are doing full matching now. So the the time scale matching now. So the the time scale matching now. So the the time scale flips right. So you start from t t

  16. L12 @ 841s

    flips right. So you start from t t flips right. So you start from t t equals z which and then you sample from equals z which and then you sample from equals z which and then you sample from your prior which could be uniform or all your prior which could be uniform or all your prior which could be uniform or all mask and then basically you just like mask and then basically you just like mask and then basically you just like try to integrate through the this OD or try to integrate through the this OD or try to integrate through the this OD or like just simulate try to simulate this like just simulate try to simulate this like just simulate try to simulate this OD simulate the particles on this OD simulate the particles on this OD simulate the particles on this probability path. Uh then you first probability path. Uh then you first probability path. Uh then you first calculate your uh transition rate by calculate your uh transition rate by calculate your uh transition rate by using your um by using your uh the using your um by using your uh the using your um by using your uh the learned uh uh cross entropy or logic learned uh uh cross entropy or logic learned uh uh cross entropy or logic prediction predictor you know so your prediction predictor you know so your prediction predictor you know so your model and then you can you can do that model and then you can you can do that model and then you can you can do that uh and then what you do is you sample uh and then what you do is you sample uh and then what you do is you sample you literally just sample from the you literally just sample from the you literally just sample from the categorical distribution um that is categorical distribution um that is categorical distribution um that is defined by you know the the the CTMC defined by you know the the the CTMC defined by you know the the the CTMC that we just saw before. Okay. And then that we just saw before. Okay. And then that we just saw before. Okay. And then you just increment the time and then you you just increment the time and then you

  17. L12 @ 885s

    you just increment the time and then you do it over over again until you reach do it over over again until you reach do it over over again until you reach the end point. Okay, make sense? This this is actually Okay, make sense? This this is actually super similar to what we have seen super similar to what we have seen super similar to what we have seen before, right? To what we have seen last before, right? To what we have seen last before, right? To what we have seen last class. class. class. It's it's just that it's like more It's it's just that it's like more It's it's just that it's like more straightforward essentially. straightforward essentially. straightforward essentially. >> Yeah. >> Yeah. >> You will still be quadratic the number >> You will still be quadratic the number >> You will still be quadratic the number of the vocabulary size, right? of the vocabulary size, right? of the vocabulary size, right? >> Yeah. >> Yeah. >> Does that make it difficult to predict >> Does that make it difficult to predict >> Does that make it difficult to predict such a large? No, because you are not such a large? No, because you are not such a large? No, because you are not predicting Q, right? You're predicting predicting Q, right? You're predicting predicting Q, right? You're predicting P. You're predicting Sx. And then you P. You're predicting Sx. And then you P. You're predicting Sx. And then you multiply by something that you can do in multiply by something that you can do in multiply by something that you can do in closed form. Yeah. Anything else? Anything else? Cool. You know, as you know, you guys Cool. You know, as you know, you guys Cool. You know, as you know, you guys already know this. This is the theme of already know this. This is the theme of already know this. This is the theme of gen modeling. the community just like gen modeling. the community just like gen modeling. the community just like seems to kind of you know converge on seems to kind of you know converge on seems to kind of you know converge on things that we developed somehow but things that we developed somehow but

  18. L12 @ 942s

    things that we developed somehow but basically there are two discreful basically there are two discreful basically there are two discreful matching papers uh this time they are matching papers uh this time they are matching papers uh this time they are not published in the same conference but not published in the same conference but not published in the same conference but they're pretty is very very similar so they're pretty is very very similar so they're pretty is very very similar so the ones that we were talking about was the ones that we were talking about was the ones that we were talking about was actually the first one uh so this guy is actually the first one uh so this guy is actually the first one uh so this guy is called Andrew Kempel this dude honestly called Andrew Kempel this dude honestly called Andrew Kempel this dude honestly is like so goatated he's like he just is like so goatated he's like he just is like so goatated he's like he just graduated from his PhD or something and graduated from his PhD or something and graduated from his PhD or something and I really do not understand why he's not I really do not understand why he's not I really do not understand why he's not like rich and famous yet. But like like rich and famous yet. But like like rich and famous yet. But like basically he um he wrote the the the basically he um he wrote the the the basically he um he wrote the the the CTMC paper that we were we've been CTMC paper that we were we've been CTMC paper that we were we've been talking about and he also wrote the one talking about and he also wrote the one talking about and he also wrote the one of the the discrete fo matching paper. of the the discrete fo matching paper. of the the discrete fo matching paper. Uh somehow right now he's doing a Uh somehow right now he's doing a Uh somehow right now he's doing a startup or something. I I don't really startup or something. I I don't really startup or something. I I don't really know but anyway it's like really like I know but anyway it's like really like I

  19. L12 @ 982s

    know but anyway it's like really like I really hope this guy get rich and famous really hope this guy get rich and famous really hope this guy get rich and famous soon. Uh anyway and then the second one soon. Uh anyway and then the second one soon. Uh anyway and then the second one is the one from Meta. Um basically they is the one from Meta. Um basically they is the one from Meta. Um basically they propose something that is a little bit propose something that is a little bit propose something that is a little bit more uh gen uh like general and then more uh gen uh like general and then more uh gen uh like general and then they also do a lot more uh you know just they also do a lot more uh you know just they also do a lot more uh you know just like nice framework and then a lot of like nice framework and then a lot of like nice framework and then a lot of connections to you know the the existing connections to you know the the existing connections to you know the the existing models and stuff like that but basically models and stuff like that but basically models and stuff like that but basically there are two discrete flow matching there are two discrete flow matching there are two discrete flow matching papers that you can read and they're all papers that you can read and they're all papers that you can read and they're all they're both very nice. they're both very nice. they're both very nice. All right, cool. So we have talked about All right, cool. So we have talked about All right, cool. So we have talked about uh pretty much all three uh uh pretty much all three uh uh pretty much all three uh parameterizations today uh already. Uh parameterizations today uh already. Uh parameterizations today uh already. Uh so uh for flow matching what we do is we so uh for flow matching what we do is we so uh for flow matching what we do is we instead of learning the velocity we sort instead of learning the velocity we sort instead of learning the velocity we sort of learn the rate matrix but we don't

  20. L12 @ 1025s

    of learn the rate matrix but we don't of learn the rate matrix but we don't actually learn the rate matrix directly actually learn the rate matrix directly actually learn the rate matrix directly we we do literally the same thing we we we do literally the same thing we we we do literally the same thing we learn the like the clean data prediction learn the like the clean data prediction learn the like the clean data prediction and then we trans and then we make it and then we trans and then we make it and then we trans and then we make it into the the rate matrix. Okay. So now into the the rate matrix. Okay. So now into the the rate matrix. Okay. So now uh are discrete diffusion models perfect uh are discrete diffusion models perfect uh are discrete diffusion models perfect now then like seems like it seems like now then like seems like it seems like now then like seems like it seems like they they they are so nice. Uh are they they they they are so nice. Uh are they they they they are so nice. Uh are they perfect? Yeah perfect? Yeah perfect? Yeah problem has been solved by fix. problem has been solved by fix. problem has been solved by fix. >> Yes. So I was going to get get you guys >> Yes. So I was going to get get you guys >> Yes. So I was going to get get you guys to discuss but I guess you don't you to discuss but I guess you don't you to discuss but I guess you don't you don't need time to think. Okay. Yes. Uh don't need time to think. Okay. Yes. Uh don't need time to think. Okay. Yes. Uh so it has fixed length problem. you you so it has fixed length problem. you you so it has fixed length problem. you you still need to like pred decide like how still need to like pred decide like how still need to like pred decide like how how long your sentence is or how long how long your sentence is or how long how long your sentence is or how long your par paragraph is in order to build your par paragraph is in order to build

  21. L12 @ 1070s

    your par paragraph is in order to build this model and this is just like super this model and this is just like super this model and this is just like super like not good right this is a very very like not good right this is a very very like not good right this is a very very not uh uh if like um realistic okay what not uh uh if like um realistic okay what not uh uh if like um realistic okay what else any anything else yeah else any anything else yeah else any anything else yeah >> I think they won't handle temporal >> I think they won't handle temporal >> I think they won't handle temporal dependency as well as auto reggressive dependency as well as auto reggressive dependency as well as auto reggressive models because like random tokens at any models because like random tokens at any models because like random tokens at any point are getting like unmasked rather point are getting like unmasked rather point are getting like unmasked rather than than than >> so so the ordering like You don't >> so so the ordering like You don't >> so so the ordering like You don't actually have an ordering of unmasking, actually have an ordering of unmasking, actually have an ordering of unmasking, right? Essentially. Yeah. Very good. right? Essentially. Yeah. Very good. right? Essentially. Yeah. Very good. What else? What else? What else? What about uh how about on like the What about uh how about on like the What about uh how about on like the efficient efficiency side? What do we efficient efficiency side? What do we efficient efficiency side? What do we think? It's supposed to be very good, think? It's supposed to be very good, think? It's supposed to be very good, but would it be actually very good in but would it be actually very good in but would it be actually very good in real life? What do we think? real life? What do we think? real life? What do we think? >> Short answer. Yeah. Yeah. Yeah. So, so >> Short answer. Yeah. Yeah. Yeah. So, so >> Short answer. Yeah. Yeah. Yeah. So, so like it's the same problem as the fixed like it's the same problem as the fixed

  22. L12 @ 1116s

    like it's the same problem as the fixed length thing, right? Yeah. length thing, right? Yeah. length thing, right? Yeah. >> I heard this from like who's doing >> I heard this from like who's doing >> I heard this from like who's doing something in this. So, like if you want something in this. So, like if you want something in this. So, like if you want to generate two words at the same time. to generate two words at the same time. to generate two words at the same time. Then these ones can't tell the Then these ones can't tell the Then these ones can't tell the difference between New York and San difference between New York and San difference between New York and San Francisco, New San Francisco. Francisco, New San Francisco. Francisco, New San Francisco. >> Ah, wait. That's actually that's >> Ah, wait. That's actually that's >> Ah, wait. That's actually that's actually so but this is kind of similar actually so but this is kind of similar actually so but this is kind of similar to like the unmasking ordering problem, to like the unmasking ordering problem, to like the unmasking ordering problem, right? That's that's a good one. Anyone right? That's that's a good one. Anyone right? That's that's a good one. Anyone else? else? else? There's there's one that is like I guess There's there's one that is like I guess There's there's one that is like I guess not very obvious, but it's related to not very obvious, but it's related to not very obvious, but it's related to efficiency. And uh if you have you know efficiency. And uh if you have you know efficiency. And uh if you have you know ever implemented LM it's like it's like ever implemented LM it's like it's like ever implemented LM it's like it's like a very very important thing. YEAH. a very very important thing. YEAH. a very very important thing. YEAH. >> KV CACHING. >> KV CACHING. >> KV CACHING. >> YEAH. YEAH. KV caching right. It's not >> YEAH. YEAH. KV caching right. It's not >> YEAH. YEAH. KV caching right. It's not possible to do KV caching right now. All possible to do KV caching right now. All possible to do KV caching right now. All right. So let's take a look at it. Uh

  23. L12 @ 1160s

    right. So let's take a look at it. Uh right. So let's take a look at it. Uh yeah. So essentially variable length you yeah. So essentially variable length you yeah. So essentially variable length you still can't do variable length still can't do variable length still can't do variable length generation. Uh the model has to use generation. Uh the model has to use generation. Uh the model has to use birectional attention because you do not birectional attention because you do not birectional attention because you do not know like where your context is right. know like where your context is right. know like where your context is right. It can come from before you. It come It can come from before you. It come It can come from before you. It come from after you. You don't know. So you from after you. You don't know. So you from after you. You don't know. So you need a full attention. So you cannot do need a full attention. So you cannot do need a full attention. So you cannot do KV caching. Uh and then also you don't KV caching. Uh and then also you don't KV caching. Uh and then also you don't really know well like basically you really know well like basically you really know well like basically you don't really have like a ordering. So don't really have like a ordering. So don't really have like a ordering. So you kind of just like depending on uh you kind of just like depending on uh you kind of just like depending on uh the model to uh to kind of figure it out the model to uh to kind of figure it out the model to uh to kind of figure it out and it can h like basically things like and it can h like basically things like and it can h like basically things like new Francisco can happen, right? And new Francisco can happen, right? And new Francisco can happen, right? And then the last thing is like not then the last thing is like not then the last thing is like not necessarily like a drawback but more of necessarily like a drawback but more of necessarily like a drawback but more of like something that we can think about, like something that we can think about,

  24. L12 @ 1197s

    like something that we can think about, right? And it's like now we have like right? And it's like now we have like right? And it's like now we have like two um like frameworks that like sort of two um like frameworks that like sort of two um like frameworks that like sort of like just like really really well like just like really really well like just like really really well connected. Uh can we make them together connected. Uh can we make them together connected. Uh can we make them together like can we merge them together? Can we like can we merge them together? Can we like can we merge them together? Can we make something that is like you know make something that is like you know make something that is like you know that's naturally multimodel and stuff that's naturally multimodel and stuff that's naturally multimodel and stuff like that right? Uh okay so let's fix like that right? Uh okay so let's fix like that right? Uh okay so let's fix them one by one essentially. them one by one essentially. them one by one essentially. All right how would you generate a All right how would you generate a All right how would you generate a variable length paragraph? Okay, so variable length paragraph? Okay, so variable length paragraph? Okay, so obviously you can use a lm to generate obviously you can use a lm to generate obviously you can use a lm to generate everything auto reggressively, right? So everything auto reggressively, right? So everything auto reggressively, right? So basically you just insert uh one token basically you just insert uh one token basically you just insert uh one token at a time at the end of the sentence, at a time at the end of the sentence, at a time at the end of the sentence, right? right? right? Is there any other way to do that? Oh, you can do multi token, but you Oh, you can do multi token, but you still insert like one at a time. Sorry still insert like one at a time. Sorry still insert like one at a time. Sorry insert at the end, right? insert at the end, right? insert at the end, right? >> Oh, you mean okay? >> Oh, you mean okay? >> Oh, you mean okay? >> Yeah. What else?

  25. L12 @ 1256s

    >> Yeah. What else? >> Yeah. What else? >> Inserting at but I saw that you can do >> Inserting at but I saw that you can do >> Inserting at but I saw that you can do decoding in blocks like you can predict decoding in blocks like you can predict decoding in blocks like you can predict it is still like predicting a block at a it is still like predicting a block at a it is still like predicting a block at a time. time. time. >> If you predict endo string then you >> If you predict endo string then you >> If you predict endo string then you discard prediction on blocks. discard prediction on blocks. discard prediction on blocks. >> Okay. So you can do a block. So you guys >> Okay. So you can do a block. So you guys >> Okay. So you can do a block. So you guys are kind of like doing thinking about are kind of like doing thinking about are kind of like doing thinking about same thing, right? Basically instead of same thing, right? Basically instead of same thing, right? Basically instead of one token at a time you predict multiple one token at a time you predict multiple one token at a time you predict multiple tokens. Uh we're going to talk about tokens. Uh we're going to talk about tokens. Uh we're going to talk about that later but not now. Yeah. just like that later but not now. Yeah. just like that later but not now. Yeah. just like tokens at the end. So then even if you tokens at the end. So then even if you tokens at the end. So then even if you have like some fixed block size, have like some fixed block size, have like some fixed block size, >> you don't have to generate for all of >> you don't have to generate for all of >> you don't have to generate for all of them. You can just have like pad at the them. You can just have like pad at the them. You can just have like pad at the end. So then you end. So then you end. So then you >> Okay, so tatting at the end. So you can

  26. L12 @ 1289s

    >> Okay, so tatting at the end. So you can >> Okay, so tatting at the end. So you can discard some of the tokens. Yeah, that discard some of the tokens. Yeah, that discard some of the tokens. Yeah, that makes sense too. What else? Yeah. makes sense too. What else? Yeah. makes sense too. What else? Yeah. >> Uh don't unmask all the tokens. >> Uh don't unmask all the tokens. >> Uh don't unmask all the tokens. >> You don't unmask all the tokens. Oh, so >> You don't unmask all the tokens. Oh, so >> You don't unmask all the tokens. Oh, so like I see I see I see that that's also like I see I see I see that that's also like I see I see I see that that's also another way. Is there anyone else? One another way. Is there anyone else? One another way. Is there anyone else? One last one last chance. last one last chance. last one last chance. By the way, this is like you kind of By the way, this is like you kind of By the way, this is like you kind of need to think out of box a little bit. need to think out of box a little bit. need to think out of box a little bit. This is like something that like we have This is like something that like we have This is like something that like we have it's like not like anything we have seen it's like not like anything we have seen it's like not like anything we have seen or like I guess it comes from like what or like I guess it comes from like what or like I guess it comes from like what we have seen but like the way of we have seen but like the way of we have seen but like the way of thinking is not >> sure what the noising in that will look >> sure what the noising in that will look like but some form of recursive like but some form of recursive like but some form of recursive diffusion. So first you generate a diffusion. So first you generate a diffusion. So first you generate a paragraph and for the next diffusion paragraph and for the next diffusion

  27. L12 @ 1335s

    paragraph and for the next diffusion process that paragraph is that one process that paragraph is that one process that paragraph is that one element for you. You noise the paragraph element for you. You noise the paragraph element for you. You noise the paragraph and there are multiple paragraphs being and there are multiple paragraphs being and there are multiple paragraphs being >> okay. So this is like kind of similar to >> okay. So this is like kind of similar to >> okay. So this is like kind of similar to what they said actually and then it's what they said actually and then it's what they said actually and then it's it's a little bit we're going to talk it's a little bit we're going to talk it's a little bit we're going to talk about something like that later. Yeah. about something like that later. Yeah. about something like that later. Yeah. OD solvers OD solvers OD solvers >> different OD solvers >> different OD solvers >> different OD solvers uh of what >> like why why would it help with the >> like why why would it help with the variable length problem I guess >> we can use like discrete flow matching >> we can use like discrete flow matching and auto reggressive like generated in and auto reggressive like generated in and auto reggressive like generated in kind of an auto reggressive way like uh kind of an auto reggressive way like uh kind of an auto reggressive way like uh kind of d noiseis aer set of tokens and kind of d noiseis aer set of tokens and kind of d noiseis aer set of tokens and then use that to den noiseise the next then use that to den noiseise the next then use that to den noiseise the next set of tokens so that we can control the set of tokens so that we can control the set of tokens so that we can control the >> so this is like all of you kind of say >> so this is like all of you kind of say >> so this is like all of you kind of say the same thing and that this is like the same thing and that this is like

  28. L12 @ 1389s

    the same thing and that this is like this is corresponding to one paper that this is corresponding to one paper that this is corresponding to one paper that we're going to talk about but not now we're going to talk about but not now we're going to talk about but not now this is this is something different okay this is this is something different okay this is this is something different okay uh anyone else anyone else want to try uh anyone else anyone else want to try uh anyone else anyone else want to try yeah yeah yeah >> start in the middle left and right >> start in the middle left and right >> start in the middle left and right okay this is the closest one okay okay this is the closest one okay okay this is the closest one okay basically basically why don't we h why do we have to insert why don't we h why do we have to insert why don't we h why do we have to insert only at the end All right. Can we insert only at the end All right. Can we insert only at the end All right. Can we insert in the middle? Like that's fine too, in the middle? Like that's fine too, in the middle? Like that's fine too, right? That's still variable length, right? That's still variable length, right? That's still variable length, right? That's still like Yeah. variable right? That's still like Yeah. variable right? That's still like Yeah. variable length generation, right? So what you length generation, right? So what you length generation, right? So what you can do is say I have like sentence like can do is say I have like sentence like can do is say I have like sentence like like a prompt or something I guess I like a prompt or something I guess I like a prompt or something I guess I don't know like I love cat. All right. don't know like I love cat. All right. don't know like I love cat. All right. And now what I can do is I can insert And now what I can do is I can insert And now what I can do is I can insert different tokens in between tokens, different tokens in between tokens,

  29. L12 @ 1429s

    different tokens in between tokens, right? I don't need to insert at the right? I don't need to insert at the right? I don't need to insert at the end. Why do I have to insert at the end? end. Why do I have to insert at the end? end. Why do I have to insert at the end? Doesn't matter, right? like and then I Doesn't matter, right? like and then I Doesn't matter, right? like and then I can do like in I can insert more tokens can do like in I can insert more tokens can do like in I can insert more tokens you know like in the in the in in this you know like in the in the in in this you know like in the in the in in this iterations of tokens right so basically iterations of tokens right so basically iterations of tokens right so basically uh this is what we call like a insertion uh this is what we call like a insertion uh this is what we call like a insertion based generation where you basically you based generation where you basically you based generation where you basically you have some like you you have some have some like you you have some have some like you you have some sequences and then instead of like auto sequences and then instead of like auto sequences and then instead of like auto reggressively this is kind of still reggressively this is kind of still reggressively this is kind of still autogressive I guess but it's like um autogressive I guess but it's like um autogressive I guess but it's like um instead of auto reggressively um like instead of auto reggressively um like instead of auto reggressively um like inserting at the end you insert you can inserting at the end you insert you can inserting at the end you insert you can insert anywhere you can insert be after insert anywhere you can insert be after insert anywhere you can insert be after any token not just the last token. Yeah. any token not just the last token. Yeah. any token not just the last token. Yeah. >> Does this work in the diffusion case? >> Does this work in the diffusion case?

  30. L12 @ 1471s

    >> Does this work in the diffusion case? >> Good question. I I'm gonna talk about >> Good question. I I'm gonna talk about >> Good question. I I'm gonna talk about it. Okay. So, how do we even model this it. Okay. So, how do we even model this it. Okay. So, how do we even model this kind of generation? Right. Uh kind of generation? Right. Uh kind of generation? Right. Uh what do we think actually let's what do we think actually let's what do we think actually let's brainstorm like if you were to design brainstorm like if you were to design brainstorm like if you were to design this kind of model how would you do it this kind of model how would you do it this kind of model how would you do it like what are the things that you need like what are the things that you need like what are the things that you need to predict in this case? to predict in this case? to predict in this case? >> Yeah. The simplest way is to have like >> Yeah. The simplest way is to have like >> Yeah. The simplest way is to have like really big like training your model on a really big like training your model on a really big like training your model on a really big >> you know what this is actually how how >> you know what this is actually how how they train it in practice. So in they train it in practice. So in they train it in practice. So in practice they just have very big and practice they just have very big and practice they just have very big and then they just like basically this is then they just like basically this is then they just like basically this is how they construct their supervision how they construct their supervision how they construct their supervision actually. So like basically if you have actually. So like basically if you have actually. So like basically if you have like empty tokens or mass tokens then like empty tokens or mass tokens then like empty tokens or mass tokens then you use it to to to essentially you use it to to to essentially

  31. L12 @ 1515s

    you use it to to to essentially construct construct the the ground truth construct construct the the ground truth construct construct the the ground truth of like what this like what the full of like what this like what the full of like what this like what the full quote unquote sequence should be but quote unquote sequence should be but quote unquote sequence should be but what is the this is a good this this is what is the this is a good this this is what is the this is a good this this is good but like we're getting there but good but like we're getting there but good but like we're getting there but what is the what is the thing that we what is the what is the thing that we what is the what is the thing that we should predict at at least right from should predict at at least right from should predict at at least right from the model like what are the things that the model like what are the things that the model like what are the things that we're predicting we're predicting we're predicting >> I was actually going to say something >> I was actually going to say something >> I was actually going to say something similar like in addition to predicting similar like in addition to predicting similar like in addition to predicting an actual English token predict a mass an actual English token predict a mass an actual English token predict a mass token that you and then further expand token that you and then further expand token that you and then further expand into either mask or another condition. into either mask or another condition. into either mask or another condition. >> Very very very close. Okay. Okay. Okay. >> Very very very close. Okay. Okay. Okay. >> Very very very close. Okay. Okay. Okay. Okay. Let's So basically basically what Okay. Let's So basically basically what Okay. Let's So basically basically what we need to predict is like we need to predict is like we need to predict is like what is the token and another thing what is the token and another thing what is the token and another thing right? What is this other thing? right? What is this other thing?

  32. L12 @ 1557s

    right? What is this other thing? >> The position or the whether to generate >> The position or the whether to generate >> The position or the whether to generate it between these it between these it between these >> whether to to insert right. Yeah. >> whether to to insert right. Yeah. >> whether to to insert right. Yeah. Basically that. So at each token what Basically that. So at each token what Basically that. So at each token what you should do is you to like basically you should do is you to like basically you should do is you to like basically what we need to decide is like basically what we need to decide is like basically what we need to decide is like basically okay so I'm looking at I I think I'm okay so I'm looking at I I think I'm okay so I'm looking at I I think I'm missing one token after I uh and then missing one token after I uh and then missing one token after I uh and then the missing token is wood just like in the missing token is wood just like in the missing token is wood just like in comparison to my ground truth here and comparison to my ground truth here and comparison to my ground truth here and then uh for love uh I'm missing two then uh for love uh I'm missing two then uh for love uh I'm missing two tokens right and then like the missing tokens right and then like the missing tokens right and then like the missing tokens can be either a or tokens can be either a or tokens can be either a or both the missing tokens after after love both the missing tokens after after love both the missing tokens after after love before cat. Right? So basically what we before cat. Right? So basically what we before cat. Right? So basically what we need to do is essentially train need to do is essentially train need to do is essentially train model to predict what we're missing and model to predict what we're missing and model to predict what we're missing and also like if we're missing right so also like if we're missing right so also like if we're missing right so basically if we're missing tokens and

  33. L12 @ 1612s

    basically if we're missing tokens and basically if we're missing tokens and like essentially like essentially like essentially note that like it's more likely or like note that like it's more likely or like note that like it's more likely or like it's like basically it's more likely to it's like basically it's more likely to it's like basically it's more likely to add more tokens after love than I in add more tokens after love than I in add more tokens after love than I in this case. You see that? Because like this case. You see that? Because like this case. You see that? Because like basically um it's like it's like because basically um it's like it's like because basically um it's like it's like because like there are more tokens that can be like there are more tokens that can be like there are more tokens that can be missing like a after love than I. So you missing like a after love than I. So you missing like a after love than I. So you should have a better chance to you know should have a better chance to you know should have a better chance to you know insert a after love like you know like insert a after love like you know like insert a after love like you know like from your model's prediction. Does it from your model's prediction. Does it from your model's prediction. Does it make sense? make sense? make sense? Okay. So what you should do is there are Okay. So what you should do is there are Okay. So what you should do is there are two things from your model that you two things from your model that you two things from your model that you should predict, right? One is predict should predict, right? One is predict should predict, right? One is predict how many tokens are missing or if how many tokens are missing or if how many tokens are missing or if there's any token missing after each there's any token missing after each there's any token missing after each token right so for each token this is token right so for each token this is token right so for each token this is what you're predicting and then the what you're predicting and then the

  34. L12 @ 1665s

    what you're predicting and then the second objective from your model is second objective from your model is second objective from your model is predict what tokens are missing so this predict what tokens are missing so this predict what tokens are missing so this is the same as LM right so basically for is the same as LM right so basically for is the same as LM right so basically for LM you only have objective two where LM you only have objective two where LM you only have objective two where like you know after what should I what like you know after what should I what like you know after what should I what should I insert after each token and should I insert after each token and should I insert after each token and then you pretend each token is the the then you pretend each token is the the then you pretend each token is the the last token Right. This is how you train last token Right. This is how you train last token Right. This is how you train it. And then for this you just add it. And then for this you just add it. And then for this you just add another objective. Yeah. another objective. Yeah. another objective. Yeah. >> But then this is predicting the tokens >> But then this is predicting the tokens >> But then this is predicting the tokens after I can see the love and cat has after I can see the love and cat has after I can see the love and cat has been generated. been generated. been generated. >> It cannot right like it can it can. >> It cannot right like it can it can. >> It cannot right like it can it can. Yeah. It can can see all the existing Yeah. It can can see all the existing Yeah. It can can see all the existing tokens. tokens. tokens. >> Yeah. So so it has to do it has to do >> Yeah. So so it has to do it has to do >> Yeah. So so it has to do it has to do insertion right and not end of things insertion right and not end of things insertion right and not end of things like I don't know. like I don't know.

  35. L12 @ 1706s

    like I don't know. >> Yeah. >> Yeah. >> Yeah. But basically what Okay. So >> Yeah. But basically what Okay. So >> Yeah. But basically what Okay. So basically what happens in the model or basically what happens in the model or basically what happens in the model or actually do I have it here? I do. I do. actually do I have it here? I do. I do. actually do I have it here? I do. I do. Okay. So what what what's happening here Okay. So what what what's happening here Okay. So what what what's happening here is that so this is like one token like is that so this is like one token like is that so this is like one token like one token in a sentence. This happens at one token in a sentence. This happens at one token in a sentence. This happens at every token. Okay. So one token in a every token. Okay. So one token in a every token. Okay. So one token in a sentence it goes into your transformers sentence it goes into your transformers sentence it goes into your transformers and then it predict two things. One is and then it predict two things. One is and then it predict two things. One is how many token that are missing. The how many token that are missing. The how many token that are missing. The second thing is like one possible tokens second thing is like one possible tokens second thing is like one possible tokens to insert. Okay. Does it make sense? to insert. Okay. Does it make sense? to insert. Okay. Does it make sense? >> Yeah. I mean how many terms are missing? >> Yeah. I mean how many terms are missing? >> Yeah. I mean how many terms are missing? You need the start and the end range, You need the start and the end range, You need the start and the end range, right? So like for example in the right? So like for example in the right? So like for example in the previous example, you need both I and previous example, you need both I and previous example, you need both I and the the the >> Yeah. Yeah. So basically what's >> Yeah. Yeah. So basically what's

  36. L12 @ 1750s

    >> Yeah. Yeah. So basically what's happening in what's hap what's happening happening in what's hap what's happening happening in what's hap what's happening in like implementation is that you are in like implementation is that you are in like implementation is that you are going to be counting how many pink going to be counting how many pink going to be counting how many pink tokens are there. That's it. tokens are there. That's it. tokens are there. That's it. >> My question was that you would need both >> My question was that you would need both >> My question was that you would need both let's say X1 and X2 as an input to be let's say X1 and X2 as an input to be let's say X1 and X2 as an input to be able to tell how many are. You don't able to tell how many are. You don't able to tell how many are. You don't because it's full attention. because it's full attention. because it's full attention. >> Yeah. Yeah. Yeah. So, so you only need >> Yeah. Yeah. Yeah. So, so you only need >> Yeah. Yeah. Yeah. So, so you only need basically the only thing you need is basically the only thing you need is basically the only thing you need is like okay after this token like add at like okay after this token like add at like okay after this token like add at this token. So remember how in this token. So remember how in this token. So remember how in transformer you're predicting something transformer you're predicting something transformer you're predicting something that has the same size as your input, that has the same size as your input, that has the same size as your input, right? So like after this token, how right? So like after this token, how right? So like after this token, how many tokens are missing after this token many tokens are missing after this token many tokens are missing after this token and before my next existing token? And and before my next existing token? And and before my next existing token? And what what is the one possible token that what what is the one possible token that what what is the one possible token that is that is missing out of all the tokens

  37. L12 @ 1795s

    is that is missing out of all the tokens is that is missing out of all the tokens that are missing? Right. Okay. So these that are missing? Right. Okay. So these that are missing? Right. Okay. So these two. All right. So how do we train two. All right. So how do we train two. All right. So how do we train these? How do we train with the these two How do we train with the these two objectives? We should know by now, objectives? We should know by now, objectives? We should know by now, right? right? I I feel like I hope we should know at I I feel like I hope we should know at I I feel like I hope we should know at least one of them. I think least one of them. I think least one of them. I think >> yeah >> yeah >> yeah like like just say next token that is like like just say next token that is like like just say next token that is just one just one just one >> that will just be at the other >> that will just be at the other >> that will just be at the other regressive models. Yeah. Yeah. regressive models. Yeah. Yeah. regressive models. Yeah. Yeah. >> Yeah. Then you can do the same thing >> Yeah. Then you can do the same thing >> Yeah. Then you can do the same thing with with with >> Mhm. Uh well okay I'll just tell you the >> Mhm. Uh well okay I'll just tell you the >> Mhm. Uh well okay I'll just tell you the answer. So the first thing is literally answer. So the first thing is literally answer. So the first thing is literally just the kale between two person right just the kale between two person right just the kale between two person right that makes sense right? So like the that makes sense right? So like the that makes sense right? So like the person between your actual missing token person between your actual missing token person between your actual missing token like how many tokens you're actually like how many tokens you're actually like how many tokens you're actually missing versus you know how many tokens missing versus you know how many tokens

  38. L12 @ 1848s

    missing versus you know how many tokens you predict to be missing and then the you predict to be missing and then the you predict to be missing and then the second thing is uh like literally just second thing is uh like literally just second thing is uh like literally just cross entropy of all possible missing cross entropy of all possible missing cross entropy of all possible missing tokens. So you just sum over all the pos tokens. So you just sum over all the pos tokens. So you just sum over all the pos all the cross entropy of pos like all the cross entropy of pos like all the cross entropy of pos like missing tokens. missing tokens. missing tokens. That's it. Yeah. That's it. Yeah. That's it. Yeah. I I don't understand at all how lower I I don't understand at all how lower I I don't understand at all how lower diffusion is coming. diffusion is coming. diffusion is coming. >> So let me I I'll I'll show you I'll show >> So let me I I'll I'll show you I'll show >> So let me I I'll I'll show you I'll show you. Uh so basically besides insertion you. Uh so basically besides insertion you. Uh so basically besides insertion uh you can also do other editing options uh you can also do other editing options uh you can also do other editing options uh other editing operations right so uh other editing operations right so uh other editing operations right so essentially what you can do is uh you essentially what you can do is uh you essentially what you can do is uh you can do deletion and substitution can do deletion and substitution can do deletion and substitution right does that make sense so besides right does that make sense so besides right does that make sense so besides insertion you can also do those things h insertion you can also do those things h insertion you can also do those things h if you add if you have all the all three if you add if you have all the all three if you add if you have all the all three operations you get something called edit operations you get something called edit operations you get something called edit flow flow

  39. L12 @ 1908s

    flow And why is it a flow model? Well, it's And why is it a flow model? Well, it's And why is it a flow model? Well, it's actually because this thing like the actually because this thing like the actually because this thing like the theory of this is coming from discre matching. Uh and okay let let's discre matching. Uh and okay let let's just like take a step back and see like just like take a step back and see like just like take a step back and see like why it is the case. So just so you know why it is the case. So just so you know why it is the case. So just so you know this is like not a rigorous proof or this is like not a rigorous proof or this is like not a rigorous proof or anything. This is basically just like anything. This is basically just like anything. This is basically just like give you some intuitions but like give you some intuitions but like give you some intuitions but like basically this is what's happening. Uh basically this is what's happening. Uh basically this is what's happening. Uh so remember how at the end of the the so remember how at the end of the the so remember how at the end of the the mass diffusion thing we we deduce mass diffusion thing we we deduce mass diffusion thing we we deduce something that looks like this right? So something that looks like this right? So something that looks like this right? So basically what what what is happening basically what what what is happening basically what what what is happening here what is the most important thing here what is the most important thing here what is the most important thing that happen here is that uh like this that happen here is that uh like this that happen here is that uh like this what this formula tells you is that if what this formula tells you is that if what this formula tells you is that if something is changing right so if you something is changing right so if you something is changing right so if you change from you know something to change from you know something to change from you know something to another thing then you should be doing

  40. L12 @ 1955s

    another thing then you should be doing another thing then you should be doing this operation where it's a ratio or this operation where it's a ratio or this operation where it's a ratio or like a rate times the clean data like a rate times the clean data like a rate times the clean data estimation right and then this thing is estimation right and then this thing is estimation right and then this thing is like this particular ratio at t limit like this particular ratio at t limit like this particular ratio at t limit right So essentially what we can do is right So essentially what we can do is right So essentially what we can do is we can parameterize the edit like the we can parameterize the edit like the we can parameterize the edit like the the basically how many tokens are you the basically how many tokens are you the basically how many tokens are you editing or like basically how frequent editing or like basically how frequent editing or like basically how frequent are you editing it like basically for are you editing it like basically for are you editing it like basically for each operation whether it's insertion or each operation whether it's insertion or each operation whether it's insertion or it's deletion or it's substitution you it's deletion or it's substitution you it's deletion or it's substitution you use this as your target for your person use this as your target for your person use this as your target for your person and then you use the same logic as your and then you use the same logic as your and then you use the same logic as your cross entropy and then you just train it cross entropy and then you just train it cross entropy and then you just train it with the person plus cross entropy loss with the person plus cross entropy loss with the person plus cross entropy loss and this is actually so the two things and this is actually so the two things and this is actually so the two things combined together uh is actually another combined together uh is actually another combined together uh is actually another divergence that is basically just a divergence that is basically just a

  41. L12 @ 2003s

    divergence that is basically just a generalized version of KL. Uh but this generalized version of KL. Uh but this generalized version of KL. Uh but this is like too complicated. We're not going is like too complicated. We're not going is like too complicated. We're not going to talk about it, but this is kind of to talk about it, but this is kind of to talk about it, but this is kind of like the intuition of what's happening. like the intuition of what's happening. like the intuition of what's happening. And like in like one specific um special And like in like one specific um special And like in like one specific um special case is that the deletion because you case is that the deletion because you case is that the deletion because you don't really need to predict another don't really need to predict another don't really need to predict another token after deletion, right? Because token after deletion, right? Because token after deletion, right? Because deletion literally just flip it to mask. deletion literally just flip it to mask. deletion literally just flip it to mask. So deletion doesn't need logic. So for So deletion doesn't need logic. So for So deletion doesn't need logic. So for deletion you only do the the person loss deletion you only do the the person loss deletion you only do the the person loss and that's it. and that's it. and that's it. Okay, any questions? So basically the Okay, any questions? So basically the Okay, any questions? So basically the reason why it's related to two reason why it's related to two reason why it's related to two diffusions because all of these is diffusions because all of these is diffusions because all of these is deduced from the discrete flow matching deduced from the discrete flow matching deduced from the discrete flow matching theory and uh the insertion based only theory and uh the insertion based only theory and uh the insertion based only the insertion only model is sort of like the insertion only model is sort of like the insertion only model is sort of like a special case of ediflow um which is a special case of ediflow um which is a special case of ediflow um which is actually like already good enough for actually like already good enough for

  42. L12 @ 2051s

    actually like already good enough for whatever you want to do. So you can whatever you want to do. So you can whatever you want to do. So you can literally just train a insertion based literally just train a insertion based literally just train a insertion based like insertion only at the flow and it's like insertion only at the flow and it's like insertion only at the flow and it's it's a valid language model. it's a valid language model. it's a valid language model. Yeah. Yeah. >> Are you assuming that the given the >> Are you assuming that the given the >> Are you assuming that the given the distribution of are like independent is distribution of are like independent is distribution of are like independent is >> oh why are you assuming the the the >> oh why are you assuming the the the >> oh why are you assuming the the the distribution of all the tokens distribution of all the tokens distribution of all the tokens independent you saying? independent you saying? independent you saying? >> Yeah. Yeah. This is just because it's >> Yeah. Yeah. This is just because it's >> Yeah. Yeah. This is just because it's easy. It's not really it's not like a easy. It's not really it's not like a easy. It's not really it's not like a good inductive bias at all. This is this good inductive bias at all. This is this good inductive bias at all. This is this is like Yeah. Obviously, this is is like Yeah. Obviously, this is is like Yeah. Obviously, this is something that you should change or or something that you should change or or something that you should change or or you can change, right? Yeah. Yeah. Yeah. you can change, right? Yeah. Yeah. Yeah. you can change, right? Yeah. Yeah. Yeah. Okay. Actually, that's a good good uh Okay. Actually, that's a good good uh Okay. Actually, that's a good good uh you know um leadway to to what we're you know um leadway to to what we're you know um leadway to to what we're going to talk about next, right? So, now going to talk about next, right? So, now going to talk about next, right? So, now we kind of fixed a veryable we kind of fixed a veryable

  43. L12 @ 2096s

    we kind of fixed a veryable length uh problem with insertion based length uh problem with insertion based length uh problem with insertion based generation. Uh we have other things to generation. Uh we have other things to generation. Uh we have other things to fix, right? The first most prominent fix, right? The first most prominent fix, right? The first most prominent thing is that like uh like we can't do thing is that like uh like we can't do thing is that like uh like we can't do KV caching. uh how can we enable KV caching. uh how can we enable KV caching. uh how can we enable caching and actually at the same time caching and actually at the same time caching and actually at the same time variable length generation. This is like variable length generation. This is like variable length generation. This is like literally what you guys just said, literally what you guys just said, literally what you guys just said, right? Basically what you can do is uh right? Basically what you can do is uh right? Basically what you can do is uh instead of like generating like a super instead of like generating like a super instead of like generating like a super large like p paragraph at once, what you large like p paragraph at once, what you large like p paragraph at once, what you can do is you can break it down into a can do is you can break it down into a can do is you can break it down into a lot of blocks and then each blocks are lot of blocks and then each blocks are lot of blocks and then each blocks are you know auto reggressive which with you know auto reggressive which with you know auto reggressive which with each other but within the blocks is like each other but within the blocks is like each other but within the blocks is like diffusion right. So this is what they diffusion right. So this is what they diffusion right. So this is what they called a block diffusion or called a block diffusion or called a block diffusion or semi-traggressive mass diffusion model. semi-traggressive mass diffusion model. semi-traggressive mass diffusion model. So like basically this thing is very So like basically this thing is very

  44. L12 @ 2140s

    So like basically this thing is very nice because you can um like essentially nice because you can um like essentially nice because you can um like essentially you can do paralyze generation like you can do paralyze generation like you can do paralyze generation like normal mass diffusion where you unmask normal mass diffusion where you unmask normal mass diffusion where you unmask like more than one token at a time. By like more than one token at a time. By like more than one token at a time. By the way um edit flow is also because all the way um edit flow is also because all the way um edit flow is also because all the tokens are independent. So you can the tokens are independent. So you can the tokens are independent. So you can unmask like or you can change you can unmask like or you can change you can unmask like or you can change you can edit uh you know each token edit uh you know each token edit uh you know each token independently. So you can you know you independently. So you can you know you independently. So you can you know you can also generate more than one token at can also generate more than one token at can also generate more than one token at a time. Uh but basically this thing is a time. Uh but basically this thing is a time. Uh but basically this thing is nice because it it preserve like the nice because it it preserve like the nice because it it preserve like the like a rough ordering like a rough auto like a rough ordering like a rough auto like a rough ordering like a rough auto reggressive ordering of your you know reggressive ordering of your you know reggressive ordering of your you know your your your your sentences. Um your your your your sentences. Um your your your your sentences. Um because of that it also enables KV because of that it also enables KV because of that it also enables KV caching. So you can cache all the things caching. So you can cache all the things caching. So you can cache all the things that you have generated like in your that you have generated like in your

  45. L12 @ 2180s

    that you have generated like in your previous blocks. The only thing you need previous blocks. The only thing you need previous blocks. The only thing you need to like do birectional attention to is to like do birectional attention to is to like do birectional attention to is your current block right? Uh and then your current block right? Uh and then your current block right? Uh and then you can also do like variable lines you can also do like variable lines you can also do like variable lines generation because you can have generation because you can have generation because you can have arbitrarily like arbitrary number of arbitrarily like arbitrary number of arbitrarily like arbitrary number of blocks and you can also predict more blocks and you can also predict more blocks and you can also predict more than one token at a time within your than one token at a time within your than one token at a time within your blocks. Uh so this thing is actually blocks. Uh so this thing is actually blocks. Uh so this thing is actually like a very good um like examples of like a very good um like examples of like a very good um like examples of like how you can sort of like like how you can sort of like like how you can sort of like interpolate uh you know two different interpolate uh you know two different interpolate uh you know two different models to get a better one. Um so yeah models to get a better one. Um so yeah models to get a better one. Um so yeah this is pretty cool. Uh so yeah so let's this is pretty cool. Uh so yeah so let's this is pretty cool. Uh so yeah so let's just look at look at work in real time. just look at look at work in real time. just look at look at work in real time. Yeah, but uh but like yeah, I'm really Yeah, but uh but like yeah, I'm really Yeah, but uh but like yeah, I'm really glad that like kind of like a lot of glad that like kind of like a lot of glad that like kind of like a lot of people think about this. Um but yeah, so

  46. L12 @ 2220s

    people think about this. Um but yeah, so people think about this. Um but yeah, so this is like a nice way to do you know this is like a nice way to do you know this is like a nice way to do you know to combine the fusion model with LMS. to combine the fusion model with LMS. to combine the fusion model with LMS. Okay, any question? Yeah, Okay, any question? Yeah, Okay, any question? Yeah, >> the diagram. Okay, I thought they were >> the diagram. Okay, I thought they were >> the diagram. Okay, I thought they were generating two blocks at the same time, generating two blocks at the same time, generating two blocks at the same time, but I thought it's all but I thought it's all but I thought it's all Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. >> Yeah. So the blocks here are like the >> Yeah. So the blocks here are like the >> Yeah. So the blocks here are like the the the the orange thing. >> It will generate a block of this EOS. >> It will generate a block of this EOS. >> No no like the EOS will is a it's a >> No no like the EOS will is a it's a >> No no like the EOS will is a it's a token that get generated in arbitrary token that get generated in arbitrary token that get generated in arbitrary one of the blocks and then you just do one of the blocks and then you just do one of the blocks and then you just do the same thing as what you would have the same thing as what you would have the same thing as what you would have done in the max diffusion just discard done in the max diffusion just discard done in the max diffusion just discard everything after that. everything after that. everything after that. >> Yeah. >> Yeah. K in that current block because we have K in that current block because we have K in that current block because we have >> we cannot >> we cannot >> we cannot >> not in the current block but but we can >> not in the current block but but we can

  47. L12 @ 2270s

    >> not in the current block but but we can K catch all the previous blocks. >> No because we don't we haven't generated >> No because we don't we haven't generated that yet, right? So like the block are that yet, right? So like the block are that yet, right? So like the block are auto reggressive. So the blocks are auto reggressive. So the blocks are auto reggressive. So the blocks are causal. So we Yeah. causal. So we Yeah. causal. So we Yeah. >> So what's generally like design choice >> So what's generally like design choice >> So what's generally like design choice for like the block size? Because if it's for like the block size? Because if it's for like the block size? Because if it's smaller than the number of diffusion smaller than the number of diffusion smaller than the number of diffusion steps then it just defeats the purpose. steps then it just defeats the purpose. steps then it just defeats the purpose. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. >> So this is like a like a like a like a >> So this is like a like a like a like a >> So this is like a like a like a like a sort of like a oblation that you need to sort of like a oblation that you need to sort of like a oblation that you need to basically it's like a hyperparameter basically it's like a hyperparameter basically it's like a hyperparameter that you kind of need to tune. Yeah. that you kind of need to tune. Yeah. that you kind of need to tune. Yeah. Yeah. You do also like basically if you Yeah. You do also like basically if you Yeah. You do also like basically if you have like a very large block then like have like a very large block then like have like a very large block then like obviously you need larger models and obviously you need larger models and obviously you need larger models and stuff like that, right? So this is kind stuff like that, right? So this is kind stuff like that, right? So this is kind of like also like but but but but if you of like also like but but but but if you

  48. L12 @ 2309s

    of like also like but but but but if you have too small of a blog then it just have too small of a blog then it just have too small of a blog then it just Yeah. It doesn't really it just is just Yeah. It doesn't really it just is just Yeah. It doesn't really it just is just auto reggressive I guess. Yeah. Yeah. auto reggressive I guess. Yeah. Yeah. auto reggressive I guess. Yeah. Yeah. >> Is like the best on all four fronts. Why >> Is like the best on all four fronts. Why >> Is like the best on all four fronts. Why is it not been adopted so far? is it not been adopted so far? is it not been adopted so far? >> Because this thing got proposed like >> Because this thing got proposed like >> Because this thing got proposed like when when did it get proposed? like last when when did it get proposed? like last when when did it get proposed? like last year. year. year. So it's pretty it's a lot of time but So it's pretty it's a lot of time but So it's pretty it's a lot of time but the world really the world really the world really >> well actually >> well actually >> well actually they do have um like AB models like that they do have um like AB models like that they do have um like AB models like that right and another thing that is like right and another thing that is like right and another thing that is like kind of tricky to do like diffusion kind of tricky to do like diffusion kind of tricky to do like diffusion language model is that uh the RL is not language model is that uh the RL is not language model is that uh the RL is not as easy as as LMS but this is like still as easy as as LMS but this is like still as easy as as LMS but this is like still like a very active research direction like a very active research direction like a very active research direction that people are investing a lot of money that people are investing a lot of money that people are investing a lot of money in I think. Yeah.

  49. L12 @ 2356s

    in I think. Yeah. in I think. Yeah. >> Do you think the second one is >> Do you think the second one is >> Do you think the second one is um like it's it's the it's a problem of um like it's it's the it's a problem of um like it's it's the it's a problem of scale then for the second one scale then for the second one scale then for the second one >> it is a problem of scale but the problem >> it is a problem of scale but the problem >> it is a problem of scale but the problem is to scale it up it's like very very is to scale it up it's like very very is to scale it up it's like very very very difficult right because like you very difficult right because like you very difficult right because like you you need full attention of all lengths you need full attention of all lengths you need full attention of all lengths so it's like quadratically hard the when so it's like quadratically hard the when so it's like quadratically hard the when you scale the the the the the length up you scale the the the the the length up you scale the the the the the length up right so this thing is just like so yeah right so this thing is just like so yeah right so this thing is just like so yeah so like discrete diffusion is actually so like discrete diffusion is actually so like discrete diffusion is actually pretty it's not trivial to scale even pretty it's not trivial to scale even pretty it's not trivial to scale even though people claim that it has better though people claim that it has better though people claim that it has better scaling scaling scaling So I don't know. >> the smallest block size is similar to >> the smallest block size is similar to >> Yeah. LM, right? It's just one one token >> Yeah. LM, right? It's just one one token >> Yeah. LM, right? It's just one one token at a time. LM. Yeah. Yeah, that's right. at a time. LM. Yeah. Yeah, that's right. at a time. LM. Yeah. Yeah, that's right. That's correct. Yeah. That's correct. Yeah.

  50. L12 @ 2411s

    That's correct. Yeah. >> For the for the second case, the mass >> For the for the second case, the mass >> For the for the second case, the mass diffusion, why can't we do something diffusion, why can't we do something diffusion, why can't we do something like sliding window to kind of like sliding window to kind of like sliding window to kind of Well, this is kind of sliding window, Well, this is kind of sliding window, Well, this is kind of sliding window, right? right? >> Without the kind of >> Without the kind of >> Without the kind of >> right. But like basically what you're >> right. But like basically what you're >> right. But like basically what you're suggesting is basically block diffusion, suggesting is basically block diffusion, suggesting is basically block diffusion, right? It's and like like what you're right? It's and like like what you're right? It's and like like what you're suggesting is actually semi-auto suggesting is actually semi-auto suggesting is actually semi-auto reggressive. I showed you already, reggressive. I showed you already, reggressive. I showed you already, right? Last class after class. So it's right? Last class after class. So it's right? Last class after class. So it's pretty much the same thing, right? It's pretty much the same thing, right? It's pretty much the same thing, right? It's pretty much just sliding window. um pretty much just sliding window. um pretty much just sliding window. um except like you don't really you don't except like you don't really you don't except like you don't really you don't really have a window like you don't really have a window like you don't really have a window like you don't change what you have generated but you change what you have generated but you change what you have generated but you could also just do sliding window that's could also just do sliding window that's could also just do sliding window that's like not against the rule at all. Yeah, like not against the rule at all. Yeah, like not against the rule at all. Yeah, that's totally fine and it's it should that's totally fine and it's it should that's totally fine and it's it should work the same. work the same. work the same. Anything else? Yep. Anything else? Yep.

  51. L12 @ 2457s

    Anything else? Yep. >> Can you overhand why mass diffusion is >> Can you overhand why mass diffusion is >> Can you overhand why mass diffusion is lower? lower? lower? >> Uh okay. Who wants to tell who who wants >> Uh okay. Who wants to tell who who wants >> Uh okay. Who wants to tell who who wants to answer this question? Why why is mass to answer this question? Why why is mass to answer this question? Why why is mass diffusion can be lower quality can have diffusion can be lower quality can have diffusion can be lower quality can have lower quality or like more difficult to lower quality or like more difficult to lower quality or like more difficult to train or more difficult to generate a train or more difficult to generate a train or more difficult to generate a high quality data? high quality data? high quality data? We kind of went over that We kind of went over that We kind of went over that the the example that you you you showed, the the example that you you you showed, the the example that you you you showed, right? Okay. Oh, yeah. Yeah. Yeah. I got right? Okay. Oh, yeah. Yeah. Yeah. I got right? Okay. Oh, yeah. Yeah. Yeah. I got them. >> Yeah. >> Yeah. Right. So, basically like auto >> Yeah. Right. So, basically like auto >> Yeah. Right. So, basically like auto reggressiveness is like a very good reggressiveness is like a very good reggressiveness is like a very good inductive bias for tax. So it's like inductive bias for tax. So it's like inductive bias for tax. So it's like very basically this is like the most very basically this is like the most very basically this is like the most naive way that you can it's the safest naive way that you can it's the safest naive way that you can it's the safest way in a sense that you can generate a way in a sense that you can generate a way in a sense that you can generate a sequence right so basically this is why sequence right so basically this is why sequence right so basically this is why yeah yeah but it doesn't mean that you you cannot but it doesn't mean that you you cannot

  52. L12 @ 2509s

    but it doesn't mean that you you cannot train a good enough diffusion model to train a good enough diffusion model to train a good enough diffusion model to blm that's not the case it's just that blm that's not the case it's just that blm that's not the case it's just that it's harder yeah cool okay so we have talked about we cool okay so we have talked about we actually kind of solved the all three actually kind of solved the all three actually kind of solved the all three with the previous thing and uh just so with the previous thing and uh just so with the previous thing and uh just so you guys know for the unmask or ordering you guys know for the unmask or ordering you guys know for the unmask or ordering thing. So basically people also have thing. So basically people also have thing. So basically people also have been developing a method to like been developing a method to like been developing a method to like basically determine basically determine basically determine the best uh unmasking order at inference the best uh unmasking order at inference the best uh unmasking order at inference time using like a pre-trained mass time using like a pre-trained mass time using like a pre-trained mass diffusion models. And if you guys are diffusion models. And if you guys are diffusion models. And if you guys are interested like feel free to Google. Um interested like feel free to Google. Um interested like feel free to Google. Um but basically what people have found is but basically what people have found is but basically what people have found is that even with like a like a fully like that even with like a like a fully like that even with like a like a fully like birectional uh mass diffusion model the birectional uh mass diffusion model the birectional uh mass diffusion model the most like optimal way to unmask is still most like optimal way to unmask is still most like optimal way to unmask is still follow like a like a semi-auto follow like a like a semi-auto follow like a like a semi-auto reggressive ordering. So like basically reggressive ordering. So like basically reggressive ordering. So like basically the the optimal that you're going to get

  53. L12 @ 2565s

    the the optimal that you're going to get the the optimal that you're going to get from whatever algorithm that they from whatever algorithm that they from whatever algorithm that they develop will look like semi-auto develop will look like semi-auto develop will look like semi-auto regggressive in a way. Yeah. So it's regggressive in a way. Yeah. So it's regggressive in a way. Yeah. So it's like pretty interesting that like you like pretty interesting that like you like pretty interesting that like you know that that that that that is like know that that that that that is like know that that that that that is like you you still need to sort of respect you you still need to sort of respect you you still need to sort of respect this um like inductive bias for for for this um like inductive bias for for for this um like inductive bias for for for text. Okay. So the last thing that you text. Okay. So the last thing that you text. Okay. So the last thing that you know we have left is the continuous like know we have left is the continuous like know we have left is the continuous like basically how we can build a multimodel basically how we can build a multimodel basically how we can build a multimodel diffusion right. So anyone want to take diffusion right. So anyone want to take diffusion right. So anyone want to take a guess this is actually very very a guess this is actually very very a guess this is actually very very straightforward. >> Yeah. image tokens like wea just the >> Yeah. image tokens like wea just the same way we wish. same way we wish. same way we wish. >> Yeah. >> Yeah. >> Yeah. That's a good but okay but I guess >> Yeah. That's a good but okay but I guess >> Yeah. That's a good but okay but I guess the question is like uh actually let's the question is like uh actually let's the question is like uh actually let's let's uh let's look at this. Uh so so let's uh let's look at this. Uh so so let's uh let's look at this. Uh so so this is not image and text but like this is not image and text but like

  54. L12 @ 2612s

    this is not image and text but like we're going to look at image and text we're going to look at image and text we're going to look at image and text later. But for those of you who are in later. But for those of you who are in later. But for those of you who are in compile I guess uh if you try to design compile I guess uh if you try to design compile I guess uh if you try to design a protein you need to design two things. a protein you need to design two things. a protein you need to design two things. one is the sequence of the the the the one is the sequence of the the the the one is the sequence of the the the the uh uh uh what is the name of the thing that the what is the name of the thing that the what is the name of the thing that the amino acid is that is that what it is? amino acid is that is that what it is? amino acid is that is that what it is? Oh, great. Thank you. I I I'm illiterate Oh, great. Thank you. I I I'm illiterate Oh, great. Thank you. I I I'm illiterate in in in bio. Anyway, and then the other in in in bio. Anyway, and then the other in in in bio. Anyway, and then the other thing that you need to design is like thing that you need to design is like thing that you need to design is like the the spatial structure of the of of the the spatial structure of the of of the the spatial structure of the of of of the protein, right? So, basically um of the protein, right? So, basically um of the protein, right? So, basically um the the sequence itself is discreet, but the the sequence itself is discreet, but the the sequence itself is discreet, but the spatial structure is continuous, the spatial structure is continuous, the spatial structure is continuous, right? So, how can we uh sort of right? So, how can we uh sort of right? So, how can we uh sort of co-generate these two things together? >> Yeah. >> What are we cogenerating? >> What are we cogenerating?

  55. L12 @ 2671s

    >> What are we cogenerating? >> So, we're co-generating something uh >> So, we're co-generating something uh >> So, we're co-generating something uh discreet and something uh continuous at discreet and something uh continuous at discreet and something uh continuous at the same time and they're kind of the same time and they're kind of the same time and they're kind of correlated. >> Yeah. Yeah. Yeah. Yeah. Literally >> Yeah. Yeah. Yeah. Yeah. Literally exactly that. So, basically what all you exactly that. So, basically what all you exactly that. So, basically what all you need to do is you need to build kind of need to do is you need to build kind of need to do is you need to build kind of two models that kind of like Yeah. two models that kind of like Yeah. two models that kind of like Yeah. There's like two models that takes each There's like two models that takes each There's like two models that takes each other as condition kind of and then for other as condition kind of and then for other as condition kind of and then for the for the continuous uh the thing you the for the continuous uh the thing you the for the continuous uh the thing you use flow matching loss for the discreing use flow matching loss for the discreing use flow matching loss for the discreing you use discrete flow matching loss you use discrete flow matching loss you use discrete flow matching loss that's it basically uh and then you can that's it basically uh and then you can that's it basically uh and then you can just like interle the basically just just like interle the basically just just like interle the basically just like oh let's d noiseise a little bit in like oh let's d noiseise a little bit in like oh let's d noiseise a little bit in the continuous space all right now let's the continuous space all right now let's the continuous space all right now let's uh you know generate something in the uh you know generate something in the uh you know generate something in the discrete and let's do that like alter discrete and let's do that like alter discrete and let's do that like alter like alternating between the two that's

  56. L12 @ 2715s

    like alternating between the two that's like alternating between the two that's pretty much that right and like pretty much that right and like pretty much that right and like basically And then if you do basically basically And then if you do basically basically And then if you do basically and and you can also do like you first and and you can also do like you first and and you can also do like you first generate the sequence fully using the generate the sequence fully using the generate the sequence fully using the discrete model uh using discrete for discrete model uh using discrete for discrete model uh using discrete for matching and then you condition on the matching and then you condition on the matching and then you condition on the generated sequence and then generated generated sequence and then generated generated sequence and then generated the structure then this is what they the structure then this is what they the structure then this is what they call forfolding in call forfolding in call forfolding in in compile and likewise you can also in compile and likewise you can also in compile and likewise you can also first generate the what we call first first generate the what we call first first generate the what we call first generate the structure then predict the generate the structure then predict the generate the structure then predict the sequence then it's called the inverse sequence then it's called the inverse sequence then it's called the inverse folding. So basically like you can just folding. So basically like you can just folding. So basically like you can just do or you can just alternate interle do or you can just alternate interle do or you can just alternate interle right that works too and um right that works too and um right that works too and um essentially what people have done essentially what people have done essentially what people have done actually let's look at this yeah so you actually let's look at this yeah so you actually let's look at this yeah so you can also do this for image and text as can also do this for image and text as can also do this for image and text as well right so what you can do is you can

  57. L12 @ 2763s

    well right so what you can do is you can well right so what you can do is you can uh have like just like I think some uh have like just like I think some uh have like just like I think some someone already said it right basically someone already said it right basically someone already said it right basically you can have text token and image tokens you can have text token and image tokens you can have text token and image tokens and then you just learn like a and then you just learn like a and then you just learn like a birectional transformer and they they birectional transformer and they they birectional transformer and they they kind of yeah you can just like kind of yeah you can just like kind of yeah you can just like co-generate your text tokens and D co-generate your text tokens and D co-generate your text tokens and D noiseis image token at the same time noiseis image token at the same time noiseis image token at the same time this kind of thing. So you can build you this kind of thing. So you can build you this kind of thing. So you can build you basically just build a birectional basically just build a birectional basically just build a birectional transformer to predict both uh and uh transformer to predict both uh and uh transformer to predict both uh and uh the for the text part you use edit flow the for the text part you use edit flow the for the text part you use edit flow loss or like insertion based flow loss loss or like insertion based flow loss loss or like insertion based flow loss and then for the image part you just use and then for the image part you just use and then for the image part you just use flow matching. All right any question? flow matching. All right any question? flow matching. All right any question? Yeah Yeah Yeah >> better than >> better than >> better than >> uh transfusion >> uh transfusion >> uh transfusion >> like the normal may even know if this >> like the normal may even know if this >> like the normal may even know if this >> I mean the problem is like this is a

  58. L12 @ 2810s

    >> I mean the problem is like this is a >> I mean the problem is like this is a research project right so it's like very research project right so it's like very research project right so it's like very difficult for it to compare with I don't difficult for it to compare with I don't difficult for it to compare with I don't know like like product I guess all this know like like product I guess all this know like like product I guess all this >> yeah but >> yeah but >> yeah but >> yeah I think they compared with >> yeah I think they compared with >> yeah I think they compared with uh I don't remember the name of the uh I don't remember the name of the uh I don't remember the name of the thing but bases things like transfusion thing but bases things like transfusion thing but bases things like transfusion in the same size and then this is like in the same size and then this is like in the same size and then this is like better basically and also I think better basically and also I think better basically and also I think another interesting thing that they uh another interesting thing that they uh another interesting thing that they uh showed in the paper is that like the the showed in the paper is that like the the showed in the paper is that like the the the tokens that it gets generated at the the tokens that it gets generated at the the tokens that it gets generated at the beginning. It's like oftentimes they'll beginning. It's like oftentimes they'll beginning. It's like oftentimes they'll generate like a lot of the most common generate like a lot of the most common generate like a lot of the most common tokens first like the something and then tokens first like the something and then tokens first like the something and then and then and then they generate like the and then and then they generate like the and then and then they generate like the most like important tokens and then and most like important tokens and then and most like important tokens and then and and then the the the image for example and then the the the image for example

  59. L12 @ 2857s

    and then the the the image for example like flower or like cat and then and like flower or like cat and then and like flower or like cat and then and then the the the the image are going to then the the the the image are going to then the the the the image are going to start to show like the flower silhouette start to show like the flower silhouette start to show like the flower silhouette and then once you have like more details and then once you have like more details and then once you have like more details about the flowers for example the color about the flowers for example the color about the flowers for example the color or I I guess color is also pretty pretty or I I guess color is also pretty pretty or I I guess color is also pretty pretty pretty primal I guess but like basically pretty primal I guess but like basically pretty primal I guess but like basically like what kind of flower or like what like what kind of flower or like what like what kind of flower or like what are the what kind of cat like what what are the what kind of cat like what what are the what kind of cat like what what is the what is it like the eye the the is the what is it like the eye the the is the what is it like the eye the the color of the eyes or something right color of the eyes or something right color of the eyes or something right then then like basically the details in then then like basically the details in then then like basically the details in the image and the text are kind of the image and the text are kind of the image and the text are kind of showing up at the same time so it's like showing up at the same time so it's like showing up at the same time so it's like kind of interesting yeah kind of interesting yeah kind of interesting yeah >> what's the point of like generating both >> what's the point of like generating both >> what's the point of like generating both text and like image at the same time

  60. L12 @ 2893s

    text and like image at the same time text and like image at the same time because like you start from nothing and because like you start from nothing and because like you start from nothing and then you ask the model to like generate then you ask the model to like generate then you ask the model to like generate random thing instead of like giving it a random thing instead of like giving it a random thing instead of like giving it a prompt to generate is it trying do like prompt to generate is it trying do like prompt to generate is it trying do like VQA. VQA. VQA. >> No, it's trying to generate based on >> No, it's trying to generate based on >> No, it's trying to generate based on prompt. So like the the the first few prompt. So like the the the first few prompt. So like the the the first few tokens are the prompts. tokens are the prompts. tokens are the prompts. >> Yeah. Yeah. Yeah. Yeah. So so yeah. So >> Yeah. Yeah. Yeah. Yeah. So so yeah. So >> Yeah. Yeah. Yeah. Yeah. So so yeah. So you you will have some prompts. You'll you you will have some prompts. You'll you you will have some prompts. You'll even have VQA, right? So you can read even have VQA, right? So you can read even have VQA, right? So you can read the image and also read prompts and the image and also read prompts and the image and also read prompts and stuff like that, right? So this is like stuff like that, right? So this is like stuff like that, right? So this is like what people call omni model uh so to what people call omni model uh so to what people call omni model uh so to speak. This is not omni yet but this is speak. This is not omni yet but this is speak. This is not omni yet but this is like kind of om like this is a towards like kind of om like this is a towards like kind of om like this is a towards the way to become omni but this is like the way to become omni but this is like

  61. L12 @ 2929s

    the way to become omni but this is like just like two two modalities I guess. just like two two modalities I guess. just like two two modalities I guess. Yeah. Yeah. Omni model just means that Yeah. Yeah. Omni model just means that Yeah. Yeah. Omni model just means that anything to anything. Yeah. All right. Cool. So, uh congratulations. All right. Cool. So, uh congratulations. you made it through this class. you made it through this class. you made it through this class. Now you're a diffusion expert. Now you're a diffusion expert. Now you're a diffusion expert. But uh basically yeah so at the But uh basically yeah so at the But uh basically yeah so at the beginning this is what we said we were beginning this is what we said we were beginning this is what we said we were going to learn you know. So hopefully going to learn you know. So hopefully going to learn you know. So hopefully you guys have learned you know enough you guys have learned you know enough you guys have learned you know enough knowledge to be you know comfortable knowledge to be you know comfortable knowledge to be you know comfortable with many of or all of these concepts. with many of or all of these concepts. with many of or all of these concepts. So like the intuition, the math, how to So like the intuition, the math, how to So like the intuition, the math, how to implement them, how to use GPU, uh you implement them, how to use GPU, uh you implement them, how to use GPU, uh you know, stuff like that. And uh just a know, stuff like that. And uh just a know, stuff like that. And uh just a couple of notes here. Basically, all the couple of notes here. Basically, all the couple of notes here. Basically, all the resources that we post for this class resources that we post for this class resources that we post for this class will remain available forever on the will remain available forever on the will remain available forever on the internet. So the recordings uh even if internet. So the recordings uh even if internet. So the recordings uh even if you cannot access the ponapto anymore, you cannot access the ponapto anymore,

  62. L12 @ 2990s

    you cannot access the ponapto anymore, it's still going to be on YouTube. Uh I it's still going to be on YouTube. Uh I it's still going to be on YouTube. Uh I kind of do some editing uh for the kind of do some editing uh for the kind of do some editing uh for the YouTube part so to to cut out some of YouTube part so to to cut out some of YouTube part so to to cut out some of the announcement parts and stuff like the announcement parts and stuff like the announcement parts and stuff like that so that you don't need to waste that so that you don't need to waste that so that you don't need to waste time. Um yeah lecture slides of course time. Um yeah lecture slides of course time. Um yeah lecture slides of course website will be there the homework will website will be there the homework will website will be there the homework will be there although I don't know if I want be there although I don't know if I want be there although I don't know if I want to share answer publicly if I want to re to share answer publicly if I want to re to share answer publicly if I want to re this class maybe this is not a good idea this class maybe this is not a good idea this class maybe this is not a good idea um but yeah and discord will just be um but yeah and discord will just be um but yeah and discord will just be there I guess all live probably not there I guess all live probably not there I guess all live probably not going to be as active but you know all going to be as active but you know all going to be as active but you know all the resources that we posted here is the resources that we posted here is the resources that we posted here is going to be there okay and now it is going to be there okay and now it is going to be there okay and now it is your turn to take what you've learned your turn to take what you've learned

  63. L12 @ 3025s

    your turn to take what you've learned into the real world so I will highly into the real world so I will highly into the real world so I will highly recommend if you are proud of what recommend if you are proud of what recommend if you are proud of what you've done especially in your you know you've done especially in your you know you've done especially in your you know homeworks three and four uh you can open homeworks three and four uh you can open homeworks three and four uh you can open source your code by uh making your source your code by uh making your source your code by uh making your GitHub repo public basically this is GitHub repo public basically this is GitHub repo public basically this is like you know one way to help the world like you know one way to help the world like you know one way to help the world or your AI agent to become a better one or your AI agent to become a better one or your AI agent to become a better one I guess um yeah another thing or I guess um yeah another thing or I guess um yeah another thing or actually I don't know because your thing actually I don't know because your thing actually I don't know because your thing is also generated by AI agent maybe is also generated by AI agent maybe is also generated by AI agent maybe anyway um but but it's a bug free anyway um but but it's a bug free anyway um but but it's a bug free version anyway another thing that you version anyway another thing that you version anyway another thing that you can consider doing is to turn your can consider doing is to turn your can consider doing is to turn your homework into a real research project so homework into a real research project so homework into a real research project so either it's like a full like numerous either it's like a full like numerous either it's like a full like numerous paper or like a new workshop paper or paper or like a new workshop paper or paper or like a new workshop paper or something like that. I think it could be

  64. L12 @ 3065s

    something like that. I think it could be something like that. I think it could be a good experience if you guys are a good experience if you guys are a good experience if you guys are interested in getting into research. And interested in getting into research. And interested in getting into research. And one thing that you can do at the poster one thing that you can do at the poster one thing that you can do at the poster session is that like basically you can session is that like basically you can session is that like basically you can go around and see if there's any people go around and see if there's any people go around and see if there's any people who are like you know doing the same who are like you know doing the same who are like you know doing the same thing as you and uh if you're interested thing as you and uh if you're interested thing as you and uh if you're interested you can like basically do joint effort. you can like basically do joint effort. you can like basically do joint effort. This way like nobody needs to do like This way like nobody needs to do like This way like nobody needs to do like like a lot of work and then you can like a lot of work and then you can like a lot of work and then you can still get like a relatively large still get like a relatively large still get like a relatively large project uh together if you guys are project uh together if you guys are project uh together if you guys are solving the same kind of problem and uh solving the same kind of problem and uh solving the same kind of problem and uh obviously uh this is what I hope that obviously uh this is what I hope that obviously uh this is what I hope that you could you know use what you have you could you know use what you have you could you know use what you have learned in this class in your own learned in this class in your own learned in this class in your own research slash you know when you go to research slash you know when you go to

  65. L12 @ 3099s

    research slash you know when you go to the AGI company you know make the AGI a the AGI company you know make the AGI a the AGI company you know make the AGI a diffusion model please we don't want to diffusion model please we don't want to diffusion model please we don't want to lose to the to the LM AGI guys you know lose to the to the LM AGI guys you know lose to the to the LM AGI guys you know just yeah yeah yeah but let let's hope just yeah yeah yeah but let let's hope just yeah yeah yeah but let let's hope that that it happens that that it happens that that it happens Okay. But uh yeah, last chance for Okay. But uh yeah, last chance for Okay. But uh yeah, last chance for questions. Is there any remaining questions. Is there any remaining questions. Is there any remaining questions that we have questions that we have questions that we have about anything? Yeah. about anything? Yeah. about anything? Yeah. >> Block diffusion. >> Block diffusion. >> Block diffusion. >> What? What did you say? >> What? What did you say? >> What? What did you say? >> Block diffusion. >> Block diffusion. >> Block diffusion. Did didn't we just go >> Block diffusion. Did didn't we just go >> Block diffusion. Did didn't we just go through that? through that? through that? >> Yeah. >> Yeah. >> Oh, okay. Yeah. >> Question was that we said that it's >> Question was that we said that it's going to be we can have like variable going to be we can have like variable going to be we can have like variable length of block diffusion, but the length of block diffusion, but the length of block diffusion, but the length of the block itself is fixed. length of the block itself is fixed. length of the block itself is fixed. Does it mean that it's always going to Does it mean that it's always going to Does it mean that it's always going to be multiple? be multiple? be multiple? >> No, because you can discard >> No, because you can discard >> No, because you can discard >> padding tokens and tokens after the OS, >> padding tokens and tokens after the OS,

  66. L12 @ 3145s

    >> padding tokens and tokens after the OS, right? right? >> Yeah. >> Yeah. >> Uh so one thing I implemented for >> Uh so one thing I implemented for >> Uh so one thing I implemented for homework 3 and like thing that took the homework 3 and like thing that took the homework 3 and like thing that took the longest time was generating the public longest time was generating the public longest time was generating the public data that that took like 5 hours data that that took like 5 hours data that that took like 5 hours >> which is like training probably took >> which is like training probably took >> which is like training probably took just two hours. just two hours. just two hours. >> So so one of the things I noticed is >> So so one of the things I noticed is >> So so one of the things I noticed is like obviously your generated data set like obviously your generated data set like obviously your generated data set is constrained by how good your 1 RF is constrained by how good your 1 RF is constrained by how good your 1 RF model is. Mhm. model is. Mhm. model is. Mhm. >> If that gets like a bad f then your >> If that gets like a bad f then your >> If that gets like a bad f then your generated sample is also pretty bad. generated sample is also pretty bad. generated sample is also pretty bad. >> So I was just wondering why do we do it >> So I was just wondering why do we do it >> So I was just wondering why do we do it this way where we sample noise and then this way where we sample noise and then this way where we sample noise and then generate an image. Can't we like take generate an image. Can't we like take generate an image. Can't we like take images from cell and it's all backwards? images from cell and it's all backwards? images from cell and it's all backwards? >> Of course you can definitely do that and >> Of course you can definitely do that and >> Of course you can definitely do that and I thought that's what they did in

  67. L12 @ 3186s

    I thought that's what they did in I thought that's what they did in refflow but maybe not. If it's not then refflow but maybe not. If it's not then refflow but maybe not. If it's not then this is a good way to improve it right. this is a good way to improve it right. this is a good way to improve it right. Obviously because like you obviously Obviously because like you obviously Obviously because like you obviously want to learn from real data, right? want to learn from real data, right? want to learn from real data, right? That that is like and also actually That that is like and also actually That that is like and also actually people have found that like if you train people have found that like if you train people have found that like if you train your diffusion model on generated on your diffusion model on generated on your diffusion model on generated on images that generate by diffusion is images that generate by diffusion is images that generate by diffusion is actually going to degrade your diffusion actually going to degrade your diffusion actually going to degrade your diffusion model's quality. Yeah. So yeah, if if model's quality. Yeah. So yeah, if if model's quality. Yeah. So yeah, if if the I thought I thought getting the the I thought I thought getting the the I thought I thought getting the noise correspondence to the real data is noise correspondence to the real data is noise correspondence to the real data is what they did in reflow. But if not then what they did in reflow. But if not then what they did in reflow. But if not then this is a good way to improve it. this is a good way to improve it. this is a good way to improve it. >> I think they said that yeah they do do >> I think they said that yeah they do do >> I think they said that yeah they do do it the forward way. That's why like a it the forward way. That's why like a it the forward way. That's why like a three RF model actually performs worse three RF model actually performs worse three RF model actually performs worse than a 2 RF model because the only

  68. L12 @ 3222s

    than a 2 RF model because the only than a 2 RF model because the only errors accumulate. errors accumulate. errors accumulate. >> Yeah. No, that's your homework. Four. >> Yeah. No, that's your homework. Four. >> Yeah. No, that's your homework. Four. Nice. Okay. Yeah. Yeah. Yeah. Totally. I Nice. Okay. Yeah. Yeah. Yeah. Totally. I Nice. Okay. Yeah. Yeah. Yeah. Totally. I think that that make make total sense. think that that make make total sense. think that that make make total sense. Yeah. You can also add some like Yeah. You can also add some like Yeah. You can also add some like Yeah. Basically, you you can I don't I Yeah. Basically, you you can I don't I Yeah. Basically, you you can I don't I don't know if like there's going to be don't know if like there's going to be don't know if like there's going to be any issues because like the the like the any issues because like the the like the any issues because like the the like the like the the X0 that you get from a like the the X0 that you get from a like the the X0 that you get from a model may or may not be well may or may model may or may not be well may or may model may or may not be well may or may not be on the gausian manifold. like you not be on the gausian manifold. like you not be on the gausian manifold. like you may need some regularization there. may need some regularization there. may need some regularization there. >> It's like easier to be on the Gausian >> It's like easier to be on the Gausian >> It's like easier to be on the Gausian than that's very true. That's very true. than that's very true. That's very true. than that's very true. That's very true. Yeah. Yeah. So So I'm just saying that Yeah. Yeah. So So I'm just saying that Yeah. Yeah. So So I'm just saying that like maybe you need some regularization like maybe you need some regularization like maybe you need some regularization but but but in general I think that's a better idea in general I think that's a better idea

  69. L12 @ 3259s

    in general I think that's a better idea than the normal the the the way that than the normal the the the way that than the normal the the the way that they they did. Okay. Any other question? they they did. Okay. Any other question? they they did. Okay. Any other question? Anything? Anything? Anything? >> Yeah. >> Yeah. >> You were hoping that diffusion promotion >> You were hoping that diffusion promotion >> You were hoping that diffusion promotion may overtake auto. may overtake auto. may overtake auto. >> I don't know what I said. No, I said >> I don't know what I said. No, I said >> I don't know what I said. No, I said well that's kind of what I said. I guess well that's kind of what I said. I guess well that's kind of what I said. I guess this was what I said but yeah continue. this was what I said but yeah continue. this was what I said but yeah continue. >> One of the strongest parts about >> One of the strongest parts about >> One of the strongest parts about anything that's generally auto testing anything that's generally auto testing anything that's generally auto testing right now is that RL part that you right now is that RL part that you right now is that RL part that you mentioned that mentioned that mentioned that >> what are like some active works that >> what are like some active works that >> what are like some active works that deal deal deal maybe more so flow matching because maybe more so flow matching because maybe more so flow matching because diffusion can apply RL to some extent. diffusion can apply RL to some extent. diffusion can apply RL to some extent. >> Are there like some recent or >> Are there like some recent or >> Are there like some recent or interesting works that you think apply interesting works that you think apply interesting works that you think apply RL through flow matching or even methods RL through flow matching or even methods RL through flow matching or even methods you spoke about today that is discreet you spoke about today that is discreet you spoke about today that is discreet diffusion discrete? diffusion discrete? diffusion discrete? >> Yeah. Yeah. So actually there multiple

  70. L12 @ 3303s

    >> Yeah. Yeah. So actually there multiple >> Yeah. Yeah. So actually there multiple uh so the most famous one in the uh so the most famous one in the uh so the most famous one in the robotics world is probably like FO I robotics world is probably like FO I robotics world is probably like FO I guess flow policy guess flow policy guess flow policy >> optimization all yeah yeah yeah good go >> optimization all yeah yeah yeah good go >> optimization all yeah yeah yeah good go I don't really know FO uh but um well I don't really know FO uh but um well I don't really know FO uh but um well this yeah and you know my opinion about this yeah and you know my opinion about this yeah and you know my opinion about it um so there's another um uh line of it um so there's another um uh line of it um so there's another um uh line of work uh that is called adjoin matching work uh that is called adjoin matching work uh that is called adjoin matching or adjoin sampling so basically what or adjoin sampling so basically what or adjoin sampling so basically what they did is that um they sort of like uh they did is that um they sort of like uh they did is that um they sort of like uh it's it's like very very similar to like it's it's like very very similar to like it's it's like very very similar to like like a gpo type of thing where um you like a gpo type of thing where um you like a gpo type of thing where um you you you have a bunch of like buffer the you you have a bunch of like buffer the you you have a bunch of like buffer the data and then you kind of like calculate data and then you kind of like calculate data and then you kind of like calculate the reward with those buffer data and the reward with those buffer data and the reward with those buffer data and then you figure out like how much then you figure out like how much then you figure out like how much gradient you should apply from those gradient you should apply from those

  71. L12 @ 3349s

    gradient you should apply from those buffer data to your current time to one buffer data to your current time to one buffer data to your current time to one of the time that you sampled. Um so this of the time that you sampled. Um so this of the time that you sampled. Um so this is like also one of the ways to do that is like also one of the ways to do that is like also one of the ways to do that like a drawing sampling addin matching. like a drawing sampling addin matching. like a drawing sampling addin matching. So this is like very very principled and So this is like very very principled and So this is like very very principled and they kind of derived from like the the they kind of derived from like the the they kind of derived from like the the stoastic control theory. Uh but it's stoastic control theory. Uh but it's stoastic control theory. Uh but it's like it's pretty like the mass is a like it's pretty like the mass is a like it's pretty like the mass is a little bit more complicated and it's little bit more complicated and it's little bit more complicated and it's also a little bit more complicated to also a little bit more complicated to also a little bit more complicated to implement. This is why and and and tune implement. This is why and and and tune implement. This is why and and and tune I guess. So but but they have like I guess. So but but they have like I guess. So but but they have like strong connection with RL. So this is strong connection with RL. So this is strong connection with RL. So this is why but but this is like a good line of why but but this is like a good line of why but but this is like a good line of work that people have done and uh they work that people have done and uh they work that people have done and uh they you you need you mainly need like an SDE you you need you mainly need like an SDE you you need you mainly need like an SDE in that framework but you can turn you

  72. L12 @ 3385s

    in that framework but you can turn you in that framework but you can turn you can essentially turn an S OD like a flow can essentially turn an S OD like a flow can essentially turn an S OD like a flow OD into a SD by yeah basically just like OD into a SD by yeah basically just like OD into a SD by yeah basically just like adding adding some noise at every time adding adding some noise at every time adding adding some noise at every time step and then you can do it. So this is step and then you can do it. So this is step and then you can do it. So this is also like kind of how uh like basically also like kind of how uh like basically also like kind of how uh like basically the like basically the uh like like the the like basically the uh like like the the like basically the uh like like the the the the trendy way of doing um you the the the trendy way of doing um you the the the trendy way of doing um you know RL fine-tuning for flow models is know RL fine-tuning for flow models is know RL fine-tuning for flow models is kind of just you add some noise in the kind of just you add some noise in the kind of just you add some noise in the middle and then it becomes soastic and middle and then it becomes soastic and middle and then it becomes soastic and then you use a gausian proxy as the as then you use a gausian proxy as the as then you use a gausian proxy as the as the lo likelihood and there's another the lo likelihood and there's another the lo likelihood and there's another thing that like we like by we I mean thing that like we like by we I mean thing that like we like by we I mean like my collaborator like me and my like my collaborator like me and my like my collaborator like me and my collaborator are trying to do is collaborator are trying to do is collaborator are trying to do is basically is there any other ways that basically is there any other ways that

  73. L12 @ 3426s

    basically is there any other ways that we can do it besides turning into a SDE we can do it besides turning into a SDE we can do it besides turning into a SDE um yeah so like let's hope we can get um yeah so like let's hope we can get um yeah so like let's hope we can get this done by nearest but uh yeah but this done by nearest but uh yeah but this done by nearest but uh yeah but this is like kind that's this is like kind that's this is like kind that's our prior work on the fast likelihood uh our prior work on the fast likelihood uh our prior work on the fast likelihood uh estimation thing. Yeah. Yeah. So so that estimation thing. Yeah. Yeah. So so that estimation thing. Yeah. Yeah. So so that that may or may not be able to get done. that may or may not be able to get done. that may or may not be able to get done. Yeah. Yeah. Okay. Uh Okay. Uh Okay. Uh any other question? Yeah, >> we learned a lot about the onestep >> we learned a lot about the onestep >> we learned a lot about the onestep generation for images being done for generation for images being done for generation for images being done for like onestep like onestep like onestep >> text diffusion. >> text diffusion. >> text diffusion. >> Yes. Wow, that's such a great question. >> Yes. Wow, that's such a great question. >> Yes. Wow, that's such a great question. Sorry. This is a very great question Sorry. This is a very great question Sorry. This is a very great question actually like three days ago. Um this is actually like three days ago. Um this is actually like three days ago. Um this is super exciting actually. So three days super exciting actually. So three days super exciting actually. So three days ago people on Twitter uh have posted a ago people on Twitter uh have posted a ago people on Twitter uh have posted a new paper called discrete flow maps. And new paper called discrete flow maps. And new paper called discrete flow maps. And essentially you just sort of like uh essentially you just sort of like uh

  74. L12 @ 3478s

    essentially you just sort of like uh they sort of extend the flow map uh uh they sort of extend the flow map uh uh they sort of extend the flow map uh uh what you call framework into the what you call framework into the what you call framework into the discrete case and then it has all the discrete case and then it has all the discrete case and then it has all the connection that we learned about today. connection that we learned about today. connection that we learned about today. And basically what they did is that like And basically what they did is that like And basically what they did is that like you can kind of convert everything into you can kind of convert everything into you can kind of convert everything into something cross entropy again and it's something cross entropy again and it's something cross entropy again and it's like super super interesting. I highly like super super interesting. I highly like super super interesting. I highly recommend everyone to read it. Uh, and I recommend everyone to read it. Uh, and I recommend everyone to read it. Uh, and I don't know if I can say this, but I hope don't know if I can say this, but I hope don't know if I can say this, but I hope at by the time that I post this online at by the time that I post this online at by the time that I post this online that it's already out there, but Nick that it's already out there, but Nick that it's already out there, but Nick Nick Buffy, uh, the professor Nick Buffy Nick Buffy, uh, the professor Nick Buffy Nick Buffy, uh, the professor Nick Buffy from MLDD told me that he's working on from MLDD told me that he's working on from MLDD told me that he's working on something that's like super duper something that's like super duper something that's like super duper related to that as well. And they related to that as well. And they related to that as well. And they supposed to post it online this week. supposed to post it online this week. supposed to post it online this week. Uh, so let's see if he actually post it Uh, so let's see if he actually post it

  75. L12 @ 3517s

    Uh, so let's see if he actually post it online this week. Uh, but yeah, I don't online this week. Uh, but yeah, I don't online this week. Uh, but yeah, I don't actually know how how he did it, but actually know how how he did it, but actually know how how he did it, but apparently we have something coming out apparently we have something coming out apparently we have something coming out of CMU that is like related to that as of CMU that is like related to that as of CMU that is like related to that as well. Yeah. Any other questions? But well. Yeah. Any other questions? But well. Yeah. Any other questions? But yeah, great question. I forgot to yeah, great question. I forgot to yeah, great question. I forgot to mention that this is like I meant to mention that this is like I meant to mention that this is like I meant to make a slide for this but I forgot. make a slide for this but I forgot. make a slide for this but I forgot. Anyway, any other things? >> Recent drifting model. >> Recent drifting model. >> I think the drifting model is super >> I think the drifting model is super >> I think the drifting model is super novel. I think this is like an novel. I think this is like an novel. I think this is like an interesting way um to think about J interesting way um to think about J interesting way um to think about J modeling. But I don't think this is like modeling. But I don't think this is like modeling. But I don't think this is like that closely related to diffusion or that closely related to diffusion or that closely related to diffusion or anything that we saw. I think it's more anything that we saw. I think it's more anything that we saw. I think it's more related to I think um the luma talk related to I think um the luma talk related to I think um the luma talk actually talked about it, right? is kind actually talked about it, right? is kind actually talked about it, right? is kind of similar to uh one of the other of similar to uh one of the other

  76. L12 @ 3559s

    of similar to uh one of the other previous uh framework and equilibrium previous uh framework and equilibrium previous uh framework and equilibrium model. So equilibrium model is something model. So equilibrium model is something model. So equilibrium model is something that also CMU developed actually is from that also CMU developed actually is from that also CMU developed actually is from my lab. Um yeah but basically what you my lab. Um yeah but basically what you my lab. Um yeah but basically what you do is like you kind of treat um do is like you kind of treat um do is like you kind of treat um basically you just kind of have like a basically you just kind of have like a basically you just kind of have like a recurrent of model like oh actually this recurrent of model like oh actually this recurrent of model like oh actually this is not even equilibrium model I guess is not even equilibrium model I guess is not even equilibrium model I guess this is like equilibrium in a different this is like equilibrium in a different this is like equilibrium in a different dimension but like basically you're dimension but like basically you're dimension but like basically you're trying to reach some sort of equilibrium trying to reach some sort of equilibrium trying to reach some sort of equilibrium in your training. Yeah, this I guess in your training. Yeah, this I guess in your training. Yeah, this I guess this is not even equilibrium model. Um, this is not even equilibrium model. Um, this is not even equilibrium model. Um, but yeah, I think this is like super but yeah, I think this is like super but yeah, I think this is like super novel. Not entirely sure if this is like novel. Not entirely sure if this is like novel. Not entirely sure if this is like super related to what we see, but I super related to what we see, but I super related to what we see, but I think the underlying uh, you know, vibe think the underlying uh, you know, vibe think the underlying uh, you know, vibe or the underlying uh, you know, uh, math or the underlying uh, you know, uh, math

  77. L12 @ 3602s

    or the underlying uh, you know, uh, math is a little bit related, but like this is a little bit related, but like this is a little bit related, but like this is definitely something new. I think is definitely something new. I think is definitely something new. I think drift model. Yeah. drift model. Yeah. drift model. Yeah. or or or something coming from some or or or something coming from some or or or something coming from some other 2015 paper as people have been on other 2015 paper as people have been on other 2015 paper as people have been on Twitter have you know have discussed. Twitter have you know have discussed. Twitter have you know have discussed. Yeah. Yeah. Any other thing? Nothing. Nothing. All right. Cool. Final reminder, come to All right. Cool. Final reminder, come to All right. Cool. Final reminder, come to poster sessions and everything. All poster sessions and everything. All poster sessions and everything. All right. Thanks everyone for joining me in right. Thanks everyone for joining me in right. Thanks everyone for joining me in this class. Now go explore the world. this class. Now go explore the world. this class. Now go explore the world. All right. That's it. All right. That's it. All right. That's it. Okay. Thank you. Thank you.

  78. L5 @ 2s

    So last time we learned about how So last time we learned about how diffusion turn noise into data by diffusion turn noise into data by diffusion turn noise into data by gradually adding noise and gradually gradually adding noise and gradually gradually adding noise and gradually denoising. denoising. denoising. H and then we learned another family of H and then we learned another family of H and then we learned another family of model called a scorebased models which model called a scorebased models which model called a scorebased models which is uh doing very very similar things and is uh doing very very similar things and is uh doing very very similar things and it turns out that they're they're pretty it turns out that they're they're pretty it turns out that they're they're pretty much the same. Um if you especially uh much the same. Um if you especially uh much the same. Um if you especially uh if you scale the number of time steps to if you scale the number of time steps to if you scale the number of time steps to infinite and then they all they both infinite and then they all they both infinite and then they all they both becomes this thing called stochastic becomes this thing called stochastic becomes this thing called stochastic process a continuous time stoastic process a continuous time stoastic process a continuous time stoastic process which can be sort of formulated process which can be sort of formulated process which can be sort of formulated into stoastic differential equations. into stoastic differential equations. into stoastic differential equations. All right. So and if you have a forward All right. So and if you have a forward All right. So and if you have a forward SDE uh then you actually directly have a SDE uh then you actually directly have a SDE uh then you actually directly have a formula to uh tell you what your reverse formula to uh tell you what your reverse formula to uh tell you what your reverse process should be which is also defined process should be which is also defined process should be which is also defined by an SDE with this score function and by an SDE with this score function and

  79. L5 @ 52s

    by an SDE with this score function and then we can just do score matching to then we can just do score matching to then we can just do score matching to learn uh how to solve the uh the learn uh how to solve the uh the learn uh how to solve the uh the reversed SDE. reversed SDE. reversed SDE. All right. So, that was like the last All right. So, that was like the last All right. So, that was like the last question question question that we had in class. It's like, is that we had in class. It's like, is that we had in class. It's like, is there a simpler way to do this? And I there a simpler way to do this? And I there a simpler way to do this? And I think someone after they took the quiz, think someone after they took the quiz, think someone after they took the quiz, they were already be like, "Oh, I I feel they were already be like, "Oh, I I feel they were already be like, "Oh, I I feel like I feel like there should be an like I feel like there should be an like I feel like there should be an easier way." Um, yeah. So, this is what easier way." Um, yeah. So, this is what easier way." Um, yeah. So, this is what we're going to learn today. Okay. So, uh we're going to learn today. Okay. So, uh we're going to learn today. Okay. So, uh let's think about this question. Let's let's think about this question. Let's let's think about this question. Let's say we're given a data point. So, we we say we're given a data point. So, we we say we're given a data point. So, we we only have like one image, right? What only have like one image, right? What only have like one image, right? What would be the simplest way to construct a would be the simplest way to construct a would be the simplest way to construct a trajectory from noise to this data trajectory from noise to this data trajectory from noise to this data point? Yeah, point? Yeah, point? Yeah, linear combination. linear combination. linear combination. That's exactly correct. Okay. Why don't That's exactly correct. Okay. Why don't

  80. L5 @ 101s

    That's exactly correct. Okay. Why don't we just do linear interpolation of the we just do linear interpolation of the we just do linear interpolation of the two things, right? Um so exact how two things, right? Um so exact how two things, right? Um so exact how exactly you can do it is basically um exactly you can do it is basically um exactly you can do it is basically um say like bas you just like at the say like bas you just like at the say like bas you just like at the beginning of time you'll have like it's beginning of time you'll have like it's beginning of time you'll have like it's so this noise is fixed and then you have so this noise is fixed and then you have so this noise is fixed and then you have like a large portion of noise mixed with like a large portion of noise mixed with like a large portion of noise mixed with some portion of data and then you get a some portion of data and then you get a some portion of data and then you get a very noisy data and then as you go on very noisy data and then as you go on very noisy data and then as you go on the portion of the noise become fewer the portion of the noise become fewer the portion of the noise become fewer like less and less and then the portion like less and less and then the portion like less and less and then the portion of data become more and more uh and then of data become more and more uh and then of data become more and more uh and then you just construct a chain of you just construct a chain of you just construct a chain of the very noisy to slightly no to the very noisy to slightly no to the very noisy to slightly no to slightly less noisy to very very clean slightly less noisy to very very clean slightly less noisy to very very clean data. Right? So you could just construct data. Right? So you could just construct data. Right? So you could just construct this trajectory in this way. Um yeah so this trajectory in this way. Um yeah so

  81. L5 @ 146s

    this trajectory in this way. Um yeah so remember the same noise same data. remember the same noise same data. remember the same noise same data. Then learning the transformation along Then learning the transformation along Then learning the transformation along this linear trajectory is super easy, this linear trajectory is super easy, this linear trajectory is super easy, right? Because literally what we can do right? Because literally what we can do right? Because literally what we can do is say now we want to uh now we want to is say now we want to uh now we want to is say now we want to uh now we want to um transfer transform from uh time 25% um transfer transform from uh time 25% um transfer transform from uh time 25% to time 50%. uh then what can we do is to time 50%. uh then what can we do is to time 50%. uh then what can we do is we can we literally just have you just we can we literally just have you just we can we literally just have you just need to add 25% more of data and then need to add 25% more of data and then need to add 25% more of data and then minus 25% minus 25% minus 25% uh of of the noise right um so basically uh of of the noise right um so basically uh of of the noise right um so basically and because the delta t is the same so and because the delta t is the same so and because the delta t is the same so you can write it in this way basically you can write it in this way basically you can write it in this way basically just like the just like the just like the uh the the noisy sample at time uh 0.5 uh the the noisy sample at time uh 0.5 uh the the noisy sample at time uh 0.5 is equal to uh the noisy sample at time is equal to uh the noisy sample at time is equal to uh the noisy sample at time 0.25 plus how much time you want to 0.25 plus how much time you want to 0.25 plus how much time you want to progress. So like 0.25 time um times uh

  82. L5 @ 208s

    progress. So like 0.25 time um times uh progress. So like 0.25 time um times uh data minus noise, right? Uh so basically data minus noise, right? Uh so basically data minus noise, right? Uh so basically uh generalizing it into t and delta t. uh generalizing it into t and delta t. uh generalizing it into t and delta t. This is like what you should get. And uh This is like what you should get. And uh This is like what you should get. And uh literally when the delta t goes to zero literally when the delta t goes to zero literally when the delta t goes to zero then you will have this velocity thing then you will have this velocity thing then you will have this velocity thing right. Basically this is just like tells right. Basically this is just like tells right. Basically this is just like tells the velocity thing. just tells you the velocity thing. just tells you the velocity thing. just tells you infinite testimony infinite testimony infinite testimony uh if you take infinite testimony small uh if you take infinite testimony small uh if you take infinite testimony small steps like what is the direction that steps like what is the direction that steps like what is the direction that you should go and no matter what t you should go and no matter what t you should go and no matter what t you're at no matter which time step you're at no matter which time step you're at no matter which time step you're at you're always going like the you're at you're always going like the you're at you're always going like the same speed same direction right so it same speed same direction right so it same speed same direction right so it just like always going to be the just like always going to be the just like always going to be the difference between the data and noise difference between the data and noise difference between the data and noise which is x1 minus minus x0 which is x1 minus minus x0 which is x1 minus minus x0 okay any questions okay any questions okay any questions pretty straightforward right okay So pretty straightforward right okay So pretty straightforward right okay So this is literally what we need to learn,

  83. L5 @ 260s

    this is literally what we need to learn, this is literally what we need to learn, right? This is like literally the only right? This is like literally the only right? This is like literally the only thing that that's stopping us from thing that that's stopping us from thing that that's stopping us from directly getting from noise to data. So directly getting from noise to data. So directly getting from noise to data. So basically the simplest way essentially basically the simplest way essentially basically the simplest way essentially to transform noise straight into data to transform noise straight into data to transform noise straight into data will be at training time. will be at training time. will be at training time. You first sample a noise and then you You first sample a noise and then you You first sample a noise and then you sample a data and then you uniformly sample a data and then you uniformly sample a data and then you uniformly sample a time step and then you compute sample a time step and then you compute sample a time step and then you compute a noisy sample by doing this linear a noisy sample by doing this linear a noisy sample by doing this linear interpolation interpolation interpolation and then you compute the velocity which and then you compute the velocity which and then you compute the velocity which is x1 - x0 just data minus noise and is x1 - x0 just data minus noise and is x1 - x0 just data minus noise and then you learn a then you learn a then you learn a to match this velocity to match this velocity to match this velocity and at sampling time after you learn and at sampling time after you learn and at sampling time after you learn this literally you can just start from t this literally you can just start from t this literally you can just start from t equals z and then sample a noise at time equals z and then sample a noise at time equals z and then sample a noise at time zero zero zero and then progress through the chain. and then progress through the chain. and then progress through the chain. Right? How do you progress through the Right? How do you progress through the

  84. L5 @ 317s

    Right? How do you progress through the chain? You just compute the displacement chain? You just compute the displacement chain? You just compute the displacement for x and then directly apply the for x and then directly apply the for x and then directly apply the displacement to your x to progress on displacement to your x to progress on displacement to your x to progress on the chain and then also progress in time the chain and then also progress in time the chain and then also progress in time steps and at the end just output x1. steps and at the end just output x1. steps and at the end just output x1. Okay. Question. Okay. Question. Okay. Question. somewhere and I'm concerned that this noise has no and I'm concerned that this noise has no semantic meaning. So would it get like semantic meaning. So would it get like semantic meaning. So would it get like like there's no reason why a particular like there's no reason why a particular like there's no reason why a particular gausian noise shouldn't amount to the gausian noise shouldn't amount to the gausian noise shouldn't amount to the rather than the house? rather than the house? rather than the house? >> Cool question. We're going to we're >> Cool question. We're going to we're >> Cool question. We're going to we're going to learn that today. Uh first of going to learn that today. Uh first of going to learn that today. Uh first of all, we're not trying to get any all, we're not trying to get any all, we're not trying to get any semantic meaning here. We're just trying semantic meaning here. We're just trying semantic meaning here. We're just trying to transform noise into data. But second to transform noise into data. But second to transform noise into data. But second of all, uh I think we're going to like of all, uh I think we're going to like of all, uh I think we're going to like answer your question um later today. answer your question um later today. answer your question um later today. Okay, any other questions? Ah, now you have flow matching. This is Ah, now you have flow matching. This is full matching. Class is over. Hey, full matching. Class is over. Hey,

  85. L5 @ 386s

    full matching. Class is over. Hey, no, not really. Okay, fine. Uh but no, not really. Okay, fine. Uh but no, not really. Okay, fine. Uh but basically the the question that we're basically the the question that we're basically the the question that we're gonna ask is that like this is so gonna ask is that like this is so gonna ask is that like this is so simple, right? Like no math, no nothing. simple, right? Like no math, no nothing. simple, right? Like no math, no nothing. Like why do we like why do we need all Like why do we like why do we need all Like why do we like why do we need all those previous diffusion math like those previous diffusion math like those previous diffusion math like stoastic differential equations? Why do stoastic differential equations? Why do stoastic differential equations? Why do we need this? Like also why is this even we need this? Like also why is this even we need this? Like also why is this even valid? Why why why is this the same? Why valid? Why why why is this the same? Why valid? Why why why is this the same? Why can we even do this? Right? If we can can we even do this? Right? If we can can we even do this? Right? If we can just do this, why do we need to do all just do this, why do we need to do all just do this, why do we need to do all the math? Uh well, turns out that thing the math? Uh well, turns out that thing the math? Uh well, turns out that thing also come from math. So we're going to also come from math. So we're going to also come from math. So we're going to learn about the math now. And to understand why this is a proper And to understand why this is a proper um probabilistic gener model we need to um probabilistic gener model we need to um probabilistic gener model we need to go back in time into the time when go back in time into the time when go back in time into the time when people start to de develop this thing. people start to de develop this thing.

  86. L5 @ 430s

    people start to de develop this thing. Okay. So basically back in 2018 there Okay. So basically back in 2018 there Okay. So basically back in 2018 there this like pretty seminal paper um uh this like pretty seminal paper um uh this like pretty seminal paper um uh called continuous normalizing or or like called continuous normalizing or or like called continuous normalizing or or like neuro neuroordinary differential neuro neuroordinary differential neuro neuroordinary differential equation and then basically um this guy equation and then basically um this guy equation and then basically um this guy Ricky who was my uh meta intermentor Ricky who was my uh meta intermentor Ricky who was my uh meta intermentor actually super legendary guy but actually super legendary guy but actually super legendary guy but basically he was like um proposed this basically he was like um proposed this basically he was like um proposed this thing called continuous normalizing thing called continuous normalizing thing called continuous normalizing flows and it's basically just like uh flows and it's basically just like uh flows and it's basically just like uh formulated a general model such that formulated a general model such that formulated a general model such that transform uh the samples from an initial transform uh the samples from an initial transform uh the samples from an initial distribution such as like a gausian to a distribution such as like a gausian to a distribution such as like a gausian to a target distribution such as the image target distribution such as the image target distribution such as the image distribution that we want by integrating distribution that we want by integrating distribution that we want by integrating through an OD through an OD through an OD and this OD is literally just the you and this OD is literally just the you and this OD is literally just the you know the infinite decimal uh change of know the infinite decimal uh change of know the infinite decimal uh change of your data should follow the velocity and your data should follow the velocity and your data should follow the velocity and at sampling time what you can do is you at sampling time what you can do is you at sampling time what you can do is you can just solve this OD by integrating

  87. L5 @ 485s

    can just solve this OD by integrating can just solve this OD by integrating through uh the velocity and then you through uh the velocity and then you through uh the velocity and then you you'll be able to get a sample or like a you'll be able to get a sample or like a you'll be able to get a sample or like a trajectory of samples. Uh trajectory of samples. Uh trajectory of samples. Uh so basically this is like literally what so basically this is like literally what so basically this is like literally what we just did, right? So you have some we just did, right? So you have some we just did, right? So you have some velocity and then you just like velocity and then you just like velocity and then you just like numerically is solve the OD at sampling numerically is solve the OD at sampling numerically is solve the OD at sampling time. time. time. Okay. So like how to interpret this, Okay. So like how to interpret this, Okay. So like how to interpret this, right? So basically just like think of right? So basically just like think of right? So basically just like think of it as like the wind flow transport some it as like the wind flow transport some it as like the wind flow transport some like water some amount of water from the like water some amount of water from the like water some amount of water from the west coast to the east coast or from west coast to the east coast or from west coast to the east coast or from like the Pacific Ocean to the uh to the like the Pacific Ocean to the uh to the like the Pacific Ocean to the uh to the east coast let's just say and um the east coast let's just say and um the east coast let's just say and um the question here is that why this is a question here is that why this is a question here is that why this is a normalizing flow normalizing flow normalizing flow first of all like what is a normalizing first of all like what is a normalizing first of all like what is a normalizing flow? Anyone else I haven't talked to today?

  88. L5 @ 535s

    Anyone else I haven't talked to today? Nobody remember what is normalizing Nobody remember what is normalizing Nobody remember what is normalizing flow? >> You can reconstruct the process. >> You can reconstruct the process. >> Yes. So it's a invertible uh generate >> Yes. So it's a invertible uh generate >> Yes. So it's a invertible uh generate model, right? Yes. And the reason why model, right? Yes. And the reason why model, right? Yes. And the reason why this is a normalizing flow is because this is a normalizing flow is because this is a normalizing flow is because the streams never cross. So imagine how the streams never cross. So imagine how the streams never cross. So imagine how like you're following a stream of river like you're following a stream of river like you're following a stream of river or like water or like the stream of the or like water or like the stream of the or like water or like the stream of the wind, right? Like if you just follow the wind, right? Like if you just follow the wind, right? Like if you just follow the stream, the stream will never like it stream, the stream will never like it stream, the stream will never like it will always point to one direction. So will always point to one direction. So will always point to one direction. So it will never like just get get it will never like just get get it will never like just get get if you just like follow the wind, you're if you just like follow the wind, you're if you just like follow the wind, you're able to basically get from one point to able to basically get from one point to able to basically get from one point to the other point guaranteed. the other point guaranteed. the other point guaranteed. So like deterministically and because So like deterministically and because So like deterministically and because this process is deterministic, if you this process is deterministic, if you this process is deterministic, if you follow this wind, if you follow this follow this wind, if you follow this follow this wind, if you follow this flow, you're actually going to get an flow, you're actually going to get an flow, you're actually going to get an invertible transformation because

  89. L5 @ 590s

    invertible transformation because invertible transformation because everything is deterministic. everything is deterministic. everything is deterministic. Okay, any question? Okay, any question? Okay, any question? Cool. You have a question. Okay. Are we Cool. You have a question. Okay. Are we Cool. You have a question. Okay. Are we saying that it's actually saying that it's actually saying that it's actually >> uh the way that we construct it is is it the way that we construct it is is it yeah it is invertible. Yeah the way that yeah it is invertible. Yeah the way that yeah it is invertible. Yeah the way that we construct it. Okay. we construct it. Okay. we construct it. Okay. Any more question? Cool. All right. Cool. All right. Uh okay. Okay. So how does this connect Uh okay. Okay. So how does this connect Uh okay. Okay. So how does this connect to probability? to probability? to probability? Well, basically in order to for this Well, basically in order to for this Well, basically in order to for this kind of like continuous flow thing model kind of like continuous flow thing model kind of like continuous flow thing model to transport between one probability to transport between one probability to transport between one probability distribution to another we need to have distribution to another we need to have distribution to another we need to have the following assumptions. The first the following assumptions. The first the following assumptions. The first thing is what we call the conservation thing is what we call the conservation thing is what we call the conservation of mass. Basically just imagine that the of mass. Basically just imagine that the of mass. Basically just imagine that the the wind carried a certain amount of the wind carried a certain amount of the wind carried a certain amount of water and this certain amount of water water and this certain amount of water water and this certain amount of water never change like the amount of water never change like the amount of water never change like the amount of water that you transporting never change. So that you transporting never change. So that you transporting never change. So no new mass uh is is going to get added no new mass uh is is going to get added

  90. L5 @ 661s

    no new mass uh is is going to get added and no mass is going to disappear and no mass is going to disappear and no mass is going to disappear getting dropped. So like translate it getting dropped. So like translate it getting dropped. So like translate it into probabilistic statement just means into probabilistic statement just means into probabilistic statement just means that the probability should always add that the probability should always add that the probability should always add up to one because otherwise it won't be up to one because otherwise it won't be up to one because otherwise it won't be a valid probability distribution. And a valid probability distribution. And a valid probability distribution. And then second thing is what we call the then second thing is what we call the then second thing is what we call the continuity equation which is like not continuity equation which is like not continuity equation which is like not only that the mass is conserved it's only that the mass is conserved it's only that the mass is conserved it's also not you cannot teleport them uh also not you cannot teleport them uh also not you cannot teleport them uh which means that everything need to move which means that everything need to move which means that everything need to move smoothly everything should move like smoothly everything should move like smoothly everything should move like continuously continuously continuously okay so basically just means that the okay so basically just means that the okay so basically just means that the probability can only get changed probability can only get changed probability can only get changed continuously you cannot just like jump continuously you cannot just like jump continuously you cannot just like jump from from point to point because it's from from point to point because it's from from point to point because it's it's a river okay any question it's a river okay any question it's a river okay any question cool all right Um so because of this we cool all right Um so because of this we cool all right Um so because of this we get to define something called a get to define something called a get to define something called a probability flux. So imagine a flux is probability flux. So imagine a flux is

  91. L5 @ 710s

    probability flux. So imagine a flux is like the amount of flow the amount of like the amount of flow the amount of like the amount of flow the amount of water uh per unit per time that's going water uh per unit per time that's going water uh per unit per time that's going through a certain a certain uh unit through a certain a certain uh unit through a certain a certain uh unit space. So if you're thinking about like space. So if you're thinking about like space. So if you're thinking about like how the probability flows is a how the probability flows is a how the probability flows is a probability flux is equal to the probability flux is equal to the probability flux is equal to the velocity times the the the density sort velocity times the the the density sort velocity times the the the density sort of. So the velocity is giving you where of. So the velocity is giving you where of. So the velocity is giving you where and how fast the probability flows and and how fast the probability flows and and how fast the probability flows and then the density is telling you how much then the density is telling you how much then the density is telling you how much probability there was so that probability there was so that probability there was so that it flows out right um okay so this is it flows out right um okay so this is it flows out right um okay so this is like a like a like a um visualization generated by vo I um visualization generated by vo I um visualization generated by vo I already tried but let's just uh look at already tried but let's just uh look at already tried but let's just uh look at it okay okay maybe I should put on okay do that maybe I should put on okay do that maybe I should put on okay do that All right. Okay. So, as you can see, as the thing Okay. So, as you can see, as the thing as the water flows, first of all, it's as the water flows, first of all, it's as the water flows, first of all, it's in a container. It's in like a contained

  92. L5 @ 775s

    in a container. It's in like a contained in a container. It's in like a contained tank. So, the probability mass never tank. So, the probability mass never tank. So, the probability mass never change. And second, like basically based change. And second, like basically based change. And second, like basically based on how fast the particle flows um like on how fast the particle flows um like on how fast the particle flows um like the probability density, which is like the probability density, which is like the probability density, which is like the water amount at this particular the water amount at this particular the water amount at this particular location, also changes. location, also changes. location, also changes. Okay, any questions? >> Flux in this case like similar to like >> Flux in this case like similar to like magnetic flu magnetic flu magnetic flu like it's like the amount of like like it's like the amount of like like it's like the amount of like >> Nope. >> Nope. >> Nope. >> It's a it just I think water is a better >> It's a it just I think water is a better >> It's a it just I think water is a better um um um I I mean I I mean I I mean >> like >> like >> like >> Yeah. Yeah. Kind of >> Yeah. Yeah. Kind of >> Yeah. Yeah. Kind of >> kind of. Yeah. It's like amount of flows >> kind of. Yeah. It's like amount of flows >> kind of. Yeah. It's like amount of flows that goes through a certain place within that goes through a certain place within that goes through a certain place within a within a unit unit amount of time. a within a unit unit amount of time. a within a unit unit amount of time. >> A sphere >> A sphere >> A sphere 0. 0. 0. >> Yeah. Yeah. Kind of kind. Okay. But >> Yeah. Yeah. Kind of kind. Okay. But >> Yeah. Yeah. Kind of kind. Okay. But anyway basically uh because of this we anyway basically uh because of this we anyway basically uh because of this we can actually write the probability flux can actually write the probability flux can actually write the probability flux in math in this way. So this is like

  93. L5 @ 840s

    in math in this way. So this is like in math in this way. So this is like basically like how much probability gets basically like how much probability gets basically like how much probability gets changed um infinite uh through time and changed um infinite uh through time and changed um infinite uh through time and then this is what we call the divergence then this is what we call the divergence then this is what we call the divergence of velocity times density and what is of velocity times density and what is of velocity times density and what is divergence divergence is what we have divergence divergence is what we have divergence divergence is what we have seen actually uh last class it's the seen actually uh last class it's the seen actually uh last class it's the trace of the jacobian what what it means trace of the jacobian what what it means trace of the jacobian what what it means is basically like how much um how like is basically like how much um how like is basically like how much um how like how much do you change in every how much do you change in every how much do you change in every direction direction direction um of the of the flow o of the of the um of the of the flow o of the of the um of the of the flow o of the of the data space. data space. data space. Okay, any questions? Okay, any questions? Okay, any questions? Everyone understand this? Yay. Yeah, this is like very physics Yay. Yeah, this is like very physics coded. Okay, any any more questions? Direction. Direction. >> Oh, D is dimension. Sorry. Yeah. So, in >> Oh, D is dimension. Sorry. Yeah. So, in >> Oh, D is dimension. Sorry. Yeah. So, in every dimension, how does this thing every dimension, how does this thing every dimension, how does this thing change? basically. Oh, uh I think uh there's a P here that Oh, uh I think uh there's a P here that is got getting uh um missed. is got getting uh um missed. is got getting uh um missed. >> Yeah. >> Yeah. >> Yeah. >> What's the second assumption? There's >> What's the second assumption? There's >> What's the second assumption? There's two assumptions. two assumptions.

  94. L5 @ 912s

    two assumptions. >> Uh two assumptions. One is that >> Uh two assumptions. One is that >> Uh two assumptions. One is that add up to one. The second one is that add up to one. The second one is that add up to one. The second one is that the the thing changes uh continuously. the the thing changes uh continuously. the the thing changes uh continuously. But how is the first one encoded in But how is the first one encoded in But how is the first one encoded in this? this? this? >> Uh it >> Uh it >> Uh it I I guess just the first one is not I I guess just the first one is not I I guess just the first one is not really encoded in this but like you just really encoded in this but like you just really encoded in this but like you just need to have a probability that add up need to have a probability that add up need to have a probability that add up to one. Yeah. >> Flux is a weird word for me. Would it be >> Flux is a weird word for me. Would it be weird to think of it as like viscosity weird to think of it as like viscosity weird to think of it as like viscosity like the amount of flow period of time like the amount of flow period of time like the amount of flow period of time and you're counting for density to like and you're counting for density to like and you're counting for density to like >> I mean the changes in space right and >> I mean the changes in space right and >> I mean the changes in space right and the viscosity should be uniform. Sure. the viscosity should be uniform. Sure. the viscosity should be uniform. Sure. But like viscosity with the thing moving But like viscosity with the thing moving But like viscosity with the thing moving over time. >> Um now flux is actually also a physics >> Um now flux is actually also a physics term and it means like the amount of term and it means like the amount of term and it means like the amount of thing that like flow out thing that like flow out

  95. L5 @ 964s

    thing that like flow out >> as physically inclined. >> as physically inclined. >> as physically inclined. >> It is okay. >> It is okay. >> It is okay. >> I'm just a civil engineer. >> I'm just a civil engineer. >> I'm just a civil engineer. >> I'm also not a physics major but uh you >> I'm also not a physics major but uh you >> I'm also not a physics major but uh you know this is like it's not really >> um I mean just a term. It's not really >> um I mean just a term. It's not really like um Yeah. Yeah. But but but like you like um Yeah. Yeah. But but but like you like um Yeah. Yeah. But but but like you can you can imagine it however way you can you can imagine it however way you can you can imagine it however way you want but that's what it means basically want but that's what it means basically want but that's what it means basically just the amount of things that it just the amount of things that it just the amount of things that it outflows in a unit time basically. outflows in a unit time basically. outflows in a unit time basically. >> Yeah, >> why is it noisy? system external noise >> why is it noisy? system external noise >> uh depending on what kind of things that >> uh depending on what kind of things that >> uh depending on what kind of things that you are. So basically if you're thinking you are. So basically if you're thinking you are. So basically if you're thinking about the path that we just constructed about the path that we just constructed about the path that we just constructed then yes each XT is noisy but this is a then yes each XT is noisy but this is a then yes each XT is noisy but this is a generalized um um like notion it it's generalized um um like notion it it's generalized um um like notion it it's like just any flow this is the this is like just any flow this is the this is like just any flow this is the this is the the definition any other question any other question yes yes yes the divergence connection again

  96. L5 @ 1028s

    the divergence connection again the divergence connection again >> yes I I think I missed a P here actually >> yes I I think I missed a P here actually >> yes I I think I missed a P here actually by the way uh but basically by the way uh but basically by the way uh but basically The divergence means that like how much The divergence means that like how much The divergence means that like how much uh probability do you change in every uh probability do you change in every uh probability do you change in every direction of the data space. Yeah. So direction of the data space. Yeah. So direction of the data space. Yeah. So that's why it's the trace of the that's why it's the trace of the that's why it's the trace of the Jacobian in every direction, every Jacobian in every direction, every Jacobian in every direction, every dimension here. Yeah. I I miss a P here. because like this case, right? because like this case, right? >> We don't we haven't talked about flow >> We don't we haven't talked about flow >> We don't we haven't talked about flow matching yet. >> All right. Any more question? >> All right. Any more question? Cool. All right. Yeah. So, this is Cool. All right. Yeah. So, this is Cool. All right. Yeah. So, this is divergence. How much probability that divergence. How much probability that divergence. How much probability that outflows from a given point per unit outflows from a given point per unit outflows from a given point per unit time in every direction? time in every direction? time in every direction? Okay. Okay. Okay. Cool. All right. All right. So uh like a Cool. All right. All right. So uh like a Cool. All right. All right. So uh like a good thing about um this thing is that good thing about um this thing is that good thing about um this thing is that we can actually transform this PDE into we can actually transform this PDE into we can actually transform this PDE into an OD. All right. So what it means is an OD. All right. So what it means is an OD. All right. So what it means is that like so this is the correct

  97. L5 @ 1101s

    that like so this is the correct that like so this is the correct divergence. Sorry that that that one I divergence. Sorry that that that one I divergence. Sorry that that that one I missed P. Um but basically we can go missed P. Um but basically we can go missed P. Um but basically we can go from here uh and basically you do the from here uh and basically you do the from here uh and basically you do the greatest uh mathematical trick of all greatest uh mathematical trick of all greatest uh mathematical trick of all time. You multiply time. You multiply time. You multiply two things at the same time from the two things at the same time from the two things at the same time from the left hand side to the right hand side left hand side to the right hand side left hand side to the right hand side and uh basically this thing because the and uh basically this thing because the and uh basically this thing because the because the the the the derivative of because the the the the derivative of because the the the the derivative of log is equal to 1 /x. So it becomes the log is equal to 1 /x. So it becomes the log is equal to 1 /x. So it becomes the derivative of the uh the log and then um derivative of the uh the log and then um derivative of the uh the log and then um basically you just apply the same thing basically you just apply the same thing basically you just apply the same thing applying the same trick here and then it applying the same trick here and then it applying the same trick here and then it becomes something like this. Um and then like essentially you basically and then like essentially you basically just do a lot of like calculus and then just do a lot of like calculus and then just do a lot of like calculus and then you're actually able to get uh an OD of you're actually able to get uh an OD of you're actually able to get uh an OD of the log probability of of the log the log probability of of the log

  98. L5 @ 1153s

    the log probability of of the log density. Okay. And this OD is actually density. Okay. And this OD is actually density. Okay. And this OD is actually equal to so the derivative of the total equal to so the derivative of the total equal to so the derivative of the total derivative of log probability is equal derivative of log probability is equal derivative of log probability is equal to the divergence of the velocity to the divergence of the velocity to the divergence of the velocity period. Okay. Any questions here? Okay. Any questions here? >> Yes. >> Yes. >> Yes. >> PFS to be zero. >> PFS to be zero. >> PFS to be zero. >> Yes. Yes. That's a that's a that's a >> Yes. Yes. That's a that's a that's a >> Yes. Yes. That's a that's a that's a great uh question. Yes. PFS is greater great uh question. Yes. PFS is greater great uh question. Yes. PFS is greater than zero everywhere. Yeah. Basically, you just do a lot of Yeah. Basically, you just do a lot of calculus and then you get this. >> No question. >> No question. >> Yes. >> Yes. >> Because it's still depending on the >> Because it's still depending on the >> Because it's still depending on the variable. variable. variable. No, basically from this line, I don't No, basically from this line, I don't No, basically from this line, I don't know if you can see my cursor. This line know if you can see my cursor. This line know if you can see my cursor. This line is literally the formula for the total is literally the formula for the total is literally the formula for the total derivative, right? derivative, right? derivative, right? Yeah. No more question. No more question. Okay, cool. Uh so basically then you can Okay, cool. Uh so basically then you can Okay, cool. Uh so basically then you can calculus then you can calculate your calculus then you can calculate your calculus then you can calculate your likelihood, right? because now you have likelihood, right? because now you have likelihood, right? because now you have a another OD that you can directly solve a another OD that you can directly solve a another OD that you can directly solve by doing numerical integration. So exactly so so

  99. L5 @ 1252s

    So exactly so so how do you train your continuous how do you train your continuous how do you train your continuous normalizing flow from here? What do we normalizing flow from here? What do we normalizing flow from here? What do we think jumping ahead. Jumping ahead. What is jumping ahead. Jumping ahead. What is the Now we have a formula for the the Now we have a formula for the the Now we have a formula for the likelihood log likelihood you can do for the the most naive thing you can do for the the most naive thing to think right is the maximum log to think right is the maximum log to think right is the maximum log likelihood. So basically uh yeah so likelihood. So basically uh yeah so likelihood. So basically uh yeah so because you have a formula to calculate because you have a formula to calculate because you have a formula to calculate the log likelihood at any point of the the log likelihood at any point of the the log likelihood at any point of the trajectory obviously you can also trajectory obviously you can also trajectory obviously you can also calculate the log likelihood of the end calculate the log likelihood of the end calculate the log likelihood of the end point right so you just do arg max of point right so you just do arg max of point right so you just do arg max of the likelihood what could be a problem the likelihood what could be a problem the likelihood what could be a problem here >> yeah yeah >> yeah yeah >> don't have access to p 0 >> don't have access to p 0 >> don't have access to p 0 >> you we can define that we define p 0 to >> you we can define that we define p 0 to >> you we can define that we define p 0 to be a gausian for example that's fine be a gausian for example that's fine be a gausian for example that's fine Very very good. So basically in order to very good. So basically in order to calculate the log likelihood of the data calculate the log likelihood of the data calculate the log likelihood of the data we actually need to do numerical we actually need to do numerical

  100. L5 @ 1324s

    we actually need to do numerical integration right. But this is at integration right. But this is at integration right. But this is at training time. So that just means that training time. So that just means that training time. So that just means that we need to do many many many many steps we need to do many many many many steps we need to do many many many many steps of integration at training time and that of integration at training time and that of integration at training time and that thing is really really expensive. thing is really really expensive. thing is really really expensive. Right. Right. Right. Okay. What what would be a better way to Okay. What what would be a better way to Okay. What what would be a better way to do this? do this? do this? >> Yeah. >> If we take the if we take the gradient >> If we take the if we take the gradient of the left hand side, we get a score of the left hand side, we get a score of the left hand side, we get a score and the left hand side removes, and the left hand side removes, and the left hand side removes, >> right? >> right? >> right? >> Uh >> Uh >> Uh even if you do that right, the even if you do that right, the even if you do that right, the divergence is still pretty difficult to divergence is still pretty difficult to divergence is still pretty difficult to calculate. calculate. calculate. >> Okay. But what else? Yes, >> we can we can use this numerator or we >> we can we can use this numerator or we can just directly sample can just directly sample can just directly sample so that it'll be uh we we'll end up like so that it'll be uh we we'll end up like so that it'll be uh we we'll end up like uh we'll end up with the summation. >> You you mean you change the integration >> You you mean you change the integration into summation? into summation? into summation? >> Yeah. Yeah. Yeah. But that's just >> Yeah. Yeah. Yeah. But that's just >> Yeah. Yeah. Yeah. But that's just numerical integration, right? So it numerical integration, right? So it

  101. L5 @ 1387s

    numerical integration, right? So it still has a lot of steps. Can we do still has a lot of steps. Can we do still has a lot of steps. Can we do something that doesn't require a lot of something that doesn't require a lot of something that doesn't require a lot of steps? Like we just have one four pass steps? Like we just have one four pass steps? Like we just have one four pass during the training time. I think during the training time. I think during the training time. I think someone said it before. Who said velocity before? Who did it? Who said velocity before? Who did it? Oh, you did? Oh, you did? Oh, you did? >> Oh, someone who said velocity. >> Oh, okay. Why? Why did you say velocity? >> Oh, okay. Why? Why did you say velocity? >> Because >> Because >> Because the model's learning velocity. Okay, it actually Okay, it actually basically if you think about it right basically if you think about it right basically if you think about it right both sampling like in order to sample or both sampling like in order to sample or both sampling like in order to sample or calculate the log likelihood you you calculate the log likelihood you you calculate the log likelihood you you pretty much only need to parameterize pretty much only need to parameterize pretty much only need to parameterize the velocity right so if you have a the velocity right so if you have a the velocity right so if you have a model for the velocity then you can model for the velocity then you can model for the velocity then you can calculate then you can do both sampling calculate then you can do both sampling calculate then you can do both sampling and calculate the log log likelihood so and calculate the log log likelihood so and calculate the log log likelihood so why don't we we just need to make sure why don't we we just need to make sure why don't we we just need to make sure that the velocity is correct right so we that the velocity is correct right so we that the velocity is correct right so we just need to make sure that the velocity just need to make sure that the velocity

  102. L5 @ 1451s

    just need to make sure that the velocity actually match with the ground truth actually match with the ground truth actually match with the ground truth velocity. velocity. velocity. So you just do L2 to match with the So you just do L2 to match with the So you just do L2 to match with the ground truth velocity. ground truth velocity. ground truth velocity. Easy, right? Easy, right? Easy, right? What is the problem? Yeah. In order to do either sampling or like In order to do either sampling or like critical estimation, we just need a critical estimation, we just need a critical estimation, we just need a velocity. So if we have a model for velocity. So if we have a model for velocity. So if we have a model for velocity, then we could can do velocity, then we could can do velocity, then we could can do integration at inference time. We don't integration at inference time. We don't integration at inference time. We don't need to do integration at training time need to do integration at training time need to do integration at training time anymore, right? But what what is the anymore, right? But what what is the anymore, right? But what what is the problem here? The ground truth, right? We don't have The ground truth, right? We don't have the ground truth. If we have the ground the ground truth. If we have the ground the ground truth. If we have the ground truth, we would know that a winter storm truth, we would know that a winter storm truth, we would know that a winter storm is coming. Apparently, it's a Saturday. is coming. Apparently, it's a Saturday. is coming. Apparently, it's a Saturday. It's a very cold day. So be prepared. It's a very cold day. So be prepared. It's a very cold day. So be prepared. Okay, winter's coming. Uh anyway, so Okay, winter's coming. Uh anyway, so Okay, winter's coming. Uh anyway, so yeah, but we do we do not know. So yeah, but we do we do not know. So yeah, but we do we do not know. So that's why the meteorologists are that's why the meteorologists are that's why the meteorologists are changing every day be like actually I changing every day be like actually I

  103. L5 @ 1512s

    changing every day be like actually I think that the the storm is moving north think that the the storm is moving north think that the the storm is moving north or maybe not. We don't know, right? So or maybe not. We don't know, right? So or maybe not. We don't know, right? So we do not know the ground velocity. How should we solve this? Well, How should we solve this? Well, basically say we don't care about basically say we don't care about basically say we don't care about everywhere else. We just care about how everywhere else. We just care about how everywhere else. We just care about how the storm is going to move to the storm is going to move to the storm is going to move to Pittsburgh. Let's just say and say now Pittsburgh. Let's just say and say now Pittsburgh. Let's just say and say now you just want to like say you now you're you just want to like say you now you're you just want to like say you now you're Poseidon. Okay, whatever. And then you Poseidon. Okay, whatever. And then you Poseidon. Okay, whatever. And then you you want to transform a bunch of water you want to transform a bunch of water you want to transform a bunch of water or a bunch of like humidity to or a bunch of like humidity to or a bunch of like humidity to Pittsburgh so that Pittsburgh would Pittsburgh so that Pittsburgh would Pittsburgh so that Pittsburgh would snow. snow. snow. What would you do? What is the easiest What would you do? What is the easiest What would you do? What is the easiest way? What is the fastest way for you to way? What is the fastest way for you to way? What is the fastest way for you to do it? How how would you do that? do it? How how would you do that? do it? How how would you do that? >> You just Yeah. >> You just Yeah. >> You just Yeah. >> Just go straight there. >> Just go straight there. >> Just go straight there. >> Just go straight there. Yeah. Yeah. >> Just go straight there. Yeah. Yeah. >> Just go straight there. Yeah. Yeah. Yeah. That's right. That's right. You Yeah. That's right. That's right. You

  104. L5 @ 1558s

    Yeah. That's right. That's right. You just go straight. just go straight. just go straight. Yeah, exactly. Uh so basically this is Yeah, exactly. Uh so basically this is Yeah, exactly. Uh so basically this is what we're going to do as well. Uh so what we're going to do as well. Uh so what we're going to do as well. Uh so let's say we fix a data point. We don't let's say we fix a data point. We don't let's say we fix a data point. We don't care about everywhere else. We just care care about everywhere else. We just care care about everywhere else. We just care about one data point. And then it's about one data point. And then it's about one data point. And then it's usually uh very easy to define a what we usually uh very easy to define a what we usually uh very easy to define a what we call continuous velocity field which uh call continuous velocity field which uh call continuous velocity field which uh sorry a conditional velocity field which sorry a conditional velocity field which sorry a conditional velocity field which means that means that means that basically it just basically it just basically it just velocity is depending on the end point velocity is depending on the end point velocity is depending on the end point right and then we call the trajectory of right and then we call the trajectory of right and then we call the trajectory of the probability distribution that gen the probability distribution that gen the probability distribution that gen get generated along the get generated along the get generated along the way of that uh velocity field the way of that uh velocity field the way of that uh velocity field the conditional probability path. So the conditional probability path. So the conditional probability path. So the conditional probability path is consists conditional probability path is consists conditional probability path is consists of like all the uh all the pts along the of like all the uh all the pts along the of like all the uh all the pts along the way. Okay. And here the conditional way. Okay. And here the conditional way. Okay. And here the conditional probability path will probability path will probability path will just start from some prior which we can

  105. L5 @ 1617s

    just start from some prior which we can just start from some prior which we can choose. So it doesn't matter we can choose. So it doesn't matter we can choose. So it doesn't matter we can choose the prior and then we will always choose the prior and then we will always choose the prior and then we will always end up either directly at Pittsburgh or end up either directly at Pittsburgh or end up either directly at Pittsburgh or somewhere around Pittsburgh. So like we somewhere around Pittsburgh. So like we somewhere around Pittsburgh. So like we just like always end up at x1 or a small just like always end up at x1 or a small just like always end up at x1 or a small small gausian that concentrate around small gausian that concentrate around small gausian that concentrate around x1. Okay. And uh by the way so this x1. Okay. And uh by the way so this x1. Okay. And uh by the way so this thing is called direct delta. So it just thing is called direct delta. So it just thing is called direct delta. So it just means that it's a point mass at at x1. means that it's a point mass at at x1. means that it's a point mass at at x1. Okay. Then uh the marginal probability Okay. Then uh the marginal probability Okay. Then uh the marginal probability path can be written in this way, right? path can be written in this way, right? path can be written in this way, right? Basically just like sum up all the Basically just like sum up all the Basically just like sum up all the possible uh sum up all the possible uh possible uh sum up all the possible uh possible uh sum up all the possible uh uh like data points and then basically uh like data points and then basically uh like data points and then basically just like yeah like and and times the just like yeah like and and times the just like yeah like and and times the the the conditional probability given the the conditional probability given the the conditional probability given the the data point then you can get a the the data point then you can get a

  106. L5 @ 1663s

    the the data point then you can get a marginal marginal marginal uh yeah and also uh just the boundary uh yeah and also uh just the boundary uh yeah and also uh just the boundary condition essentially uh the probability condition essentially uh the probability condition essentially uh the probability uh at time uh at time uh at time should recover the data probability. should recover the data probability. should recover the data probability. This is basically what we're trying to This is basically what we're trying to This is basically what we're trying to construct. construct. construct. Okay. >> All right. Cool. Um so say now we have a >> All right. Cool. Um so say now we have a conditional velocity here. uh then we conditional velocity here. uh then we conditional velocity here. uh then we can also define a marginal velocity. can also define a marginal velocity. can also define a marginal velocity. What does what does this mean? So What does what does this mean? So What does what does this mean? So basically this just means that uh we are basically this just means that uh we are basically this just means that uh we are sort of like we're like sort of like sort of like we're like sort of like sort of like we're like sort of like apply a bay theorem on on your uh apply a bay theorem on on your uh apply a bay theorem on on your uh conditional velocity. So that basically conditional velocity. So that basically conditional velocity. So that basically it's like reweighted it's like reweighted it's like reweighted uh like um taking account into like uh like um taking account into like uh like um taking account into like basically um how likely is your current basically um how likely is your current basically um how likely is your current uh noisy point like your intermediate uh noisy point like your intermediate uh noisy point like your intermediate data point along the the the conditional data point along the the the conditional data point along the the the conditional probability path. How likely is the data probability path. How likely is the data probability path. How likely is the data point that defined this conditional point that defined this conditional point that defined this conditional probability path and also just like how

  107. L5 @ 1743s

    probability path and also just like how probability path and also just like how like and reweed by how likely in general like and reweed by how likely in general like and reweed by how likely in general of this current intermediate point. Does of this current intermediate point. Does of this current intermediate point. Does this make sense? All right. So basically you can sort of All right. So basically you can sort of like intuitively understand this as like like intuitively understand this as like like intuitively understand this as like this is the average conditional velocity this is the average conditional velocity this is the average conditional velocity at this current location in this current at this current location in this current at this current location in this current time weighted by how likely uh is the time weighted by how likely uh is the time weighted by how likely uh is the data point for this current location in data point for this current location in data point for this current location in time. part. Then in the next slide when we're part. Then in the next slide when we're doing it for the velocity, is it doing it for the velocity, is it doing it for the velocity, is it something we're doing from that step or something we're doing from that step or something we're doing from that step or is it just like separate entirely? is it just like separate entirely? is it just like separate entirely? >> Separate. I'll go level resurrected. Uh yeah. So, I'll go level resurrected. Uh yeah. So, so, so this is kind of like what we so, so this is kind of like what we so, so this is kind of like what we defined. It's not like it's two defined. It's not like it's two defined. It's not like it's two different notions different notions different notions and like you can intuitively understand and like you can intuitively understand and like you can intuitively understand it as like sort of like the um like the it as like sort of like the um like the it as like sort of like the um like the average uh conditional velocity and then average uh conditional velocity and then average uh conditional velocity and then this average is like averaging by like

  108. L5 @ 1838s

    this average is like averaging by like this average is like averaging by like how likely is the data point given your how likely is the data point given your how likely is the data point given your current position and the data points are available. and the data points are available. All right, does this make sense? we'll get to it. We'll get to it. Um all we'll get to it. We'll get to it. Um all right. Uh and a very interesting thing right. Uh and a very interesting thing right. Uh and a very interesting thing here is that the marginal velocity we here is that the marginal velocity we here is that the marginal velocity we just saw actually generates the marginal just saw actually generates the marginal just saw actually generates the marginal probability path that we want. Okay. So what it means is what it means by So what it means is what it means by generate is basically just means that generate is basically just means that generate is basically just means that like the the continuity equation is like the the continuity equation is like the the continuity equation is satisfies the the PTE is satisfied and satisfies the the PTE is satisfied and satisfies the the PTE is satisfied and why does it satisfy is because if you why does it satisfy is because if you why does it satisfy is because if you just break everything down uh and then just break everything down uh and then just break everything down uh and then you are able to get basically the the you are able to get basically the the you are able to get basically the the divergence back. Uh so this is just divergence back. Uh so this is just divergence back. Uh so this is just like step by step how you will get a like step by step how you will get a like step by step how you will get a divergence back. Uh so you first um swap divergence back. Uh so you first um swap divergence back. Uh so you first um swap in the the the derivative with some in the the the derivative with some in the the the derivative with some regularization condition. This can can regularization condition. This can can

  109. L5 @ 1904s

    regularization condition. This can can can be done. Uh and then uh basically can be done. Uh and then uh basically can be done. Uh and then uh basically you uh input your um because we know you uh input your um because we know you uh input your um because we know that the marginal sorry the conditional that the marginal sorry the conditional that the marginal sorry the conditional velocity generates the conditional path. velocity generates the conditional path. velocity generates the conditional path. So we can uh input the continuity So we can uh input the continuity So we can uh input the continuity equation there and then it becomes this equation there and then it becomes this equation there and then it becomes this whole thing with divergence. Uh and then whole thing with divergence. Uh and then whole thing with divergence. Uh and then plug in the definition that we just saw plug in the definition that we just saw plug in the definition that we just saw before we can actually get something before we can actually get something before we can actually get something like this and again pulling out the the like this and again pulling out the the like this and again pulling out the the the the derivative here because because the the derivative here because because the the derivative here because because of some regularization condition that we of some regularization condition that we of some regularization condition that we can satisfy it. uh and this thing is can satisfy it. uh and this thing is can satisfy it. uh and this thing is literally just the divergence of the literally just the divergence of the literally just the divergence of the marginal uh velocity times the marginal marginal uh velocity times the marginal marginal uh velocity times the marginal density. density. density. So all in all this is to say that the So all in all this is to say that the So all in all this is to say that the marginal velocity that we defined before marginal velocity that we defined before marginal velocity that we defined before from the conditional velocity that we from the conditional velocity that we from the conditional velocity that we can construct can generate the marginal can construct can generate the marginal

  110. L5 @ 1966s

    can construct can generate the marginal property path which is what we want. property path which is what we want. property path which is what we want. Oh, I saw so many confusion. Oh, I saw so many confusion. Oh, I saw so many confusion. All good. All good. We're getting there. All good. All good. We're getting there. All good. All good. We're getting there. We're getting there. I promise. Ah, We're getting there. I promise. Ah, We're getting there. I promise. Ah, okay. The other thing which is the most okay. The other thing which is the most okay. The other thing which is the most important thing and the most magical important thing and the most magical important thing and the most magical part is that actually part is that actually part is that actually so this is the the the the equation that so this is the the the the equation that so this is the the the the equation that we want, right? So this is the equation we want, right? So this is the equation we want, right? So this is the equation that we that that that we came up with. that we that that that we came up with. that we that that that we came up with. You just need to match the ground truth You just need to match the ground truth You just need to match the ground truth marginal velocity. But we do not have marginal velocity. But we do not have marginal velocity. But we do not have the groundous marginal velocity, right? the groundous marginal velocity, right? the groundous marginal velocity, right? And it actually turns out that if you And it actually turns out that if you And it actually turns out that if you just match the conditional velocity, just match the conditional velocity, just match the conditional velocity, you're actually going to end up with the you're actually going to end up with the you're actually going to end up with the same uh global optimal. You're like same uh global optimal. You're like same uh global optimal. You're like basically just like matching the ground basically just like matching the ground basically just like matching the ground truth marginal velocity is equivalent to truth marginal velocity is equivalent to truth marginal velocity is equivalent to ma matching the conditional velocity.

  111. L5 @ 2030s

    Think about that. Isn't that pretty Think about that. Isn't that pretty cool? Okay. Why? Why? Why is that? Okay. cool? Okay. Why? Why? Why is that? Okay. cool? Okay. Why? Why? Why is that? Okay. So, the reason why is because we can So, the reason why is because we can So, the reason why is because we can break the two L2 down into this thing, break the two L2 down into this thing, break the two L2 down into this thing, right? We we saw this yesterday, not right? We we saw this yesterday, not right? We we saw this yesterday, not yesterday, on Tuesday already, right? yesterday, on Tuesday already, right? yesterday, on Tuesday already, right? And the first term for both equation are And the first term for both equation are And the first term for both equation are constant with respect to our learn constant with respect to our learn constant with respect to our learn parameter. So, we're just going to parameter. So, we're just going to parameter. So, we're just going to cancel it out, right? cancel it out, right? cancel it out, right? Uh and then now we get to two equations Uh and then now we get to two equations Uh and then now we get to two equations each with two components. Right? So each with two components. Right? So each with two components. Right? So we're going to look at the first we're going to look at the first we're going to look at the first components together first. All right. So components together first. All right. So components together first. All right. So the first component if you if you look the first component if you if you look the first component if you if you look at the conditional flow matching uh at the conditional flow matching uh at the conditional flow matching uh equation here uh basically this thing equation here uh basically this thing equation here uh basically this thing you can um decompose it into two you can um decompose it into two you can um decompose it into two integrals and then the these two integrals and then the these two integrals and then the these two integrals integrals integrals um can be um can be um can be uh uh uh Like basically you if you swap uh the

  112. L5 @ 2086s

    Like basically you if you swap uh the Like basically you if you swap uh the dx1 term inside so you swap the order of dx1 term inside so you swap the order of dx1 term inside so you swap the order of the integration you're actually going to the integration you're actually going to the integration you're actually going to get the first term of the first equation get the first term of the first equation get the first term of the first equation back. All right. This part is clear. Cool. All right. All right. Now look at Cool. All right. All right. Now look at the second part. The second part is the second part. The second part is the second part. The second part is pretty The second part is pretty pretty The second part is pretty pretty The second part is pretty similar. Basically what you do is you similar. Basically what you do is you similar. Basically what you do is you just uh starting from the second part of just uh starting from the second part of just uh starting from the second part of the first the first the first uh equation and then you just um plug in uh equation and then you just um plug in uh equation and then you just um plug in what we have as definition and then you what we have as definition and then you what we have as definition and then you do a lot of integrals in between and do a lot of integrals in between and do a lot of integrals in between and then you're actually going to wind up then you're actually going to wind up then you're actually going to wind up at the second part of the second at the second part of the second at the second part of the second equation. equation. equation. Any question? Yes. No. No. Okay. understand everything Yes. No. No. Okay. understand everything everyone. everyone. everyone. >> It's not so it's not even that we're >> It's not so it's not even that we're >> It's not so it's not even that we're sort of having a elbow type thing where sort of having a elbow type thing where sort of having a elbow type thing where we're compressing the bullet. we're compressing the bullet.

  113. L5 @ 2150s

    we're compressing the bullet. >> It's literally the same. It's literally >> It's literally the same. It's literally >> It's literally the same. It's literally the same. It's equivalent. It's not even the same. It's equivalent. It's not even the same. It's equivalent. It's not even elbow. No question. All right. No question. All right. So, this just means that like we don't So, this just means that like we don't So, this just means that like we don't need to match to the thing that we don't need to match to the thing that we don't need to match to the thing that we don't know. we actually can match something know. we actually can match something know. we actually can match something that we do know and do know how to that we do know and do know how to that we do know and do know how to construct which is the conditional construct which is the conditional construct which is the conditional velocity. velocity. Okay. >> Yes, pretty much right. And then you >> Yes, pretty much right. And then you average out and then it is actually the average out and then it is actually the average out and then it is actually the same. Um so uh now let's just look at same. Um so uh now let's just look at same. Um so uh now let's just look at the example that we have right. So now the example that we have right. So now the example that we have right. So now let's suppose our conditional let's suppose our conditional let's suppose our conditional probability path is to transport uh probability path is to transport uh probability path is to transport uh trans transform a gausian a standard trans transform a gausian a standard trans transform a gausian a standard normal gausian straight to a single normal gausian straight to a single normal gausian straight to a single point with constant speed. So this is a point with constant speed. So this is a point with constant speed. So this is a conditional right. So that's why we can conditional right. So that's why we can conditional right. So that's why we can have a single point here. What do we do? Well we first have the the the endpoint

  114. L5 @ 2213s

    Well we first have the the the endpoint distribution and because we're distribution and because we're distribution and because we're transforming it straight to the point transforming it straight to the point transforming it straight to the point with constant speed right? So just with constant speed right? So just with constant speed right? So just linearly very straight. Um so we can linearly very straight. Um so we can linearly very straight. Um so we can literally have a formula for each literally have a formula for each literally have a formula for each intermediate point as a linear intermediate point as a linear intermediate point as a linear combination of uh of the data and the combination of uh of the data and the combination of uh of the data and the noise and this linear combination will noise and this linear combination will noise and this linear combination will add up to one right and uh the add up to one right and uh the add up to one right and uh the probability path can be written in this probability path can be written in this probability path can be written in this way. And most importantly, way. And most importantly, way. And most importantly, the velocity is literally just xt taking the velocity is literally just xt taking the velocity is literally just xt taking derivative of t, which is equal to theta derivative of t, which is equal to theta derivative of t, which is equal to theta minus noise. That's it. And this is what we're That's it. And this is what we're matching to at training time, which we matching to at training time, which we matching to at training time, which we have already seen. >> Yeah. >> First question. So this us >> First question. So this us >> First question. So this us parameterizing it as this linear parameterizing it as this linear parameterizing it as this linear combination that fact is not used really combination that fact is not used really combination that fact is not used really in any of the other. in any of the other. in any of the other. >> Yeah. No no this is what we chose. This >> Yeah. No no this is what we chose. This

  115. L5 @ 2272s

    >> Yeah. No no this is what we chose. This is a design choice. So that so this is a design choice. So that so this is a design choice. So that so this thing is generalizable to any kind of thing is generalizable to any kind of thing is generalizable to any kind of path. Yeah. path. Yeah. path. Yeah. >> How is the ptxt given x1 that >> How is the ptxt given x1 that >> How is the ptxt given x1 that >> how is what >> how is what >> how is what >> the third length? How is the third? >> the third length? How is the third? >> the third length? How is the third? Oh, here. Oh, here. Oh, here. >> Well, because x0 is zero mean gausian. >> Well, because x0 is zero mean gausian. >> Well, because x0 is zero mean gausian. >> Yeah. >> Yeah. All right. Any other? Yeah. All right. Any other? Yeah. All right. Any other? Yeah. >> I don't see any like >> I don't see any like >> I don't see any like gausian assumption. gausian assumption. gausian assumption. >> Oh, there's no gausian assumption. You >> Oh, there's no gausian assumption. You >> Oh, there's no gausian assumption. You don't need gian assumption. don't need gian assumption. don't need gian assumption. >> Just assume that the noise is any >> Just assume that the noise is any >> Just assume that the noise is any distribution. It still works. distribution. It still works. distribution. It still works. >> Yes. Well, um, well, Well, um, well, yeah, pretty much. Yeah, you you don't yeah, pretty much. Yeah, you you don't yeah, pretty much. Yeah, you you don't you don't really need Gausian here. >> Any reason other than simplicity for >> Any reason other than simplicity for choosing? choosing? choosing? Um Um Um here here here not really like but like what what is not really like but like what what is not really like but like what what is the other uh uh what is the other the other uh uh what is the other the other uh uh what is the other distribution that you can think of that distribution that you can think of that distribution that you can think of that we can directly sample from we can directly sample from

  116. L5 @ 2349s

    we can directly sample from >> therefor you >> like previous lectures like gausian >> like previous lectures like gausian assumption is used like this closure on assumption is used like this closure on assumption is used like this closure on the gian used in the derivation but in the gian used in the derivation but in the gian used in the derivation but in flow matching nowhere is flow matching nowhere is flow matching nowhere is >> yeah >> yeah >> yeah it's just here it's just that basically >> like we have previously seen that we can >> like we have previously seen that we can get some kind of stoastic differential get some kind of stoastic differential get some kind of stoastic differential equation for the equation for the equation for the for our scores. So here, don't we just for our scores. So here, don't we just for our scores. So here, don't we just have like a parameterization of that have like a parameterization of that have like a parameterization of that same equation that will get us the same same equation that will get us the same same equation that will get us the same result? result? result? >> Great question. We're going to talk >> Great question. We're going to talk >> Great question. We're going to talk about it later. about it later. about it later. >> Yeah. >> Yeah. >> Assuming that there's an assumption that >> Assuming that there's an assumption that >> Assuming that there's an assumption that I think we assumed earlier to the come I think we assumed earlier to the come I think we assumed earlier to the come earlier that the you said the velocity earlier that the you said the velocity earlier that the you said the velocity is constant. So is constant. So is constant. So >> yeah, the velocity yes is constant >> yeah, the velocity yes is constant >> yeah, the velocity yes is constant speed. Yeah. But is it actually speed. Yeah. But is it actually speed. Yeah. But is it actually constant? Because last lecture I think constant? Because last lecture I think constant? Because last lecture I think we talked about stuff when we u when we we talked about stuff when we u when we

  117. L5 @ 2422s

    we talked about stuff when we u when we actually uh decrease the the velocity at actually uh decrease the the velocity at actually uh decrease the the velocity at which we adding noise. which we adding noise. which we adding noise. >> Here the velocity is different from >> Here the velocity is different from >> Here the velocity is different from previous class. previous class. previous class. So this is two different models. Um and So this is two different models. Um and So this is two different models. Um and the reason why this the reason why this the reason why this constant speed is because literally you constant speed is because literally you constant speed is because literally you have the t there, right? And then like have the t there, right? And then like have the t there, right? And then like literally if you derive the velocity is literally if you derive the velocity is literally if you derive the velocity is literally a constant literally a constant literally a constant right you have question. you have question. >> Oh, okay. Cool. Uh yes. Can >> Oh, okay. Cool. Uh yes. Can >> Oh, okay. Cool. Uh yes. Can >> we use this to do likelihood estimation >> we use this to do likelihood estimation >> we use this to do likelihood estimation as well or? as well or? as well or? >> Yes. We're going to talk about it later. >> Yes. We're going to talk about it later. >> Yes. We're going to talk about it later. Yeah. Yeah. Yeah. Anyone else? Anyone else? Anyone else? Cool. Yeah. And this is the this is the Cool. Yeah. And this is the this is the Cool. Yeah. And this is the this is the uh the loss function. Literally uh the loss function. Literally uh the loss function. Literally literally just you learn the model literally just you learn the model literally just you learn the model should predict data minus noise. Yeah. should predict data minus noise. Yeah. should predict data minus noise. Yeah. >> The velocity stays constant. >> The velocity stays constant. >> The velocity stays constant. >> Um the the velocity is constant only on >> Um the the velocity is constant only on >> Um the the velocity is constant only on this particular path. Only on this

  118. L5 @ 2486s

    this particular path. Only on this this particular path. Only on this particular x0 to x1 path. But it's not particular x0 to x1 path. But it's not particular x0 to x1 path. But it's not in general. in general it's not actually in general. in general it's not actually in general. in general it's not actually I think this may answer your question I think this may answer your question I think this may answer your question yeah so basically in like so the left yeah so basically in like so the left yeah so basically in like so the left hand side is like the the path that you hand side is like the the path that you hand side is like the the path that you construct at training time so everything construct at training time so everything construct at training time so everything is straight right everything is straight is straight right everything is straight is straight right everything is straight but the learn the function will you have but the learn the function will you have but the learn the function will you have this averaging effect right so you learn this averaging effect right so you learn this averaging effect right so you learn the the the the entire field so you the the the the entire field so you the the the the entire field so you don't don't don't have this straight line anymore actually have this straight line anymore actually have this straight line anymore actually but you have this like curve curve the but you have this like curve curve the but you have this like curve curve the slightly curved average field still normalizing flow right yeah still normalizing flow right yeah because because you have the one to one because because you have the one to one because because you have the one to one you have the the the invertible you have the the the invertible you have the the the invertible invertability invertability invertability okay any question okay any question okay any question >> yes >> yes >> yes >> why is it slightly collapsing in the >> why is it slightly collapsing in the >> why is it slightly collapsing in the middle middle middle >> because if you look at the if you look

  119. L5 @ 2544s

    >> because if you look at the if you look >> because if you look at the if you look at the this the the left hand side right at the this the the left hand side right at the this the the left hand side right so like you can see how the field is not so like you can see how the field is not so like you can see how the field is not like the field has a collapsing. So like like the field has a collapsing. So like like the field has a collapsing. So like the the the margin so we're later on the the the margin so we're later on the the the margin so we're later on when when when sampling we're using it as if it's the sampling we're using it as if it's the sampling we're using it as if it's the marginal and the marginal has this like marginal and the marginal has this like marginal and the marginal has this like slight slight concentration in the in slight slight concentration in the in slight slight concentration in the in the middle. Yep. the middle. Yep. the middle. Yep. >> So in this case if I were to pass the >> So in this case if I were to pass the >> So in this case if I were to pass the same gshian input twice to the model I same gshian input twice to the model I same gshian input twice to the model I get the same image out. get the same image out. get the same image out. >> Yes. It's deterministic completely which >> Yes. It's deterministic completely which >> Yes. It's deterministic completely which is different from diffusion right? is different from diffusion right? is different from diffusion right? >> Yes. >> Yes. Okay, any more question? Cool. Actually, this thing is like super Cool. Actually, this thing is like super cool because the same algorithm was cool because the same algorithm was cool because the same algorithm was developed by three different group of developed by three different group of developed by three different group of people at literally the same time like people at literally the same time like people at literally the same time like literally the same conference and they literally the same conference and they

  120. L5 @ 2596s

    literally the same conference and they all got into the same conference. all got into the same conference. all got into the same conference. So yeah, so the first group is a group So yeah, so the first group is a group So yeah, so the first group is a group in Metafair and in Metafair and in Metafair and this is the group that I interned at this is the group that I interned at this is the group that I interned at actually. Um and then the second group I actually. Um and then the second group I actually. Um and then the second group I believe is a group from UT Austin and believe is a group from UT Austin and believe is a group from UT Austin and then the third group is a group from uh then the third group is a group from uh then the third group is a group from uh NYU and they all derive the same NYU and they all derive the same NYU and they all derive the same algorithm from different uh theoretical algorithm from different uh theoretical algorithm from different uh theoretical perspective and all submit to the same perspective and all submit to the same perspective and all submit to the same conference and all got in. conference and all got in. conference and all got in. So this is like sort of like a beautiful So this is like sort of like a beautiful So this is like sort of like a beautiful moment of science where you just kind of moment of science where you just kind of moment of science where you just kind of have like a collapsing ideas but like have like a collapsing ideas but like have like a collapsing ideas but like they're from different points you know they're from different points you know they're from different points you know it's just like our you know flow it's just like our you know flow it's just like our you know flow matching everyone goes to the same point matching everyone goes to the same point matching everyone goes to the same point anyway. Yeah, pretty pretty magical. Um, anyway. Yeah, pretty pretty magical. Um, anyway. Yeah, pretty pretty magical. Um, but you may have this question, right? I but you may have this question, right? I

  121. L5 @ 2640s

    but you may have this question, right? I feel like some people are having this feel like some people are having this feel like some people are having this question but they didn't ask. So like question but they didn't ask. So like question but they didn't ask. So like the marginal flow has some dips to it, the marginal flow has some dips to it, the marginal flow has some dips to it, right? So it doesn't it's not completely right? So it doesn't it's not completely right? So it doesn't it's not completely straight. It's not the straightest. So straight. It's not the straightest. So straight. It's not the straightest. So is there a way that we can like let's is there a way that we can like let's is there a way that we can like let's say now we have trained a flow matching say now we have trained a flow matching say now we have trained a flow matching model now. So now we have this this this model now. So now we have this this this model now. So now we have this this this um like this invertability this coupling um like this invertability this coupling um like this invertability this coupling already. Is there a way to already. Is there a way to already. Is there a way to make this even straighter? Is it make this even straighter? Is it make this even straighter? Is it possible directly move from your P 0 to directly move from your P 0 to >> Yeah, pretty much. Pretty much. Um, so >> Yeah, pretty much. Pretty much. Um, so >> Yeah, pretty much. Pretty much. Um, so this thing is what we call reflow. So this thing is what we call reflow. So this thing is what we call reflow. So this is a technique that developed by by this is a technique that developed by by this is a technique that developed by by the second group of people who proposed the second group of people who proposed the second group of people who proposed the same thing. Uh so basically what you the same thing. Uh so basically what you the same thing. Uh so basically what you do is now let's say you you after do is now let's say you you after

  122. L5 @ 2698s

    do is now let's say you you after training with the linear interpolation training with the linear interpolation training with the linear interpolation you already got this coupling. So the you already got this coupling. So the you already got this coupling. So the blue coupling and the the green coupling blue coupling and the the green coupling blue coupling and the the green coupling and then what you can do is you can just and then what you can do is you can just and then what you can do is you can just like use this coupling as your new like use this coupling as your new like use this coupling as your new ground truth and then instead of like ground truth and then instead of like ground truth and then instead of like randomly sampling from a gausian you randomly sampling from a gausian you randomly sampling from a gausian you sample from the coupling and then you sample from the coupling and then you sample from the coupling and then you can get the straight points can get the straight points can get the straight points and then it become more straight. Let me explain again. Let me explain again. >> Oh, okay. One more time. One more time. >> Oh, okay. One more time. One more time. >> Oh, okay. One more time. One more time. So, basically this is like your first So, basically this is like your first So, basically this is like your first time uh flow matching. So, basically time uh flow matching. So, basically time uh flow matching. So, basically after computing all the the the the in after computing all the the the the in after computing all the the the the in the uh linear interpolations, you get the uh linear interpolations, you get the uh linear interpolations, you get this like slightly dipped um marginals, this like slightly dipped um marginals, this like slightly dipped um marginals, right? You get this slightly dipped right? You get this slightly dipped right? You get this slightly dipped version, but now you have a a coupling version, but now you have a a coupling version, but now you have a a coupling because the thing is deterministic, because the thing is deterministic, because the thing is deterministic, right? So you can map from one point in

  123. L5 @ 2750s

    right? So you can map from one point in right? So you can map from one point in P 0 to another point in P1 and then you P 0 to another point in P1 and then you P 0 to another point in P1 and then you just use that coupling as your new just use that coupling as your new just use that coupling as your new ground truth. So instead of like ground truth. So instead of like ground truth. So instead of like randomly sampling your your your randomly sampling your your your randomly sampling your your your coupling like what we do what we did in coupling like what we do what we did in coupling like what we do what we did in the original flow matching you just use the original flow matching you just use the original flow matching you just use that coupling as your ground truth and that coupling as your ground truth and that coupling as your ground truth and train for matching again and then it's train for matching again and then it's train for matching again and then it's going to get straightened up. >> Yeah. Go ahead. Because when you sample >> Yeah. Go ahead. Because when you sample um the more the more linear your clothes um the more the more linear your clothes um the more the more linear your clothes are, the less test. are, the less test. are, the less test. >> Yes, exactly. So, think about it, right? >> Yes, exactly. So, think about it, right? >> Yes, exactly. So, think about it, right? Um like if you have say if you have this Um like if you have say if you have this Um like if you have say if you have this dip, right? You have you have this dip dip, right? You have you have this dip dip, right? You have you have this dip then like like the safer bet for you to then like like the safer bet for you to then like like the safer bet for you to sample is to take two steps, right? sample is to take two steps, right? sample is to take two steps, right? because you don't know like basically if because you don't know like basically if

  124. L5 @ 2794s

    because you don't know like basically if you if you only do one step there can be you if you only do one step there can be you if you only do one step there can be like a lot of um uh uh like like like a lot of um uh uh like like like a lot of um uh uh like like discretization error that is happening discretization error that is happening discretization error that is happening because of this dip. But if you have a because of this dip. But if you have a because of this dip. But if you have a very straight line then you can just do very straight line then you can just do very straight line then you can just do one step and it's straight. So one step one step and it's straight. So one step one step and it's straight. So one step is fine right? So straighter you are uh is fine right? So straighter you are uh is fine right? So straighter you are uh the better you are at this like few step the better you are at this like few step the better you are at this like few step sampling. How do you decide what to cover? How do you decide what to cover? >> By your previous model, right? By the >> By your previous model, right? By the >> By your previous model, right? By the model that you learned. So, model that you learned. So, model that you learned. So, so this blue this this this blue and so this blue this this this blue and so this blue this this this blue and green are the coupling induced by your green are the coupling induced by your green are the coupling induced by your learn the flow matching model and then learn the flow matching model and then learn the flow matching model and then you just use this this coupling and then you just use this this coupling and then you just use this this coupling and then do flow matching again on the new model. do flow matching again on the new model. do flow matching again on the new model. Yeah. >> It won't collide because it's >> It won't collide because it's It's a flow. It never cross. Yeah.

  125. L5 @ 2857s

    >> The third image is the linear >> The third image is the linear interpolation of the new coupling. interpolation of the new coupling. interpolation of the new coupling. >> That what is that doing? >> That what is that doing? >> That what is that doing? >> Oh, this thing is just like show you the >> Oh, this thing is just like show you the >> Oh, this thing is just like show you the marginal kind of. marginal kind of. marginal kind of. >> Yeah. >> Yeah. >> The green is going up. But so on the >> The green is going up. But so on the >> The green is going up. But so on the >> Yeah. Yeah. Yeah. No. Remember how this >> Yeah. Yeah. Yeah. No. Remember how this >> Yeah. Yeah. Yeah. No. Remember how this you have a you have a concentration in you have a you have a concentration in you have a you have a concentration in the middle. you have a dip, right? So, the middle. you have a dip, right? So, the middle. you have a dip, right? So, it's sort of the same thing here. it's sort of the same thing here. it's sort of the same thing here. >> Is it when you're doing the sampling? >> Is it when you're doing the sampling? >> Is it when you're doing the sampling? >> No, no. Linear interpolation is just >> No, no. Linear interpolation is just >> No, no. Linear interpolation is just coupling. So, like you you you choose coupling. So, like you you you choose coupling. So, like you you you choose you you choose a you you choose a sample you you choose a you you choose a sample you you choose a you you choose a sample here, you choose a sample here and then here, you choose a sample here and then here, you choose a sample here and then you straight line and then you connect you straight line and then you connect you straight line and then you connect them with a straight line. them with a straight line. them with a straight line. >> Sample from both side. We mostly >> Sample from both side. We mostly >> Sample from both side. We mostly connected this side to that side and

  126. L5 @ 2897s

    connected this side to that side and connected this side to that side and that side to this side. There's no that side to this side. There's no that side to this side. There's no there's no this anymore because because there's no this anymore because because there's no this anymore because because it won't be basically the the the it won't be basically the the the it won't be basically the the the coupling tells you that you will not go coupling tells you that you will not go coupling tells you that you will not go to the other side anymore because to the other side anymore because to the other side anymore because otherwise it will cross and then your otherwise it will cross and then your otherwise it will cross and then your stream will never cross. stream will never cross. stream will never cross. >> Yeah. >> Yeah. >> You have a data set that you know has >> You have a data set that you know has >> You have a data set that you know has multiple classes. Could you use this to multiple classes. Could you use this to multiple classes. Could you use this to disentangle them? disentangle them? disentangle them? >> Yes. And also actually um there's >> Yes. And also actually um there's >> Yes. And also actually um there's another paper I didn't put it here that another paper I didn't put it here that another paper I didn't put it here that if you already have a coupling in your if you already have a coupling in your if you already have a coupling in your data set you can just directly use it to data set you can just directly use it to data set you can just directly use it to train flow matching. You don't have to train flow matching. You don't have to train flow matching. You don't have to do random sampling. Yeah, I think do random sampling. Yeah, I think do random sampling. Yeah, I think someone has Yeah. someone has Yeah. someone has Yeah. >> If your flow matching model is not >> If your flow matching model is not >> If your flow matching model is not perfectly trained into this process, perfectly trained into this process,

  127. L5 @ 2936s

    perfectly trained into this process, >> then you do it again. Yeah. Yeah. So, >> then you do it again. Yeah. Yeah. So, >> then you do it again. Yeah. Yeah. So, this is just you you get it this is just you you get it this is just you you get it straightened. It's not guaranteed that straightened. It's not guaranteed that straightened. It's not guaranteed that you're going to get straight uh like you're going to get straight uh like you're going to get straight uh like path on on on the on one go. Just like path on on on the on one go. Just like path on on on the on one go. Just like you're just going to get straighter and you're just going to get straighter and you're just going to get straighter and straighter kind of. Yeah, it it's straighter kind of. Yeah, it it's straighter kind of. Yeah, it it's definitely possible. Yeah. Oh. No. Okay. No, nobody. No. Okay. No, nobody. Cool. All right. So, just to summarize Cool. All right. So, just to summarize Cool. All right. So, just to summarize everything. So, just what's the everything. So, just what's the everything. So, just what's the difference between diffusion and flow difference between diffusion and flow difference between diffusion and flow matching in this context, right? So, matching in this context, right? So, matching in this context, right? So, diffusion is sort of like you are diffusion is sort of like you are diffusion is sort of like you are wandering in the wood and with a compass wandering in the wood and with a compass wandering in the wood and with a compass uh like in your hand. So, you know like uh like in your hand. So, you know like uh like in your hand. So, you know like you kind of know which direction you're you kind of know which direction you're you kind of know which direction you're going, but you're still like you're kind going, but you're still like you're kind going, but you're still like you're kind of just like exploring a little bit as of just like exploring a little bit as of just like exploring a little bit as well. you're just like kind of random well. you're just like kind of random

  128. L5 @ 2984s

    well. you're just like kind of random random walking a little bit. Whereas for random walking a little bit. Whereas for random walking a little bit. Whereas for flow matching, you don't really do flow matching, you don't really do flow matching, you don't really do random walking anymore. You just like random walking anymore. You just like random walking anymore. You just like sit on the boat and then go go down the sit on the boat and then go go down the sit on the boat and then go go down the stream uh on the river. So this is kind stream uh on the river. So this is kind stream uh on the river. So this is kind of what like to conceptually what is of what like to conceptually what is of what like to conceptually what is happening. Yeah. happening. Yeah. happening. Yeah. >> Like is this used now? Like if it's so >> Like is this used now? Like if it's so >> Like is this used now? Like if it's so good this good this good this >> great question. >> great question. >> great question. >> Do you know flux? Do you know the the >> Do you know flux? Do you know the the >> Do you know flux? Do you know the the black for that's full matching? Yeah. black for that's full matching? Yeah. black for that's full matching? Yeah. This is like the state of the art This is like the state of the art This is like the state of the art basically. It's actually crazy good and basically. It's actually crazy good and basically. It's actually crazy good and it's so simple, right? Okay. So, so far it's so simple, right? Okay. So, so far it's so simple, right? Okay. So, so far we have seen so many gen models and we we have seen so many gen models and we we have seen so many gen models and we know that the diffusion model and know that the diffusion model and know that the diffusion model and scorebased model are same thing and scorebased model are same thing and scorebased model are same thing and today we learned flow matching. Now the today we learned flow matching. Now the today we learned flow matching. Now the question is is flow matching also the

  129. L5 @ 3027s

    question is is flow matching also the question is is flow matching also the same thing because if you think about it same thing because if you think about it same thing because if you think about it you're also like adding a lot of noise you're also like adding a lot of noise you're also like adding a lot of noise like and then d noiseis is kind of like and then d noiseis is kind of like and then d noiseis is kind of dinoising right you're just like a lot dinoising right you're just like a lot dinoising right you're just like a lot of noisy samples in the middle is are of noisy samples in the middle is are of noisy samples in the middle is are they the same thing? they the same thing? they the same thing? What do we think kind of actually? So basically what you kind of actually? So basically what you can do so this is what we had before can do so this is what we had before can do so this is what we had before right uh we have this like very straight right uh we have this like very straight right uh we have this like very straight line but what if you define something line but what if you define something line but what if you define something because the flow matching theory can be because the flow matching theory can be because the flow matching theory can be applied to any path right like what applied to any path right like what applied to any path right like what Dwanch was asking so we can also define Dwanch was asking so we can also define Dwanch was asking so we can also define a path from what we just learned uh on a path from what we just learned uh on a path from what we just learned uh on Tuesday right so you can define a a v Tuesday right so you can define a a v Tuesday right so you can define a a v path a v uh like diffusion path and you path a v uh like diffusion path and you path a v uh like diffusion path and you can write everything as if it is flow

  130. L5 @ 3072s

    can write everything as if it is flow can write everything as if it is flow matching as well and then you can matching as well and then you can matching as well and then you can actually just recover for V SD with actually just recover for V SD with actually just recover for V SD with obviously obviously obviously with with the noise adding to it. with with the noise adding to it. with with the noise adding to it. And what about if we do what can can And what about if we do what can can And what about if we do what can can what about the other way around, right? what about the other way around, right? what about the other way around, right? Can we do score SDE Can we do score SDE Can we do score SDE transform into an OD a flow OD? Can we transform into an OD a flow OD? Can we transform into an OD a flow OD? Can we do that? What do we think by looking at do that? What do we think by looking at do that? What do we think by looking at this this this formula? Okay, just get rid of the the G part, Okay, just get rid of the the G part, right? Is that what you said? All right. right? Is that what you said? All right. right? Is that what you said? All right. So, basically at this part look like a So, basically at this part look like a So, basically at this part look like a velocity, right? velocity, right? velocity, right? But the problem is like the G part, the But the problem is like the G part, the But the problem is like the G part, the the brown in motion part, this part the brown in motion part, this part the brown in motion part, this part actually induce some probability because actually induce some probability because actually induce some probability because it's a gausian, right? So, we still it's a gausian, right? So, we still it's a gausian, right? So, we still induce some probability there. So, we induce some probability there. So, we induce some probability there. So, we still need to take care of that part. still need to take care of that part.

  131. L5 @ 3123s

    still need to take care of that part. But it turns out that we actually do But it turns out that we actually do But it turns out that we actually do know how to take care of that part. And know how to take care of that part. And know how to take care of that part. And this thing is called focal plank PDE. this thing is called focal plank PDE. this thing is called focal plank PDE. All right? uh the the name is really All right? uh the the name is really All right? uh the the name is really fancy but basically what it means is fancy but basically what it means is fancy but basically what it means is that like if you if you have some that like if you if you have some that like if you if you have some probability in it the focal point PTE um probability in it the focal point PTE um probability in it the focal point PTE um it's going to tell you it's going to tell you it's going to tell you how your probability changes through how your probability changes through how your probability changes through this through this PTE that you can this through this PTE that you can this through this PTE that you can derive uh and if you're interested in derive uh and if you're interested in derive uh and if you're interested in the derivation of this PD for the the derivation of this PD for the the derivation of this PD for the interest of time we're not going to go interest of time we're not going to go interest of time we're not going to go over it but it's in the in the the over it but it's in the in the the over it but it's in the in the the principle of diffusion model book so you principle of diffusion model book so you principle of diffusion model book so you can take a look at it um but basically can take a look at it um but basically can take a look at it um but basically Okay. And now we have probability flow Okay. And now we have probability flow Okay. And now we have probability flow OD, right? So this is the just um say we

  132. L5 @ 3164s

    OD, right? So this is the just um say we OD, right? So this is the just um say we have this particular OD and it turns out have this particular OD and it turns out have this particular OD and it turns out that it's contin continuity equation that it's contin continuity equation that it's contin continuity equation which is like the the PD of the change which is like the the PD of the change which is like the the PD of the change of um probability which is what we of um probability which is what we of um probability which is what we learned today by using the divergence is learned today by using the divergence is learned today by using the divergence is actually so if you just like plug actually so if you just like plug actually so if you just like plug everything in and then you do a lot of everything in and then you do a lot of everything in and then you do a lot of algebra and and calculus uh you are algebra and and calculus uh you are algebra and and calculus uh you are actually going to get the same thing actually going to get the same thing actually going to get the same thing literally the same thing. So this is literally the same thing. So this is literally the same thing. So this is just saying that the rate of like the just saying that the rate of like the just saying that the rate of like the rate of change in probability rate of change in probability rate of change in probability from this score SD or from this 4 SD from this score SD or from this 4 SD from this score SD or from this 4 SD they share the same uh which is the same they share the same uh which is the same they share the same uh which is the same as this probability flow OD as this probability flow OD as this probability flow OD this and the same PD for the rate of this and the same PD for the rate of this and the same PD for the rate of change for the probability means that change for the probability means that

  133. L5 @ 3210s

    change for the probability means that the marginalss are all the same. So this the marginalss are all the same. So this the marginalss are all the same. So this just means that now if you have your just means that now if you have your just means that now if you have your score function instead of sampling from score function instead of sampling from score function instead of sampling from an SD you can do OD samples now. Yay. Yay. Yay. Yay. Well, no excitement. It's okay. Um what Well, no excitement. It's okay. Um what Well, no excitement. It's okay. Um what else can you else can you else can you with OD? Right. I think someone someone with OD? Right. I think someone someone with OD? Right. I think someone someone said before that like we can do density said before that like we can do density said before that like we can do density estimation with this, right? So estimation with this, right? So estimation with this, right? So literally this is true, right? So you literally this is true, right? So you literally this is true, right? So you can literally just um um swap in the the can literally just um um swap in the the can literally just um um swap in the the velocity here and then you just like go velocity here and then you just like go velocity here and then you just like go back in time through your OD and then back in time through your OD and then back in time through your OD and then you're able to calculate the log density you're able to calculate the log density you're able to calculate the log density of your data point. of your data point. of your data point. Cool. And Cool. And Cool. And what about density estimation for what about density estimation for what about density estimation for diffusion, right? diffusion, right? diffusion, right? Oh, density estimation means that like Oh, density estimation means that like Oh, density estimation means that like just like to calculate uh the likelihood just like to calculate uh the likelihood just like to calculate uh the likelihood of the the the the day. of the the the the day. of the the the the day. >> That's a great question. Um so for

  134. L5 @ 3276s

    >> That's a great question. Um so for >> That's a great question. Um so for example, I don't know if you guys example, I don't know if you guys example, I don't know if you guys familiar with the RL. familiar with the RL. familiar with the RL. >> All right, cool. Anyway, >> All right, cool. Anyway, >> All right, cool. Anyway, and so for density estimation for and so for density estimation for and so for density estimation for diffusion, uh the easiest thing is that diffusion, uh the easiest thing is that diffusion, uh the easiest thing is that because the loss function that we because the loss function that we because the loss function that we derived from two lectures ago is a derived from two lectures ago is a derived from two lectures ago is a elbow, right? So we can just directly elbow, right? So we can just directly elbow, right? So we can just directly use this elbow as an estimate of the use this elbow as an estimate of the use this elbow as an estimate of the likelihood. That's fine. But this is not likelihood. That's fine. But this is not likelihood. That's fine. But this is not accurate because it's a elbow. It's a accurate because it's a elbow. It's a accurate because it's a elbow. It's a lower bound. It's it's not really the lower bound. It's it's not really the lower bound. It's it's not really the the real log log likelihood, right? So the real log log likelihood, right? So the real log log likelihood, right? So is there another way to do it? I feel like someone someone here knows I feel like someone someone here knows already. Okay. What do we think? What do we Okay. What do we think? What do we think? We know we know how to calculate the We know we know how to calculate the exact log likelihood from OD. What do we exact log likelihood from OD. What do we exact log likelihood from OD. What do we do if we have a score SDE? What do we do if we have a score SDE? What do we do if we have a score SDE? What do we do? do? do? Use the use the corresponding PF OD of

  135. L5 @ 3340s

    Use the use the corresponding PF OD of Use the use the corresponding PF OD of the SDE. Right? So you just plug it in the SDE. Right? So you just plug it in the SDE. Right? So you just plug it in and then it's the same and then it's the same and then it's the same as for matching. Yes, Yes, >> it's like that. >> it's like that. >> it's like that. >> Um, >> Um, >> Um, one one one one one >> Okay. >> Okay. >> Okay. >> Yeah, >> Yeah, >> Yeah, you can sample with an OD. What is like you can sample with an OD. What is like you can sample with an OD. What is like sampling? sampling? sampling? >> Oh, yes. Yes. So you can sample with >> Oh, yes. Yes. So you can sample with >> Oh, yes. Yes. So you can sample with this OD meaning that like you can you this OD meaning that like you can you this OD meaning that like you can you can literally do uh let's see you can can literally do uh let's see you can can literally do uh let's see you can you can literally just do numerical you can literally just do numerical you can literally just do numerical integration from x0 and then you do x0 integration from x0 and then you do x0 integration from x0 and then you do x0 plus um integrate from zero to one plus um integrate from zero to one plus um integrate from zero to one velocity and this velocity is just this velocity and this velocity is just this velocity and this velocity is just this odz >> uh what do you mean the sampling is only >> uh what do you mean the sampling is only x0 oh you mean like to sample from a x0 oh you mean like to sample from a x0 oh you mean like to sample from a distribution yes that's right Yeah. distribution yes that's right Yeah. distribution yes that's right Yeah. Cool. Cool. Oh, yeah. Yeah. Yeah. Oh, yeah. Yeah. Yeah. >> Yeah. >> Yeah. >> A lot of uh uh lot of PD getting >> A lot of uh uh lot of PD getting

  136. L5 @ 3427s

    >> A lot of uh uh lot of PD getting converted to OD. Is it are we just doing converted to OD. Is it are we just doing converted to OD. Is it are we just doing it because it's simpler or do we you it because it's simpler or do we you it because it's simpler or do we you want a better velocity? want a better velocity? want a better velocity? >> Uh no you mean here here it's because >> Uh no you mean here here it's because >> Uh no you mean here here it's because it's exact. So in in uh in comparison to it's exact. So in in uh in comparison to it's exact. So in in uh in comparison to the elbow right if you compare to the the elbow right if you compare to the the elbow right if you compare to the elbow the elbow is a lower bound of your elbow the elbow is a lower bound of your elbow the elbow is a lower bound of your um of your lo likelihood which is not um of your lo likelihood which is not um of your lo likelihood which is not exact but here we can actually get an exact but here we can actually get an exact but here we can actually get an exact log likelihood. exact log likelihood. exact log likelihood. Yeah. Yeah. >> Intrinsically Asian suffer from >> Intrinsically Asian suffer from >> Intrinsically Asian suffer from diversity, right? Because diversity, right? Because diversity, right? Because >> it suffers from uh discretization error. >> it suffers from uh discretization error. >> it suffers from uh discretization error. Is that what you mean? Like basically if Is that what you mean? Like basically if Is that what you mean? Like basically if you take the step and then you take the step and then you take the step and then because uh because uh because uh >> ah uh well well here you're calculating >> ah uh well well here you're calculating >> ah uh well well here you're calculating the the the log likelihood. So it's not the the the log likelihood. So it's not the the the log likelihood. So it's not diversity but but yes so like diversity but but yes so like

  137. L5 @ 3473s

    diversity but but yes so like essentially um if you're sampling from a essentially um if you're sampling from a essentially um if you're sampling from a SDE then it's going to be yeah because SDE then it's going to be yeah because SDE then it's going to be yeah because you have exploration right so it's more you have exploration right so it's more you have exploration right so it's more diverse than sampling from OD which is diverse than sampling from OD which is diverse than sampling from OD which is completely deterministic completely deterministic completely deterministic >> yeah that's right Cool. Cool. Not cool. Okay. Not cool. Okay. Not cool. Okay. >> So, if if I want to calculate the >> So, if if I want to calculate the >> So, if if I want to calculate the likelihood of X1, likelihood of X1, likelihood of X1, >> I first need to find the corresponding >> I first need to find the corresponding >> I first need to find the corresponding X0 for me to start integrating. And X0 for me to start integrating. And X0 for me to start integrating. And >> hey, you have everyone you should know >> hey, you have everyone you should know >> hey, you have everyone you should know this. But anyway, continue. Yeah. Yeah. this. But anyway, continue. Yeah. Yeah. this. But anyway, continue. Yeah. Yeah. Yeah. Yeah. Oh yeah. So yeah. So essentially what Oh yeah. So yeah. So essentially what Oh yeah. So yeah. So essentially what you do is you you remember how you you you do is you you remember how you you you do is you you remember how you you you reverse the chain, right? >> Yeah, that is right. >> Yeah, that is right. >> That is right. And which is why this x0 >> That is right. And which is why this x0 >> That is right. And which is why this x0 k has a hat on it. So this is a reverse k has a hat on it. So this is a reverse k has a hat on it. So this is a reverse the x0. So you first reverse your the x0. So you first reverse your the x0. So you first reverse your sampling chain through your by by by

  138. L5 @ 3546s

    sampling chain through your by by by sampling chain through your by by by backward integrating your velocity and backward integrating your velocity and backward integrating your velocity and then during the backward integration you then during the backward integration you then during the backward integration you can alo calculate the divergence at the can alo calculate the divergence at the can alo calculate the divergence at the same time. Yeah. So you you need to find the Yeah. So you you need to find the corresponding chain which is corresponding chain which is corresponding chain which is deterministic. But uh But uh >> p of x0 should be constant. >> p of x0 should be constant. >> p of x0 should be constant. >> Oh, you mean p of x? Yeah, p 0 is >> Oh, you mean p of x? Yeah, p 0 is >> Oh, you mean p of x? Yeah, p 0 is constant. Yeah, that's right. Yeah. constant. Yeah, that's right. Yeah. constant. Yeah, that's right. Yeah. Yeah. If if if it's a uniform. Yeah. If if if it's a uniform. Yeah. If if if it's a uniform. All right. All right. All right. More question. More question. More question. All good. All right. All good. All right. All good. All right. Okay. So Okay. So Okay. So to summarize everything and we have seen to summarize everything and we have seen to summarize everything and we have seen three models so far. Uh diffusion model, three models so far. Uh diffusion model, three models so far. Uh diffusion model, squarebased models, flow matching all squarebased models, flow matching all squarebased models, flow matching all model. They are more or less all the model. They are more or less all the model. They are more or less all the same thing. Same same different. same thing. Same same different. same thing. Same same different. I feel like saying this I'm going to I feel like saying this I'm going to I feel like saying this I'm going to offend a lot of people but but yeah know offend a lot of people but but yeah know offend a lot of people but but yeah know you know just like you know you just add you know just like you know you just add

  139. L5 @ 3612s

    you know just like you know you just add something divide something you it's the something divide something you it's the something divide something you it's the same thing. Okay. same thing. Okay. same thing. Okay. >> Yeah everything is the same thing. Uh >> Yeah everything is the same thing. Uh >> Yeah everything is the same thing. Uh but yeah um so yeah congrat oh okay we but yeah um so yeah congrat oh okay we but yeah um so yeah congrat oh okay we haven't congrat Okay. haven't congrat Okay. haven't congrat Okay. >> Uh so if we can still compute the >> Uh so if we can still compute the >> Uh so if we can still compute the likelihood using flow matching why does likelihood using flow matching why does likelihood using flow matching why does it come under likelihood free like it come under likelihood free like it come under likelihood free like what's the classification? Oh here what's the classification? Oh here what's the classification? Oh here likelihood of free means that the likelihood of free means that the likelihood of free means that the training loss doesn't have likelihood training loss doesn't have likelihood training loss doesn't have likelihood it's not maximizing the likelihood but it's not maximizing the likelihood but it's not maximizing the likelihood but you can still compute likelihood right you can still compute likelihood right you can still compute likelihood right score same for scorebased model you can score same for scorebased model you can score same for scorebased model you can compute the likelihood compute the likelihood compute the likelihood through the same o pfod thing right yeah through the same o pfod thing right yeah through the same o pfod thing right yeah all right all right so congratulations all right all right so congratulations all right all right so congratulations we gone through the basics now yay now we gone through the basics now yay now we gone through the basics now yay now we know what is diffusion one is full we know what is diffusion one is full we know what is diffusion one is full matching yay um yeah so starting from matching yay um yeah so starting from matching yay um yeah so starting from next week we're gonna be able to see next week we're gonna be able to see

  140. L5 @ 3661s

    next week we're gonna be able to see like like like uh different options to further uh different options to further uh different options to further improve what we have learned. Um so for improve what we have learned. Um so for improve what we have learned. Um so for example like I I guess you guys have if example like I I guess you guys have if example like I I guess you guys have if you guys have done the homework you you guys have done the homework you you guys have done the homework you should know this by now there's so many should know this by now there's so many should know this by now there's so many knobs that you can tune right for knobs that you can tune right for knobs that you can tune right for diffusion models like what are you use diffusion models like what are you use diffusion models like what are you use how many number of steps and stuff like how many number of steps and stuff like how many number of steps and stuff like that right so you know we're gonna we're that right so you know we're gonna we're that right so you know we're gonna we're going to learn about that like basically going to learn about that like basically going to learn about that like basically uh there's a paper that like just uh there's a paper that like just uh there's a paper that like just systematically study this uh and we are systematically study this uh and we are systematically study this uh and we are like for those of you who did uh the like for those of you who did uh the like for those of you who did uh the homework you should also notice that homework you should also notice that homework you should also notice that diffusion models sample really really diffusion models sample really really diffusion models sample really really slowly and it's like just like slowly and it's like just like slowly and it's like just like It takes so much time. So how do we make It takes so much time. So how do we make It takes so much time. So how do we make the generation faster? And we're going the generation faster? And we're going

  141. L5 @ 3701s

    the generation faster? And we're going to learn uh to do it with or without to learn uh to do it with or without to learn uh to do it with or without training retraining of the model. And training retraining of the model. And training retraining of the model. And lastly, right now we all the thing that lastly, right now we all the thing that lastly, right now we all the thing that we learn are unconditional generations. we learn are unconditional generations. we learn are unconditional generations. So how can we make the generation more So how can we make the generation more So how can we make the generation more controllable or conditional, right? And controllable or conditional, right? And controllable or conditional, right? And we are actually going to learn it uh to we are actually going to learn it uh to we are actually going to learn it uh to do it with training and also without do it with training and also without do it with training and also without training which is very interesting. All training which is very interesting. All training which is very interesting. All right. Uh that's it for today. I'll see right. Uh that's it for today. I'll see right. Uh that's it for today. I'll see you next week. Uh stay warm this uh this you next week. Uh stay warm this uh this you next week. Uh stay warm this uh this uh this this uh weekend. There's a lot uh this this uh weekend. There's a lot uh this this uh weekend. There's a lot of flow going on.

diffusion-principles-vizuara

  1. L1 @ 4s

    Hello everyone and welcome to the next Hello everyone and welcome to the next lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion models. diffusion models. diffusion models. In the last lecture we started by In the last lecture we started by In the last lecture we started by introducing flow models. introducing flow models. introducing flow models. In particular we first understood the In particular we first understood the In particular we first understood the meaning of flow itself and later we meaning of flow itself and later we meaning of flow itself and later we moved on to understanding what is the moved on to understanding what is the moved on to understanding what is the meaning of flow modeling. meaning of flow modeling. meaning of flow modeling. Let us do a brief recap. Let us do a brief recap. Let us do a brief recap. of what we covered in the last lecture. of what we covered in the last lecture. of what we covered in the last lecture. Before we move ahead to this lecture, and this box includes a lot of tiny and this box includes a lot of tiny particles. particles. particles. So this is particle P1. Let's focus on So this is particle P1. Let's focus on So this is particle P1. Let's focus on that particle. that particle. that particle. Now imagine that for some reason these Now imagine that for some reason these Now imagine that for some reason these particles are moving inside the box. particles are moving inside the box. particles are moving inside the box. So the particles are following specific So the particles are following specific So the particles are following specific trajectories. trajectories. trajectories. For example, this is the trajectory For example, this is the trajectory For example, this is the trajectory which is followed by P1. Then we take another point P2 Then we take another point P2 and we find that this is the trajectory and we find that this is the trajectory and we find that this is the trajectory which is followed by the point P2. which is followed by the point P2. which is followed by the point P2. Similarly, every single point in the box Similarly, every single point in the box

  2. L1 @ 87s

    Similarly, every single point in the box is following a specific trajectory is following a specific trajectory is following a specific trajectory and the particles are moving as and the particles are moving as and the particles are moving as [clears throat] time is progressing [clears throat] time is progressing [clears throat] time is progressing ahead. ahead. ahead. You can imagine this to be like a fluid You can imagine this to be like a fluid You can imagine this to be like a fluid which is moving with time. which is moving with time. which is moving with time. [snorts] Okay. Now [snorts] Okay. Now [snorts] Okay. Now we ask a simple question. What is making these particles move What is making these particles move according to the specific trajectories? For example, For example, at every point, at every point, at every point, [snorts] we know that there is a [snorts] we know that there is a [snorts] we know that there is a velocity vector which is causing the velocity vector which is causing the velocity vector which is causing the particle to move in that direction. >> [snorts] >> [snorts] >> So we try to think about >> So we try to think about >> So we try to think about what is what is responsible for the what is what is responsible for the what is what is responsible for the trajectories of the particles to behave trajectories of the particles to behave trajectories of the particles to behave in a certain way. in a certain way. in a certain way. It's almost like a detective who is It's almost like a detective who is It's almost like a detective who is trying to find out the reason behind a trying to find out the reason behind a trying to find out the reason behind a mysterious behavior. mysterious behavior. mysterious behavior. Now our detective tells us that the Now our detective tells us that the Now our detective tells us that the reason the particles move in this way is reason the particles move in this way is reason the particles move in this way is because because because there is a velocity field which is there is a velocity field which is there is a velocity field which is changing changing changing with not just space but it is changing

  3. L1 @ 169s

    with not just space but it is changing with not just space but it is changing with time also. [snorts] Why it is with time also. [snorts] Why it is with time also. [snorts] Why it is changing with space? It's because the changing with space? It's because the changing with space? It's because the trajectories for all all the particles trajectories for all all the particles trajectories for all all the particles in the box are different. So it has to in the box are different. So it has to in the box are different. So it has to change with space. change with space. change with space. Well, it is not completely necessary Well, it is not completely necessary Well, it is not completely necessary that it changes with time as well. But that it changes with time as well. But that it changes with time as well. But we'll consider a more general case where we'll consider a more general case where we'll consider a more general case where the velocity field [snorts] which is the the velocity field [snorts] which is the the velocity field [snorts] which is the culprit which the the detective has culprit which the the detective has culprit which the the detective has found out is changing with both space found out is changing with both space found out is changing with both space and with time as well. [snorts] and with time as well. [snorts] and with time as well. [snorts] Let us denote this velocity field as u Let us denote this velocity field as u Let us denote this velocity field as u of of of x t. x t. x t. [snorts] The detective has done its job. The The detective has done its job. The detective has found that uh there is a detective has found that uh there is a detective has found that uh there is a velocity field which is um responsible velocity field which is um responsible velocity field which is um responsible for the movement of the trajectories or for the movement of the trajectories or for the movement of the trajectories or the movement of the particles and the the movement of the particles and the the movement of the particles and the generation of these trajectories. generation of these trajectories. generation of these trajectories. Now we approach another person. Let's

  4. L1 @ 224s

    Now we approach another person. Let's Now we approach another person. Let's call that person uh detective number call that person uh detective number call that person uh detective number two. And we ask the question well uh two. And we ask the question well uh two. And we ask the question well uh let's say I give you a velocity field. let's say I give you a velocity field. let's say I give you a velocity field. [snorts] [snorts] [snorts] Can you tell me the trajectories of all Can you tell me the trajectories of all Can you tell me the trajectories of all the particles in the box and how exactly the particles in the box and how exactly the particles in the box and how exactly they will move? they will move? they will move? The detective thinks and says um okay so The detective thinks and says um okay so The detective thinks and says um okay so this looks like a different problem now this looks like a different problem now this looks like a different problem now that you have used the services of that you have used the services of that you have used the services of detective one and understood that there detective one and understood that there detective one and understood that there is an underlying velocity field which is is an underlying velocity field which is is an underlying velocity field which is responsible for the movement of these responsible for the movement of these responsible for the movement of these particles now you're trying to find out particles now you're trying to find out particles now you're trying to find out [snorts] given the velocity field can I [snorts] given the velocity field can I [snorts] given the velocity field can I generate the trajectory so essentially generate the trajectory so essentially generate the trajectory so essentially you're trying to solve the reverse you're trying to solve the reverse you're trying to solve the reverse problem problem problem [snorts] uh so detective one first [snorts] uh so detective one first [snorts] uh so detective one first detective two thinks that okay initially detective two thinks that okay initially detective two thinks that okay initially These people are thinking about going These people are thinking about going These people are thinking about going from trajectories and then calculating

  5. L1 @ 276s

    from trajectories and then calculating from trajectories and then calculating the velocity field. [snorts] Now they the velocity field. [snorts] Now they the velocity field. [snorts] Now they are thinking in the reverse way from are thinking in the reverse way from are thinking in the reverse way from given the velocity field can I calculate given the velocity field can I calculate given the velocity field can I calculate the trajectories of these particles. [snorts] The detective too remembers uh [snorts] The detective too remembers uh some of the some of the some of the theory he had learned in mathematics in theory he had learned in mathematics in theory he had learned in mathematics in college. and comes up with an ordinary college. and comes up with an ordinary college. and comes up with an ordinary differential equation and thinks that differential equation and thinks that differential equation and thinks that okay this is straightforward. We can okay this is straightforward. We can okay this is straightforward. We can simply write down dx of t of t dt dt dt is equal to u of x t. is equal to u of x t. is equal to u of x t. [snorts] [snorts] A simple differential equation which is A simple differential equation which is A simple differential equation which is an ordinary differential equation is an ordinary differential equation is an ordinary differential equation is written to written to written to [snorts] calculate the trajectories. [snorts] calculate the trajectories. [snorts] calculate the trajectories. But then how do we go from here to But then how do we go from here to But then how do we go from here to calculating the trajectories in the calculating the trajectories in the calculating the trajectories in the first place? Well, what you do is you first place? Well, what you do is you first place? Well, what you do is you discretise this discretise this discretise this by saying delta x by delta t is equal to by saying delta x by delta t is equal to by saying delta x by delta t is equal to mu of u of x t >> [snorts] >> So let's say you start at >> So let's say you start at >> So let's say you start at x0 x0 x0 which is a specific point in space.

  6. L1 @ 364s

    which is a specific point in space. which is a specific point in space. [snorts] Let's say this is x0. Now what is x1? X1 is equal to Now what is x1? X1 is equal to X0 + delta X X0 + delta X X0 + delta X and delta X can be written as and delta X can be written as and delta X can be written as delta T * delta T * delta T * U of the velocity at that point X0 and U of the velocity at that point X0 and U of the velocity at that point X0 and the initial time stamp. So let's call it the initial time stamp. So let's call it the initial time stamp. So let's call it zero. zero. zero. So now you get X1. So now you get X1. So now you get X1. So the particle has moved from X0 to X1. So the particle has moved from X0 to X1. So the particle has moved from X0 to X1. Similarly, you can calculate x2 Similarly, you can calculate x2 Similarly, you can calculate x2 x3 and you can get the entire trajectory x3 and you can get the entire trajectory x3 and you can get the entire trajectory of the particle. [snorts] of the particle. [snorts] of the particle. [snorts] Why are we doing this? Well, this means Why are we doing this? Well, this means Why are we doing this? Well, this means that if I know the velocity field and that if I know the velocity field and that if I know the velocity field and how it changes with space and time, I how it changes with space and time, I how it changes with space and time, I can exactly pinpoint the trajectories of can exactly pinpoint the trajectories of can exactly pinpoint the trajectories of all the single part all the particles in all the single part all the particles in all the single part all the particles in the box. the box. the box. So, okay fine. So, that is great. Now so So, okay fine. So, that is great. Now so So, okay fine. So, that is great. Now so far we have looked at individual far we have looked at individual far we have looked at individual particles. We have looked at their

  7. L1 @ 431s

    particles. We have looked at their particles. We have looked at their trajectories. We have looked at the trajectories. We have looked at the trajectories. We have looked at the mysterious force that is causing these mysterious force that is causing these mysterious force that is causing these trajectories. Then we looked at the trajectories. Then we looked at the trajectories. Then we looked at the reverse problem. If we know this reverse problem. If we know this reverse problem. If we know this mysterious force, can we calculate the mysterious force, can we calculate the mysterious force, can we calculate the trajectories? And the answer is yes. trajectories? And the answer is yes. trajectories? And the answer is yes. Now let's come to the main part of uh Now let's come to the main part of uh Now let's come to the main part of uh the discussion which is flow. And now the discussion which is flow. And now the discussion which is flow. And now once we understand this understanding once we understand this understanding once we understand this understanding flow is fairly straightforward. Flow is flow is fairly straightforward. Flow is flow is fairly straightforward. Flow is defined as a collection of the defined as a collection of the defined as a collection of the trajectories of all these particles. trajectories of all these particles. trajectories of all these particles. Every single particle moves in a certain Every single particle moves in a certain Every single particle moves in a certain way, right? So you aggregate the way, right? So you aggregate the way, right? So you aggregate the trajectories of all the particles and trajectories of all the particles and trajectories of all the particles and you call that to be flow. you call that to be flow. you call that to be flow. So it's fairly intuitive what flow So it's fairly intuitive what flow So it's fairly intuitive what flow means. means. means. So a visual description of the flow for So a visual description of the flow for So a visual description of the flow for this same box would look as something this same box would look as something this same box would look as something like follows. like follows. like follows. you have uh all the particles following you have uh all the particles following you have uh all the particles following their trajectories their trajectories

  8. L1 @ 488s

    their trajectories and you collect all these trajectories and you collect all these trajectories and you collect all these trajectories and you call them as a flow map or just and you call them as a flow map or just and you call them as a flow map or just flow. flow. flow. A very [snorts] intuitive way to A very [snorts] intuitive way to A very [snorts] intuitive way to visualize this is to um take a square visualize this is to um take a square visualize this is to um take a square grid, put the square grid on the box and grid, put the square grid on the box and grid, put the square grid on the box and then see how the square grid deforms then see how the square grid deforms then see how the square grid deforms with time. We actually saw this in the with time. We actually saw this in the with time. We actually saw this in the uh last lecture uh last lecture uh last lecture where uh the square grid is nothing but where uh the square grid is nothing but where uh the square grid is nothing but a representation of let's say it's a 4x4 a representation of let's say it's a 4x4 a representation of let's say it's a 4x4 grid 16 points in space and then the grid 16 points in space and then the grid 16 points in space and then the points deform but then you exactly see points deform but then you exactly see points deform but then you exactly see how the grid deforms with time. So it it how the grid deforms with time. So it it how the grid deforms with time. So it it helps you to visually understand what helps you to visually understand what helps you to visually understand what the flow is doing. Is it a spiral flow? the flow is doing. Is it a spiral flow? the flow is doing. Is it a spiral flow? Is it a flow which is um you know Is it a flow which is um you know Is it a flow which is um you know converging towards a sink etc. So you converging towards a sink etc. So you converging towards a sink etc. So you can get a fair idea about what the flow

  9. L1 @ 541s

    can get a fair idea about what the flow can get a fair idea about what the flow is representing is representing is representing by by by having this square grid and having this square grid and having this square grid and understanding how individual points in understanding how individual points in understanding how individual points in this square grid are deforming with this square grid are deforming with this square grid are deforming with time. We saw a very nice example of this time. We saw a very nice example of this time. We saw a very nice example of this in the last lecture. Let me just show in the last lecture. Let me just show in the last lecture. Let me just show that to you. that to you. that to you. Yeah, we saw this example where this is Yeah, we saw this example where this is Yeah, we saw this example where this is the square grid. Imagine you immerse the the square grid. Imagine you immerse the the square grid. Imagine you immerse the square grid in the box and then see how square grid in the box and then see how square grid in the box and then see how it deforms with time. So [snorts] this it deforms with time. So [snorts] this it deforms with time. So [snorts] this is exactly how you can visualize the is exactly how you can visualize the is exactly how you can visualize the flow. flow. Okay. Now this is great. Uh we have Okay. Now this is great. Uh we have Okay. Now this is great. Uh we have understood what flow is. Now the understood what flow is. Now the understood what flow is. Now the question is what is the meaning of a question is what is the meaning of a question is what is the meaning of a flow model? [snorts] flow model? [snorts] flow model? [snorts] Uh and this this brings us to the whole Uh and this this brings us to the whole Uh and this this brings us to the whole objective of why we are focusing on flow objective of why we are focusing on flow objective of why we are focusing on flow flow models in the first place. So far flow models in the first place. So far

  10. L1 @ 590s

    flow models in the first place. So far whatever we have discussed can be whatever we have discussed can be whatever we have discussed can be understood by a physics student easily understood by a physics student easily understood by a physics student easily with absolutely no background in AI or with absolutely no background in AI or with absolutely no background in AI or ML right because this is fairly ML right because this is fairly ML right because this is fairly intuitive stuff in the field of fluid intuitive stuff in the field of fluid intuitive stuff in the field of fluid mechanics or physics or [snorts] even mechanics or physics or [snorts] even mechanics or physics or [snorts] even basic for that matter. Now we relate basic for that matter. Now we relate basic for that matter. Now we relate this to the broader problem which we are this to the broader problem which we are this to the broader problem which we are trying to solve. Remember the broader trying to solve. Remember the broader trying to solve. Remember the broader problem that we are trying to solve is problem that we are trying to solve is problem that we are trying to solve is we want to calculate the probability we want to calculate the probability we want to calculate the probability distribution of the underlying data so distribution of the underlying data so distribution of the underlying data so that we can sample from it. So that we can sample from it. So that we can sample from it. So essentially we are in the domain of deep essentially we are in the domain of deep essentially we are in the domain of deep generative modeling. So [snorts] what we are trying to do So [snorts] what we are trying to do let's say you are given images of cats let's say you are given images of cats let's say you are given images of cats you are given one to 100 images of cats you are given one to 100 images of cats you are given one to 100 images of cats and someone tells you and someone tells you and someone tells you okay generate a new image of a cat. okay generate a new image of a cat.

  11. L1 @ 647s

    okay generate a new image of a cat. Now the way you solve this problem is Now the way you solve this problem is Now the way you solve this problem is you try to find a distribution you try to find a distribution you try to find a distribution which which which governs the behavior or the properties governs the behavior or the properties governs the behavior or the properties of all these images of cats and later of all these images of cats and later of all these images of cats and later you just sample from that distribution. So [snorts] the biggest challenge that So [snorts] the biggest challenge that we are trying to solve is we want to go we are trying to solve is we want to go we are trying to solve is we want to go from data to distribution which is not from data to distribution which is not from data to distribution which is not easy because we have absolutely no idea easy because we have absolutely no idea easy because we have absolutely no idea what the underlying distribution is what the underlying distribution is what the underlying distribution is going to look like. [snorts] Okay. Now uh we have seen how to solve Okay. Now uh we have seen how to solve Okay. Now uh we have seen how to solve this classes of problems. We have the this classes of problems. We have the this classes of problems. We have the first model which we used to solve this first model which we used to solve this first model which we used to solve this was variational autoenccoders. was variational autoenccoders. was variational autoenccoders. Then we looked at diffusion models which Then we looked at diffusion models which Then we looked at diffusion models which is another class of models to solve this is another class of models to solve this is another class of models to solve this problem. What typically happens in problem. What typically happens in problem. What typically happens in diffusion models is diffusion models is diffusion models is to find this distribution you start with to find this distribution you start with to find this distribution you start with a gshian distribution that is noise and a gshian distribution that is noise and

  12. L1 @ 710s

    a gshian distribution that is noise and you end up with the actual distribution. you end up with the actual distribution. you end up with the actual distribution. So the path that we are going to follow So the path that we are going to follow So the path that we are going to follow here is that in in flow models is here is that in in flow models is here is that in in flow models is [snorts] [snorts] um we are going to assume that we have um we are going to assume that we have um we are going to assume that we have an initial distribution which is denoted an initial distribution which is denoted an initial distribution which is denoted by x of0 by x of0 by x of0 which is given by pinet. which is given by pinet. which is given by pinet. This is the distribution that we are This is the distribution that we are This is the distribution that we are aware about. It is probably a gshian aware about. It is probably a gshian aware about. It is probably a gshian distribution which is uniformly distribution which is uniformly distribution which is uniformly distributed in space distributed in space distributed in space [snorts] and we want to end up [snorts] and we want to end up [snorts] and we want to end up at the distribution of the data that we at the distribution of the data that we at the distribution of the data that we want to predict. want to predict. want to predict. Here 0 represents the time step t=0 and Here 0 represents the time step t=0 and Here 0 represents the time step t=0 and 1 represents the time t= 1. [snorts] So 1 represents the time t= 1. [snorts] So 1 represents the time t= 1. [snorts] So we want to do something which changes we want to do something which changes we want to do something which changes the initial distribution to the final the initial distribution to the final the initial distribution to the final distribution. distribution. distribution. >> [snorts] >> [snorts] >> Now >> Now >> Now people thought of casting this problem people thought of casting this problem people thought of casting this problem in a way which is similar to how the

  13. L1 @ 775s

    in a way which is similar to how the in a way which is similar to how the particles are moving in space and time. particles are moving in space and time. particles are moving in space and time. What do I mean by that? Well, imagine What do I mean by that? Well, imagine What do I mean by that? Well, imagine that that that your pinet which is the initial your pinet which is the initial your pinet which is the initial distribution distribution distribution is just given by a bunch of particles is just given by a bunch of particles is just given by a bunch of particles which are arranged uniformly which are arranged uniformly which are arranged uniformly symmetrically in space like this. symmetrically in space like this. symmetrically in space like this. uniform, completely uniform. uniform, completely uniform. uniform, completely uniform. And then you make these particles, let's And then you make these particles, let's And then you make these particles, let's say there are thousand of these say there are thousand of these say there are thousand of these particles, you make these particles move particles, you make these particles move particles, you make these particles move in such a way that you change it to the distribution that you change it to the distribution that you want to predict. So this is a spiral you want to predict. So this is a spiral you want to predict. So this is a spiral distribution, let's say. distribution, let's say. distribution, let's say. So let's let's look at a figure to So let's let's look at a figure to So let's let's look at a figure to understand this or a animation. So you understand this or a animation. So you understand this or a animation. So you see here we are initially starting with see here we are initially starting with see here we are initially starting with a gshian distribution and we slowly move a gshian distribution and we slowly move a gshian distribution and we slowly move to the distribution of a heart shape to the distribution of a heart shape to the distribution of a heart shape that we want to predict. that we want to predict. that we want to predict. So we are casting the deep generative So we are casting the deep generative

  14. L1 @ 840s

    So we are casting the deep generative modeling problem modeling problem modeling problem as a problem where we are trying to as a problem where we are trying to as a problem where we are trying to predict the flow map from the initial predict the flow map from the initial predict the flow map from the initial distribution of the particles to the distribution of the particles to the distribution of the particles to the final distribution of the particles. final distribution of the particles. final distribution of the particles. [snorts] Now from the initial discussion [snorts] Now from the initial discussion [snorts] Now from the initial discussion we had your question might be okay how we had your question might be okay how we had your question might be okay how how do I generate this map in the first how do I generate this map in the first how do I generate this map in the first place? How do I generate this flow? And place? How do I generate this flow? And place? How do I generate this flow? And we just discussed that the trajectories we just discussed that the trajectories we just discussed that the trajectories of the particles of the particles of the particles completely depend on the velocity field. completely depend on the velocity field. completely depend on the velocity field. Right? So what if you write down the Right? So what if you write down the Right? So what if you write down the equation as dx by dt. equation as dx by dt. equation as dx by dt. Now here x represents the evolution of Now here x represents the evolution of Now here x represents the evolution of the probability field the probability field the probability field which is similar to the trajecolution of which is similar to the trajecolution of which is similar to the trajecolution of the trajectories followed by every the trajectories followed by every the trajectories followed by every single particle which are representing single particle which are representing single particle which are representing that field according to a that field according to a that field according to a velocity field which depends on x velocity field which depends on x velocity field which depends on x [snorts] and it depends on time as well. [snorts] and it depends on time as well.

  15. L1 @ 905s

    [snorts] and it depends on time as well. So this is exactly the same equation So this is exactly the same equation So this is exactly the same equation that we looked at before. that we looked at before. that we looked at before. Now this is something if we know this Now this is something if we know this Now this is something if we know this velocity field we can calculate exactly velocity field we can calculate exactly velocity field we can calculate exactly how the trajectories will move and we how the trajectories will move and we how the trajectories will move and we want to find the field which can take us want to find the field which can take us want to find the field which can take us from the initial distribution to the from the initial distribution to the from the initial distribution to the final data distribution. final data distribution. final data distribution. So [snorts] we have cast the problem of So [snorts] we have cast the problem of So [snorts] we have cast the problem of deep generative modeling into a problem deep generative modeling into a problem deep generative modeling into a problem where we want to predict the velocity where we want to predict the velocity where we want to predict the velocity field which governs the trajectories of field which governs the trajectories of field which governs the trajectories of these particles. these particles. these particles. Now this is exactly what is called as a Now this is exactly what is called as a Now this is exactly what is called as a flow model. The name is slightly misleading because The name is slightly misleading because I think this should be called as a I think this should be called as a I think this should be called as a velocity field model because in the end velocity field model because in the end velocity field model because in the end what people do is that they approximate what people do is that they approximate what people do is that they approximate this with a neural network and they try this with a neural network and they try this with a neural network and they try to predict this. So this is approximated to predict this. So this is approximated

  16. L1 @ 954s

    to predict this. So this is approximated by a neural network. by a neural network. by a neural network. Okay. And these are the conditions we Okay. And these are the conditions we Okay. And these are the conditions we want to satisfy. X of0 is the initial want to satisfy. X of0 is the initial want to satisfy. X of0 is the initial gshian distribution and we want to move gshian distribution and we want to move gshian distribution and we want to move to our actual data distribution. to our actual data distribution. to our actual data distribution. If you want to revise this properly, If you want to revise this properly, If you want to revise this properly, please go ahead and refer to the please go ahead and refer to the please go ahead and refer to the previous lecture where I have discussed previous lecture where I have discussed previous lecture where I have discussed this in complete detail. this in complete detail. this in complete detail. [snorts] [snorts] Okay. So, uh now the question is Okay. So, uh now the question is Okay. So, uh now the question is [snorts] [snorts] we want to simulate an OD which looks as we want to simulate an OD which looks as we want to simulate an OD which looks as follows. follows. follows. dxt by dt is equal to dxt by dt is equal to dxt by dt is equal to the velocity field the velocity field the velocity field >> [snorts] >> [snorts] >> uh at at that time and at the location >> uh at at that time and at the location >> uh at at that time and at the location in space. in space. in space. [snorts] Conceptually this looks [snorts] Conceptually this looks [snorts] Conceptually this looks excellent but the main question is that excellent but the main question is that excellent but the main question is that how do we go ahead and find this ideal how do we go ahead and find this ideal how do we go ahead and find this ideal velocity field that transports our velocity field that transports our velocity field that transports our initial probability distribution to the initial probability distribution to the initial probability distribution to the distribution that we want to predict in distribution that we want to predict in

  17. L1 @ 1016s

    distribution that we want to predict in the first place. the first place. the first place. And it turns out that this is at the end And it turns out that this is at the end And it turns out that this is at the end you will see this is very very easy and you will see this is very very easy and you will see this is very very easy and straightforward. straightforward. straightforward. But as I said in the first lecture we But as I said in the first lecture we But as I said in the first lecture we are going to take a path which is going are going to take a path which is going are going to take a path which is going to help us understand and appreciate the to help us understand and appreciate the to help us understand and appreciate the final easy formulation of this in a much final easy formulation of this in a much final easy formulation of this in a much better way. Okay. So the first thing that we think Okay. So the first thing that we think about is let's say we u about is let's say we u about is let's say we u use a neural network to approximate this use a neural network to approximate this use a neural network to approximate this velocity field and we randomly velocity field and we randomly velocity field and we randomly initialize the parameters theta of the initialize the parameters theta of the initialize the parameters theta of the neural network. neural network. neural network. Well then the OD will just produce Well then the OD will just produce Well then the OD will just produce nonsense. You will start with a washian nonsense. You will start with a washian nonsense. You will start with a washian distribution but you might end up with distribution but you might end up with distribution but you might end up with something which is completely random. something which is completely random. something which is completely random. So you're trying to flow the So you're trying to flow the So you're trying to flow the distribution in a specific way, right? distribution in a specific way, right? distribution in a specific way, right? But what if you start with a gshian

  18. L1 @ 1072s

    But what if you start with a gshian But what if you start with a gshian distribution and you end up with distribution and you end up with distribution and you end up with something that does not represent your something that does not represent your something that does not represent your data at all. So you can't init randomly data at all. So you can't init randomly data at all. So you can't init randomly initialize the weights. initialize the weights. initialize the weights. This is where we come to the main This is where we come to the main This is where we come to the main concept in machine learning that we need concept in machine learning that we need concept in machine learning that we need a t we need a target to train our neural a t we need a target to train our neural a t we need a target to train our neural network. network. network. Now usually the target in let's say Now usually the target in let's say Now usually the target in let's say supervised learning task is known to us. supervised learning task is known to us. supervised learning task is known to us. For example, if we are trying to uh For example, if we are trying to uh For example, if we are trying to uh predict predict predict the house prices which are going to the house prices which are going to the house prices which are going to change in the coming years. We generally change in the coming years. We generally change in the coming years. We generally use a regression model to solve this use a regression model to solve this use a regression model to solve this problem where we have a previous set of problem where we have a previous set of problem where we have a previous set of data and we extrapolate to understand data and we extrapolate to understand data and we extrapolate to understand what happens in the next 5 years. what happens in the next 5 years. what happens in the next 5 years. So the target is very clear. The target So the target is very clear. The target So the target is very clear. The target is the price of the house. In this case,

  19. L1 @ 1125s

    is the price of the house. In this case, is the price of the house. In this case, we want to minimize the loss function so we want to minimize the loss function so we want to minimize the loss function so that that the predicted velocity vector field the predicted velocity vector field the predicted velocity vector field matches the target velocity vector matches the target velocity vector matches the target velocity vector field. field. field. [snorts] All of us intuitively [snorts] All of us intuitively [snorts] All of us intuitively understand this. This is very similar to understand this. This is very similar to understand this. This is very similar to what we do in all types of machine what we do in all types of machine what we do in all types of machine learning problems where we are trying to learning problems where we are trying to learning problems where we are trying to predict the parameters of a network and predict the parameters of a network and predict the parameters of a network and there is a target which we are trying to there is a target which we are trying to there is a target which we are trying to match. So in most of the cases in match. So in most of the cases in match. So in most of the cases in supervised learning tasks this target is supervised learning tasks this target is supervised learning tasks this target is absolutely known to us. So it becomes absolutely known to us. So it becomes absolutely known to us. So it becomes easy to find the parameter so that the easy to find the parameter so that the easy to find the parameter so that the loss between our model and the target is loss between our model and the target is loss between our model and the target is minimized. minimized. minimized. Now as a mathematical formulation this Now as a mathematical formulation this Now as a mathematical formulation this looks very clean but let us come to the looks very clean but let us come to the looks very clean but let us come to the core problem. The core problem here is that predicting The core problem here is that predicting this target is not very easy. In fact we this target is not very easy. In fact we

  20. L1 @ 1182s

    this target is not very easy. In fact we have absolutely no idea what this target have absolutely no idea what this target have absolutely no idea what this target is. is. is. And that is going to be the focus of And that is going to be the focus of And that is going to be the focus of this lecture is to make it somewhat easy this lecture is to make it somewhat easy this lecture is to make it somewhat easy for us to identify this target. Now Now at the end you will see that the final at the end you will see that the final at the end you will see that the final training target is just a linear training target is just a linear training target is just a linear equation. It is very easy but we are equation. It is very easy but we are equation. It is very easy but we are going to take some time to come to that going to take some time to come to that going to take some time to come to that training target. training target. training target. So in this lecture our goal is to find So in this lecture our goal is to find So in this lecture our goal is to find an equation for the training target. an equation for the training target. an equation for the training target. Maybe we'll not be able to complete it Maybe we'll not be able to complete it Maybe we'll not be able to complete it in this specific lecture but by the next in this specific lecture but by the next in this specific lecture but by the next lecture we will have the complete flow lecture we will have the complete flow lecture we will have the complete flow modeling pipeline ready. Before we get to the training target and Before we get to the training target and uh how we predict the training target, uh how we predict the training target, uh how we predict the training target, we need to understand some terminologies we need to understand some terminologies we need to understand some terminologies which are very critical to understand which are very critical to understand which are very critical to understand this this this uh how the training target is predicted

  21. L1 @ 1246s

    uh how the training target is predicted uh how the training target is predicted in the first place. in the first place. in the first place. And whatever I'm going to discuss next And whatever I'm going to discuss next And whatever I'm going to discuss next is going to form the critical piece of is going to form the critical piece of is going to form the critical piece of training flow models which is used in training flow models which is used in training flow models which is used in most of the modern uh diffusion models most of the modern uh diffusion models most of the modern uh diffusion models or flow-based models which are either or flow-based models which are either or flow-based models which are either used for image generation, video used for image generation, video used for image generation, video generation or even in robotics for that generation or even in robotics for that generation or even in robotics for that matter. matter. matter. [snorts] [snorts] Okay. So remember our goal is to convert Okay. So remember our goal is to convert Okay. So remember our goal is to convert the initial probability distribution to a final data distribution. So here to a final data distribution. So here [snorts] [snorts] this is the initial distribution which this is the initial distribution which this is the initial distribution which is uh noise. This is simply noise. It is uh noise. This is simply noise. It is uh noise. This is simply noise. It does not have any structure to it. does not have any structure to it. does not have any structure to it. But uh for the example of cats that we But uh for the example of cats that we But uh for the example of cats that we have referred to several times in this have referred to several times in this have referred to several times in this series, series, series, we want to get to a distribution which we want to get to a distribution which we want to get to a distribution which looks like this where you pick any point and you get a where you pick any point and you get a cat which is maybe specific to that cat which is maybe specific to that cat which is maybe specific to that breed. breed.

  22. L1 @ 1311s

    breed. So we have different So we have different So we have different we have a probability distribution that we have a probability distribution that we have a probability distribution that you know that is grounded in data which you know that is grounded in data which you know that is grounded in data which actually makes sense which you can actually makes sense which you can actually makes sense which you can sample from and generate images of cats. sample from and generate images of cats. sample from and generate images of cats. Now a naive thinking would tell you that Now a naive thinking would tell you that Now a naive thinking would tell you that well to go from noise to data somewhere well to go from noise to data somewhere well to go from noise to data somewhere we need we need to use the data right we need we need to use the data right we need we need to use the data right otherwise and that has to be a big part otherwise and that has to be a big part otherwise and that has to be a big part of our training target otherwise how of our training target otherwise how of our training target otherwise how will our model know that the final will our model know that the final will our model know that the final distribution that we want to morph the distribution that we want to morph the distribution that we want to morph the initial noisy distribution to represents initial noisy distribution to represents initial noisy distribution to represents the distribution of cats in this case. the distribution of cats in this case. the distribution of cats in this case. So uh okay so here we will use a So uh okay so here we will use a So uh okay so here we will use a [snorts] most general and most simple [snorts] most general and most simple [snorts] most general and most simple technique which is in line with most of technique which is in line with most of technique which is in line with most of the state-of-the-art models what is the state-of-the-art models what is the state-of-the-art models what is going to follow next and uh we are going going to follow next and uh we are going

  23. L1 @ 1366s

    going to follow next and uh we are going to go through a little bit of to go through a little bit of to go through a little bit of mathematics right now so please don't be mathematics right now so please don't be mathematics right now so please don't be repelled by the maths which you are repelled by the maths which you are repelled by the maths which you are about to see it is very intuitive I'm about to see it is very intuitive I'm about to see it is very intuitive I'm not going to introduce any equations not going to introduce any equations not going to introduce any equations it's just going to be some terminologies it's just going to be some terminologies it's just going to be some terminologies which I'm going to introduce which I'm going to introduce which I'm going to introduce but once we understand these but once we understand these but once we understand these terminologies it will help us understand terminologies it will help us understand terminologies it will help us understand flow models in a much better way and in flow models in a much better way and in flow models in a much better way and in fact we'll also be drawing some fact we'll also be drawing some fact we'll also be drawing some parallels with the diffusion models parallels with the diffusion models parallels with the diffusion models which we have looked at before in the which we have looked at before in the which we have looked at before in the previous lectures. previous lectures. previous lectures. [snorts] So if you are new to the field [snorts] So if you are new to the field [snorts] So if you are new to the field of deep generative modeling um don't of deep generative modeling um don't of deep generative modeling um don't worry about diffusion models for now. We worry about diffusion models for now. We worry about diffusion models for now. We will make sure that the following will make sure that the following will make sure that the following terminologies can be understood by terminologies can be understood by terminologies can be understood by everyone who is new to the field as everyone who is new to the field as everyone who is new to the field as well.

  24. L1 @ 1418s

    Okay. So, first concept that we will Okay. So, first concept that we will discuss is called as So let's let's try to understand what is So let's let's try to understand what is the meaning of [snorts] conditional the meaning of [snorts] conditional the meaning of [snorts] conditional probability path. probability path. probability path. Conditional probability path simply Conditional probability path simply Conditional probability path simply means that given a point zed. means that given a point zed. means that given a point zed. We define the conditional probability We define the conditional probability We define the conditional probability path as a transformation from the path as a transformation from the path as a transformation from the initial distri the initial distribution initial distri the initial distribution initial distri the initial distribution pinet pinet pinet to the point zed. to the point zed. to the point zed. So we are not trying here to morph the So we are not trying here to morph the So we are not trying here to morph the initial distribution to the final initial distribution to the final initial distribution to the final unknown distribution that we have no unknown distribution that we have no unknown distribution that we have no idea about. What we have idea about is idea about. What we have idea about is idea about. What we have idea about is the data. I know individual images of the data. I know individual images of the data. I know individual images of the cats, right? the cats, right? the cats, right? So there is some data which is given to So there is some data which is given to So there is some data which is given to me and the first thought that comes to me and the first thought that comes to me and the first thought that comes to my mind is instead of taking the initial my mind is instead of taking the initial my mind is instead of taking the initial gshian distribution to a distribution gshian distribution to a distribution gshian distribution to a distribution which I do not know. What if I take it which I do not know. What if I take it which I do not know. What if I take it to something I do know even if it is to something I do know even if it is

  25. L1 @ 1491s

    to something I do know even if it is like just one simple single data point like just one simple single data point like just one simple single data point [snorts] and that is exactly what [snorts] and that is exactly what [snorts] and that is exactly what conditional probability path means. Why conditional probability path means. Why conditional probability path means. Why conditional? because it is conditional. conditional? because it is conditional. conditional? because it is conditional. It is conditioned on that data sample. It is conditioned on that data sample. It is conditioned on that data sample. We are not predicting the We are not predicting the We are not predicting the complete probability distribution or the complete probability distribution or the complete probability distribution or the complete path but we are pro we are complete path but we are pro we are complete path but we are pro we are predicting the path which is conditioned predicting the path which is conditioned predicting the path which is conditioned on the point Z. on the point Z. on the point Z. [snorts] Let's try to visualize this to [snorts] Let's try to visualize this to [snorts] Let's try to visualize this to make sure we understand this properly. make sure we understand this properly. make sure we understand this properly. Now in the figure you can see that Now in the figure you can see that Now in the figure you can see that the big circle is the initial the big circle is the initial the big circle is the initial distribution and the green is the point distribution and the green is the point distribution and the green is the point where we want to move the initial where we want to move the initial where we want to move the initial distribution towards. distribution towards. distribution towards. Okay. So this is how our transformation is going this is how our transformation is going to look like. [snorts] And the good part to look like. [snorts] And the good part to look like. [snorts] And the good part about this is that we know point Z. We about this is that we know point Z. We about this is that we know point Z. We know the data which is given to us. We know the data which is given to us. We

  26. L1 @ 1551s

    know the data which is given to us. We know this initial distribution also. know this initial distribution also. know this initial distribution also. So it is very rational to think that we So it is very rational to think that we So it is very rational to think that we can probably predict this we can probably predict the velocity we can probably predict the velocity field that field that field that transforms this initial distribution and transforms this initial distribution and transforms this initial distribution and takes it to the final point Z. So how how do we construct this path? So how how do we construct this path? Let's first talk about the probability Let's first talk about the probability Let's first talk about the probability path. What do I mean by a probability path. What do I mean by a probability path. What do I mean by a probability path? Well, let's see. path? Well, let's see. path? Well, let's see. The probability at time t =0, the The probability at time t =0, the The probability at time t =0, the probability distribution is let's say probability distribution is let's say probability distribution is let's say uniform. uniform. uniform. And at time t = 1, there is no distribution. It is just one there is no distribution. It is just one single point zed. Right? single point zed. Right? single point zed. Right? Now what probability path tells us is Now what probability path tells us is Now what probability path tells us is that at every time step in this that at every time step in this that at every time step in this transformation transformation transformation can you generate a probability can you generate a probability can you generate a probability distribution for me. That is the meaning distribution for me. That is the meaning distribution for me. That is the meaning of probability path. And why conditional of probability path. And why conditional of probability path. And why conditional probability path? Because we have probability path? Because we have probability path? Because we have already know that the finally it should already know that the finally it should already know that the finally it should go to the point Z. So it is conditioned go to the point Z. So it is conditioned

  27. L1 @ 1625s

    go to the point Z. So it is conditioned on the point Z. on the point Z. So what can I do so that each individual So what can I do so that each individual So what can I do so that each individual distribution I know and I can take from distribution I know and I can take from distribution I know and I can take from the initial distribution to the final the initial distribution to the final distribution and what we do is very simple in in and what we do is very simple in in reality what is done is let's say the initial distribution is let's say the initial distribution is such that the mean is zero such that the mean is zero such that the mean is zero and the standard deviation is And now the final distribution is such that now the final distribution is such that the mean is zed the mean is zed the mean is zed and the deviation is zero because every and the deviation is zero because every and the deviation is zero because every time you sample you should get zed. It's time you sample you should get zed. It's time you sample you should get zed. It's a fixed point. Right? [snorts] a fixed point. Right? [snorts] a fixed point. Right? [snorts] Now what if I define Now what if I define Now what if I define every single every single every single time step to be a gshian such that the time step to be a gshian such that the time step to be a gshian such that the mu starts from zero and it ends up with mu starts from zero and it ends up with mu starts from zero and it ends up with zed zed zed at time t = 1. at time t = 1. at time t = 1. This is time t equal to0 and uh this is z and uh this is z and the variance starts with one one and the variance ends at zero. What this will do is that it will What this will do is that it will provide me with the probability path provide me with the probability path provide me with the probability path such that at every time step I will have

  28. L1 @ 1726s

    such that at every time step I will have such that at every time step I will have a gshian and a gshian and a gshian and the probability distribution will change the probability distribution will change the probability distribution will change in such a way that the gshian will in such a way that the gshian will in such a way that the gshian will slowly contract and be displaced towards slowly contract and be displaced towards slowly contract and be displaced towards the point z. the point z. the point z. I can visualize this very nicely here. I can visualize this very nicely here. I can visualize this very nicely here. Initially it starts like this with a Initially it starts like this with a Initially it starts like this with a mean zero and a deviation of one. mean zero and a deviation of one. mean zero and a deviation of one. [snorts] Now this is the point which I [snorts] Now this is the point which I [snorts] Now this is the point which I have conditioned on. I have to move the have conditioned on. I have to move the have conditioned on. I have to move the distribution finally to this point. So distribution finally to this point. So distribution finally to this point. So you can see the mean is shifting as we you can see the mean is shifting as we you can see the mean is shifting as we go from time step t=0 to time step t= 1. go from time step t=0 to time step t= 1. go from time step t=0 to time step t= 1. The mean is slowly shifting towards the The mean is slowly shifting towards the The mean is slowly shifting towards the right and it's becoming narrow and right and it's becoming narrow and right and it's becoming narrow and narrower and narrower. So the deviation narrower and narrower. So the deviation narrower and narrower. So the deviation is reducing and finally I get a single is reducing and finally I get a single is reducing and finally I get a single point which is zed. point which is zed. point which is zed. So I have defined a conditional So I have defined a conditional So I have defined a conditional probability path which is linear in

  29. L1 @ 1779s

    probability path which is linear in probability path which is linear in time. time. time. So the mean increases from zero to the So the mean increases from zero to the So the mean increases from zero to the value of zed and the deviation decreases value of zed and the deviation decreases value of zed and the deviation decreases from 1 to 0. So [snorts] this can be visualized as So [snorts] this can be visualized as follows. Here also we have taken the follows. Here also we have taken the follows. Here also we have taken the example of Z= 3. The mean increases from example of Z= 3. The mean increases from example of Z= 3. The mean increases from 0 to 3 and the variance decreases from 1 0 to 3 and the variance decreases from 1 0 to 3 and the variance decreases from 1 to 0. to 0. to 0. So now you know I am I know the So now you know I am I know the So now you know I am I know the trajectories which are followed by every trajectories which are followed by every trajectories which are followed by every single particle. single particle. single particle. And uh And uh And uh because I know the trajectories which because I know the trajectories which because I know the trajectories which are followed by every single particle, are followed by every single particle, are followed by every single particle, you might refer to our discussion about you might refer to our discussion about you might refer to our discussion about the detectives and say that well then the detectives and say that well then the detectives and say that well then maybe I can predict the velocity field maybe I can predict the velocity field maybe I can predict the velocity field here, right? here, right? here, right? [snorts] Maybe I know the velocity field [snorts] Maybe I know the velocity field [snorts] Maybe I know the velocity field because since I know the trajectories because since I know the trajectories because since I know the trajectories very clearly, very clearly, very clearly, I can find the velocity field which is I can find the velocity field which is I can find the velocity field which is going to generate those trajectories. going to generate those trajectories.

  30. L1 @ 1839s

    going to generate those trajectories. uh and this is exactly what we are going uh and this is exactly what we are going uh and this is exactly what we are going to do next. But before we move, I want to do next. But before we move, I want to do next. But before we move, I want you to clearly understand what we have you to clearly understand what we have you to clearly understand what we have done so far. This does not mean we have done so far. This does not mean we have done so far. This does not mean we have solved the problem. We have simply let's solved the problem. We have simply let's solved the problem. We have simply let's say there are thousand images of cats, say there are thousand images of cats, say there are thousand images of cats, right? That are given to us. Now for right? That are given to us. Now for right? That are given to us. Now for every single image, we can construct a every single image, we can construct a every single image, we can construct a probability path which transforms the probability path which transforms the probability path which transforms the initial gshian distribution and takes it initial gshian distribution and takes it initial gshian distribution and takes it to that single image. We do this for all to that single image. We do this for all to that single image. We do this for all the thousand images. So far we have only the thousand images. So far we have only the thousand images. So far we have only discussed this much. We have nowhere discussed this much. We have nowhere discussed this much. We have nowhere discussed about discussed about discussed about uh how to construct a training target. uh how to construct a training target. uh how to construct a training target. But it turns out that this is exactly But it turns out that this is exactly But it turns out that this is exactly what we use for the training target. But what we use for the training target. But what we use for the training target. But anyways uh for now just think about this anyways uh for now just think about this

  31. L1 @ 1883s

    anyways uh for now just think about this as being a conditional probability path as being a conditional probability path as being a conditional probability path which is conditioned on the point Z which is conditioned on the point Z which is conditioned on the point Z head. head. head. Now this technique of conditioning is at Now this technique of conditioning is at Now this technique of conditioning is at the heart of deep generative modeling the heart of deep generative modeling the heart of deep generative modeling which I'm also realizing as I'm which I'm also realizing as I'm which I'm also realizing as I'm researching this topic. It is coming up researching this topic. It is coming up researching this topic. It is coming up everywhere to design training targets everywhere to design training targets everywhere to design training targets even for score based models diffusion even for score based models diffusion even for score based models diffusion models. Wherever we are not able to models. Wherever we are not able to models. Wherever we are not able to construct a training target people construct a training target people construct a training target people condition it on the data sample and then condition it on the data sample and then condition it on the data sample and then magically we can construct the training magically we can construct the training magically we can construct the training target. target. target. We will come to that how the We will come to that how the We will come to that how the construction appears but for now just construction appears but for now just construction appears but for now just make sure you understand the conditional make sure you understand the conditional make sure you understand the conditional probability path and understand the the probability path and understand the the probability path and understand the the terms probability path why because terms probability path why because terms probability path why because probability path is nothing but a probability path is nothing but a probability path is nothing but a trajectory. We are transforming the trajectory. We are transforming the trajectory. We are transforming the initial gshian distribution to the final initial gshian distribution to the final initial gshian distribution to the final single point that is the probability single point that is the probability

  32. L1 @ 1935s

    single point that is the probability path and we want to know at every single path and we want to know at every single path and we want to know at every single time step in this transition how does time step in this transition how does time step in this transition how does the probability distribution look like. the probability distribution look like. the probability distribution look like. Why conditional is because this is not Why conditional is because this is not Why conditional is because this is not the actual probability path. It is the actual probability path. It is the actual probability path. It is conditioned on the point's head. conditioned on the point's head. conditioned on the point's head. [snorts] [snorts] Okay. Now as we discussed before because Okay. Now as we discussed before because Okay. Now as we discussed before because now we know the trajectories of each now we know the trajectories of each now we know the trajectories of each particle we we know that there should be particle we we know that there should be particle we we know that there should be a velocity field associated with it a velocity field associated with it a velocity field associated with it right. right. right. So for any conditional probability path So for any conditional probability path So for any conditional probability path there exists an equivalent vector field. [snorts] So instead of [snorts] So instead of sampling from probabilities at each time sampling from probabilities at each time sampling from probabilities at each time step, we can simply sample a point from step, we can simply sample a point from step, we can simply sample a point from P in it and we can just follow the P in it and we can just follow the P in it and we can just follow the vector field. This is exactly what we vector field. This is exactly what we vector field. This is exactly what we discussed at the beginning of the discussed at the beginning of the discussed at the beginning of the lecture. Remember if we want to go from lecture. Remember if we want to go from lecture. Remember if we want to go from the velocity vector field to the the velocity vector field to the the velocity vector field to the trajectories the reverse problem

  33. L1 @ 1997s

    trajectories the reverse problem trajectories the reverse problem we simply sample the initial point which we simply sample the initial point which we simply sample the initial point which was x0 in our case and we use the oiler was x0 in our case and we use the oiler was x0 in our case and we use the oiler discretization to discretization to discretization to find the next point x1 then we go to x2 find the next point x1 then we go to x2 find the next point x1 then we go to x2 and similarly we can we know the entire and similarly we can we know the entire and similarly we can we know the entire trajectory of the particle. trajectory of the particle. trajectory of the particle. [snorts] So that's what we are trying to [snorts] So that's what we are trying to [snorts] So that's what we are trying to say in this piece of text that once you say in this piece of text that once you say in this piece of text that once you know the velocity field you can use that know the velocity field you can use that know the velocity field you can use that velocity field to get the entire velocity field to get the entire velocity field to get the entire trajectory of a particle trajectory of a particle trajectory of a particle and uh here you can actually see the and uh here you can actually see the and uh here you can actually see the conditional probability path and the conditional probability path and the conditional probability path and the conditional vector field. [snorts] So conditional vector field. [snorts] So conditional vector field. [snorts] So first let's look at the conditional first let's look at the conditional first let's look at the conditional probability path. Every single particle probability path. Every single particle probability path. Every single particle is being [snorts] transformed from the is being [snorts] transformed from the is being [snorts] transformed from the initial distribution uh to the final initial distribution uh to the final initial distribution uh to the final point zed. Now who is responsible for point zed. Now who is responsible for point zed. Now who is responsible for this transformation? There is a vector this transformation? There is a vector

  34. L1 @ 2046s

    this transformation? There is a vector field which is responsible for this field which is responsible for this field which is responsible for this transformation transformation and [snorts] and [snorts] and [snorts] rightfully so the vector field is rightfully so the vector field is rightfully so the vector field is pointing you can see if I zoom in here pointing you can see if I zoom in here pointing you can see if I zoom in here this is the vector field all the vector this is the vector field all the vector this is the vector field all the vector points all the vectors are pointing points all the vectors are pointing points all the vectors are pointing towards the direction of zed because towards the direction of zed because towards the direction of zed because this is what we want to move the points this is what we want to move the points this is what we want to move the points towards. towards. towards. [snorts] So these are the vector field [snorts] So these are the vector field [snorts] So these are the vector field lines and the white lines are the lines and the white lines are the lines and the white lines are the trajectories which are followed by the trajectories which are followed by the trajectories which are followed by the individual green particles. Now here is another example where this Now here is another example where this is the target. So we want to move our is the target. So we want to move our is the target. So we want to move our distribution which is uh shown over here distribution which is uh shown over here distribution which is uh shown over here to this target. to this target. to this target. And you can see as time proceeds the And you can see as time proceeds the And you can see as time proceeds the distribution slowly moves towards the distribution slowly moves towards the distribution slowly moves towards the target. But I want you to focus on these target. But I want you to focus on these target. But I want you to focus on these arrows which are the velocity vectors. [snorts] First of all note that how [snorts] First of all note that how every single point the velocity vector every single point the velocity vector

  35. L1 @ 2106s

    every single point the velocity vector is pointed towards them. And And for points which are far away the for points which are far away the for points which are far away the vectors are longer in magnitude. As you vectors are longer in magnitude. As you vectors are longer in magnitude. As you move closer the vectors are reducing in move closer the vectors are reducing in move closer the vectors are reducing in in in magnitude. in in magnitude. in in magnitude. And the formula for this is actually And the formula for this is actually And the formula for this is actually given very simply by this. given very simply by this. given very simply by this. So zed is our point which is our target. So zed is our point which is our target. So zed is our point which is our target. If x is very far from zed the magnitude If x is very far from zed the magnitude If x is very far from zed the magnitude is very high. If x is very close to zed is very high. If x is very close to zed is very high. If x is very close to zed it's almost zero. You can see here. it's almost zero. You can see here. it's almost zero. You can see here. But not just that there is a variation But not just that there is a variation But not just that there is a variation with time also. As as time goes close to with time also. As as time goes close to with time also. As as time goes close to one as you go towards the final one as you go towards the final one as you go towards the final distribution you can see the magnitude distribution you can see the magnitude distribution you can see the magnitude increases. So [snorts] increases. So [snorts] increases. So [snorts] from here to here you can see the arrows from here to here you can see the arrows from here to here you can see the arrows increasing in the magnitude increasing in the magnitude increasing in the magnitude from left to right from left to right from left to right and [snorts] uh and [snorts] uh and [snorts] uh this is because as as as time proceeds

  36. L1 @ 2165s

    this is because as as as time proceeds this is because as as as time proceeds we probably want to uh move the velocity we probably want to uh move the velocity we probably want to uh move the velocity field in a more aggressive manner field in a more aggressive manner field in a more aggressive manner towards our target. So first of all the towards our target. So first of all the towards our target. So first of all the first observation is that wherever your first observation is that wherever your first observation is that wherever your point is the velocity field is targeted point is the velocity field is targeted point is the velocity field is targeted towards the location's head and the towards the location's head and the towards the location's head and the magnitude is proportional to how far magnitude is proportional to how far magnitude is proportional to how far your point is from the target your point is from the target your point is from the target which probably makes sense because the which probably makes sense because the which probably makes sense because the further your point is from the target further your point is from the target further your point is from the target you want to push the particle more with you want to push the particle more with you want to push the particle more with with more force. So in reality what with more force. So in reality what with more force. So in reality what happens is that every single particle is happens is that every single particle is happens is that every single particle is is pushed towards your target. So that's is pushed towards your target. So that's is pushed towards your target. So that's why you get this conditional probability why you get this conditional probability why you get this conditional probability path [snorts] and this this formula for path [snorts] and this this formula for path [snorts] and this this formula for the velocity field is is uh once we know the velocity field is is uh once we know the velocity field is is uh once we know this velocity field formula this velocity field formula this velocity field formula we don't need anything else using this we don't need anything else using this

  37. L1 @ 2216s

    we don't need anything else using this formula we can simply calculate the formula we can simply calculate the formula we can simply calculate the trajectories of each and every point so trajectories of each and every point so trajectories of each and every point so [snorts] that every single point goes [snorts] that every single point goes [snorts] that every single point goes from the initial gshian distribution to from the initial gshian distribution to from the initial gshian distribution to your point Z. your point Z. your point Z. So the velocity field is given by the So the velocity field is given by the So the velocity field is given by the ratio of two quantities. The numerator ratio of two quantities. The numerator ratio of two quantities. The numerator is simply a vector from the point to the is simply a vector from the point to the is simply a vector from the point to the target and the denominator captures the target and the denominator captures the target and the denominator captures the evolution of time. evolution of time. evolution of time. So you observe something very critical So you observe something very critical So you observe something very critical here. Uh as we move forward in time the here. Uh as we move forward in time the here. Uh as we move forward in time the value of denominator decreases which value of denominator decreases which value of denominator decreases which means the magnitude of the velocity means the magnitude of the velocity means the magnitude of the velocity field lines increase as as we go forward field lines increase as as we go forward field lines increase as as we go forward in time. This can be seen from the above in time. This can be seen from the above in time. This can be seen from the above figures as the arrows get bigger and figures as the arrows get bigger and figures as the arrows get bigger and bigger as time proceeds. So this this bigger as time proceeds. So this this bigger as time proceeds. So this this this overall intuition will be critical this overall intuition will be critical this overall intuition will be critical for us as we move ahead. for us as we move ahead. for us as we move ahead. Now this u of t is is called as

  38. L1 @ 2266s

    Now this u of t is is called as Now this u of t is is called as conditional So you you might look at this and think So you you might look at this and think uh well if you look at the training uh well if you look at the training uh well if you look at the training target target target wasn't our training target also a wasn't our training target also a wasn't our training target also a velocity field so can we directly use velocity field so can we directly use velocity field so can we directly use this as a training target [snorts] well this as a training target [snorts] well this as a training target [snorts] well my first answer to that is you can't my first answer to that is you can't my first answer to that is you can't directly use it because this is not our directly use it because this is not our directly use it because this is not our training target remember it is training target remember it is training target remember it is conditioned on the data z conditioned on the data z conditioned on the data z uh so at a first glance it looks like uh so at a first glance it looks like uh so at a first glance it looks like okay this is very different from the okay this is very different from the okay this is very different from the actual velocity field because there the actual velocity field because there the actual velocity field because there the the target which I looked at before that the target which I looked at before that the target which I looked at before that is not conditioned on the point Z head is not conditioned on the point Z head is not conditioned on the point Z head but here we know the exact formula for but here we know the exact formula for but here we know the exact formula for this condition velocity field. So this condition velocity field. So this condition velocity field. So ideally we can we can replace this in in ideally we can we can replace this in in ideally we can we can replace this in in our target and train our neural network our target and train our neural network

  39. L1 @ 2320s

    our target and train our neural network to predict it for every single point to predict it for every single point to predict it for every single point zed. zed. zed. It turns out that what we will do is It turns out that what we will do is It turns out that what we will do is exactly that. But then let me not exactly that. But then let me not exactly that. But then let me not explain to you right now why that explain to you right now why that explain to you right now why that happens. And that's why this formulation happens. And that's why this formulation happens. And that's why this formulation is very easy. But for now for our is very easy. But for now for our is very easy. But for now for our where we are right now in terms of where we are right now in terms of where we are right now in terms of accessing this material is that we know accessing this material is that we know accessing this material is that we know that this is the conditional vector that this is the conditional vector that this is the conditional vector field and we can completely predict this field and we can completely predict this field and we can completely predict this conditional vector field. Okay, so now so far we have looked at Okay, so now so far we have looked at the conditional probability path and the the conditional probability path and the the conditional probability path and the conditional vector field and [snorts] we conditional vector field and [snorts] we conditional vector field and [snorts] we can predict the conditional probability can predict the conditional probability can predict the conditional probability path as a sequence of gshians which path as a sequence of gshians which path as a sequence of gshians which takes us from the initial gshian takes us from the initial gshian takes us from the initial gshian distribution to a final point and what distribution to a final point and what distribution to a final point and what is responsible for this path is the is responsible for this path is the is responsible for this path is the velocity vector field which is pointed velocity vector field which is pointed

  40. L1 @ 2377s

    velocity vector field which is pointed from our data to the data point where we from our data to the data point where we from our data to the data point where we want to go. So it's it's a it's arrow want to go. So it's it's a it's arrow want to go. So it's it's a it's arrow which directly goes from where we are which directly goes from where we are which directly goes from where we are right now x and where we want to go that right now x and where we want to go that right now x and where we want to go that is zed. Okay. So now we will go to marginal Okay. So now we will go to marginal probability path. probability path. So now marginal is exactly the opposite So now marginal is exactly the opposite So now marginal is exactly the opposite of conditional. Imagine that instead of of conditional. Imagine that instead of of conditional. Imagine that instead of a single point zed we have a complex a single point zed we have a complex a single point zed we have a complex data distribution p data. And for every data distribution p data. And for every data distribution p data. And for every point in this distribution which is point in this distribution which is point in this distribution which is given to us, we can build a conditional given to us, we can build a conditional given to us, we can build a conditional probability path. And a collection of probability path. And a collection of probability path. And a collection of all the conditional probability path all the conditional probability path all the conditional probability path gives us the marginal probability path. gives us the marginal probability path. gives us the marginal probability path. So it's like once we understand So it's like once we understand So it's like once we understand conditional probability path we can conditional probability path we can conditional probability path we can think in this direction that if you think in this direction that if you think in this direction that if you aggregate all these paths together the aggregate all these paths together the aggregate all these paths together the final flow that you get the the final final flow that you get the the final

  41. L1 @ 2432s

    final flow that you get the the final trajectories that you get those are the trajectories that you get those are the trajectories that you get those are the marginal probability paths. marginal probability paths. marginal probability paths. We have in fact defined this as flow in We have in fact defined this as flow in We have in fact defined this as flow in the last lecture but now we are giving the last lecture but now we are giving the last lecture but now we are giving it another name which is a marginal it another name which is a marginal it another name which is a marginal probability path. probability path. So you can imagine the distribution to So you can imagine the distribution to So you can imagine the distribution to be made up of a collection of zeds and a be made up of a collection of zeds and a be made up of a collection of zeds and a marginal path is a set of distributions marginal path is a set of distributions marginal path is a set of distributions which transforms the initial which transforms the initial which transforms the initial distribution pinet to P data. distribution pinet to P data. distribution pinet to P data. And if you recall this is exactly what And if you recall this is exactly what And if you recall this is exactly what we want to predict. we want to predict. we want to predict. [snorts] But here the interesting thing [snorts] But here the interesting thing [snorts] But here the interesting thing is that P data is not known to us. So we is that P data is not known to us. So we is that P data is not known to us. So we are just discussing this for a are just discussing this for a are just discussing this for a conceptual understanding of the conceptual understanding of the conceptual understanding of the difference between uh conditional difference between uh conditional difference between uh conditional probability path and marginal probability path and marginal probability path and marginal probability path probability path probability path but in reality the p data is not known but in reality the p data is not known but in reality the p data is not known to us. So here if you see

  42. L1 @ 2484s

    to us. So here if you see to us. So here if you see the final distribution that we get which the final distribution that we get which the final distribution that we get which is which are these um is which are these um is which are these um eight clusters right that is the eight clusters right that is the eight clusters right that is the distribution that we want to predict. If distribution that we want to predict. If distribution that we want to predict. If that was known to us, we could directly that was known to us, we could directly that was known to us, we could directly construct this marginal probability construct this marginal probability construct this marginal probability path. path. path. But all we know is probably just But all we know is probably just But all we know is probably just thousand points within that thousand points within that thousand points within that distribution. We don't know the final distribution. We don't know the final distribution. We don't know the final overall distribution yet. overall distribution yet. overall distribution yet. And similarly, uh we can actually And similarly, uh we can actually And similarly, uh we can actually understand this. Let's let's understand this. Let's let's understand this. Let's let's go to this Google Collab notebook. >> Okay. So >> Okay. So I'm going to run this piece of code so I'm going to run this piece of code so I'm going to run this piece of code so that we understand what is the meaning that we understand what is the meaning that we understand what is the meaning of a marginal probability path. of a marginal probability path. of a marginal probability path. So we'll run this first piece of code So we'll run this first piece of code So we'll run this first piece of code over here. So this is very similar to the final So this is very similar to the final animation which I showed you. Uh animation which I showed you. Uh animation which I showed you. Uh so here this is the target data where we so here this is the target data where we so here this is the target data where we want to reach which is a sample of eight want to reach which is a sample of eight

  43. L1 @ 2570s

    want to reach which is a sample of eight washians washians washians and initially we start with a random and initially we start with a random and initially we start with a random noise. noise. noise. So here you see the animation is now So here you see the animation is now So here you see the animation is now available to us. available to us. available to us. So we go from the initial distribution So we go from the initial distribution So we go from the initial distribution of random noise to a final distribution of random noise to a final distribution of random noise to a final distribution where these eight gshians are where these eight gshians are where these eight gshians are distributed in space. distributed in space. distributed in space. Now that is this is exactly what we want Now that is this is exactly what we want Now that is this is exactly what we want to predict the marginal probability path to predict the marginal probability path to predict the marginal probability path in in reality the distribution of cats in in reality the distribution of cats in in reality the distribution of cats will also be very complex right will also be very complex right will also be very complex right and uh along with the marginal and uh along with the marginal and uh along with the marginal probability path now you know that there probability path now you know that there probability path now you know that there will be a field also associated with it will be a field also associated with it will be a field also associated with it similar to how there was a vector field similar to how there was a vector field similar to how there was a vector field associated with the conditional associated with the conditional associated with the conditional probability path. Similarly for marginal probability path. Similarly for marginal probability path. Similarly for marginal probability path there will be a vector probability path there will be a vector probability path there will be a vector field associated and ideally this vector field associated and ideally this vector field associated and ideally this vector field is our training target. field is our training target. field is our training target. Now uh here let's play this Now uh here let's play this

  44. L1 @ 2625s

    Now uh here let's play this piece of code to understand this piece of code to understand this piece of code to understand this u and you'll probably get something like and you'll probably get something like this at the end. So here you see the this at the end. So here you see the this at the end. So here you see the velocity vector field. How they change velocity vector field. How they change velocity vector field. How they change with time, how they evolve with time. So with time, how they evolve with time. So with time, how they evolve with time. So it's a vector field which is dependent it's a vector field which is dependent it's a vector field which is dependent on space and time. But it is marginal on space and time. But it is marginal on space and time. But it is marginal vector field because we are not just vector field because we are not just vector field because we are not just finding a vector field for one single finding a vector field for one single finding a vector field for one single point which takes from a distribution to point which takes from a distribution to point which takes from a distribution to that point but rather we have a that point but rather we have a that point but rather we have a collection of points in space. So and we collection of points in space. So and we collection of points in space. So and we are we are calculating the vector field are we are calculating the vector field are we are calculating the vector field for all those points which make them for all those points which make them for all those points which make them evolve according to their probability evolve according to their probability evolve according to their probability paths. [snorts] paths. [snorts] paths. [snorts] So if we had access to this there would So if we had access to this there would So if we had access to this there would not be any question that we replace this not be any question that we replace this not be any question that we replace this in our target. But uh the the issue is in our target. But uh the the issue is

  45. L1 @ 2679s

    in our target. But uh the the issue is that we do not have access to this that we do not have access to this that we do not have access to this because we only have access to our because we only have access to our because we only have access to our individual data points. Okay. Okay. So here is another example Okay. Okay. So here is another example to understand the difference between to understand the difference between to understand the difference between conditional and marginal paths. So here conditional and marginal paths. So here conditional and marginal paths. So here you see you see you see u u u our actual distribution is are these our actual distribution is are these our actual distribution is are these five spheres that I want to go to right five spheres that I want to go to right five spheres that I want to go to right which which are shown in these circles. which which are shown in these circles. which which are shown in these circles. This is the distribution which is a This is the distribution which is a This is the distribution which is a collection of these five spheres. But collection of these five spheres. But collection of these five spheres. But I'm not aware of these distributions. I I'm not aware of these distributions. I I'm not aware of these distributions. I I I just know a single point zed. I I just know a single point zed. I I just know a single point zed. So I take my original gshian and I move So I take my original gshian and I move So I take my original gshian and I move it towards this single point zed it towards this single point zed it towards this single point zed according to these trajectories according to these trajectories according to these trajectories which I get by solving this conditional which I get by solving this conditional which I get by solving this conditional distribution uh this this conditional od distribution uh this this conditional od distribution uh this this conditional od where we have seen that this mu is given where we have seen that this mu is given where we have seen that this mu is given by z minus x divided by 1 minus t.

  46. L1 @ 2739s

    by z minus x divided by 1 minus t. by z minus x divided by 1 minus t. [snorts] So these these these field [snorts] So these these these field [snorts] So these these these field lines are almost linear from the source lines are almost linear from the source lines are almost linear from the source to the target over here. to the target over here. to the target over here. So we are calculating the trajectories So we are calculating the trajectories So we are calculating the trajectories from the original gshian to the single from the original gshian to the single from the original gshian to the single point Z which has been sampled from our point Z which has been sampled from our point Z which has been sampled from our distribution. distribution. This is what is the meaning of a This is what is the meaning of a This is what is the meaning of a conditional probability path. conditional probability path. conditional probability path. Whereas [snorts] Whereas [snorts] Whereas [snorts] marginal probability path is something marginal probability path is something marginal probability path is something very different. Marginal probability very different. Marginal probability very different. Marginal probability path is uh something which takes us from path is uh something which takes us from path is uh something which takes us from our original uh our original uh our original uh distribution which is the gshian distribution which is the gshian distribution which is the gshian distribution over here and then you see distribution over here and then you see distribution over here and then you see how we are learning all these how we are learning all these how we are learning all these trajectories. So basically you sample a trajectories. So basically you sample a trajectories. So basically you sample a lot of zed from the uh and and if you lot of zed from the uh and and if you lot of zed from the uh and and if you have access to infinite samples then have access to infinite samples then have access to infinite samples then obviously you will go down this path but obviously you will go down this path but obviously you will go down this path but we do not have access to infinite we do not have access to infinite

  47. L1 @ 2790s

    we do not have access to infinite samples. we have access to finite samples. we have access to finite samples. we have access to finite samples samples samples and uh this this uh credit goes to uh and uh this this uh credit goes to uh and uh this this uh credit goes to uh the MIT course by Peter Bodarit. I have the MIT course by Peter Bodarit. I have the MIT course by Peter Bodarit. I have link this in in the original flow model link this in in the original flow model link this in in the original flow model lecture also. So this this makes the lecture also. So this this makes the lecture also. So this this makes the difference between the conditional paths difference between the conditional paths difference between the conditional paths and the marginal paths very clear. and the marginal paths very clear. and the marginal paths very clear. Now uh the reason why we went into Now uh the reason why we went into Now uh the reason why we went into conditional and marginal paths is conditional and marginal paths is conditional and marginal paths is because it will help us understand the because it will help us understand the because it will help us understand the training target and what compromise we training target and what compromise we training target and what compromise we have done to to get to the training have done to to get to the training have done to to get to the training target. The ideal training target should target. The ideal training target should target. The ideal training target should be the marginal vector field but uh we be the marginal vector field but uh we be the marginal vector field but uh we will understand that we make a will understand that we make a will understand that we make a compromise to it and we use a very compromise to it and we use a very compromise to it and we use a very different type of training target. different type of training target. different type of training target. So now uh let's let's try to broadly So now uh let's let's try to broadly So now uh let's let's try to broadly understand what we are doing here. understand what we are doing here.

  48. L1 @ 2838s

    understand what we are doing here. Essentially we are treating data Essentially we are treating data Essentially we are treating data generation as a flow problem. We know generation as a flow problem. We know generation as a flow problem. We know mathematically that mathematically that mathematically that for any complex data set whether it's for any complex data set whether it's for any complex data set whether it's pictures of cats, robotics policies or pictures of cats, robotics policies or pictures of cats, robotics policies or Giblly scenes there exists a vector Giblly scenes there exists a vector Giblly scenes there exists a vector field that connects a cloud of random field that connects a cloud of random field that connects a cloud of random noise directly to that data and this is noise directly to that data and this is noise directly to that data and this is the marginal vector field. Okay. Now if we had a map of this Okay. Now if we had a map of this current uh generating a new image would current uh generating a new image would current uh generating a new image would be simple and deterministic. So we be simple and deterministic. So we be simple and deterministic. So we already know this if we have access to already know this if we have access to already know this if we have access to this vector field uh we we simply use this vector field uh we we simply use this vector field uh we we simply use this vector field and uh we calculate this vector field and uh we calculate this vector field and uh we calculate the trajectory using this vector field the trajectory using this vector field the trajectory using this vector field using the oiler method. This is what using the oiler method. This is what using the oiler method. This is what detective 2 said at the start of today's detective 2 said at the start of today's detective 2 said at the start of today's lecture. lecture. lecture. But the biggest problem is that we do But the biggest problem is that we do But the biggest problem is that we do not have the formula for this vector not have the formula for this vector not have the formula for this vector field is impossibly complex. field is impossibly complex.

  49. L1 @ 2897s

    field is impossibly complex. You cannot write down an equation that You cannot write down an equation that You cannot write down an equation that defines what a Giblly movie looks like. defines what a Giblly movie looks like. defines what a Giblly movie looks like. So we we cannot calculate the vector So we we cannot calculate the vector So we we cannot calculate the vector field but we can approximate it. And uh field but we can approximate it. And uh field but we can approximate it. And uh in today's lecture we are not going to in today's lecture we are not going to in today's lecture we are not going to look at how exactly the target of the look at how exactly the target of the look at how exactly the target of the velocity field is constructed. But from velocity field is constructed. But from velocity field is constructed. But from a mathematical point of view if you look a mathematical point of view if you look a mathematical point of view if you look at it today's lecture was a bit involved at it today's lecture was a bit involved at it today's lecture was a bit involved into condition and marginal path. The into condition and marginal path. The into condition and marginal path. The next lecture is going to be very easy next lecture is going to be very easy next lecture is going to be very easy for us because it is simply going to for us because it is simply going to for us because it is simply going to build upon today's lecture and in fact build upon today's lecture and in fact build upon today's lecture and in fact we will understand how easy it is to we will understand how easy it is to we will understand how easy it is to construct the training target. Uh and construct the training target. Uh and construct the training target. Uh and and you might think okay so does this and you might think okay so does this and you might think okay so does this actually work? It it sounds too good to actually work? It it sounds too good to actually work? It it sounds too good to be true and it does actually work. We we

  50. L1 @ 2946s

    be true and it does actually work. We we be true and it does actually work. We we have seen this in other uh aspects of have seen this in other uh aspects of have seen this in other uh aspects of deep generative models like diffusion deep generative models like diffusion deep generative models like diffusion models and uh uh score based models and uh uh score based models and uh uh score based generative models that conditional generative models that conditional generative models that conditional models or if you condition it on the models or if you condition it on the models or if you condition it on the data you you get a training target which data you you get a training target which data you you get a training target which is feasible and magically it just it is feasible and magically it just it is feasible and magically it just it just works. So we are going to just works. So we are going to just works. So we are going to understand how to construct this understand how to construct this understand how to construct this training target which is going to be training target which is going to be training target which is going to be exceedingly simple and then we are going exceedingly simple and then we are going exceedingly simple and then we are going to understand how to set up our whole to understand how to set up our whole to understand how to set up our whole flow pipeline and uh we will also look flow pipeline and uh we will also look flow pipeline and uh we will also look at a practical example of how to at a practical example of how to at a practical example of how to generate a flow model uh how to train a generate a flow model uh how to train a generate a flow model uh how to train a flow model using uh the training target flow model using uh the training target flow model using uh the training target and how to sample from it. [snorts] So and how to sample from it. [snorts] So and how to sample from it. [snorts] So uh I I hope this lecture was interesting uh I I hope this lecture was interesting

  51. L1 @ 2994s

    uh I I hope this lecture was interesting for all of you. I haven't yet given the for all of you. I haven't yet given the for all of you. I haven't yet given the final reveal. Uh I'm I'm delaying it final reveal. Uh I'm I'm delaying it final reveal. Uh I'm I'm delaying it purposefully. As I said at the start of purposefully. As I said at the start of purposefully. As I said at the start of the first lecture in the flow series, I the first lecture in the flow series, I the first lecture in the flow series, I could have wrapped this entire series in could have wrapped this entire series in could have wrapped this entire series in just maybe 40 minutes. But then you will just maybe 40 minutes. But then you will just maybe 40 minutes. But then you will not appreciate the kind of thinking not appreciate the kind of thinking not appreciate the kind of thinking which has gone into constructing this which has gone into constructing this which has gone into constructing this training target. And uh today's lecture training target. And uh today's lecture training target. And uh today's lecture I wanted to specifically devote to that I wanted to specifically devote to that I wanted to specifically devote to that so that all of you understand uh how so that all of you understand uh how so that all of you understand uh how these targets are constructed. Before we these targets are constructed. Before we these targets are constructed. Before we go ahead, I just want to uh motivate all go ahead, I just want to uh motivate all go ahead, I just want to uh motivate all of you by showing this paper pi 0.5 of you by showing this paper pi 0.5 of you by showing this paper pi 0.5 or let's look at pi 0. or let's look at pi 0. or let's look at pi 0. So this is a robotics foundational model So this is a robotics foundational model So this is a robotics foundational model and uh the reason I'm showing this to and uh the reason I'm showing this to and uh the reason I'm showing this to you right now is you right now is you right now is this this came out last year

  52. L1 @ 3045s

    this this came out last year this this came out last year and this is based on uh flow matching. and this is based on uh flow matching. and this is based on uh flow matching. So here essentially what we are doing is So here essentially what we are doing is So here essentially what we are doing is uh when you see flow matching now all of uh when you see flow matching now all of uh when you see flow matching now all of you can understand that we are trying to you can understand that we are trying to you can understand that we are trying to find the trajectories from the initial find the trajectories from the initial find the trajectories from the initial gshian distribution to a final gshian distribution to a final gshian distribution to a final distribution and we [snorts] are trying distribution and we [snorts] are trying distribution and we [snorts] are trying to find a velocity field that takes us to find a velocity field that takes us to find a velocity field that takes us from the initial to the final from the initial to the final from the initial to the final distribution. The only thing to think distribution. The only thing to think distribution. The only thing to think about is what is this distribution about is what is this distribution about is what is this distribution representing. So, so far we have seen representing. So, so far we have seen representing. So, so far we have seen distributions representing images. But distributions representing images. But distributions representing images. But distributions can also represent robotic distributions can also represent robotic distributions can also represent robotic actions which is exactly what is actions which is exactly what is actions which is exactly what is happening in this in this paper which is happening in this in this paper which is happening in this in this paper which is amazing. If you see the applications of amazing. If you see the applications of amazing. If you see the applications of flow they they have a flow expert which flow they they have a flow expert which flow they they have a flow expert which is which is the action expert here which is which is the action expert here which

  53. L1 @ 3092s

    is which is the action expert here which is a it's based on flow matching and uh is a it's based on flow matching and uh is a it's based on flow matching and uh the training target that you see here is the training target that you see here is the training target that you see here is exceedingly simple. Maybe I can show exceedingly simple. Maybe I can show exceedingly simple. Maybe I can show that to you uh which is written in in in that to you uh which is written in in in that to you uh which is written in in in in this section. We will we will look at in this section. We will we will look at in this section. We will we will look at this in detail in the next lecture. But this in detail in the next lecture. But this in detail in the next lecture. But once you go through the next lecture, once you go through the next lecture, once you go through the next lecture, you will be able to understand these you will be able to understand these you will be able to understand these sections which are coming in any paper. sections which are coming in any paper. sections which are coming in any paper. So that whenever you see flow matching, So that whenever you see flow matching, So that whenever you see flow matching, it will suddenly not be very it will suddenly not be very it will suddenly not be very inaccessible field to you, but you can inaccessible field to you, but you can inaccessible field to you, but you can relate to what we have seen in the relate to what we have seen in the relate to what we have seen in the lecture and build a very strong lecture and build a very strong lecture and build a very strong intuition for it. Thank you very much intuition for it. Thank you very much intuition for it. Thank you very much everyone and I will see you in the next everyone and I will see you in the next everyone and I will see you in the next lecture which is going to be like a lecture which is going to be like a

  54. L1 @ 3129s

    lecture which is going to be like a magician's reveal. Thank you.

  55. L2 @ 3s

    Good morning everyone. Good morning everyone. Today we are starting with the next Today we are starting with the next Today we are starting with the next lecture of our course which is lecture of our course which is lecture of our course which is introduction to flow models. introduction to flow models. introduction to flow models. I have decided to divide this topic into I have decided to divide this topic into I have decided to divide this topic into a series of three to four lectures a series of three to four lectures a series of three to four lectures so that I can cover the entire depth in so that I can cover the entire depth in so that I can cover the entire depth in a very comprehensive manner. a very comprehensive manner. a very comprehensive manner. One interesting thing about the theory One interesting thing about the theory One interesting thing about the theory behind flow models is that behind flow models is that behind flow models is that you can teach the same topic in two you can teach the same topic in two you can teach the same topic in two different ways. different ways. different ways. You can teach this entire topic within a You can teach this entire topic within a You can teach this entire topic within a span of one lecture span of one lecture span of one lecture or you can teach it within a span of or you can teach it within a span of or you can teach it within a span of four lectures. Now that is quite paradoxical that how Now that is quite paradoxical that how you can teach the same concept you can teach the same concept you can teach the same concept with these two format types. The main with these two format types. The main with these two format types. The main reason is that if you go through the reason is that if you go through the reason is that if you go through the difficult route of the four lectures, difficult route of the four lectures, difficult route of the four lectures, you end up at a place which is quite you end up at a place which is quite you end up at a place which is quite simple which you could have got in the

  56. L2 @ 70s

    simple which you could have got in the simple which you could have got in the first lecture itself. first lecture itself. first lecture itself. But I don't want to show you that route But I don't want to show you that route But I don't want to show you that route that easily because you won't appreciate that easily because you won't appreciate that easily because you won't appreciate the the the work which has been put behind work which has been put behind work which has been put behind the easy route. So in essence flow model the easy route. So in essence flow model the easy route. So in essence flow model is a very very easy concept and it can is a very very easy concept and it can is a very very easy concept and it can be explained in a very simple and be explained in a very simple and be explained in a very simple and intuitive manner. intuitive manner. intuitive manner. But because But because But because reaching there is not that reaching there is not that reaching there is not that straightforward, we might have to spend straightforward, we might have to spend straightforward, we might have to spend some more time to understand the simple some more time to understand the simple some more time to understand the simple concept. concept. concept. In other words, imagine that we are In other words, imagine that we are In other words, imagine that we are going we want to go from point A to going we want to go from point A to going we want to go from point A to point B, but instead of going through point B, but instead of going through point B, but instead of going through the straight line, we are going to take the straight line, we are going to take the straight line, we are going to take a curve and reach point B. a curve and reach point B. a curve and reach point B. It will be slightly mathematical uh It will be slightly mathematical uh It will be slightly mathematical uh today's lecture but uh I have tried to today's lecture but uh I have tried to today's lecture but uh I have tried to keep the mathematics as simple as keep the mathematics as simple as

  57. L2 @ 126s

    keep the mathematics as simple as possible [snorts] and I have taken possible [snorts] and I have taken possible [snorts] and I have taken inspiration from an uh MIT course to inspiration from an uh MIT course to inspiration from an uh MIT course to structure these lectures. I will put the structure these lectures. I will put the structure these lectures. I will put the link to that course in the description link to that course in the description link to that course in the description and their lecture notes as well. and their lecture notes as well. and their lecture notes as well. So before we get started for those of So before we get started for those of So before we get started for those of you who haven't heard of flow models and you who haven't heard of flow models and you who haven't heard of flow models and uh what these models represent uh what these models represent uh what these models represent first of all let us divide this two first of all let us divide this two first of all let us divide this two words into two distinct words. The first words into two distinct words. The first words into two distinct words. The first word is flow. Now what does flow mean? word is flow. Now what does flow mean? word is flow. Now what does flow mean? When I hear flow the first thing that When I hear flow the first thing that When I hear flow the first thing that comes to my mind is probably a fluid. comes to my mind is probably a fluid. comes to my mind is probably a fluid. Right? So let's click on this and see Right? So let's click on this and see Right? So let's click on this and see something something moving in in space something something moving in in space something something moving in in space something like this something something something like this something something something like this something something like a fluid motion like a fluid motion like a fluid motion and model is of course we are aware a and model is of course we are aware a and model is of course we are aware a model is a representation of a phenomena model is a representation of a phenomena

  58. L2 @ 181s

    model is a representation of a phenomena a neural network is an example of a a neural network is an example of a a neural network is an example of a model or any mathematical function which model or any mathematical function which model or any mathematical function which maps an input to an output is a model. maps an input to an output is a model. maps an input to an output is a model. [snorts] [snorts] [snorts] Now the question is how we are going to Now the question is how we are going to Now the question is how we are going to use these physical notions of the use these physical notions of the use these physical notions of the concept of flow and concept of flow and concept of flow and u solve the problem that we have set out u solve the problem that we have set out u solve the problem that we have set out to solve at the beginning of this to solve at the beginning of this to solve at the beginning of this series. If you remember the problem that series. If you remember the problem that series. If you remember the problem that we have is we have is we have is called as deep generative modeling. >> Now what does this mean? >> Now what does this mean? We are given a bunch of data. Let's say we are given these images of Let's say we are given these images of cats cats cats and we want to find a distribution using and we want to find a distribution using and we want to find a distribution using which we can sample images which look which we can sample images which look which we can sample images which look exactly like these images of cats. So exactly like these images of cats. So exactly like these images of cats. So from data from data from data We want to move to distribution and that is the broad problem that we and that is the broad problem that we want to solve want to solve want to solve in in this whole whole series and so far in in this whole whole series and so far in in this whole whole series and so far we have looked at different techniques we have looked at different techniques

  59. L2 @ 262s

    we have looked at different techniques to solve this broader problem. It to solve this broader problem. It to solve this broader problem. It started out with variational started out with variational started out with variational autoenccoders. Then we move to diffusion autoenccoders. Then we move to diffusion autoenccoders. Then we move to diffusion models and now we are discussing another models and now we are discussing another models and now we are discussing another type of deep generative models which are type of deep generative models which are type of deep generative models which are called as flow models. So we discussed VAS, So we discussed VAS, we discussed diffusion models we discussed diffusion models we discussed diffusion models [clears throat] [clears throat] [clears throat] and now we are going to discuss flow and now we are going to discuss flow and now we are going to discuss flow models. So in our mind we know that okay models So in our mind we know that okay models probably represent deep generative probably represent deep generative probably represent deep generative models and models and models and flow is something like a fluid which flow is something like a fluid which flow is something like a fluid which transports particles from one initial transports particles from one initial transports particles from one initial location to a final location. So so location to a final location. So so location to a final location. So so there is a movement of particles in there is a movement of particles in there is a movement of particles in space space space and something is changing with time and something is changing with time and something is changing with time also. So we want to combine these also. So we want to combine these also. So we want to combine these physical intuitions into a deep physical intuitions into a deep physical intuitions into a deep generative model that helps us map a generative model that helps us map a generative model that helps us map a data to a probability distribution. So let's understand uh before we dive So let's understand uh before we dive deep into the theory. Flow models are deep into the theory. Flow models are deep into the theory. Flow models are one of the most consequential techniques one of the most consequential techniques

  60. L2 @ 334s

    one of the most consequential techniques in modern [snorts] in modern [snorts] in modern [snorts] generative modeling. And the reason it generative modeling. And the reason it generative modeling. And the reason it is very consequential it is it is used is very consequential it is it is used is very consequential it is it is used in a lot of image generation models. Lot in a lot of image generation models. Lot in a lot of image generation models. Lot of state-of-the-art image generation of state-of-the-art image generation of state-of-the-art image generation tools use flow models. It is also used tools use flow models. It is also used tools use flow models. It is also used in robotics. There is a policy which is in robotics. There is a policy which is in robotics. There is a policy which is called as pi zero foundational model. Now this is a policy which is a robotics Now this is a policy which is a robotics foundational model which means that foundational model which means that foundational model which means that similar to how LLMs are trained on similar to how LLMs are trained on similar to how LLMs are trained on massive amounts of web corpus data these massive amounts of web corpus data these massive amounts of web corpus data these models are trained on massive amounts of models are trained on massive amounts of models are trained on massive amounts of videos. So the robot learns from these videos. So the robot learns from these videos. So the robot learns from these static or static videos and learns to static or static videos and learns to static or static videos and learns to perform actions in the real world by perform actions in the real world by perform actions in the real world by learning from these trajectories which learning from these trajectories which learning from these trajectories which have been recorded. have been recorded. have been recorded. Now this policy uses something called as Now this policy uses something called as Now this policy uses something called as a flow model u in in the architecture a flow model u in in the architecture a flow model u in in the architecture and it works exceptionally well. You can and it works exceptionally well. You can

  61. L2 @ 393s

    and it works exceptionally well. You can see here they have this action expert see here they have this action expert see here they have this action expert and this action expert is built [snorts] and this action expert is built [snorts] and this action expert is built [snorts] on top of a flow model. So understanding on top of a flow model. So understanding on top of a flow model. So understanding flow model is extremely critical for us flow model is extremely critical for us flow model is extremely critical for us to understand all these applications of to understand all these applications of to understand all these applications of uh these models in different domains uh these models in different domains uh these models in different domains including robotics including uh image including robotics including uh image including robotics including uh image generation and video generation as well. generation and video generation as well. generation and video generation as well. [snorts] In fact a lot of protein [snorts] In fact a lot of protein [snorts] In fact a lot of protein synthesis synthesis of protein molecules synthesis synthesis of protein molecules synthesis synthesis of protein molecules uh or drug discovery is now being done uh or drug discovery is now being done uh or drug discovery is now being done using flow models. So there are using flow models. So there are using flow models. So there are innumerable number of applications and innumerable number of applications and innumerable number of applications and as I said in the beginning the final as I said in the beginning the final as I said in the beginning the final formulation is so simple that we'll formulation is so simple that we'll formulation is so simple that we'll think back and say why did we spend think back and say why did we spend think back and say why did we spend three lectures on this this could have three lectures on this this could have three lectures on this this could have been maybe done in half an hour but we been maybe done in half an hour but we been maybe done in half an hour but we really want to appreciate how this really want to appreciate how this really want to appreciate how this formula comes about which is very formula comes about which is very

  62. L2 @ 446s

    formula comes about which is very simple. So flow modeling at the heart of simple. So flow modeling at the heart of simple. So flow modeling at the heart of it is very simple and it has huge number it is very simple and it has huge number it is very simple and it has huge number of applications. of applications. of applications. But surprisingly this is the model which But surprisingly this is the model which But surprisingly this is the model which came out the came out the came out the uh came out late compared to other uh came out late compared to other uh came out late compared to other models. For example score based models, models. For example score based models, models. For example score based models, diffusion models all these models came diffusion models all these models came diffusion models all these models came early and then flow models came after early and then flow models came after early and then flow models came after that. [snorts] Okay. So first let's start with a simple Okay. So first let's start with a simple Okay. So first let's start with a simple mathematical description mathematical description mathematical description [snorts] [snorts] where where where we are looking at something called as we are looking at something called as we are looking at something called as time dependent vector fields. [snorts] First let us understand what is a vector First let us understand what is a vector First let us understand what is a vector field. Okay. So imagine that you have field. Okay. So imagine that you have field. Okay. So imagine that you have this 2D two-dimensional grid x-axis and y-axis. two-dimensional grid x-axis and y-axis. [snorts] A vector field is something [snorts] A vector field is something [snorts] A vector field is something where given any point where given any point where given any point it gives a vector as an output. it gives a vector as an output. it gives a vector as an output. So for example, this is a vector field. So for example, this is a vector field. So for example, this is a vector field. [snorts] [snorts] So let's take an example. Imagine that So let's take an example. Imagine that So let's take an example. Imagine that maybe you're sitting on a table right

  63. L2 @ 525s

    maybe you're sitting on a table right maybe you're sitting on a table right now or you have a table in front of you. now or you have a table in front of you. now or you have a table in front of you. Uh you place a bunch of magnets on the Uh you place a bunch of magnets on the Uh you place a bunch of magnets on the table and the magnets table and the magnets table and the magnets create a vector field all around them. create a vector field all around them. create a vector field all around them. Uh or the magnets create a magnetic Uh or the magnets create a magnetic Uh or the magnets create a magnetic field all around them and you take any field all around them and you take any field all around them and you take any point on the table and you ask what is point on the table and you ask what is point on the table and you ask what is the magnetic field at that location. the magnetic field at that location. the magnetic field at that location. That [snorts] is an example of a vector That [snorts] is an example of a vector That [snorts] is an example of a vector field. Given any point in space, it field. Given any point in space, it field. Given any point in space, it gives you a magnitude and a direction. Now here what we are considering is a Now here what we are considering is a timed dependent vector field. timed dependent vector field. timed dependent vector field. So what does a timed dependent vector So what does a timed dependent vector So what does a timed dependent vector field do? Instead of looking at one field do? Instead of looking at one field do? Instead of looking at one single snapshot, so this is maybe a single snapshot, so this is maybe a single snapshot, so this is maybe a snapshot at time t equal to0. snapshot at time t equal to0. snapshot at time t equal to0. This vector field is going to change This vector field is going to change This vector field is going to change with time. So the magnets that we have with time. So the magnets that we have

  64. L2 @ 591s

    with time. So the magnets that we have placed on the table, what if these placed on the table, what if these placed on the table, what if these magnets are constantly moving? magnets are constantly moving? magnets are constantly moving? Then the magnetic field created by them Then the magnetic field created by them Then the magnetic field created by them will also change with time, right? And will also change with time, right? And will also change with time, right? And because of that you will have these because of that you will have these because of that you will have these vector fields which are also now vector fields which are also now vector fields which are also now changing with time. depends on two things. depends on two things. [snorts] [snorts] It depends on first space It depends on first space It depends on first space which is at at what point in this which is at at what point in this which is at at what point in this cartian grid you are looking at and cartian grid you are looking at and cartian grid you are looking at and second it depends on time as well. So [snorts] whatever I have shown here So [snorts] whatever I have shown here it's this is like a single snapshot. it's this is like a single snapshot. it's this is like a single snapshot. So imagine the magnetic field on your So imagine the magnetic field on your So imagine the magnetic field on your table is varying with time. You take a table is varying with time. You take a table is varying with time. You take a camera and you take a photo of the camera and you take a photo of the camera and you take a photo of the magnetic field. So you freeze time and magnetic field. So you freeze time and magnetic field. So you freeze time and then the magnetic field will only vary then the magnetic field will only vary then the magnetic field will only vary with space. Now these photos you can with space. Now these photos you can with space. Now these photos you can take at multiple time steps and hence take at multiple time steps and hence

  65. L2 @ 659s

    take at multiple time steps and hence you will get a vector field which is you will get a vector field which is you will get a vector field which is changing with time. So now we are discussing time dependent So now we are discussing time dependent vector fields. vector fields. vector fields. A time dependent vector field maps a A time dependent vector field maps a A time dependent vector field maps a point x in space point x in space point x in space and time t to a velocity vector. and time t to a velocity vector. and time t to a velocity vector. And mathematically we define such a And mathematically we define such a And mathematically we define such a field of as u of x t. For example, if I field of as u of x t. For example, if I field of as u of x t. For example, if I say u of say u of say u of um um um let's say this x is a vector. So I point let's say this x is a vector. So I point let's say this x is a vector. So I point it as 5 it as 5 it as 5 e ex let's say it's a vector only in the e ex let's say it's a vector only in the e ex let's say it's a vector only in the x direction, x direction, x direction, 0. 0. 0. This means that I'm taking a snapshot at This means that I'm taking a snapshot at This means that I'm taking a snapshot at time t=0 time t=0 time t=0 and I am asking what is the velocity and I am asking what is the velocity and I am asking what is the velocity field at this point field at this point field at this point and and and this is given by a vector which can this is given by a vector which can this is given by a vector which can point in any direction. point in any direction. point in any direction. So the notation u u is a vector which So the notation u u is a vector which So the notation u u is a vector which depends on not just the position in depends on not just the position in

  66. L2 @ 732s

    depends on not just the position in space but also it depends on the time space but also it depends on the time space but also it depends on the time snapshot. So it is a timed dependent snapshot. So it is a timed dependent snapshot. So it is a timed dependent vector field. This is something which is vector field. This is something which is vector field. This is something which is new for some people and it takes a bit new for some people and it takes a bit new for some people and it takes a bit of a time to understand it. But you can of a time to understand it. But you can of a time to understand it. But you can imagine something like imagine something like imagine something like let's say you have a u let's say you have a u let's say you have a u you have a bank or let's say a container you have a bank or let's say a container you have a bank or let's say a container in which you pour water in which you pour water in which you pour water and this water is and you are moving the and this water is and you are moving the and this water is and you are moving the container left and right. So the water container left and right. So the water container left and right. So the water is also moving left and right. Now you is also moving left and right. Now you is also moving left and right. Now you take any point in the container, the take any point in the container, the take any point in the container, the velocity at that point is going to velocity at that point is going to velocity at that point is going to change with time because the container change with time because the container change with time because the container itself is moving and every point in the itself is moving and every point in the itself is moving and every point in the container is going to have different container is going to have different container is going to have different velocities. So this is an example of a velocities. So this is an example of a velocities. So this is an example of a timed dependent vector field.

  67. L2 @ 787s

    timed dependent vector field. So another way of defining this is So another way of defining this is So another way of defining this is people usually write this t as a people usually write this t as a people usually write this t as a subscript and x is written in the subscript and x is written in the subscript and x is written in the bracket. So this means that the velocity bracket. So this means that the velocity bracket. So this means that the velocity field at time snapshot t and at a space field at time snapshot t and at a space field at time snapshot t and at a space position which is denoted as x. position which is denoted as x. position which is denoted as x. Okay. So now the Okay. So now the Okay. So now the interesting part about this velocity interesting part about this velocity interesting part about this velocity field or the vector field is that the field or the vector field is that the field or the vector field is that the trajectories of the points are changing trajectories of the points are changing trajectories of the points are changing according to it. according to it. according to it. So up until now what I have said is that So up until now what I have said is that So up until now what I have said is that all the points in space are having some all the points in space are having some all the points in space are having some vector. vector. vector. Okay. Now I'm saying something else. I'm Okay. Now I'm saying something else. I'm Okay. Now I'm saying something else. I'm saying that these vectors represent the saying that these vectors represent the saying that these vectors represent the velocities of these particles which velocities of these particles which velocities of these particles which means that the means that the means that the position of this particle is going to position of this particle is going to position of this particle is going to change because of these velocities. change because of these velocities. change because of these velocities. So for example, if we take this example So for example, if we take this example

  68. L2 @ 851s

    So for example, if we take this example at at the top at time t equal to0, this at at the top at time t equal to0, this at at the top at time t equal to0, this is what my velocity field looks like. is what my velocity field looks like. is what my velocity field looks like. And because of these this velocity field And because of these this velocity field And because of these this velocity field my particles which is let's say I focus my particles which is let's say I focus my particles which is let's say I focus on this green particle right now the on this green particle right now the on this green particle right now the position of this particle is going to position of this particle is going to change because this velocity vector is change because this velocity vector is change because this velocity vector is acts upon it. So maybe at the next time acts upon it. So maybe at the next time acts upon it. So maybe at the next time step this particle moves here this step this particle moves here this step this particle moves here this particle moves here. This particle moves particle moves here. This particle moves particle moves here. This particle moves here. here. here. So the location of all the particles is So the location of all the particles is So the location of all the particles is going to change because of the velocity going to change because of the velocity going to change because of the velocity which acts upon them and then we move to the next time and then we move to the next time snapshot. Now here again there might be snapshot. Now here again there might be snapshot. Now here again there might be some other velocity field vector which some other velocity field vector which some other velocity field vector which is imposed on these points. So again the is imposed on these points. So again the is imposed on these points. So again the position of these points will change. position of these points will change. position of these points will change. So now we are asking the question how do So now we are asking the question how do

  69. L2 @ 916s

    So now we are asking the question how do the position of particles which are the position of particles which are the position of particles which are inside the space change according to inside the space change according to inside the space change according to this velocity vector field this velocity vector field this velocity vector field and that is given by an equation which and that is given by an equation which and that is given by an equation which looks like this. looks like this. looks like this. [snorts] Now this is very similar to the velocity Now this is very similar to the velocity formula. All of us know that velocity is formula. All of us know that velocity is formula. All of us know that velocity is given by given by given by distance upon time which can also be written as change in which can also be written as change in the position of the particle the position of the particle the position of the particle divided by change in time which is delta divided by change in time which is delta divided by change in time which is delta x divided by delta t. And if these x divided by delta t. And if these x divided by delta t. And if these changes are small enough this can be changes are small enough this can be changes are small enough this can be approximated as dx by dt. approximated as dx by dt. approximated as dx by dt. So given a velocity vector field at any So given a velocity vector field at any So given a velocity vector field at any point if I know a small incremental point if I know a small incremental point if I know a small incremental change in the time I can predict by how change in the time I can predict by how change in the time I can predict by how much the particle is going to move in much the particle is going to move in much the particle is going to move in space. space. space. So I can predict the displacement of the So I can predict the displacement of the So I can predict the displacement of the particle if I know the velocity vector particle if I know the velocity vector

  70. L2 @ 984s

    particle if I know the velocity vector field. This is exactly what this formula field. This is exactly what this formula field. This is exactly what this formula says. says. says. So if I have a particle located at So if I have a particle located at So if I have a particle located at position x at time t, position x at time t, position x at time t, then the change in the location of that then the change in the location of that then the change in the location of that particle is given by is given by the velocity vector field at that same the velocity vector field at that same the velocity vector field at that same point and at time t. This is what this point and at time t. This is what this point and at time t. This is what this formula represents. and we have schematically drawn it in in and we have schematically drawn it in in this figure itself. If we want to this figure itself. If we want to this figure itself. If we want to exactly calculate this displacement, exactly calculate this displacement, exactly calculate this displacement, let [snorts] us do that using this let [snorts] us do that using this let [snorts] us do that using this formula. Let's let's focus on this formula. Let's let's focus on this formula. Let's let's focus on this point. So let's say this velocity is u point. So let's say this velocity is u point. So let's say this velocity is u and now the velocity u of x comma 0. and now the velocity u of x comma 0. and now the velocity u of x comma 0. Okay. So let's say this velocity is Okay. So let's say this velocity is Okay. So let's say this velocity is given by given by 3. Okay. Now I'm interested to find the Okay. Now I'm interested to find the [clears throat] dx by dt [clears throat] dx by dt [clears throat] dx by dt the change in velo the change in the the change in velo the change in the the change in velo the change in the position of this point with time and I know this is equal to u with time and I know this is equal to u of x0 of x0

  71. L2 @ 1079s

    of x0 okay so now this if I write as delta x okay so now this if I write as delta x okay so now this if I write as delta x by delta t by delta t by delta t is equal to mu of u of x0. is equal to mu of u of x0. is equal to mu of u of x0. Now if let's say delta t is equal to 1 Now if let's say delta t is equal to 1 Now if let's say delta t is equal to 1 then delta x is equal to u which is 3 e then delta x is equal to u which is 3 e then delta x is equal to u which is 3 e x + 4 ey. Now this is of course an approximate Now this is of course an approximate usually delta t will be very small in usually delta t will be very small in usually delta t will be very small in magnitude. So what that means is that magnitude. So what that means is that magnitude. So what that means is that this particle will move three places in the x direction and four three places in the x direction and four places in the y direction. places in the y direction. places in the y direction. So if we know the velocity field at all So if we know the velocity field at all So if we know the velocity field at all points at all times snapshots points at all times snapshots points at all times snapshots theoretically we can calculate the exact theoretically we can calculate the exact theoretically we can calculate the exact trajectories of each and every particle trajectories of each and every particle trajectories of each and every particle in the fluid. in the fluid. in the fluid. So imagine that you have a river and you So imagine that you have a river and you So imagine that you have a river and you place a small box in the river place a small box in the river place a small box in the river and you ask the question where will this and you ask the question where will this and you ask the question where will this box end up at time t equal to 10 let's

  72. L2 @ 1148s

    box end up at time t equal to 10 let's box end up at time t equal to 10 let's say. Now if you know the exact velocity say. Now if you know the exact velocity say. Now if you know the exact velocity at all points in the river at all times at all points in the river at all times at all points in the river at all times you can map the trajectory of this box you can map the trajectory of this box you can map the trajectory of this box and how it moves with time and where it and how it moves with time and where it and how it moves with time and where it ends up at time t equal to 10. That is ends up at time t equal to 10. That is ends up at time t equal to 10. That is exactly what we are saying in this exactly what we are saying in this exactly what we are saying in this mathematical description. mathematical description. mathematical description. and uh this is represented by this and uh this is represented by this and uh this is represented by this simple formula which is also a simple formula which is also a simple formula which is also a differential equation. Now the central question that we want to Now the central question that we want to ask is if a point starts at x0 ask is if a point starts at x0 ask is if a point starts at x0 at time t =0 at time t =0 at time t =0 and follows the vector field then where and follows the vector field then where and follows the vector field then where does the particle end up at a later does the particle end up at a later does the particle end up at a later time? So time? So time? So if if [snorts] I have a box and I place if if [snorts] I have a box and I place if if [snorts] I have a box and I place it in the river at time t equal to0, the it in the river at time t equal to0, the it in the river at time t equal to0, the box follows the vector field and the

  73. L2 @ 1202s

    box follows the vector field and the box follows the vector field and the question I'm trying to address is where question I'm trying to address is where question I'm trying to address is where does the box end up later in time does the box end up later in time does the box end up later in time because of the flow of the river. Now this is nicely illustrated in this Now this is nicely illustrated in this uh image. uh image. uh image. So we start at x x0 and you can see how So we start at x x0 and you can see how So we start at x x0 and you can see how the particle moves. the particle moves. the particle moves. Now you can see two things here. The Now you can see two things here. The Now you can see two things here. The green dot is the particle and the white green dot is the particle and the white green dot is the particle and the white line is the trajectory followed by that line is the trajectory followed by that line is the trajectory followed by that particle. particle. particle. But in the background what do we see? We But in the background what do we see? We But in the background what do we see? We see these arrows are constantly changing see these arrows are constantly changing see these arrows are constantly changing with time. This is because with time. This is because with time. This is because as the particle moves the time is also as the particle moves the time is also as the particle moves the time is also changing and the velocity vector field changing and the velocity vector field changing and the velocity vector field at every time is different. So these at every time is different. So these at every time is different. So these arrows represent the vector field. arrows represent the vector field. arrows represent the vector field. Maybe I can uh denote it using a label Maybe I can uh denote it using a label Maybe I can uh denote it using a label over here. So these arrows over here. So these arrows over here. So these arrows they represent the vector field which is they represent the vector field which is

  74. L2 @ 1266s

    they represent the vector field which is time dependent. That is why the arrows time dependent. That is why the arrows time dependent. That is why the arrows are changing with time. And if we know are changing with time. And if we know are changing with time. And if we know the time dependent vector field by using the time dependent vector field by using the time dependent vector field by using the above differential equation, we can the above differential equation, we can the above differential equation, we can exactly pinpoint how each particle will exactly pinpoint how each particle will exactly pinpoint how each particle will move according to these time dependent move according to these time dependent move according to these time dependent vector fields. So to answer this question, we need to So to answer this question, we need to solve the above differential equation. solve the above differential equation. solve the above differential equation. Now the idea is simple. We use velocity Now the idea is simple. We use velocity Now the idea is simple. We use velocity vectors to advance the point in small vectors to advance the point in small vectors to advance the point in small time increments. This is something we time increments. This is something we time increments. This is something we saw in the previous example also that if saw in the previous example also that if saw in the previous example also that if we have the velocity vector [snorts] uh we have the velocity vector [snorts] uh we have the velocity vector [snorts] uh we use we we use we we use we convert dx by dt as delta x by delta t convert dx by dt as delta x by delta t convert dx by dt as delta x by delta t and then we can calculate the individual and then we can calculate the individual and then we can calculate the individual displacements of each point because of displacements of each point because of displacements of each point because of these vector fields. the white line is something that we have the white line is something that we have got as the trajectory of the particle got as the trajectory of the particle got as the trajectory of the particle but it is not quite correct because at

  75. L2 @ 1341s

    but it is not quite correct because at but it is not quite correct because at the end of it we are doing an the end of it we are doing an the end of it we are doing an approximation. approximation. approximation. We are not calculating the exact vector We are not calculating the exact vector We are not calculating the exact vector field but we are replacing dx by dt by field but we are replacing dx by dt by field but we are replacing dx by dt by delta x by delta t. delta x by delta t. delta x by delta t. And that is why we are calculating an And that is why we are calculating an And that is why we are calculating an approximate approximate approximate trajectory of the particle. So you can trajectory of the particle. So you can trajectory of the particle. So you can see the difference between these white see the difference between these white see the difference between these white lines and the gray lines. lines and the gray lines. lines and the gray lines. And this discrepancy appears because of And this discrepancy appears because of And this discrepancy appears because of the way we have discretized the the way we have discretized the the way we have discretized the differential equation. Okay. So this is the first part which is Okay. So this is the first part which is understanding time dependent vector understanding time dependent vector understanding time dependent vector fields. fields. fields. Understanding how these vector fields Understanding how these vector fields Understanding how these vector fields can be used to find the trajectories of can be used to find the trajectories of can be used to find the trajectories of all the particles in space using a all the particles in space using a all the particles in space using a simple ordinary differential equation. Okay, let's move ahead now and we will Okay, let's move ahead now and we will define [snorts] what is flow. define [snorts] what is flow. define [snorts] what is flow. In the literature, you will see flow In the literature, you will see flow In the literature, you will see flow models as the terminology used models as the terminology used models as the terminology used everywhere in a lot of papers. But flow

  76. L2 @ 1418s

    everywhere in a lot of papers. But flow everywhere in a lot of papers. But flow means something very specific. In our means something very specific. In our means something very specific. In our mind, we know flow means something is mind, we know flow means something is mind, we know flow means something is moving. But we will moving. But we will moving. But we will u try to understand it in a bit more u try to understand it in a bit more u try to understand it in a bit more rigorous manner to understand what rigorous manner to understand what rigorous manner to understand what exactly is flow. it will turn out that exactly is flow. it will turn out that exactly is flow. it will turn out that it is exactly the same. Uh it it really it is exactly the same. Uh it it really it is exactly the same. Uh it it really matches our physical intuition. matches our physical intuition. matches our physical intuition. So what my physical intuition tells me So what my physical intuition tells me So what my physical intuition tells me is that let's say is that let's say is that let's say um you have um you have um you have a collection of thousand particles and a collection of thousand particles and a collection of thousand particles and these thousand particles are moving in these thousand particles are moving in these thousand particles are moving in space. So the flow is something which space. So the flow is something which space. So the flow is something which tells you the trajectories of these tells you the trajectories of these tells you the trajectories of these thousand particles and it it's it's thousand particles and it it's it's thousand particles and it it's it's basically a collection of the basically a collection of the basically a collection of the trajectories of all of these particles trajectories of all of these particles trajectories of all of these particles in in space and that is exactly true in in space and that is exactly true in in space and that is exactly true according to the formal definition the according to the formal definition the according to the formal definition the flow is a collection of trajectories flow is a collection of trajectories

  77. L2 @ 1474s

    flow is a collection of trajectories which evolve according to the velocity which evolve according to the velocity which evolve according to the velocity field. We have just seen that if we know field. We have just seen that if we know field. We have just seen that if we know the velocity vector field a time the velocity vector field a time the velocity vector field a time dependent velocity vector field we can dependent velocity vector field we can dependent velocity vector field we can calculate the trajectories of every calculate the trajectories of every calculate the trajectories of every individual particle. Now if we do this individual particle. Now if we do this individual particle. Now if we do this for all the particles in the space for all the particles in the space for all the particles in the space we will get the collective motion of the we will get the collective motion of the we will get the collective motion of the object which is called as flow. object which is called as flow. object which is called as flow. Let us understand this using a Let us understand this using a Let us understand this using a visualization. So visualization. So visualization. So here at the background in in gray what here at the background in in gray what here at the background in in gray what you see is the vector field which is you see is the vector field which is you see is the vector field which is marked in gray. Now I have taken a marked in gray. Now I have taken a marked in gray. Now I have taken a snapshot of one single time. Let's say snapshot of one single time. Let's say snapshot of one single time. Let's say this is at time t equal to0. I have this is at time t equal to0. I have this is at time t equal to0. I have taken a snapshot. taken a snapshot. taken a snapshot. As time changes the gray velocity vector As time changes the gray velocity vector As time changes the gray velocity vector field is also going to change. But for field is also going to change. But for field is also going to change. But for now let us look at one single snapshot now let us look at one single snapshot

  78. L2 @ 1530s

    now let us look at one single snapshot because we have we have restricted with because we have we have restricted with because we have we have restricted with the screen in in front of us. Now you the screen in in front of us. Now you the screen in in front of us. Now you can see there are these blue dots. can see there are these blue dots. can see there are these blue dots. These blue dots are particles. These blue dots are particles. These blue dots are particles. Now I have for simplicity denoted a Now I have for simplicity denoted a Now I have for simplicity denoted a collection of 20 particles over here. collection of 20 particles over here. collection of 20 particles over here. But you can imagine these particles to But you can imagine these particles to But you can imagine these particles to be much much more. be much much more. be much much more. Now every particle starts at the blue Now every particle starts at the blue Now every particle starts at the blue point and it it evolves. This is the point and it it evolves. This is the point and it it evolves. This is the trajectory the way the particle moves trajectory the way the particle moves trajectory the way the particle moves with time. Now this trajectory is given with time. Now this trajectory is given with time. Now this trajectory is given or we can calculate this trajectory or we can calculate this trajectory or we can calculate this trajectory because we know the vector field. So for because we know the vector field. So for because we know the vector field. So for every single particle we know how the every single particle we know how the every single particle we know how the trajectory changes with time because we trajectory changes with time because we trajectory changes with time because we know the velocity vector field. And now know the velocity vector field. And now know the velocity vector field. And now you see we have collected all these you see we have collected all these you see we have collected all these trajectories and we get a overall sense trajectories and we get a overall sense

  79. L2 @ 1581s

    trajectories and we get a overall sense of the flow. You can see that this this of the flow. You can see that this this of the flow. You can see that this this is probably something which is swirling. is probably something which is swirling. is probably something which is swirling. It's like a vortex. Um let's say you It's like a vortex. Um let's say you It's like a vortex. Um let's say you take a bucket of water and you put your take a bucket of water and you put your take a bucket of water and you put your hand inside and you make a swirl. Then hand inside and you make a swirl. Then hand inside and you make a swirl. Then you will see something like this. Now you will see something like this. Now you will see something like this. Now it's it it's very simple to visualize it's it it's very simple to visualize it's it it's very simple to visualize what flow is. But if someone asks you what flow is. But if someone asks you what flow is. But if someone asks you how do you define flow more rigorously, how do you define flow more rigorously, how do you define flow more rigorously, you can say that flow is a collection of you can say that flow is a collection of you can say that flow is a collection of all trajectories for all particles which all trajectories for all particles which all trajectories for all particles which are there in that bucket of water. are there in that bucket of water. are there in that bucket of water. Okay. So uh another interesting way to Okay. So uh another interesting way to Okay. So uh another interesting way to visualize how flow is is to place a visualize how flow is is to place a visualize how flow is is to place a small grid on space and see how it small grid on space and see how it small grid on space and see how it deforms. So for this same above example, deforms. So for this same above example, deforms. So for this same above example, let's let's see how the grid deforms. let's let's see how the grid deforms.

  80. L2 @ 1629s

    let's let's see how the grid deforms. Now note that the grid is going to Now note that the grid is going to Now note that the grid is going to deform in a way that the vector field deform in a way that the vector field deform in a way that the vector field exists in the space. So because we have exists in the space. So because we have exists in the space. So because we have a swirling flow, every single point in a swirling flow, every single point in a swirling flow, every single point in the grid is going to move according to a the grid is going to move according to a the grid is going to move according to a swirling trajectory. swirling trajectory. swirling trajectory. And uh we see a cumulative And uh we see a cumulative And uh we see a cumulative deformation of the grid. It looks kind deformation of the grid. It looks kind deformation of the grid. It looks kind of weird here right now, but that is of weird here right now, but that is of weird here right now, but that is what flow is. A flow is something which what flow is. A flow is something which what flow is. A flow is something which deforms a uniformly placed grid into deforms a uniformly placed grid into deforms a uniformly placed grid into something else because of which is something else because of which is something else because of which is governed by the velocity vector field. So this is a technique which uh is So this is a technique which uh is generally used to visualize the flows. generally used to visualize the flows. generally used to visualize the flows. If if we place a square grid, so square If if we place a square grid, so square If if we place a square grid, so square grid is nothing but just uniformly grid is nothing but just uniformly grid is nothing but just uniformly placed points in in the space and every placed points in in the space and every placed points in in the space and every single point is going to evolve single point is going to evolve single point is going to evolve according to a trajectory which is according to a trajectory which is

  81. L2 @ 1689s

    according to a trajectory which is governed by an ordinary differential governed by an ordinary differential governed by an ordinary differential equation and if we collect all these equation and if we collect all these equation and if we collect all these trajectories together we get the flow. trajectories together we get the flow. trajectories together we get the flow. Now uh Now uh Now uh let's look at an example to visualize let's look at an example to visualize let's look at an example to visualize this. So in this example we are going to So in this example we are going to visualize the flow using code. visualize the flow using code. visualize the flow using code. So I'll just run this piece of cell and So I'll just run this piece of cell and So I'll just run this piece of cell and u this is exactly the same animation which I had shown me. So this animation which I had shown me. So this is the flow field and uh here is the is the flow field and uh here is the is the flow field and uh here is the code where I have defined the velocity code where I have defined the velocity code where I have defined the velocity vector field and the particles are vector field and the particles are vector field and the particles are evolving according to that velocity evolving according to that velocity evolving according to that velocity vector field over here. vector field over here. vector field over here. And then uh if you run the second piece And then uh if you run the second piece And then uh if you run the second piece of cell you'll be able to see the of cell you'll be able to see the of cell you'll be able to see the deformation of the square grid as well. deformation of the square grid as well. deformation of the square grid as well. Probably this might not work. So I will Probably this might not work. So I will Probably this might not work. So I will u run the third piece of cell to u run the third piece of cell to u run the third piece of cell to understand how the grid deforms. Yeah. understand how the grid deforms. Yeah.

  82. L2 @ 1766s

    understand how the grid deforms. Yeah. So this this does work. So this this does work. So this this does work. Okay. So you can actually change these Okay. So you can actually change these Okay. So you can actually change these velocity vector fields uh in the velocity vector fields uh in the velocity vector fields uh in the description and uh you can you know play description and uh you can you know play description and uh you can you know play around with it and see how the flow around with it and see how the flow around with it and see how the flow changes according to the velocity vector changes according to the velocity vector changes according to the velocity vector fields which is rightly so because the fields which is rightly so because the fields which is rightly so because the velocity vector fields are governing the velocity vector fields are governing the velocity vector fields are governing the trajectories which are in turn governing trajectories which are in turn governing trajectories which are in turn governing the flow. So if you were to you know uh the flow. So if you were to you know uh the flow. So if you were to you know uh draw a sequence you can say that the draw a sequence you can say that the draw a sequence you can say that the sequence looks as follows. sequence looks as follows. sequence looks as follows. First is a velocity field. From the velocity field velocity field. From the velocity field you get trajectories and from the trajectories you get the and from the trajectories you get the flow. flow. flow. All these three things are related to All these three things are related to All these three things are related to each other. Okay. So this is the first step. Okay. So this is the first step. All of us at this point should All of us at this point should All of us at this point should understand three things. First is the understand three things. First is the understand three things. First is the velocity field which I'm interchangeably velocity field which I'm interchangeably velocity field which I'm interchangeably using the word vector field also. Second using the word vector field also. Second

  83. L2 @ 1834s

    using the word vector field also. Second is the trajectories which are calculated is the trajectories which are calculated is the trajectories which are calculated from the velocity field using the simple from the velocity field using the simple from the velocity field using the simple ordinary differential equation. And the ordinary differential equation. And the ordinary differential equation. And the third is flow which is the collection of third is flow which is the collection of third is flow which is the collection of all the trajectories in space. all the trajectories in space. all the trajectories in space. These three concepts are going to be These three concepts are going to be These three concepts are going to be very fundamental for us as we understand very fundamental for us as we understand very fundamental for us as we understand uh uh uh flow models in more depth. So once we flow models in more depth. So once we flow models in more depth. So once we understand this everything is built on understand this everything is built on understand this everything is built on top of these three concepts. Okay. Now let us take an example Okay. Now let us take an example and uh try to understand the flow in and uh try to understand the flow in and uh try to understand the flow in more depth using some practical more depth using some practical more depth using some practical examples. Okay. Okay. So the example that we will Okay. Okay. So the example that we will take is suppose we have a velocity field take is suppose we have a velocity field take is suppose we have a velocity field which is given by So first thing we can see is this So first thing we can see is this velocity field is dependent on velocity field is dependent on velocity field is dependent on um it's it's just a one-dimensional um it's it's just a one-dimensional um it's it's just a one-dimensional velocity field. In the above example we velocity field. In the above example we velocity field. In the above example we were considering two dimensions X and Y. were considering two dimensions X and Y. were considering two dimensions X and Y. It is only one dimensional minus theta x It is only one dimensional minus theta x

  84. L2 @ 1906s

    It is only one dimensional minus theta x and it is independent of time. So at and it is independent of time. So at and it is independent of time. So at every time snapshot you take a camera every time snapshot you take a camera every time snapshot you take a camera you take a photo of the space the you take a photo of the space the you take a photo of the space the velocity field will be exactly the same velocity field will be exactly the same velocity field will be exactly the same in all the photos which you have taken. in all the photos which you have taken. in all the photos which you have taken. Now uh [snorts] Now uh [snorts] Now uh [snorts] okay so okay so okay so we have to check whether the flow field we have to check whether the flow field we have to check whether the flow field is given by this formula. is given by this formula. is given by this formula. So given the velocity field then the So given the velocity field then the So given the velocity field then the flow field is given by this formula. But let us see whether that is actually But let us see whether that is actually true or not. true or not. true or not. Okay. So what do we know? Because the Okay. So what do we know? Because the Okay. So what do we know? Because the velocity field is given by this formula, velocity field is given by this formula, velocity field is given by this formula, we can use the differential equation we can use the differential equation we can use the differential equation that we have been talking about in this that we have been talking about in this that we have been talking about in this lecture multiple times. So that lecture multiple times. So that lecture multiple times. So that differential equation simply says dx by differential equation simply says dx by differential equation simply says dx by dt is equal to the velocity field which dt is equal to the velocity field which dt is equal to the velocity field which is dxt by dt is equal to minus theta of is dxt by dt is equal to minus theta of

  85. L2 @ 1972s

    is dxt by dt is equal to minus theta of xt. xt. xt. Now let's see if this is indeed the Now let's see if this is indeed the Now let's see if this is indeed the flow. flow. If we substitute this in xt, it should If we substitute this in xt, it should If we substitute this in xt, it should satisfy this equation. So let us see if satisfy this equation. So let us see if satisfy this equation. So let us see if that actually happens or not. If xt is that actually happens or not. If xt is that actually happens or not. If xt is equal to equal to equal to e^ minus theta tx0, then dxt by dt then dxt by dt [clears throat] [clears throat] is equal to minus theta e raus theta t is equal to minus theta e raus theta t is equal to minus theta e raus theta t x0 x0 x0 which is equal to minus theta xt. which is equal to minus theta xt. which is equal to minus theta xt. So it does satisfy the differential So it does satisfy the differential So it does satisfy the differential equation. equation. equation. Now remember we said that the flow is a Now remember we said that the flow is a Now remember we said that the flow is a collection of trajectories. This is the collection of trajectories. This is the collection of trajectories. This is the collection of trajectories. Depending on the initial position Depending on the initial position you exactly get how the particle changes you exactly get how the particle changes you exactly get how the particle changes or moves with time. Let's see how to or moves with time. Let's see how to or moves with time. Let's see how to visualize this. It's it's very visualize this. It's it's very visualize this. It's it's very interesting. So if x0 is 0, interesting. So if x0 is 0, interesting. So if x0 is 0, you simply get a straight line because you simply get a straight line because you simply get a straight line because theta is equal to zero or this this this theta is equal to zero or this this this theta is equal to zero or this this this is itself completely zero. So the is itself completely zero. So the

  86. L2 @ 2046s

    is itself completely zero. So the position stays zero. But position stays zero. But position stays zero. But let's look at yeah if x0 is any any any let's look at yeah if x0 is any any any let's look at yeah if x0 is any any any positive number then it will simply look positive number then it will simply look positive number then it will simply look like an exponential curve. For example, like an exponential curve. For example, like an exponential curve. For example, if x0 is 3, then it will be 3 e to minus if x0 is 3, then it will be 3 e to minus if x0 is 3, then it will be 3 e to minus theta t. theta t. theta t. So it will decrease like this. And So it will decrease like this. And So it will decrease like this. And similarly for positive and negative x we similarly for positive and negative x we similarly for positive and negative x we can check. So this is how the flow looks can check. So this is how the flow looks can check. So this is how the flow looks like for a simple linear ordinary like for a simple linear ordinary like for a simple linear ordinary differential equation. differential equation. differential equation. Now why is this a flow? Because this is Now why is this a flow? Because this is Now why is this a flow? Because this is a collection of trajectories. Given the a collection of trajectories. Given the a collection of trajectories. Given the initial position of my particle, I can initial position of my particle, I can initial position of my particle, I can exactly pinpoint how the particle moves exactly pinpoint how the particle moves exactly pinpoint how the particle moves with time. and a collection of all these with time. and a collection of all these with time. and a collection of all these trajectories is going to help me trajectories is going to help me trajectories is going to help me identify what my flow is. So we can uh visualize this using the So we can uh visualize this using the same Google collab notebook and um you can see here I have taken the same you can see here I have taken the same example example

  87. L2 @ 2112s

    example x0 into e to minus theta t and we get the exact same plot. and we get the exact same plot. So we can actually play this and see how So we can actually play this and see how So we can actually play this and see how the uh particles move with time. So the the uh particles move with time. So the the uh particles move with time. So the particles start at so all these particles start at so all these particles start at so all these particles are let's say attached on a particles are let's say attached on a particles are let's say attached on a string they start at a different string they start at a different string they start at a different location and all of them are moving and location and all of them are moving and location and all of them are moving and um going towards [clears throat] zero um going towards [clears throat] zero um going towards [clears throat] zero finally all of them are going to end up finally all of them are going to end up finally all of them are going to end up at zero. So this is almost like a sink. at zero. So this is almost like a sink. at zero. So this is almost like a sink. Imagine you have a sink and you start Imagine you have a sink and you start Imagine you have a sink and you start the particles anywhere. Every particle the particles anywhere. Every particle the particles anywhere. Every particle moves towards the sink and ends up in moves towards the sink and ends up in moves towards the sink and ends up in the sink. So that is the flow. the sink. So that is the flow. the sink. So that is the flow. Okay. So now with this example uh I hope Okay. So now with this example uh I hope Okay. So now with this example uh I hope we have understood what flow is how the we have understood what flow is how the we have understood what flow is how the flow is related to the velocity vector flow is related to the velocity vector flow is related to the velocity vector field and how the flow is a collection field and how the flow is a collection

  88. L2 @ 2163s

    field and how the flow is a collection of the trajectories. of the trajectories. of the trajectories. [snorts] Okay. Now uh once we are getting Okay. Now uh once we are getting confident with the definitions of confident with the definitions of confident with the definitions of velocity fields and flow let us move on. velocity fields and flow let us move on. velocity fields and flow let us move on. So the question is and we have already So the question is and we have already So the question is and we have already looked at this briefly. If we have a OD looked at this briefly. If we have a OD looked at this briefly. If we have a OD which is an ordinary differential which is an ordinary differential which is an ordinary differential equation, how do we simulate it and find equation, how do we simulate it and find equation, how do we simulate it and find the trajectories? I have said before that um once you have I have said before that um once you have the vector field you'll be able to find the vector field you'll be able to find the vector field you'll be able to find the trajectories and I have briefly the trajectories and I have briefly the trajectories and I have briefly explained it here as well how that can explained it here as well how that can explained it here as well how that can be done but maybe we can uh explain it a be done but maybe we can uh explain it a be done but maybe we can uh explain it a bit more formally. bit more formally. bit more formally. So the method to do that is called as So the method to do that is called as So the method to do that is called as the oiler method. the oiler method. the oiler method. And what I oiler method says is we And what I oiler method says is we And what I oiler method says is we simply take steps in the direction of simply take steps in the direction of simply take steps in the direction of the velocity field. the velocity field. the velocity field. We take small steps in the direction of We take small steps in the direction of

  89. L2 @ 2226s

    We take small steps in the direction of the velocity field and we see how the the velocity field and we see how the the velocity field and we see how the particle trajectory changes with time. particle trajectory changes with time. particle trajectory changes with time. So for example, if we start here, we So for example, if we start here, we So for example, if we start here, we take a tiny step, we go here. We take take a tiny step, we go here. We take take a tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. So the another tiny step, we go here. So the another tiny step, we go here. So the velocity field will help us to take velocity field will help us to take velocity field will help us to take these tiny steps. these tiny steps. these tiny steps. And this is the same thing which we have And this is the same thing which we have And this is the same thing which we have outlined here. It looks slightly outlined here. It looks slightly outlined here. It looks slightly detailed, but let's understand this. detailed, but let's understand this. detailed, but let's understand this. Okay. So as an input, we have a vector Okay. So as an input, we have a vector Okay. So as an input, we have a vector field which changes with time. We have field which changes with time. We have field which changes with time. We have initial condition x0 initial condition x0 initial condition x0 and uh we have the number of steps n and uh we have the number of steps n and uh we have the number of steps n which is how many times you are going to which is how many times you are going to which is how many times you are going to update the particle trajectory for how update the particle trajectory for how update the particle trajectory for how long is your simulation carried on. long is your simulation carried on.

  90. L2 @ 2276s

    long is your simulation carried on. Okay. So we initialize time as t equal Okay. So we initialize time as t equal Okay. So we initialize time as t equal to0. The step size which is delta t is 1 to0. The step size which is delta t is 1 to0. The step size which is delta t is 1 by n. by n. by n. And uh the basic idea is which is given And uh the basic idea is which is given And uh the basic idea is which is given here. dx by dt is equal to mu here. dx by dt is equal to mu here. dx by dt is equal to mu u rather. So delta x by delta t is equal u rather. So delta x by delta t is equal u rather. So delta x by delta t is equal to d u. So we can write delta x as u * to d u. So we can write delta x as u * to d u. So we can write delta x as u * delta t. delta t. delta t. Okay. And delta x is x t + delta t minus Okay. And delta x is x t + delta t minus Okay. And delta x is x t + delta t minus xt is equal to u * delta t. xt is equal to u * delta t. xt is equal to u * delta t. Now here delta t is replaced by h. Now here delta t is replaced by h. Now here delta t is replaced by h. So x of t + h is equal to xt + h into u So x of t + h is equal to xt + h into u So x of t + h is equal to xt + h into u which is what is mentioned over here. which is what is mentioned over here. which is what is mentioned over here. This is how we update the position of This is how we update the position of This is how we update the position of the particle to calculate the next the particle to calculate the next the particle to calculate the next position. That is all is mentioned in position. That is all is mentioned in

  91. L2 @ 2336s

    position. That is all is mentioned in the flowchart. So basically the flowchart. So basically the flowchart. So basically if we have a particle and if we know the if we have a particle and if we know the if we have a particle and if we know the velocity vector field we take small velocity vector field we take small velocity vector field we take small increments in time we change this delta increments in time we change this delta increments in time we change this delta x this moves here now now we again x this moves here now now we again x this moves here now now we again change it we again change it we again change it we again change it we again change it we again change it we again change it and we do it till the time we change it and we do it till the time we change it and we do it till the time we are ending the simulation are ending the simulation are ending the simulation this is also called as the oiler method this is also called as the oiler method this is also called as the oiler method so the reason why we are describing this so the reason why we are describing this so the reason why we are describing this is if we know the velocity vector field is if we know the velocity vector field is if we know the velocity vector field at every time we can calculate the at every time we can calculate the at every time we can calculate the trajectories of the particles very trajectories of the particles very trajectories of the particles very precisely precisely precisely and this is something we should keep in and this is something we should keep in and this is something we should keep in mind. We are going to utilize this mind. We are going to utilize this mind. We are going to utilize this later. This is what is being used in the later. This is what is being used in the later. This is what is being used in the inference for the flow models. inference for the flow models. inference for the flow models. For example, when we are using flow For example, when we are using flow

  92. L2 @ 2385s

    For example, when we are using flow models for robotics or protein models for robotics or protein models for robotics or protein synthesis, synthesis, synthesis, people calculate the flow which is the people calculate the flow which is the people calculate the flow which is the trajectories of the particles but or the trajectories of the particles but or the trajectories of the particles but or the velocity vector field which governs the velocity vector field which governs the velocity vector field which governs the trajectories of the particles. But from trajectories of the particles. But from trajectories of the particles. But from there, how do you calculate the there, how do you calculate the there, how do you calculate the trajectories itself? trajectories itself? trajectories itself? Then you have to solve that differential Then you have to solve that differential Then you have to solve that differential equation which gives you the equation which gives you the equation which gives you the trajectories of these individual trajectories of these individual trajectories of these individual particles. particles. particles. So the inference happens by solving an So the inference happens by solving an So the inference happens by solving an ordinary differential equation which is ordinary differential equation which is ordinary differential equation which is a very unique property of the flow a very unique property of the flow a very unique property of the flow models. models. models. Remember this never happened in Remember this never happened in Remember this never happened in diffusion. This never happened in VA. diffusion. This never happened in VA. diffusion. This never happened in VA. In diffusion the inference happened by In diffusion the inference happened by In diffusion the inference happened by repeated sampling from noise to data. We repeated sampling from noise to data. We repeated sampling from noise to data. We did not sample a differential equation. did not sample a differential equation. did not sample a differential equation. In uh variational autoenccoders we In uh variational autoenccoders we In uh variational autoenccoders we trained a neural network to predict trained a neural network to predict trained a neural network to predict images. There also we did not have a images. There also we did not have a images. There also we did not have a differential equation. This is what differential equation. This is what differential equation. This is what makes flow models unique.

  93. L2 @ 2449s

    Now we have defined the word flow in the Now we have defined the word flow in the initial part of the lecture. We will initial part of the lecture. We will initial part of the lecture. We will understand what is the meaning of flow understand what is the meaning of flow understand what is the meaning of flow models models models which is the whole title of today's which is the whole title of today's which is the whole title of today's lecture is flow models. Right? We have lecture is flow models. Right? We have lecture is flow models. Right? We have just looked at the meaning of the word just looked at the meaning of the word just looked at the meaning of the word flow. flow. So let's start with our original goal of So let's start with our original goal of So let's start with our original goal of deep generative models. deep generative models. deep generative models. Given we have let's say images of cats. Given we have let's say images of cats. Given we have let's say images of cats. We want to find the distribution which We want to find the distribution which We want to find the distribution which governs these images. governs these images. governs these images. So our goal is to find a complex So our goal is to find a complex So our goal is to find a complex distribution which is P data. But we do distribution which is P data. But we do distribution which is P data. But we do not have P data. All we have is a simple not have P data. All we have is a simple not have P data. All we have is a simple distribution which is P init which maybe distribution which is P init which maybe distribution which is P init which maybe it is a Gshian distribution. it is a Gshian distribution. it is a Gshian distribution. Remember what we did in diffusion Remember what we did in diffusion Remember what we did in diffusion models. We started with noise and we models. We started with noise and we models. We started with noise and we ended with the data distribution. ended with the data distribution. ended with the data distribution. So this is exactly what we are going to

  94. L2 @ 2501s

    So this is exactly what we are going to So this is exactly what we are going to do in flow models. Also our goal is to do in flow models. Also our goal is to do in flow models. Also our goal is to convert a simple distribution pinet to a convert a simple distribution pinet to a convert a simple distribution pinet to a complex distribution which is P data. complex distribution which is P data. complex distribution which is P data. But we do not know what is P data. Okay. So now let us understand how we Okay. So now let us understand how we are going to convert the simple are going to convert the simple are going to convert the simple distribution to a complex distribution distribution to a complex distribution distribution to a complex distribution and the simulation of a ordinary and the simulation of a ordinary and the simulation of a ordinary differential equation is a natural differential equation is a natural differential equation is a natural choice for this transformation. A flow model is described by the A flow model is described by the following ordinary differential following ordinary differential following ordinary differential equation. Okay. So Okay. So how the ordinary differential is how the ordinary differential is how the ordinary differential is equation is given by is equation is given by is equation is given by is x0 is the initial distribution that we x0 is the initial distribution that we x0 is the initial distribution that we start out with. It can be complete start out with. It can be complete start out with. It can be complete noise. It can be a gshian distribution. noise. It can be a gshian distribution. noise. It can be a gshian distribution. And And And now instead of making the particles now instead of making the particles now instead of making the particles evolve with time according to evolve with time according to evolve with time according to differential equation. differential equation. Now the main difference here is that we Now the main difference here is that we Now the main difference here is that we make the probability distribution itself make the probability distribution itself make the probability distribution itself change with time. change with time. change with time. So we use the same ordinary differential

  95. L2 @ 2585s

    So we use the same ordinary differential So we use the same ordinary differential equation dx by dt is equal to u. But equation dx by dt is equal to u. But equation dx by dt is equal to u. But this time the main difference is that so this time the main difference is that so this time the main difference is that so far we have been looking at individual far we have been looking at individual far we have been looking at individual particles right. We have been looking at particles right. We have been looking at particles right. We have been looking at the movement of those particles. the movement of those particles. the movement of those particles. Now what we are doing is Now what we are doing is Now what we are doing is we still have particles but these we still have particles but these we still have particles but these particles are a part of a distribution particles are a part of a distribution particles are a part of a distribution of a probability distribution. of a probability distribution. of a probability distribution. So we are still evolving the So we are still evolving the So we are still evolving the trajectories of the particles but now trajectories of the particles but now trajectories of the particles but now these particles as a unit are these particles as a unit are these particles as a unit are representing something. representing something. representing something. For example let's say we have 100 For example let's say we have 100 For example let's say we have 100 particles in a space in a box. Okay. And particles in a space in a box. Okay. And particles in a space in a box. Okay. And these particles are evolving with time. these particles are evolving with time. these particles are evolving with time. they are going in space according to the they are going in space according to the they are going in space according to the velocity field. We have just understood velocity field. We have just understood velocity field. We have just understood how that works. Now the only thing which how that works. Now the only thing which how that works. Now the only thing which we have added in flow model is that the

  96. L2 @ 2638s

    we have added in flow model is that the we have added in flow model is that the collection of these particles represent collection of these particles represent collection of these particles represent a distribution. a distribution. a distribution. So initially we start with a normal So initially we start with a normal So initially we start with a normal distribution which is a gshian distribution which is a gshian distribution which is a gshian distribution and then we have to make distribution and then we have to make distribution and then we have to make the particles flow such that from the the particles flow such that from the the particles flow such that from the normal distribution you move to the normal distribution you move to the normal distribution you move to the distribution that you want to calculate. distribution that you want to calculate. distribution that you want to calculate. That is why it is called a flow model. A That is why it is called a flow model. A That is why it is called a flow model. A flow is a collection of particles flow is a collection of particles flow is a collection of particles evolving according to trajectories. A evolving according to trajectories. A evolving according to trajectories. A flow model is a collection of particles flow model is a collection of particles flow model is a collection of particles which take the initial distribution and which take the initial distribution and which take the initial distribution and convert it into the actual probability convert it into the actual probability convert it into the actual probability distribution which we want to predict. distribution which we want to predict. distribution which we want to predict. So this is what I have mentioned here. So this is what I have mentioned here. So this is what I have mentioned here. Our goal is to make the end point x1 of Our goal is to make the end point x1 of Our goal is to make the end point x1 of the trajectory have the distribution p the trajectory have the distribution p the trajectory have the distribution p data. data. data. In all of these models, the time always In all of these models, the time always In all of these models, the time always evolves from t =0 to t = 1.

  97. L2 @ 2694s

    evolves from t =0 to t = 1. evolves from t =0 to t = 1. So this is sort of what we want. We want So this is sort of what we want. We want So this is sort of what we want. We want to start from the initial distribution to start from the initial distribution to start from the initial distribution and we want to move at x1. We want to and we want to move at x1. We want to and we want to move at x1. We want to move to the data distribution. move to the data distribution. move to the data distribution. Now all of you might have some ideas in Now all of you might have some ideas in Now all of you might have some ideas in your mind related to flow models and you your mind related to flow models and you your mind related to flow models and you might have visualized this in a certain might have visualized this in a certain might have visualized this in a certain way in your mind but we cannot truly way in your mind but we cannot truly way in your mind but we cannot truly understand this unless we look at some understand this unless we look at some understand this unless we look at some practical examples. practical examples. practical examples. So there is an extremely interesting So there is an extremely interesting So there is an extremely interesting Google collab notebook which uh Google collab notebook which uh Google collab notebook which uh I have opened up I have opened up I have opened up here. Now here the interesting thing is here. Now here the interesting thing is here. Now here the interesting thing is first let us run the first cell. So I'm going to show you how to convert So I'm going to show you how to convert probability distributions from an probability distributions from an probability distributions from an initial distribution to a final initial distribution to a final initial distribution to a final distribution which you want to predict. Okay. So now here you can see this is Okay. So now here you can see this is the start distribution which is a simple the start distribution which is a simple

  98. L2 @ 2757s

    the start distribution which is a simple gshian. It is uniformly distributed. gshian. It is uniformly distributed. gshian. It is uniformly distributed. There are maybe around 2,000 particles There are maybe around 2,000 particles There are maybe around 2,000 particles here. So there are 2,000 particles in here. So there are 2,000 particles in here. So there are 2,000 particles in the space. And these particles represent the space. And these particles represent the space. And these particles represent a distribution. They represent a uniform a distribution. They represent a uniform a distribution. They represent a uniform distribution. distribution. distribution. Okay. And now Okay. And now Okay. And now let us say the distribution that we want let us say the distribution that we want let us say the distribution that we want to predict which is X1. It looks like to predict which is X1. It looks like to predict which is X1. It looks like this. It is the shape of a heart. Now this. It is the shape of a heart. Now this. It is the shape of a heart. Now this also has 200 points. But it has a this also has 200 points. But it has a this also has 200 points. But it has a very specific distribution, right? It is very specific distribution, right? It is very specific distribution, right? It is it it is not like uniform distribution it it is not like uniform distribution it it is not like uniform distribution but it is our data distribution. but it is our data distribution. but it is our data distribution. In reality we will not be knowing this In reality we will not be knowing this In reality we will not be knowing this also but now I'm assuming that we know also but now I'm assuming that we know also but now I'm assuming that we know the start distribution and we know the the start distribution and we know the the start distribution and we know the end distribution and we are going to end distribution and we are going to end distribution and we are going to understand how a flow model takes you understand how a flow model takes you understand how a flow model takes you from the start distribution to the end from the start distribution to the end

  99. L2 @ 2807s

    from the start distribution to the end distribution. Now here there are a lot of options. We Now here there are a lot of options. We can convert can convert can convert we can change the end distribution to we can change the end distribution to we can change the end distribution to maybe a spiral data also. We can change maybe a spiral data also. We can change maybe a spiral data also. We can change it to uh goshian data. But again gshian it to uh goshian data. But again gshian it to uh goshian data. But again gshian data will not make sense. So we have two data will not make sense. So we have two data will not make sense. So we have two options maybe spiral data and hard data options maybe spiral data and hard data options maybe spiral data and hard data over here. over here. over here. Now uh there are some parts here which Now uh there are some parts here which Now uh there are some parts here which will become clear as we go along in the will become clear as we go along in the will become clear as we go along in the next two lectures. next two lectures. next two lectures. For now here I have defined the flow For now here I have defined the flow For now here I have defined the flow model by a neural network which takes a model by a neural network which takes a model by a neural network which takes a position x and time t and outputs a position x and time t and outputs a position x and time t and outputs a velocity vector. This is the velocity velocity vector. This is the velocity velocity vector. This is the velocity vector field which we have been talking vector field which we have been talking vector field which we have been talking about since the start of the lecture. about since the start of the lecture. about since the start of the lecture. Now one crucial thing we have done here Now one crucial thing we have done here Now one crucial thing we have done here is the velocity vector field has been is the velocity vector field has been is the velocity vector field has been replaced by a neural network.

  100. L2 @ 2862s

    replaced by a neural network. replaced by a neural network. So there is another word for this in So there is another word for this in So there is another word for this in literature which is also called as a literature which is also called as a literature which is also called as a neural OD neural OD neural OD which is a neural ordinary differential which is a neural ordinary differential which is a neural ordinary differential equation. equation. It is very similar to what we discussed It is very similar to what we discussed It is very similar to what we discussed before. The only thing is that the input before. The only thing is that the input before. The only thing is that the input is the same X and T. The output is still is the same X and T. The output is still is the same X and T. The output is still the velocity vector but inside you have the velocity vector but inside you have the velocity vector but inside you have a bunch of neurons. Now this velocity field is defined here. Now this velocity field is defined here. So let me run this. Um and then we carry So let me run this. Um and then we carry So let me run this. Um and then we carry the training process. The training the training process. The training the training process. The training process is very interesting and it's process is very interesting and it's process is very interesting and it's extremely simple but as I said before to extremely simple but as I said before to extremely simple but as I said before to reach this simple point is going to take reach this simple point is going to take reach this simple point is going to take us some time. So right now I don't us some time. So right now I don't us some time. So right now I don't expect you to understand the training expect you to understand the training expect you to understand the training process but here you can see the flow process but here you can see the flow process but here you can see the flow model is being trained and the loss is model is being trained and the loss is

  101. L2 @ 2909s

    model is being trained and the loss is going down. What is the meaning of this going down. What is the meaning of this going down. What is the meaning of this statement? The flow model is going to be statement? The flow model is going to be statement? The flow model is going to be trained. The meaning is that we are trained. The meaning is that we are trained. The meaning is that we are training the neural network given by the training the neural network given by the training the neural network given by the velocity vector field which are going to velocity vector field which are going to velocity vector field which are going to take these 2,000 particles in the take these 2,000 particles in the take these 2,000 particles in the initial uniform shape. They are going to initial uniform shape. They are going to initial uniform shape. They are going to evolve the trajectories according to our evolve the trajectories according to our evolve the trajectories according to our differential equation such that the differential equation such that the differential equation such that the final shape will be the heart shape. Next step is to simulate the OD. Now Next step is to simulate the OD. Now this is where it gets very interesting this is where it gets very interesting this is where it gets very interesting and we use the uler step here. You see and we use the uler step here. You see and we use the uler step here. You see here how we are changing the here how we are changing the here how we are changing the uh trajectories of the particles uh trajectories of the particles uh trajectories of the particles according to the oiler step according to the oiler step according to the oiler step and we can see the real data and the and we can see the real data and the and we can see the real data and the generated data. A great way to visualize generated data. A great way to visualize generated data. A great way to visualize this is to see the animation and we can this is to see the animation and we can this is to see the animation and we can see the animation over here once this see the animation over here once this

  102. L2 @ 2965s

    see the animation over here once this code runs. Yeah. So let's see uh as expected we Yeah. So let's see uh as expected we start with the initial gshian start with the initial gshian start with the initial gshian distribution and then slowly the distribution and then slowly the distribution and then slowly the particles evolve and we start to she see particles evolve and we start to she see particles evolve and we start to she see the heart-shaped the heart-shaped the heart-shaped structure. So structure. So structure. So you can see how the particles are slowly you can see how the particles are slowly you can see how the particles are slowly compressing inside moving in and then compressing inside moving in and then compressing inside moving in and then also changing their shape or or moving also changing their shape or or moving also changing their shape or or moving according to lines where they start to according to lines where they start to according to lines where they start to resemble a heart shape. resemble a heart shape. resemble a heart shape. So this is this is very interesting and So this is this is very interesting and So this is this is very interesting and it's almost like what we have done here it's almost like what we have done here it's almost like what we have done here is we have predicted a velocity field is we have predicted a velocity field is we have predicted a velocity field which can take a uniform distribution to which can take a uniform distribution to which can take a uniform distribution to a predicted distribution which we want a predicted distribution which we want a predicted distribution which we want and we can imagine the flow. The and we can imagine the flow. The and we can imagine the flow. The particles are flowing in in time. particles are flowing in in time. particles are flowing in in time. In variation autoenccoders we had an In variation autoenccoders we had an In variation autoenccoders we had an encoder we had a decoder. In diffusion encoder we had a decoder. In diffusion encoder we had a decoder. In diffusion models we had a number of encoders and a models we had a number of encoders and a

  103. L2 @ 3030s

    models we had a number of encoders and a number of decoders. number of decoders. number of decoders. So they were discrete. Here something a So they were discrete. Here something a So they were discrete. Here something a process happens which is continuous. So process happens which is continuous. So process happens which is continuous. So I find this transformation from the I find this transformation from the I find this transformation from the discrete process to a continuous process discrete process to a continuous process discrete process to a continuous process very very interesting in these flow very very interesting in these flow very very interesting in these flow models. models. And uh now what we'll do is let's change And uh now what we'll do is let's change And uh now what we'll do is let's change this target distribution and we'll see this target distribution and we'll see this target distribution and we'll see what happens with the spiral data. Yeah. So now the end distribution is a Yeah. So now the end distribution is a spiral data. We'll again define the spiral data. We'll again define the spiral data. We'll again define the neural vector field. We'll train this neural vector field. We'll train this neural vector field. We'll train this velocity field now. velocity field now. velocity field now. And uh we will see the loss decreases as And uh we will see the loss decreases as And uh we will see the loss decreases as the number of epochs increase. The training is complete. Now we will The training is complete. Now we will simulate the OD simulate the OD simulate the OD using the oiler method. And let's see using the oiler method. And let's see using the oiler method. And let's see the animation. [snorts] Let's see. Let's see. Let's see. So now you see the particles are So now you see the particles are So now you see the particles are revolving and slowly moving into a shape revolving and slowly moving into a shape revolving and slowly moving into a shape which represents a spiral. which represents a spiral. which represents a spiral. Instead of the heart-shaped uh there are Instead of the heart-shaped uh there are Instead of the heart-shaped uh there are the particles who are compressing inside the particles who are compressing inside

  104. L2 @ 3123s

    the particles who are compressing inside here they are moving out and here they are moving out and here they are moving out and slowly resembling that spiral shape. So slowly resembling that spiral shape. So slowly resembling that spiral shape. So this is exactly what we mean by flow this is exactly what we mean by flow this is exactly what we mean by flow models. models. Okay. Let us recap what we did in Okay. Let us recap what we did in Okay. Let us recap what we did in today's lecture. today's lecture. today's lecture. In today's lecture, we first started In today's lecture, we first started In today's lecture, we first started with the concept of time dependent with the concept of time dependent with the concept of time dependent velocity fields. Why we started with the velocity fields. Why we started with the velocity fields. Why we started with the concept of that? Because once we have a concept of that? Because once we have a concept of that? Because once we have a timed dependent velocity field, we can timed dependent velocity field, we can timed dependent velocity field, we can calculate the trajectories of all the calculate the trajectories of all the calculate the trajectories of all the particles in space according to that particles in space according to that particles in space according to that velocity field. How do we do that? We velocity field. How do we do that? We velocity field. How do we do that? We use the the oiler method to do that use the the oiler method to do that use the the oiler method to do that where we discretize the differential where we discretize the differential where we discretize the differential equation and we see how the particles equation and we see how the particles equation and we see how the particles move in time. move in time. move in time. Now the question is okay this is fine we Now the question is okay this is fine we Now the question is okay this is fine we have defined the velocity vector field have defined the velocity vector field have defined the velocity vector field but how how do we solve our problem but how how do we solve our problem but how how do we solve our problem initial problem of converting the uh

  105. L2 @ 3176s

    initial problem of converting the uh initial problem of converting the uh gshian distribution to our actual gshian distribution to our actual gshian distribution to our actual distribution. What we say is that we use distribution. What we say is that we use distribution. What we say is that we use the same concept of a velocity field the same concept of a velocity field the same concept of a velocity field and instead of changing the trajectories and instead of changing the trajectories and instead of changing the trajectories of the particles here we will change the of the particles here we will change the of the particles here we will change the trajectories of particles but we will trajectories of particles but we will trajectories of particles but we will transform the original distribution to a transform the original distribution to a transform the original distribution to a final distribution final distribution final distribution according to the trajectories of the according to the trajectories of the according to the trajectories of the particles which govern that particles which govern that particles which govern that distribution. distribution. And we have a technical word for this And we have a technical word for this And we have a technical word for this which governs the trajectories of all which governs the trajectories of all which governs the trajectories of all these particles which is called as a these particles which is called as a these particles which is called as a flow. flow. And what is a flow model? A flow model And what is a flow model? A flow model And what is a flow model? A flow model is a model which gives us the velocity is a model which gives us the velocity is a model which gives us the velocity vector field vector field vector field that will be used to calculate the that will be used to calculate the that will be used to calculate the trajectories of all the particles trajectories of all the particles trajectories of all the particles and this velocity vector field is often and this velocity vector field is often and this velocity vector field is often times a neural network. times a neural network. times a neural network. So the differential equation is also So the differential equation is also

  106. L2 @ 3230s

    So the differential equation is also termed as a neural ordinary differential termed as a neural ordinary differential termed as a neural ordinary differential equation. So the terms that I have equation. So the terms that I have equation. So the terms that I have introduced in this lecture are first is introduced in this lecture are first is introduced in this lecture are first is ordinary differential equation, neural ordinary differential equation, neural ordinary differential equation, neural ordinary differential equation, vector ordinary differential equation, vector ordinary differential equation, vector field, velocity vector field, timed field, velocity vector field, timed field, velocity vector field, timed dependent velocity vector field, flow dependent velocity vector field, flow dependent velocity vector field, flow and flow models. All of these terms mean and flow models. All of these terms mean and flow models. All of these terms mean something very specific and I want you something very specific and I want you something very specific and I want you to understand these terminologies in to understand these terminologies in to understand these terminologies in detail. detail. detail. Also to give you a vision of where we Also to give you a vision of where we Also to give you a vision of where we are heading something happened in this are heading something happened in this are heading something happened in this training step which allowed us to and training step which allowed us to and training step which allowed us to and this is so simple you can see it's this is so simple you can see it's this is so simple you can see it's hardly 20 lines hardly 20 lines hardly 20 lines and that is exactly the simplicity of and that is exactly the simplicity of and that is exactly the simplicity of the flow modeling approach which we will the flow modeling approach which we will the flow modeling approach which we will get at slowly but we will do it in a get at slowly but we will do it in a get at slowly but we will do it in a very step-by-step manner. very step-by-step manner. very step-by-step manner. Thank you very much everyone and uh this Thank you very much everyone and uh this Thank you very much everyone and uh this ends our first lecture and in the next

  107. L2 @ 3283s

    ends our first lecture and in the next ends our first lecture and in the next lecture we will understand what is the lecture we will understand what is the lecture we will understand what is the target of our flow model. How do we target of our flow model. How do we target of our flow model. How do we define the target and how do we train define the target and how do we train define the target and how do we train the flow models as well.