← back to concept

Continuous Normalizing Flow — across courses

cmu-10799-diffusion-flow

  1. L5 @ 2s

    So last time we learned about how So last time we learned about how diffusion turn noise into data by diffusion turn noise into data by diffusion turn noise into data by gradually adding noise and gradually gradually adding noise and gradually gradually adding noise and gradually denoising. denoising. denoising. H and then we learned another family of H and then we learned another family of H and then we learned another family of model called a scorebased models which model called a scorebased models which model called a scorebased models which is uh doing very very similar things and is uh doing very very similar things and is uh doing very very similar things and it turns out that they're they're pretty it turns out that they're they're pretty it turns out that they're they're pretty much the same. Um if you especially uh much the same. Um if you especially uh much the same. Um if you especially uh if you scale the number of time steps to if you scale the number of time steps to if you scale the number of time steps to infinite and then they all they both infinite and then they all they both infinite and then they all they both becomes this thing called stochastic becomes this thing called stochastic becomes this thing called stochastic process a continuous time stoastic process a continuous time stoastic process a continuous time stoastic process which can be sort of formulated process which can be sort of formulated process which can be sort of formulated into stoastic differential equations. into stoastic differential equations. into stoastic differential equations. All right. So and if you have a forward All right. So and if you have a forward All right. So and if you have a forward SDE uh then you actually directly have a SDE uh then you actually directly have a SDE uh then you actually directly have a formula to uh tell you what your reverse formula to uh tell you what your reverse formula to uh tell you what your reverse process should be which is also defined process should be which is also defined process should be which is also defined by an SDE with this score function and by an SDE with this score function and

  2. L5 @ 52s

    by an SDE with this score function and then we can just do score matching to then we can just do score matching to then we can just do score matching to learn uh how to solve the uh the learn uh how to solve the uh the learn uh how to solve the uh the reversed SDE. reversed SDE. reversed SDE. All right. So, that was like the last All right. So, that was like the last All right. So, that was like the last question question question that we had in class. It's like, is that we had in class. It's like, is that we had in class. It's like, is there a simpler way to do this? And I there a simpler way to do this? And I there a simpler way to do this? And I think someone after they took the quiz, think someone after they took the quiz, think someone after they took the quiz, they were already be like, "Oh, I I feel they were already be like, "Oh, I I feel they were already be like, "Oh, I I feel like I feel like there should be an like I feel like there should be an like I feel like there should be an easier way." Um, yeah. So, this is what easier way." Um, yeah. So, this is what easier way." Um, yeah. So, this is what we're going to learn today. Okay. So, uh we're going to learn today. Okay. So, uh we're going to learn today. Okay. So, uh let's think about this question. Let's let's think about this question. Let's let's think about this question. Let's say we're given a data point. So, we we say we're given a data point. So, we we say we're given a data point. So, we we only have like one image, right? What only have like one image, right? What only have like one image, right? What would be the simplest way to construct a would be the simplest way to construct a would be the simplest way to construct a trajectory from noise to this data trajectory from noise to this data trajectory from noise to this data point? Yeah, point? Yeah, point? Yeah, linear combination. linear combination. linear combination. That's exactly correct. Okay. Why don't That's exactly correct. Okay. Why don't

  3. L5 @ 101s

    That's exactly correct. Okay. Why don't we just do linear interpolation of the we just do linear interpolation of the we just do linear interpolation of the two things, right? Um so exact how two things, right? Um so exact how two things, right? Um so exact how exactly you can do it is basically um exactly you can do it is basically um exactly you can do it is basically um say like bas you just like at the say like bas you just like at the say like bas you just like at the beginning of time you'll have like it's beginning of time you'll have like it's beginning of time you'll have like it's so this noise is fixed and then you have so this noise is fixed and then you have so this noise is fixed and then you have like a large portion of noise mixed with like a large portion of noise mixed with like a large portion of noise mixed with some portion of data and then you get a some portion of data and then you get a some portion of data and then you get a very noisy data and then as you go on very noisy data and then as you go on very noisy data and then as you go on the portion of the noise become fewer the portion of the noise become fewer the portion of the noise become fewer like less and less and then the portion like less and less and then the portion like less and less and then the portion of data become more and more uh and then of data become more and more uh and then of data become more and more uh and then you just construct a chain of you just construct a chain of you just construct a chain of the very noisy to slightly no to the very noisy to slightly no to the very noisy to slightly no to slightly less noisy to very very clean slightly less noisy to very very clean slightly less noisy to very very clean data. Right? So you could just construct data. Right? So you could just construct data. Right? So you could just construct this trajectory in this way. Um yeah so this trajectory in this way. Um yeah so

  4. L5 @ 146s

    this trajectory in this way. Um yeah so remember the same noise same data. remember the same noise same data. remember the same noise same data. Then learning the transformation along Then learning the transformation along Then learning the transformation along this linear trajectory is super easy, this linear trajectory is super easy, this linear trajectory is super easy, right? Because literally what we can do right? Because literally what we can do right? Because literally what we can do is say now we want to uh now we want to is say now we want to uh now we want to is say now we want to uh now we want to um transfer transform from uh time 25% um transfer transform from uh time 25% um transfer transform from uh time 25% to time 50%. uh then what can we do is to time 50%. uh then what can we do is to time 50%. uh then what can we do is we can we literally just have you just we can we literally just have you just we can we literally just have you just need to add 25% more of data and then need to add 25% more of data and then need to add 25% more of data and then minus 25% minus 25% minus 25% uh of of the noise right um so basically uh of of the noise right um so basically uh of of the noise right um so basically and because the delta t is the same so and because the delta t is the same so and because the delta t is the same so you can write it in this way basically you can write it in this way basically you can write it in this way basically just like the just like the just like the uh the the noisy sample at time uh 0.5 uh the the noisy sample at time uh 0.5 uh the the noisy sample at time uh 0.5 is equal to uh the noisy sample at time is equal to uh the noisy sample at time is equal to uh the noisy sample at time 0.25 plus how much time you want to 0.25 plus how much time you want to 0.25 plus how much time you want to progress. So like 0.25 time um times uh

  5. L5 @ 208s

    progress. So like 0.25 time um times uh progress. So like 0.25 time um times uh data minus noise, right? Uh so basically data minus noise, right? Uh so basically data minus noise, right? Uh so basically uh generalizing it into t and delta t. uh generalizing it into t and delta t. uh generalizing it into t and delta t. This is like what you should get. And uh This is like what you should get. And uh This is like what you should get. And uh literally when the delta t goes to zero literally when the delta t goes to zero literally when the delta t goes to zero then you will have this velocity thing then you will have this velocity thing then you will have this velocity thing right. Basically this is just like tells right. Basically this is just like tells right. Basically this is just like tells the velocity thing. just tells you the velocity thing. just tells you the velocity thing. just tells you infinite testimony infinite testimony infinite testimony uh if you take infinite testimony small uh if you take infinite testimony small uh if you take infinite testimony small steps like what is the direction that steps like what is the direction that steps like what is the direction that you should go and no matter what t you should go and no matter what t you should go and no matter what t you're at no matter which time step you're at no matter which time step you're at no matter which time step you're at you're always going like the you're at you're always going like the you're at you're always going like the same speed same direction right so it same speed same direction right so it same speed same direction right so it just like always going to be the just like always going to be the just like always going to be the difference between the data and noise difference between the data and noise difference between the data and noise which is x1 minus minus x0 which is x1 minus minus x0 which is x1 minus minus x0 okay any questions okay any questions okay any questions pretty straightforward right okay So pretty straightforward right okay So pretty straightforward right okay So this is literally what we need to learn,

  6. L5 @ 260s

    this is literally what we need to learn, this is literally what we need to learn, right? This is like literally the only right? This is like literally the only right? This is like literally the only thing that that's stopping us from thing that that's stopping us from thing that that's stopping us from directly getting from noise to data. So directly getting from noise to data. So directly getting from noise to data. So basically the simplest way essentially basically the simplest way essentially basically the simplest way essentially to transform noise straight into data to transform noise straight into data to transform noise straight into data will be at training time. will be at training time. will be at training time. You first sample a noise and then you You first sample a noise and then you You first sample a noise and then you sample a data and then you uniformly sample a data and then you uniformly sample a data and then you uniformly sample a time step and then you compute sample a time step and then you compute sample a time step and then you compute a noisy sample by doing this linear a noisy sample by doing this linear a noisy sample by doing this linear interpolation interpolation interpolation and then you compute the velocity which and then you compute the velocity which and then you compute the velocity which is x1 - x0 just data minus noise and is x1 - x0 just data minus noise and is x1 - x0 just data minus noise and then you learn a then you learn a then you learn a to match this velocity to match this velocity to match this velocity and at sampling time after you learn and at sampling time after you learn and at sampling time after you learn this literally you can just start from t this literally you can just start from t this literally you can just start from t equals z and then sample a noise at time equals z and then sample a noise at time equals z and then sample a noise at time zero zero zero and then progress through the chain. and then progress through the chain. and then progress through the chain. Right? How do you progress through the Right? How do you progress through the

  7. L5 @ 317s

    Right? How do you progress through the chain? You just compute the displacement chain? You just compute the displacement chain? You just compute the displacement for x and then directly apply the for x and then directly apply the for x and then directly apply the displacement to your x to progress on displacement to your x to progress on displacement to your x to progress on the chain and then also progress in time the chain and then also progress in time the chain and then also progress in time steps and at the end just output x1. steps and at the end just output x1. steps and at the end just output x1. Okay. Question. Okay. Question. Okay. Question. somewhere and I'm concerned that this noise has no and I'm concerned that this noise has no semantic meaning. So would it get like semantic meaning. So would it get like semantic meaning. So would it get like like there's no reason why a particular like there's no reason why a particular like there's no reason why a particular gausian noise shouldn't amount to the gausian noise shouldn't amount to the gausian noise shouldn't amount to the rather than the house? rather than the house? rather than the house? >> Cool question. We're going to we're >> Cool question. We're going to we're >> Cool question. We're going to we're going to learn that today. Uh first of going to learn that today. Uh first of going to learn that today. Uh first of all, we're not trying to get any all, we're not trying to get any all, we're not trying to get any semantic meaning here. We're just trying semantic meaning here. We're just trying semantic meaning here. We're just trying to transform noise into data. But second to transform noise into data. But second to transform noise into data. But second of all, uh I think we're going to like of all, uh I think we're going to like of all, uh I think we're going to like answer your question um later today. answer your question um later today. answer your question um later today. Okay, any other questions? Ah, now you have flow matching. This is Ah, now you have flow matching. This is full matching. Class is over. Hey, full matching. Class is over. Hey,

  8. L5 @ 386s

    full matching. Class is over. Hey, no, not really. Okay, fine. Uh but no, not really. Okay, fine. Uh but no, not really. Okay, fine. Uh but basically the the question that we're basically the the question that we're basically the the question that we're gonna ask is that like this is so gonna ask is that like this is so gonna ask is that like this is so simple, right? Like no math, no nothing. simple, right? Like no math, no nothing. simple, right? Like no math, no nothing. Like why do we like why do we need all Like why do we like why do we need all Like why do we like why do we need all those previous diffusion math like those previous diffusion math like those previous diffusion math like stoastic differential equations? Why do stoastic differential equations? Why do stoastic differential equations? Why do we need this? Like also why is this even we need this? Like also why is this even we need this? Like also why is this even valid? Why why why is this the same? Why valid? Why why why is this the same? Why valid? Why why why is this the same? Why can we even do this? Right? If we can can we even do this? Right? If we can can we even do this? Right? If we can just do this, why do we need to do all just do this, why do we need to do all just do this, why do we need to do all the math? Uh well, turns out that thing the math? Uh well, turns out that thing the math? Uh well, turns out that thing also come from math. So we're going to also come from math. So we're going to also come from math. So we're going to learn about the math now. And to understand why this is a proper And to understand why this is a proper um probabilistic gener model we need to um probabilistic gener model we need to um probabilistic gener model we need to go back in time into the time when go back in time into the time when go back in time into the time when people start to de develop this thing. people start to de develop this thing.

  9. L5 @ 430s

    people start to de develop this thing. Okay. So basically back in 2018 there Okay. So basically back in 2018 there Okay. So basically back in 2018 there this like pretty seminal paper um uh this like pretty seminal paper um uh this like pretty seminal paper um uh called continuous normalizing or or like called continuous normalizing or or like called continuous normalizing or or like neuro neuroordinary differential neuro neuroordinary differential neuro neuroordinary differential equation and then basically um this guy equation and then basically um this guy equation and then basically um this guy Ricky who was my uh meta intermentor Ricky who was my uh meta intermentor Ricky who was my uh meta intermentor actually super legendary guy but actually super legendary guy but actually super legendary guy but basically he was like um proposed this basically he was like um proposed this basically he was like um proposed this thing called continuous normalizing thing called continuous normalizing thing called continuous normalizing flows and it's basically just like uh flows and it's basically just like uh flows and it's basically just like uh formulated a general model such that formulated a general model such that formulated a general model such that transform uh the samples from an initial transform uh the samples from an initial transform uh the samples from an initial distribution such as like a gausian to a distribution such as like a gausian to a distribution such as like a gausian to a target distribution such as the image target distribution such as the image target distribution such as the image distribution that we want by integrating distribution that we want by integrating distribution that we want by integrating through an OD through an OD through an OD and this OD is literally just the you and this OD is literally just the you and this OD is literally just the you know the infinite decimal uh change of know the infinite decimal uh change of know the infinite decimal uh change of your data should follow the velocity and your data should follow the velocity and your data should follow the velocity and at sampling time what you can do is you at sampling time what you can do is you at sampling time what you can do is you can just solve this OD by integrating

  10. L5 @ 485s

    can just solve this OD by integrating can just solve this OD by integrating through uh the velocity and then you through uh the velocity and then you through uh the velocity and then you you'll be able to get a sample or like a you'll be able to get a sample or like a you'll be able to get a sample or like a trajectory of samples. Uh trajectory of samples. Uh trajectory of samples. Uh so basically this is like literally what so basically this is like literally what so basically this is like literally what we just did, right? So you have some we just did, right? So you have some we just did, right? So you have some velocity and then you just like velocity and then you just like velocity and then you just like numerically is solve the OD at sampling numerically is solve the OD at sampling numerically is solve the OD at sampling time. time. time. Okay. So like how to interpret this, Okay. So like how to interpret this, Okay. So like how to interpret this, right? So basically just like think of right? So basically just like think of right? So basically just like think of it as like the wind flow transport some it as like the wind flow transport some it as like the wind flow transport some like water some amount of water from the like water some amount of water from the like water some amount of water from the west coast to the east coast or from west coast to the east coast or from west coast to the east coast or from like the Pacific Ocean to the uh to the like the Pacific Ocean to the uh to the like the Pacific Ocean to the uh to the east coast let's just say and um the east coast let's just say and um the east coast let's just say and um the question here is that why this is a question here is that why this is a question here is that why this is a normalizing flow normalizing flow normalizing flow first of all like what is a normalizing first of all like what is a normalizing first of all like what is a normalizing flow? Anyone else I haven't talked to today?

  11. L5 @ 535s

    Anyone else I haven't talked to today? Nobody remember what is normalizing Nobody remember what is normalizing Nobody remember what is normalizing flow? >> You can reconstruct the process. >> You can reconstruct the process. >> Yes. So it's a invertible uh generate >> Yes. So it's a invertible uh generate >> Yes. So it's a invertible uh generate model, right? Yes. And the reason why model, right? Yes. And the reason why model, right? Yes. And the reason why this is a normalizing flow is because this is a normalizing flow is because this is a normalizing flow is because the streams never cross. So imagine how the streams never cross. So imagine how the streams never cross. So imagine how like you're following a stream of river like you're following a stream of river like you're following a stream of river or like water or like the stream of the or like water or like the stream of the or like water or like the stream of the wind, right? Like if you just follow the wind, right? Like if you just follow the wind, right? Like if you just follow the stream, the stream will never like it stream, the stream will never like it stream, the stream will never like it will always point to one direction. So will always point to one direction. So will always point to one direction. So it will never like just get get it will never like just get get it will never like just get get if you just like follow the wind, you're if you just like follow the wind, you're if you just like follow the wind, you're able to basically get from one point to able to basically get from one point to able to basically get from one point to the other point guaranteed. the other point guaranteed. the other point guaranteed. So like deterministically and because So like deterministically and because So like deterministically and because this process is deterministic, if you this process is deterministic, if you this process is deterministic, if you follow this wind, if you follow this follow this wind, if you follow this follow this wind, if you follow this flow, you're actually going to get an flow, you're actually going to get an flow, you're actually going to get an invertible transformation because

  12. L5 @ 590s

    invertible transformation because invertible transformation because everything is deterministic. everything is deterministic. everything is deterministic. Okay, any question? Okay, any question? Okay, any question? Cool. You have a question. Okay. Are we Cool. You have a question. Okay. Are we Cool. You have a question. Okay. Are we saying that it's actually saying that it's actually saying that it's actually >> uh the way that we construct it is is it the way that we construct it is is it yeah it is invertible. Yeah the way that yeah it is invertible. Yeah the way that yeah it is invertible. Yeah the way that we construct it. Okay. we construct it. Okay. we construct it. Okay. Any more question? Cool. All right. Cool. All right. Uh okay. Okay. So how does this connect Uh okay. Okay. So how does this connect Uh okay. Okay. So how does this connect to probability? to probability? to probability? Well, basically in order to for this Well, basically in order to for this Well, basically in order to for this kind of like continuous flow thing model kind of like continuous flow thing model kind of like continuous flow thing model to transport between one probability to transport between one probability to transport between one probability distribution to another we need to have distribution to another we need to have distribution to another we need to have the following assumptions. The first the following assumptions. The first the following assumptions. The first thing is what we call the conservation thing is what we call the conservation thing is what we call the conservation of mass. Basically just imagine that the of mass. Basically just imagine that the of mass. Basically just imagine that the the wind carried a certain amount of the wind carried a certain amount of the wind carried a certain amount of water and this certain amount of water water and this certain amount of water water and this certain amount of water never change like the amount of water never change like the amount of water never change like the amount of water that you transporting never change. So that you transporting never change. So that you transporting never change. So no new mass uh is is going to get added no new mass uh is is going to get added

  13. L5 @ 661s

    no new mass uh is is going to get added and no mass is going to disappear and no mass is going to disappear and no mass is going to disappear getting dropped. So like translate it getting dropped. So like translate it getting dropped. So like translate it into probabilistic statement just means into probabilistic statement just means into probabilistic statement just means that the probability should always add that the probability should always add that the probability should always add up to one because otherwise it won't be up to one because otherwise it won't be up to one because otherwise it won't be a valid probability distribution. And a valid probability distribution. And a valid probability distribution. And then second thing is what we call the then second thing is what we call the then second thing is what we call the continuity equation which is like not continuity equation which is like not continuity equation which is like not only that the mass is conserved it's only that the mass is conserved it's only that the mass is conserved it's also not you cannot teleport them uh also not you cannot teleport them uh also not you cannot teleport them uh which means that everything need to move which means that everything need to move which means that everything need to move smoothly everything should move like smoothly everything should move like smoothly everything should move like continuously continuously continuously okay so basically just means that the okay so basically just means that the okay so basically just means that the probability can only get changed probability can only get changed probability can only get changed continuously you cannot just like jump continuously you cannot just like jump continuously you cannot just like jump from from point to point because it's from from point to point because it's from from point to point because it's it's a river okay any question it's a river okay any question it's a river okay any question cool all right Um so because of this we cool all right Um so because of this we cool all right Um so because of this we get to define something called a get to define something called a get to define something called a probability flux. So imagine a flux is probability flux. So imagine a flux is

  14. L5 @ 710s

    probability flux. So imagine a flux is like the amount of flow the amount of like the amount of flow the amount of like the amount of flow the amount of water uh per unit per time that's going water uh per unit per time that's going water uh per unit per time that's going through a certain a certain uh unit through a certain a certain uh unit through a certain a certain uh unit space. So if you're thinking about like space. So if you're thinking about like space. So if you're thinking about like how the probability flows is a how the probability flows is a how the probability flows is a probability flux is equal to the probability flux is equal to the probability flux is equal to the velocity times the the the density sort velocity times the the the density sort velocity times the the the density sort of. So the velocity is giving you where of. So the velocity is giving you where of. So the velocity is giving you where and how fast the probability flows and and how fast the probability flows and and how fast the probability flows and then the density is telling you how much then the density is telling you how much then the density is telling you how much probability there was so that probability there was so that probability there was so that it flows out right um okay so this is it flows out right um okay so this is it flows out right um okay so this is like a like a like a um visualization generated by vo I um visualization generated by vo I um visualization generated by vo I already tried but let's just uh look at already tried but let's just uh look at already tried but let's just uh look at it okay okay maybe I should put on okay do that maybe I should put on okay do that maybe I should put on okay do that All right. Okay. So, as you can see, as the thing Okay. So, as you can see, as the thing as the water flows, first of all, it's as the water flows, first of all, it's as the water flows, first of all, it's in a container. It's in like a contained

  15. L5 @ 775s

    in a container. It's in like a contained in a container. It's in like a contained tank. So, the probability mass never tank. So, the probability mass never tank. So, the probability mass never change. And second, like basically based change. And second, like basically based change. And second, like basically based on how fast the particle flows um like on how fast the particle flows um like on how fast the particle flows um like the probability density, which is like the probability density, which is like the probability density, which is like the water amount at this particular the water amount at this particular the water amount at this particular location, also changes. location, also changes. location, also changes. Okay, any questions? >> Flux in this case like similar to like >> Flux in this case like similar to like magnetic flu magnetic flu magnetic flu like it's like the amount of like like it's like the amount of like like it's like the amount of like >> Nope. >> Nope. >> Nope. >> It's a it just I think water is a better >> It's a it just I think water is a better >> It's a it just I think water is a better um um um I I mean I I mean I I mean >> like >> like >> like >> Yeah. Yeah. Kind of >> Yeah. Yeah. Kind of >> Yeah. Yeah. Kind of >> kind of. Yeah. It's like amount of flows >> kind of. Yeah. It's like amount of flows >> kind of. Yeah. It's like amount of flows that goes through a certain place within that goes through a certain place within that goes through a certain place within a within a unit unit amount of time. a within a unit unit amount of time. a within a unit unit amount of time. >> A sphere >> A sphere >> A sphere 0. 0. 0. >> Yeah. Yeah. Kind of kind. Okay. But >> Yeah. Yeah. Kind of kind. Okay. But >> Yeah. Yeah. Kind of kind. Okay. But anyway basically uh because of this we anyway basically uh because of this we anyway basically uh because of this we can actually write the probability flux can actually write the probability flux can actually write the probability flux in math in this way. So this is like

  16. L5 @ 840s

    in math in this way. So this is like in math in this way. So this is like basically like how much probability gets basically like how much probability gets basically like how much probability gets changed um infinite uh through time and changed um infinite uh through time and changed um infinite uh through time and then this is what we call the divergence then this is what we call the divergence then this is what we call the divergence of velocity times density and what is of velocity times density and what is of velocity times density and what is divergence divergence is what we have divergence divergence is what we have divergence divergence is what we have seen actually uh last class it's the seen actually uh last class it's the seen actually uh last class it's the trace of the jacobian what what it means trace of the jacobian what what it means trace of the jacobian what what it means is basically like how much um how like is basically like how much um how like is basically like how much um how like how much do you change in every how much do you change in every how much do you change in every direction direction direction um of the of the flow o of the of the um of the of the flow o of the of the um of the of the flow o of the of the data space. data space. data space. Okay, any questions? Okay, any questions? Okay, any questions? Everyone understand this? Yay. Yeah, this is like very physics Yay. Yeah, this is like very physics coded. Okay, any any more questions? Direction. Direction. >> Oh, D is dimension. Sorry. Yeah. So, in >> Oh, D is dimension. Sorry. Yeah. So, in >> Oh, D is dimension. Sorry. Yeah. So, in every dimension, how does this thing every dimension, how does this thing every dimension, how does this thing change? basically. Oh, uh I think uh there's a P here that Oh, uh I think uh there's a P here that is got getting uh um missed. is got getting uh um missed. is got getting uh um missed. >> Yeah. >> Yeah. >> Yeah. >> What's the second assumption? There's >> What's the second assumption? There's >> What's the second assumption? There's two assumptions. two assumptions.

  17. L5 @ 912s

    two assumptions. >> Uh two assumptions. One is that >> Uh two assumptions. One is that >> Uh two assumptions. One is that add up to one. The second one is that add up to one. The second one is that add up to one. The second one is that the the thing changes uh continuously. the the thing changes uh continuously. the the thing changes uh continuously. But how is the first one encoded in But how is the first one encoded in But how is the first one encoded in this? this? this? >> Uh it >> Uh it >> Uh it I I guess just the first one is not I I guess just the first one is not I I guess just the first one is not really encoded in this but like you just really encoded in this but like you just really encoded in this but like you just need to have a probability that add up need to have a probability that add up need to have a probability that add up to one. Yeah. >> Flux is a weird word for me. Would it be >> Flux is a weird word for me. Would it be weird to think of it as like viscosity weird to think of it as like viscosity weird to think of it as like viscosity like the amount of flow period of time like the amount of flow period of time like the amount of flow period of time and you're counting for density to like and you're counting for density to like and you're counting for density to like >> I mean the changes in space right and >> I mean the changes in space right and >> I mean the changes in space right and the viscosity should be uniform. Sure. the viscosity should be uniform. Sure. the viscosity should be uniform. Sure. But like viscosity with the thing moving But like viscosity with the thing moving But like viscosity with the thing moving over time. >> Um now flux is actually also a physics >> Um now flux is actually also a physics term and it means like the amount of term and it means like the amount of term and it means like the amount of thing that like flow out thing that like flow out

  18. L5 @ 964s

    thing that like flow out >> as physically inclined. >> as physically inclined. >> as physically inclined. >> It is okay. >> It is okay. >> It is okay. >> I'm just a civil engineer. >> I'm just a civil engineer. >> I'm just a civil engineer. >> I'm also not a physics major but uh you >> I'm also not a physics major but uh you >> I'm also not a physics major but uh you know this is like it's not really >> um I mean just a term. It's not really >> um I mean just a term. It's not really like um Yeah. Yeah. But but but like you like um Yeah. Yeah. But but but like you like um Yeah. Yeah. But but but like you can you can imagine it however way you can you can imagine it however way you can you can imagine it however way you want but that's what it means basically want but that's what it means basically want but that's what it means basically just the amount of things that it just the amount of things that it just the amount of things that it outflows in a unit time basically. outflows in a unit time basically. outflows in a unit time basically. >> Yeah, >> why is it noisy? system external noise >> why is it noisy? system external noise >> uh depending on what kind of things that >> uh depending on what kind of things that >> uh depending on what kind of things that you are. So basically if you're thinking you are. So basically if you're thinking you are. So basically if you're thinking about the path that we just constructed about the path that we just constructed about the path that we just constructed then yes each XT is noisy but this is a then yes each XT is noisy but this is a then yes each XT is noisy but this is a generalized um um like notion it it's generalized um um like notion it it's generalized um um like notion it it's like just any flow this is the this is like just any flow this is the this is like just any flow this is the this is the the definition any other question any other question yes yes yes the divergence connection again

  19. L5 @ 1028s

    the divergence connection again the divergence connection again >> yes I I think I missed a P here actually >> yes I I think I missed a P here actually >> yes I I think I missed a P here actually by the way uh but basically by the way uh but basically by the way uh but basically The divergence means that like how much The divergence means that like how much The divergence means that like how much uh probability do you change in every uh probability do you change in every uh probability do you change in every direction of the data space. Yeah. So direction of the data space. Yeah. So direction of the data space. Yeah. So that's why it's the trace of the that's why it's the trace of the that's why it's the trace of the Jacobian in every direction, every Jacobian in every direction, every Jacobian in every direction, every dimension here. Yeah. I I miss a P here. because like this case, right? because like this case, right? >> We don't we haven't talked about flow >> We don't we haven't talked about flow >> We don't we haven't talked about flow matching yet. >> All right. Any more question? >> All right. Any more question? Cool. All right. Yeah. So, this is Cool. All right. Yeah. So, this is Cool. All right. Yeah. So, this is divergence. How much probability that divergence. How much probability that divergence. How much probability that outflows from a given point per unit outflows from a given point per unit outflows from a given point per unit time in every direction? time in every direction? time in every direction? Okay. Okay. Okay. Cool. All right. All right. So uh like a Cool. All right. All right. So uh like a Cool. All right. All right. So uh like a good thing about um this thing is that good thing about um this thing is that good thing about um this thing is that we can actually transform this PDE into we can actually transform this PDE into we can actually transform this PDE into an OD. All right. So what it means is an OD. All right. So what it means is an OD. All right. So what it means is that like so this is the correct

  20. L5 @ 1101s

    that like so this is the correct that like so this is the correct divergence. Sorry that that that one I divergence. Sorry that that that one I divergence. Sorry that that that one I missed P. Um but basically we can go missed P. Um but basically we can go missed P. Um but basically we can go from here uh and basically you do the from here uh and basically you do the from here uh and basically you do the greatest uh mathematical trick of all greatest uh mathematical trick of all greatest uh mathematical trick of all time. You multiply time. You multiply time. You multiply two things at the same time from the two things at the same time from the two things at the same time from the left hand side to the right hand side left hand side to the right hand side left hand side to the right hand side and uh basically this thing because the and uh basically this thing because the and uh basically this thing because the because the the the the derivative of because the the the the derivative of because the the the the derivative of log is equal to 1 /x. So it becomes the log is equal to 1 /x. So it becomes the log is equal to 1 /x. So it becomes the derivative of the uh the log and then um derivative of the uh the log and then um derivative of the uh the log and then um basically you just apply the same thing basically you just apply the same thing basically you just apply the same thing applying the same trick here and then it applying the same trick here and then it applying the same trick here and then it becomes something like this. Um and then like essentially you basically and then like essentially you basically just do a lot of like calculus and then just do a lot of like calculus and then just do a lot of like calculus and then you're actually able to get uh an OD of you're actually able to get uh an OD of you're actually able to get uh an OD of the log probability of of the log the log probability of of the log

  21. L5 @ 1153s

    the log probability of of the log density. Okay. And this OD is actually density. Okay. And this OD is actually density. Okay. And this OD is actually equal to so the derivative of the total equal to so the derivative of the total equal to so the derivative of the total derivative of log probability is equal derivative of log probability is equal derivative of log probability is equal to the divergence of the velocity to the divergence of the velocity to the divergence of the velocity period. Okay. Any questions here? Okay. Any questions here? >> Yes. >> Yes. >> Yes. >> PFS to be zero. >> PFS to be zero. >> PFS to be zero. >> Yes. Yes. That's a that's a that's a >> Yes. Yes. That's a that's a that's a >> Yes. Yes. That's a that's a that's a great uh question. Yes. PFS is greater great uh question. Yes. PFS is greater great uh question. Yes. PFS is greater than zero everywhere. Yeah. Basically, you just do a lot of Yeah. Basically, you just do a lot of calculus and then you get this. >> No question. >> No question. >> Yes. >> Yes. >> Because it's still depending on the >> Because it's still depending on the >> Because it's still depending on the variable. variable. variable. No, basically from this line, I don't No, basically from this line, I don't No, basically from this line, I don't know if you can see my cursor. This line know if you can see my cursor. This line know if you can see my cursor. This line is literally the formula for the total is literally the formula for the total is literally the formula for the total derivative, right? derivative, right? derivative, right? Yeah. No more question. No more question. Okay, cool. Uh so basically then you can Okay, cool. Uh so basically then you can Okay, cool. Uh so basically then you can calculus then you can calculate your calculus then you can calculate your calculus then you can calculate your likelihood, right? because now you have likelihood, right? because now you have likelihood, right? because now you have a another OD that you can directly solve a another OD that you can directly solve a another OD that you can directly solve by doing numerical integration. So exactly so so

  22. L5 @ 1252s

    So exactly so so how do you train your continuous how do you train your continuous how do you train your continuous normalizing flow from here? What do we normalizing flow from here? What do we normalizing flow from here? What do we think jumping ahead. Jumping ahead. What is jumping ahead. Jumping ahead. What is the Now we have a formula for the the Now we have a formula for the the Now we have a formula for the likelihood log likelihood you can do for the the most naive thing you can do for the the most naive thing to think right is the maximum log to think right is the maximum log to think right is the maximum log likelihood. So basically uh yeah so likelihood. So basically uh yeah so likelihood. So basically uh yeah so because you have a formula to calculate because you have a formula to calculate because you have a formula to calculate the log likelihood at any point of the the log likelihood at any point of the the log likelihood at any point of the trajectory obviously you can also trajectory obviously you can also trajectory obviously you can also calculate the log likelihood of the end calculate the log likelihood of the end calculate the log likelihood of the end point right so you just do arg max of point right so you just do arg max of point right so you just do arg max of the likelihood what could be a problem the likelihood what could be a problem the likelihood what could be a problem here >> yeah yeah >> yeah yeah >> don't have access to p 0 >> don't have access to p 0 >> don't have access to p 0 >> you we can define that we define p 0 to >> you we can define that we define p 0 to >> you we can define that we define p 0 to be a gausian for example that's fine be a gausian for example that's fine be a gausian for example that's fine Very very good. So basically in order to very good. So basically in order to calculate the log likelihood of the data calculate the log likelihood of the data calculate the log likelihood of the data we actually need to do numerical we actually need to do numerical

  23. L5 @ 1324s

    we actually need to do numerical integration right. But this is at integration right. But this is at integration right. But this is at training time. So that just means that training time. So that just means that training time. So that just means that we need to do many many many many steps we need to do many many many many steps we need to do many many many many steps of integration at training time and that of integration at training time and that of integration at training time and that thing is really really expensive. thing is really really expensive. thing is really really expensive. Right. Right. Right. Okay. What what would be a better way to Okay. What what would be a better way to Okay. What what would be a better way to do this? do this? do this? >> Yeah. >> If we take the if we take the gradient >> If we take the if we take the gradient of the left hand side, we get a score of the left hand side, we get a score of the left hand side, we get a score and the left hand side removes, and the left hand side removes, and the left hand side removes, >> right? >> right? >> right? >> Uh >> Uh >> Uh even if you do that right, the even if you do that right, the even if you do that right, the divergence is still pretty difficult to divergence is still pretty difficult to divergence is still pretty difficult to calculate. calculate. calculate. >> Okay. But what else? Yes, >> we can we can use this numerator or we >> we can we can use this numerator or we can just directly sample can just directly sample can just directly sample so that it'll be uh we we'll end up like so that it'll be uh we we'll end up like so that it'll be uh we we'll end up like uh we'll end up with the summation. >> You you mean you change the integration >> You you mean you change the integration into summation? into summation? into summation? >> Yeah. Yeah. Yeah. But that's just >> Yeah. Yeah. Yeah. But that's just >> Yeah. Yeah. Yeah. But that's just numerical integration, right? So it numerical integration, right? So it

  24. L5 @ 1387s

    numerical integration, right? So it still has a lot of steps. Can we do still has a lot of steps. Can we do still has a lot of steps. Can we do something that doesn't require a lot of something that doesn't require a lot of something that doesn't require a lot of steps? Like we just have one four pass steps? Like we just have one four pass steps? Like we just have one four pass during the training time. I think during the training time. I think during the training time. I think someone said it before. Who said velocity before? Who did it? Who said velocity before? Who did it? Oh, you did? Oh, you did? Oh, you did? >> Oh, someone who said velocity. >> Oh, okay. Why? Why did you say velocity? >> Oh, okay. Why? Why did you say velocity? >> Because >> Because >> Because the model's learning velocity. Okay, it actually Okay, it actually basically if you think about it right basically if you think about it right basically if you think about it right both sampling like in order to sample or both sampling like in order to sample or both sampling like in order to sample or calculate the log likelihood you you calculate the log likelihood you you calculate the log likelihood you you pretty much only need to parameterize pretty much only need to parameterize pretty much only need to parameterize the velocity right so if you have a the velocity right so if you have a the velocity right so if you have a model for the velocity then you can model for the velocity then you can model for the velocity then you can calculate then you can do both sampling calculate then you can do both sampling calculate then you can do both sampling and calculate the log log likelihood so and calculate the log log likelihood so and calculate the log log likelihood so why don't we we just need to make sure why don't we we just need to make sure why don't we we just need to make sure that the velocity is correct right so we that the velocity is correct right so we that the velocity is correct right so we just need to make sure that the velocity just need to make sure that the velocity

  25. L5 @ 1451s

    just need to make sure that the velocity actually match with the ground truth actually match with the ground truth actually match with the ground truth velocity. velocity. velocity. So you just do L2 to match with the So you just do L2 to match with the So you just do L2 to match with the ground truth velocity. ground truth velocity. ground truth velocity. Easy, right? Easy, right? Easy, right? What is the problem? Yeah. In order to do either sampling or like In order to do either sampling or like critical estimation, we just need a critical estimation, we just need a critical estimation, we just need a velocity. So if we have a model for velocity. So if we have a model for velocity. So if we have a model for velocity, then we could can do velocity, then we could can do velocity, then we could can do integration at inference time. We don't integration at inference time. We don't integration at inference time. We don't need to do integration at training time need to do integration at training time need to do integration at training time anymore, right? But what what is the anymore, right? But what what is the anymore, right? But what what is the problem here? The ground truth, right? We don't have The ground truth, right? We don't have the ground truth. If we have the ground the ground truth. If we have the ground the ground truth. If we have the ground truth, we would know that a winter storm truth, we would know that a winter storm truth, we would know that a winter storm is coming. Apparently, it's a Saturday. is coming. Apparently, it's a Saturday. is coming. Apparently, it's a Saturday. It's a very cold day. So be prepared. It's a very cold day. So be prepared. It's a very cold day. So be prepared. Okay, winter's coming. Uh anyway, so Okay, winter's coming. Uh anyway, so Okay, winter's coming. Uh anyway, so yeah, but we do we do not know. So yeah, but we do we do not know. So yeah, but we do we do not know. So that's why the meteorologists are that's why the meteorologists are that's why the meteorologists are changing every day be like actually I changing every day be like actually I

  26. L5 @ 1512s

    changing every day be like actually I think that the the storm is moving north think that the the storm is moving north think that the the storm is moving north or maybe not. We don't know, right? So or maybe not. We don't know, right? So or maybe not. We don't know, right? So we do not know the ground velocity. How should we solve this? Well, How should we solve this? Well, basically say we don't care about basically say we don't care about basically say we don't care about everywhere else. We just care about how everywhere else. We just care about how everywhere else. We just care about how the storm is going to move to the storm is going to move to the storm is going to move to Pittsburgh. Let's just say and say now Pittsburgh. Let's just say and say now Pittsburgh. Let's just say and say now you just want to like say you now you're you just want to like say you now you're you just want to like say you now you're Poseidon. Okay, whatever. And then you Poseidon. Okay, whatever. And then you Poseidon. Okay, whatever. And then you you want to transform a bunch of water you want to transform a bunch of water you want to transform a bunch of water or a bunch of like humidity to or a bunch of like humidity to or a bunch of like humidity to Pittsburgh so that Pittsburgh would Pittsburgh so that Pittsburgh would Pittsburgh so that Pittsburgh would snow. snow. snow. What would you do? What is the easiest What would you do? What is the easiest What would you do? What is the easiest way? What is the fastest way for you to way? What is the fastest way for you to way? What is the fastest way for you to do it? How how would you do that? do it? How how would you do that? do it? How how would you do that? >> You just Yeah. >> You just Yeah. >> You just Yeah. >> Just go straight there. >> Just go straight there. >> Just go straight there. >> Just go straight there. Yeah. Yeah. >> Just go straight there. Yeah. Yeah. >> Just go straight there. Yeah. Yeah. Yeah. That's right. That's right. You Yeah. That's right. That's right. You

  27. L5 @ 1558s

    Yeah. That's right. That's right. You just go straight. just go straight. just go straight. Yeah, exactly. Uh so basically this is Yeah, exactly. Uh so basically this is Yeah, exactly. Uh so basically this is what we're going to do as well. Uh so what we're going to do as well. Uh so what we're going to do as well. Uh so let's say we fix a data point. We don't let's say we fix a data point. We don't let's say we fix a data point. We don't care about everywhere else. We just care care about everywhere else. We just care care about everywhere else. We just care about one data point. And then it's about one data point. And then it's about one data point. And then it's usually uh very easy to define a what we usually uh very easy to define a what we usually uh very easy to define a what we call continuous velocity field which uh call continuous velocity field which uh call continuous velocity field which uh sorry a conditional velocity field which sorry a conditional velocity field which sorry a conditional velocity field which means that means that means that basically it just basically it just basically it just velocity is depending on the end point velocity is depending on the end point velocity is depending on the end point right and then we call the trajectory of right and then we call the trajectory of right and then we call the trajectory of the probability distribution that gen the probability distribution that gen the probability distribution that gen get generated along the get generated along the get generated along the way of that uh velocity field the way of that uh velocity field the way of that uh velocity field the conditional probability path. So the conditional probability path. So the conditional probability path. So the conditional probability path is consists conditional probability path is consists conditional probability path is consists of like all the uh all the pts along the of like all the uh all the pts along the of like all the uh all the pts along the way. Okay. And here the conditional way. Okay. And here the conditional way. Okay. And here the conditional probability path will probability path will probability path will just start from some prior which we can

  28. L5 @ 1617s

    just start from some prior which we can just start from some prior which we can choose. So it doesn't matter we can choose. So it doesn't matter we can choose. So it doesn't matter we can choose the prior and then we will always choose the prior and then we will always choose the prior and then we will always end up either directly at Pittsburgh or end up either directly at Pittsburgh or end up either directly at Pittsburgh or somewhere around Pittsburgh. So like we somewhere around Pittsburgh. So like we somewhere around Pittsburgh. So like we just like always end up at x1 or a small just like always end up at x1 or a small just like always end up at x1 or a small small gausian that concentrate around small gausian that concentrate around small gausian that concentrate around x1. Okay. And uh by the way so this x1. Okay. And uh by the way so this x1. Okay. And uh by the way so this thing is called direct delta. So it just thing is called direct delta. So it just thing is called direct delta. So it just means that it's a point mass at at x1. means that it's a point mass at at x1. means that it's a point mass at at x1. Okay. Then uh the marginal probability Okay. Then uh the marginal probability Okay. Then uh the marginal probability path can be written in this way, right? path can be written in this way, right? path can be written in this way, right? Basically just like sum up all the Basically just like sum up all the Basically just like sum up all the possible uh sum up all the possible uh possible uh sum up all the possible uh possible uh sum up all the possible uh uh like data points and then basically uh like data points and then basically uh like data points and then basically just like yeah like and and times the just like yeah like and and times the just like yeah like and and times the the the conditional probability given the the conditional probability given the the conditional probability given the the data point then you can get a the the data point then you can get a

  29. L5 @ 1663s

    the the data point then you can get a marginal marginal marginal uh yeah and also uh just the boundary uh yeah and also uh just the boundary uh yeah and also uh just the boundary condition essentially uh the probability condition essentially uh the probability condition essentially uh the probability uh at time uh at time uh at time should recover the data probability. should recover the data probability. should recover the data probability. This is basically what we're trying to This is basically what we're trying to This is basically what we're trying to construct. construct. construct. Okay. >> All right. Cool. Um so say now we have a >> All right. Cool. Um so say now we have a conditional velocity here. uh then we conditional velocity here. uh then we conditional velocity here. uh then we can also define a marginal velocity. can also define a marginal velocity. can also define a marginal velocity. What does what does this mean? So What does what does this mean? So What does what does this mean? So basically this just means that uh we are basically this just means that uh we are basically this just means that uh we are sort of like we're like sort of like sort of like we're like sort of like sort of like we're like sort of like apply a bay theorem on on your uh apply a bay theorem on on your uh apply a bay theorem on on your uh conditional velocity. So that basically conditional velocity. So that basically conditional velocity. So that basically it's like reweighted it's like reweighted it's like reweighted uh like um taking account into like uh like um taking account into like uh like um taking account into like basically um how likely is your current basically um how likely is your current basically um how likely is your current uh noisy point like your intermediate uh noisy point like your intermediate uh noisy point like your intermediate data point along the the the conditional data point along the the the conditional data point along the the the conditional probability path. How likely is the data probability path. How likely is the data probability path. How likely is the data point that defined this conditional point that defined this conditional point that defined this conditional probability path and also just like how

  30. L5 @ 1743s

    probability path and also just like how probability path and also just like how like and reweed by how likely in general like and reweed by how likely in general like and reweed by how likely in general of this current intermediate point. Does of this current intermediate point. Does of this current intermediate point. Does this make sense? All right. So basically you can sort of All right. So basically you can sort of like intuitively understand this as like like intuitively understand this as like like intuitively understand this as like this is the average conditional velocity this is the average conditional velocity this is the average conditional velocity at this current location in this current at this current location in this current at this current location in this current time weighted by how likely uh is the time weighted by how likely uh is the time weighted by how likely uh is the data point for this current location in data point for this current location in data point for this current location in time. part. Then in the next slide when we're part. Then in the next slide when we're doing it for the velocity, is it doing it for the velocity, is it doing it for the velocity, is it something we're doing from that step or something we're doing from that step or something we're doing from that step or is it just like separate entirely? is it just like separate entirely? is it just like separate entirely? >> Separate. I'll go level resurrected. Uh yeah. So, I'll go level resurrected. Uh yeah. So, so, so this is kind of like what we so, so this is kind of like what we so, so this is kind of like what we defined. It's not like it's two defined. It's not like it's two defined. It's not like it's two different notions different notions different notions and like you can intuitively understand and like you can intuitively understand and like you can intuitively understand it as like sort of like the um like the it as like sort of like the um like the it as like sort of like the um like the average uh conditional velocity and then average uh conditional velocity and then average uh conditional velocity and then this average is like averaging by like

  31. L5 @ 1838s

    this average is like averaging by like this average is like averaging by like how likely is the data point given your how likely is the data point given your how likely is the data point given your current position and the data points are available. and the data points are available. All right, does this make sense? we'll get to it. We'll get to it. Um all we'll get to it. We'll get to it. Um all right. Uh and a very interesting thing right. Uh and a very interesting thing right. Uh and a very interesting thing here is that the marginal velocity we here is that the marginal velocity we here is that the marginal velocity we just saw actually generates the marginal just saw actually generates the marginal just saw actually generates the marginal probability path that we want. Okay. So what it means is what it means by So what it means is what it means by generate is basically just means that generate is basically just means that generate is basically just means that like the the continuity equation is like the the continuity equation is like the the continuity equation is satisfies the the PTE is satisfied and satisfies the the PTE is satisfied and satisfies the the PTE is satisfied and why does it satisfy is because if you why does it satisfy is because if you why does it satisfy is because if you just break everything down uh and then just break everything down uh and then just break everything down uh and then you are able to get basically the the you are able to get basically the the you are able to get basically the the divergence back. Uh so this is just divergence back. Uh so this is just divergence back. Uh so this is just like step by step how you will get a like step by step how you will get a like step by step how you will get a divergence back. Uh so you first um swap divergence back. Uh so you first um swap divergence back. Uh so you first um swap in the the the derivative with some in the the the derivative with some in the the the derivative with some regularization condition. This can can regularization condition. This can can

  32. L5 @ 1904s

    regularization condition. This can can can be done. Uh and then uh basically can be done. Uh and then uh basically can be done. Uh and then uh basically you uh input your um because we know you uh input your um because we know you uh input your um because we know that the marginal sorry the conditional that the marginal sorry the conditional that the marginal sorry the conditional velocity generates the conditional path. velocity generates the conditional path. velocity generates the conditional path. So we can uh input the continuity So we can uh input the continuity So we can uh input the continuity equation there and then it becomes this equation there and then it becomes this equation there and then it becomes this whole thing with divergence. Uh and then whole thing with divergence. Uh and then whole thing with divergence. Uh and then plug in the definition that we just saw plug in the definition that we just saw plug in the definition that we just saw before we can actually get something before we can actually get something before we can actually get something like this and again pulling out the the like this and again pulling out the the like this and again pulling out the the the the derivative here because because the the derivative here because because the the derivative here because because of some regularization condition that we of some regularization condition that we of some regularization condition that we can satisfy it. uh and this thing is can satisfy it. uh and this thing is can satisfy it. uh and this thing is literally just the divergence of the literally just the divergence of the literally just the divergence of the marginal uh velocity times the marginal marginal uh velocity times the marginal marginal uh velocity times the marginal density. density. density. So all in all this is to say that the So all in all this is to say that the So all in all this is to say that the marginal velocity that we defined before marginal velocity that we defined before marginal velocity that we defined before from the conditional velocity that we from the conditional velocity that we from the conditional velocity that we can construct can generate the marginal can construct can generate the marginal

  33. L5 @ 1966s

    can construct can generate the marginal property path which is what we want. property path which is what we want. property path which is what we want. Oh, I saw so many confusion. Oh, I saw so many confusion. Oh, I saw so many confusion. All good. All good. We're getting there. All good. All good. We're getting there. All good. All good. We're getting there. We're getting there. I promise. Ah, We're getting there. I promise. Ah, We're getting there. I promise. Ah, okay. The other thing which is the most okay. The other thing which is the most okay. The other thing which is the most important thing and the most magical important thing and the most magical important thing and the most magical part is that actually part is that actually part is that actually so this is the the the the equation that so this is the the the the equation that so this is the the the the equation that we want, right? So this is the equation we want, right? So this is the equation we want, right? So this is the equation that we that that that we came up with. that we that that that we came up with. that we that that that we came up with. You just need to match the ground truth You just need to match the ground truth You just need to match the ground truth marginal velocity. But we do not have marginal velocity. But we do not have marginal velocity. But we do not have the groundous marginal velocity, right? the groundous marginal velocity, right? the groundous marginal velocity, right? And it actually turns out that if you And it actually turns out that if you And it actually turns out that if you just match the conditional velocity, just match the conditional velocity, just match the conditional velocity, you're actually going to end up with the you're actually going to end up with the you're actually going to end up with the same uh global optimal. You're like same uh global optimal. You're like same uh global optimal. You're like basically just like matching the ground basically just like matching the ground basically just like matching the ground truth marginal velocity is equivalent to truth marginal velocity is equivalent to truth marginal velocity is equivalent to ma matching the conditional velocity.

  34. L5 @ 2030s

    Think about that. Isn't that pretty Think about that. Isn't that pretty cool? Okay. Why? Why? Why is that? Okay. cool? Okay. Why? Why? Why is that? Okay. cool? Okay. Why? Why? Why is that? Okay. So, the reason why is because we can So, the reason why is because we can So, the reason why is because we can break the two L2 down into this thing, break the two L2 down into this thing, break the two L2 down into this thing, right? We we saw this yesterday, not right? We we saw this yesterday, not right? We we saw this yesterday, not yesterday, on Tuesday already, right? yesterday, on Tuesday already, right? yesterday, on Tuesday already, right? And the first term for both equation are And the first term for both equation are And the first term for both equation are constant with respect to our learn constant with respect to our learn constant with respect to our learn parameter. So, we're just going to parameter. So, we're just going to parameter. So, we're just going to cancel it out, right? cancel it out, right? cancel it out, right? Uh and then now we get to two equations Uh and then now we get to two equations Uh and then now we get to two equations each with two components. Right? So each with two components. Right? So each with two components. Right? So we're going to look at the first we're going to look at the first we're going to look at the first components together first. All right. So components together first. All right. So components together first. All right. So the first component if you if you look the first component if you if you look the first component if you if you look at the conditional flow matching uh at the conditional flow matching uh at the conditional flow matching uh equation here uh basically this thing equation here uh basically this thing equation here uh basically this thing you can um decompose it into two you can um decompose it into two you can um decompose it into two integrals and then the these two integrals and then the these two integrals and then the these two integrals integrals integrals um can be um can be um can be uh uh uh Like basically you if you swap uh the

  35. L5 @ 2086s

    Like basically you if you swap uh the Like basically you if you swap uh the dx1 term inside so you swap the order of dx1 term inside so you swap the order of dx1 term inside so you swap the order of the integration you're actually going to the integration you're actually going to the integration you're actually going to get the first term of the first equation get the first term of the first equation get the first term of the first equation back. All right. This part is clear. Cool. All right. All right. Now look at Cool. All right. All right. Now look at the second part. The second part is the second part. The second part is the second part. The second part is pretty The second part is pretty pretty The second part is pretty pretty The second part is pretty similar. Basically what you do is you similar. Basically what you do is you similar. Basically what you do is you just uh starting from the second part of just uh starting from the second part of just uh starting from the second part of the first the first the first uh equation and then you just um plug in uh equation and then you just um plug in uh equation and then you just um plug in what we have as definition and then you what we have as definition and then you what we have as definition and then you do a lot of integrals in between and do a lot of integrals in between and do a lot of integrals in between and then you're actually going to wind up then you're actually going to wind up then you're actually going to wind up at the second part of the second at the second part of the second at the second part of the second equation. equation. equation. Any question? Yes. No. No. Okay. understand everything Yes. No. No. Okay. understand everything everyone. everyone. everyone. >> It's not so it's not even that we're >> It's not so it's not even that we're >> It's not so it's not even that we're sort of having a elbow type thing where sort of having a elbow type thing where sort of having a elbow type thing where we're compressing the bullet. we're compressing the bullet.

  36. L5 @ 2150s

    we're compressing the bullet. >> It's literally the same. It's literally >> It's literally the same. It's literally >> It's literally the same. It's literally the same. It's equivalent. It's not even the same. It's equivalent. It's not even the same. It's equivalent. It's not even elbow. No question. All right. No question. All right. So, this just means that like we don't So, this just means that like we don't So, this just means that like we don't need to match to the thing that we don't need to match to the thing that we don't need to match to the thing that we don't know. we actually can match something know. we actually can match something know. we actually can match something that we do know and do know how to that we do know and do know how to that we do know and do know how to construct which is the conditional construct which is the conditional construct which is the conditional velocity. velocity. Okay. >> Yes, pretty much right. And then you >> Yes, pretty much right. And then you average out and then it is actually the average out and then it is actually the average out and then it is actually the same. Um so uh now let's just look at same. Um so uh now let's just look at same. Um so uh now let's just look at the example that we have right. So now the example that we have right. So now the example that we have right. So now let's suppose our conditional let's suppose our conditional let's suppose our conditional probability path is to transport uh probability path is to transport uh probability path is to transport uh trans transform a gausian a standard trans transform a gausian a standard trans transform a gausian a standard normal gausian straight to a single normal gausian straight to a single normal gausian straight to a single point with constant speed. So this is a point with constant speed. So this is a point with constant speed. So this is a conditional right. So that's why we can conditional right. So that's why we can conditional right. So that's why we can have a single point here. What do we do? Well we first have the the the endpoint

  37. L5 @ 2213s

    Well we first have the the the endpoint distribution and because we're distribution and because we're distribution and because we're transforming it straight to the point transforming it straight to the point transforming it straight to the point with constant speed right? So just with constant speed right? So just with constant speed right? So just linearly very straight. Um so we can linearly very straight. Um so we can linearly very straight. Um so we can literally have a formula for each literally have a formula for each literally have a formula for each intermediate point as a linear intermediate point as a linear intermediate point as a linear combination of uh of the data and the combination of uh of the data and the combination of uh of the data and the noise and this linear combination will noise and this linear combination will noise and this linear combination will add up to one right and uh the add up to one right and uh the add up to one right and uh the probability path can be written in this probability path can be written in this probability path can be written in this way. And most importantly, way. And most importantly, way. And most importantly, the velocity is literally just xt taking the velocity is literally just xt taking the velocity is literally just xt taking derivative of t, which is equal to theta derivative of t, which is equal to theta derivative of t, which is equal to theta minus noise. That's it. And this is what we're That's it. And this is what we're matching to at training time, which we matching to at training time, which we matching to at training time, which we have already seen. >> Yeah. >> First question. So this us >> First question. So this us >> First question. So this us parameterizing it as this linear parameterizing it as this linear parameterizing it as this linear combination that fact is not used really combination that fact is not used really combination that fact is not used really in any of the other. in any of the other. in any of the other. >> Yeah. No no this is what we chose. This >> Yeah. No no this is what we chose. This

  38. L5 @ 2272s

    >> Yeah. No no this is what we chose. This is a design choice. So that so this is a design choice. So that so this is a design choice. So that so this thing is generalizable to any kind of thing is generalizable to any kind of thing is generalizable to any kind of path. Yeah. path. Yeah. path. Yeah. >> How is the ptxt given x1 that >> How is the ptxt given x1 that >> How is the ptxt given x1 that >> how is what >> how is what >> how is what >> the third length? How is the third? >> the third length? How is the third? >> the third length? How is the third? Oh, here. Oh, here. Oh, here. >> Well, because x0 is zero mean gausian. >> Well, because x0 is zero mean gausian. >> Well, because x0 is zero mean gausian. >> Yeah. >> Yeah. All right. Any other? Yeah. All right. Any other? Yeah. All right. Any other? Yeah. >> I don't see any like >> I don't see any like >> I don't see any like gausian assumption. gausian assumption. gausian assumption. >> Oh, there's no gausian assumption. You >> Oh, there's no gausian assumption. You >> Oh, there's no gausian assumption. You don't need gian assumption. don't need gian assumption. don't need gian assumption. >> Just assume that the noise is any >> Just assume that the noise is any >> Just assume that the noise is any distribution. It still works. distribution. It still works. distribution. It still works. >> Yes. Well, um, well, Well, um, well, yeah, pretty much. Yeah, you you don't yeah, pretty much. Yeah, you you don't yeah, pretty much. Yeah, you you don't you don't really need Gausian here. >> Any reason other than simplicity for >> Any reason other than simplicity for choosing? choosing? choosing? Um Um Um here here here not really like but like what what is not really like but like what what is not really like but like what what is the other uh uh what is the other the other uh uh what is the other the other uh uh what is the other distribution that you can think of that distribution that you can think of that distribution that you can think of that we can directly sample from we can directly sample from

  39. L5 @ 2349s

    we can directly sample from >> therefor you >> like previous lectures like gausian >> like previous lectures like gausian assumption is used like this closure on assumption is used like this closure on assumption is used like this closure on the gian used in the derivation but in the gian used in the derivation but in the gian used in the derivation but in flow matching nowhere is flow matching nowhere is flow matching nowhere is >> yeah >> yeah >> yeah it's just here it's just that basically >> like we have previously seen that we can >> like we have previously seen that we can get some kind of stoastic differential get some kind of stoastic differential get some kind of stoastic differential equation for the equation for the equation for the for our scores. So here, don't we just for our scores. So here, don't we just for our scores. So here, don't we just have like a parameterization of that have like a parameterization of that have like a parameterization of that same equation that will get us the same same equation that will get us the same same equation that will get us the same result? result? result? >> Great question. We're going to talk >> Great question. We're going to talk >> Great question. We're going to talk about it later. about it later. about it later. >> Yeah. >> Yeah. >> Assuming that there's an assumption that >> Assuming that there's an assumption that >> Assuming that there's an assumption that I think we assumed earlier to the come I think we assumed earlier to the come I think we assumed earlier to the come earlier that the you said the velocity earlier that the you said the velocity earlier that the you said the velocity is constant. So is constant. So is constant. So >> yeah, the velocity yes is constant >> yeah, the velocity yes is constant >> yeah, the velocity yes is constant speed. Yeah. But is it actually speed. Yeah. But is it actually speed. Yeah. But is it actually constant? Because last lecture I think constant? Because last lecture I think constant? Because last lecture I think we talked about stuff when we u when we we talked about stuff when we u when we

  40. L5 @ 2422s

    we talked about stuff when we u when we actually uh decrease the the velocity at actually uh decrease the the velocity at actually uh decrease the the velocity at which we adding noise. which we adding noise. which we adding noise. >> Here the velocity is different from >> Here the velocity is different from >> Here the velocity is different from previous class. previous class. previous class. So this is two different models. Um and So this is two different models. Um and So this is two different models. Um and the reason why this the reason why this the reason why this constant speed is because literally you constant speed is because literally you constant speed is because literally you have the t there, right? And then like have the t there, right? And then like have the t there, right? And then like literally if you derive the velocity is literally if you derive the velocity is literally if you derive the velocity is literally a constant literally a constant literally a constant right you have question. you have question. >> Oh, okay. Cool. Uh yes. Can >> Oh, okay. Cool. Uh yes. Can >> Oh, okay. Cool. Uh yes. Can >> we use this to do likelihood estimation >> we use this to do likelihood estimation >> we use this to do likelihood estimation as well or? as well or? as well or? >> Yes. We're going to talk about it later. >> Yes. We're going to talk about it later. >> Yes. We're going to talk about it later. Yeah. Yeah. Yeah. Anyone else? Anyone else? Anyone else? Cool. Yeah. And this is the this is the Cool. Yeah. And this is the this is the Cool. Yeah. And this is the this is the uh the loss function. Literally uh the loss function. Literally uh the loss function. Literally literally just you learn the model literally just you learn the model literally just you learn the model should predict data minus noise. Yeah. should predict data minus noise. Yeah. should predict data minus noise. Yeah. >> The velocity stays constant. >> The velocity stays constant. >> The velocity stays constant. >> Um the the velocity is constant only on >> Um the the velocity is constant only on >> Um the the velocity is constant only on this particular path. Only on this

  41. L5 @ 2486s

    this particular path. Only on this this particular path. Only on this particular x0 to x1 path. But it's not particular x0 to x1 path. But it's not particular x0 to x1 path. But it's not in general. in general it's not actually in general. in general it's not actually in general. in general it's not actually I think this may answer your question I think this may answer your question I think this may answer your question yeah so basically in like so the left yeah so basically in like so the left yeah so basically in like so the left hand side is like the the path that you hand side is like the the path that you hand side is like the the path that you construct at training time so everything construct at training time so everything construct at training time so everything is straight right everything is straight is straight right everything is straight is straight right everything is straight but the learn the function will you have but the learn the function will you have but the learn the function will you have this averaging effect right so you learn this averaging effect right so you learn this averaging effect right so you learn the the the the entire field so you the the the the entire field so you the the the the entire field so you don't don't don't have this straight line anymore actually have this straight line anymore actually have this straight line anymore actually but you have this like curve curve the but you have this like curve curve the but you have this like curve curve the slightly curved average field still normalizing flow right yeah still normalizing flow right yeah because because you have the one to one because because you have the one to one because because you have the one to one you have the the the invertible you have the the the invertible you have the the the invertible invertability invertability invertability okay any question okay any question okay any question >> yes >> yes >> yes >> why is it slightly collapsing in the >> why is it slightly collapsing in the >> why is it slightly collapsing in the middle middle middle >> because if you look at the if you look

  42. L5 @ 2544s

    >> because if you look at the if you look >> because if you look at the if you look at the this the the left hand side right at the this the the left hand side right at the this the the left hand side right so like you can see how the field is not so like you can see how the field is not so like you can see how the field is not like the field has a collapsing. So like like the field has a collapsing. So like like the field has a collapsing. So like the the the margin so we're later on the the the margin so we're later on the the the margin so we're later on when when when sampling we're using it as if it's the sampling we're using it as if it's the sampling we're using it as if it's the marginal and the marginal has this like marginal and the marginal has this like marginal and the marginal has this like slight slight concentration in the in slight slight concentration in the in slight slight concentration in the in the middle. Yep. the middle. Yep. the middle. Yep. >> So in this case if I were to pass the >> So in this case if I were to pass the >> So in this case if I were to pass the same gshian input twice to the model I same gshian input twice to the model I same gshian input twice to the model I get the same image out. get the same image out. get the same image out. >> Yes. It's deterministic completely which >> Yes. It's deterministic completely which >> Yes. It's deterministic completely which is different from diffusion right? is different from diffusion right? is different from diffusion right? >> Yes. >> Yes. Okay, any more question? Cool. Actually, this thing is like super Cool. Actually, this thing is like super cool because the same algorithm was cool because the same algorithm was cool because the same algorithm was developed by three different group of developed by three different group of developed by three different group of people at literally the same time like people at literally the same time like people at literally the same time like literally the same conference and they literally the same conference and they

  43. L5 @ 2596s

    literally the same conference and they all got into the same conference. all got into the same conference. all got into the same conference. So yeah, so the first group is a group So yeah, so the first group is a group So yeah, so the first group is a group in Metafair and in Metafair and in Metafair and this is the group that I interned at this is the group that I interned at this is the group that I interned at actually. Um and then the second group I actually. Um and then the second group I actually. Um and then the second group I believe is a group from UT Austin and believe is a group from UT Austin and believe is a group from UT Austin and then the third group is a group from uh then the third group is a group from uh then the third group is a group from uh NYU and they all derive the same NYU and they all derive the same NYU and they all derive the same algorithm from different uh theoretical algorithm from different uh theoretical algorithm from different uh theoretical perspective and all submit to the same perspective and all submit to the same perspective and all submit to the same conference and all got in. conference and all got in. conference and all got in. So this is like sort of like a beautiful So this is like sort of like a beautiful So this is like sort of like a beautiful moment of science where you just kind of moment of science where you just kind of moment of science where you just kind of have like a collapsing ideas but like have like a collapsing ideas but like have like a collapsing ideas but like they're from different points you know they're from different points you know they're from different points you know it's just like our you know flow it's just like our you know flow it's just like our you know flow matching everyone goes to the same point matching everyone goes to the same point matching everyone goes to the same point anyway. Yeah, pretty pretty magical. Um, anyway. Yeah, pretty pretty magical. Um, anyway. Yeah, pretty pretty magical. Um, but you may have this question, right? I but you may have this question, right? I

  44. L5 @ 2640s

    but you may have this question, right? I feel like some people are having this feel like some people are having this feel like some people are having this question but they didn't ask. So like question but they didn't ask. So like question but they didn't ask. So like the marginal flow has some dips to it, the marginal flow has some dips to it, the marginal flow has some dips to it, right? So it doesn't it's not completely right? So it doesn't it's not completely right? So it doesn't it's not completely straight. It's not the straightest. So straight. It's not the straightest. So straight. It's not the straightest. So is there a way that we can like let's is there a way that we can like let's is there a way that we can like let's say now we have trained a flow matching say now we have trained a flow matching say now we have trained a flow matching model now. So now we have this this this model now. So now we have this this this model now. So now we have this this this um like this invertability this coupling um like this invertability this coupling um like this invertability this coupling already. Is there a way to already. Is there a way to already. Is there a way to make this even straighter? Is it make this even straighter? Is it make this even straighter? Is it possible directly move from your P 0 to directly move from your P 0 to >> Yeah, pretty much. Pretty much. Um, so >> Yeah, pretty much. Pretty much. Um, so >> Yeah, pretty much. Pretty much. Um, so this thing is what we call reflow. So this thing is what we call reflow. So this thing is what we call reflow. So this is a technique that developed by by this is a technique that developed by by this is a technique that developed by by the second group of people who proposed the second group of people who proposed the second group of people who proposed the same thing. Uh so basically what you the same thing. Uh so basically what you the same thing. Uh so basically what you do is now let's say you you after do is now let's say you you after

  45. L5 @ 2698s

    do is now let's say you you after training with the linear interpolation training with the linear interpolation training with the linear interpolation you already got this coupling. So the you already got this coupling. So the you already got this coupling. So the blue coupling and the the green coupling blue coupling and the the green coupling blue coupling and the the green coupling and then what you can do is you can just and then what you can do is you can just and then what you can do is you can just like use this coupling as your new like use this coupling as your new like use this coupling as your new ground truth and then instead of like ground truth and then instead of like ground truth and then instead of like randomly sampling from a gausian you randomly sampling from a gausian you randomly sampling from a gausian you sample from the coupling and then you sample from the coupling and then you sample from the coupling and then you can get the straight points can get the straight points can get the straight points and then it become more straight. Let me explain again. Let me explain again. >> Oh, okay. One more time. One more time. >> Oh, okay. One more time. One more time. >> Oh, okay. One more time. One more time. So, basically this is like your first So, basically this is like your first So, basically this is like your first time uh flow matching. So, basically time uh flow matching. So, basically time uh flow matching. So, basically after computing all the the the the in after computing all the the the the in after computing all the the the the in the uh linear interpolations, you get the uh linear interpolations, you get the uh linear interpolations, you get this like slightly dipped um marginals, this like slightly dipped um marginals, this like slightly dipped um marginals, right? You get this slightly dipped right? You get this slightly dipped right? You get this slightly dipped version, but now you have a a coupling version, but now you have a a coupling version, but now you have a a coupling because the thing is deterministic, because the thing is deterministic, because the thing is deterministic, right? So you can map from one point in

  46. L5 @ 2750s

    right? So you can map from one point in right? So you can map from one point in P 0 to another point in P1 and then you P 0 to another point in P1 and then you P 0 to another point in P1 and then you just use that coupling as your new just use that coupling as your new just use that coupling as your new ground truth. So instead of like ground truth. So instead of like ground truth. So instead of like randomly sampling your your your randomly sampling your your your randomly sampling your your your coupling like what we do what we did in coupling like what we do what we did in coupling like what we do what we did in the original flow matching you just use the original flow matching you just use the original flow matching you just use that coupling as your ground truth and that coupling as your ground truth and that coupling as your ground truth and train for matching again and then it's train for matching again and then it's train for matching again and then it's going to get straightened up. >> Yeah. Go ahead. Because when you sample >> Yeah. Go ahead. Because when you sample um the more the more linear your clothes um the more the more linear your clothes um the more the more linear your clothes are, the less test. are, the less test. are, the less test. >> Yes, exactly. So, think about it, right? >> Yes, exactly. So, think about it, right? >> Yes, exactly. So, think about it, right? Um like if you have say if you have this Um like if you have say if you have this Um like if you have say if you have this dip, right? You have you have this dip dip, right? You have you have this dip dip, right? You have you have this dip then like like the safer bet for you to then like like the safer bet for you to then like like the safer bet for you to sample is to take two steps, right? sample is to take two steps, right? sample is to take two steps, right? because you don't know like basically if because you don't know like basically if

  47. L5 @ 2794s

    because you don't know like basically if you if you only do one step there can be you if you only do one step there can be you if you only do one step there can be like a lot of um uh uh like like like a lot of um uh uh like like like a lot of um uh uh like like discretization error that is happening discretization error that is happening discretization error that is happening because of this dip. But if you have a because of this dip. But if you have a because of this dip. But if you have a very straight line then you can just do very straight line then you can just do very straight line then you can just do one step and it's straight. So one step one step and it's straight. So one step one step and it's straight. So one step is fine right? So straighter you are uh is fine right? So straighter you are uh is fine right? So straighter you are uh the better you are at this like few step the better you are at this like few step the better you are at this like few step sampling. How do you decide what to cover? How do you decide what to cover? >> By your previous model, right? By the >> By your previous model, right? By the >> By your previous model, right? By the model that you learned. So, model that you learned. So, model that you learned. So, so this blue this this this blue and so this blue this this this blue and so this blue this this this blue and green are the coupling induced by your green are the coupling induced by your green are the coupling induced by your learn the flow matching model and then learn the flow matching model and then learn the flow matching model and then you just use this this coupling and then you just use this this coupling and then you just use this this coupling and then do flow matching again on the new model. do flow matching again on the new model. do flow matching again on the new model. Yeah. >> It won't collide because it's >> It won't collide because it's It's a flow. It never cross. Yeah.

  48. L5 @ 2857s

    >> The third image is the linear >> The third image is the linear interpolation of the new coupling. interpolation of the new coupling. interpolation of the new coupling. >> That what is that doing? >> That what is that doing? >> That what is that doing? >> Oh, this thing is just like show you the >> Oh, this thing is just like show you the >> Oh, this thing is just like show you the marginal kind of. marginal kind of. marginal kind of. >> Yeah. >> Yeah. >> The green is going up. But so on the >> The green is going up. But so on the >> The green is going up. But so on the >> Yeah. Yeah. Yeah. No. Remember how this >> Yeah. Yeah. Yeah. No. Remember how this >> Yeah. Yeah. Yeah. No. Remember how this you have a you have a concentration in you have a you have a concentration in you have a you have a concentration in the middle. you have a dip, right? So, the middle. you have a dip, right? So, the middle. you have a dip, right? So, it's sort of the same thing here. it's sort of the same thing here. it's sort of the same thing here. >> Is it when you're doing the sampling? >> Is it when you're doing the sampling? >> Is it when you're doing the sampling? >> No, no. Linear interpolation is just >> No, no. Linear interpolation is just >> No, no. Linear interpolation is just coupling. So, like you you you choose coupling. So, like you you you choose coupling. So, like you you you choose you you choose a you you choose a sample you you choose a you you choose a sample you you choose a you you choose a sample here, you choose a sample here and then here, you choose a sample here and then here, you choose a sample here and then you straight line and then you connect you straight line and then you connect you straight line and then you connect them with a straight line. them with a straight line. them with a straight line. >> Sample from both side. We mostly >> Sample from both side. We mostly >> Sample from both side. We mostly connected this side to that side and

  49. L5 @ 2897s

    connected this side to that side and connected this side to that side and that side to this side. There's no that side to this side. There's no that side to this side. There's no there's no this anymore because because there's no this anymore because because there's no this anymore because because it won't be basically the the the it won't be basically the the the it won't be basically the the the coupling tells you that you will not go coupling tells you that you will not go coupling tells you that you will not go to the other side anymore because to the other side anymore because to the other side anymore because otherwise it will cross and then your otherwise it will cross and then your otherwise it will cross and then your stream will never cross. stream will never cross. stream will never cross. >> Yeah. >> Yeah. >> You have a data set that you know has >> You have a data set that you know has >> You have a data set that you know has multiple classes. Could you use this to multiple classes. Could you use this to multiple classes. Could you use this to disentangle them? disentangle them? disentangle them? >> Yes. And also actually um there's >> Yes. And also actually um there's >> Yes. And also actually um there's another paper I didn't put it here that another paper I didn't put it here that another paper I didn't put it here that if you already have a coupling in your if you already have a coupling in your if you already have a coupling in your data set you can just directly use it to data set you can just directly use it to data set you can just directly use it to train flow matching. You don't have to train flow matching. You don't have to train flow matching. You don't have to do random sampling. Yeah, I think do random sampling. Yeah, I think do random sampling. Yeah, I think someone has Yeah. someone has Yeah. someone has Yeah. >> If your flow matching model is not >> If your flow matching model is not >> If your flow matching model is not perfectly trained into this process, perfectly trained into this process,

  50. L5 @ 2936s

    perfectly trained into this process, >> then you do it again. Yeah. Yeah. So, >> then you do it again. Yeah. Yeah. So, >> then you do it again. Yeah. Yeah. So, this is just you you get it this is just you you get it this is just you you get it straightened. It's not guaranteed that straightened. It's not guaranteed that straightened. It's not guaranteed that you're going to get straight uh like you're going to get straight uh like you're going to get straight uh like path on on on the on one go. Just like path on on on the on one go. Just like path on on on the on one go. Just like you're just going to get straighter and you're just going to get straighter and you're just going to get straighter and straighter kind of. Yeah, it it's straighter kind of. Yeah, it it's straighter kind of. Yeah, it it's definitely possible. Yeah. Oh. No. Okay. No, nobody. No. Okay. No, nobody. Cool. All right. So, just to summarize Cool. All right. So, just to summarize Cool. All right. So, just to summarize everything. So, just what's the everything. So, just what's the everything. So, just what's the difference between diffusion and flow difference between diffusion and flow difference between diffusion and flow matching in this context, right? So, matching in this context, right? So, matching in this context, right? So, diffusion is sort of like you are diffusion is sort of like you are diffusion is sort of like you are wandering in the wood and with a compass wandering in the wood and with a compass wandering in the wood and with a compass uh like in your hand. So, you know like uh like in your hand. So, you know like uh like in your hand. So, you know like you kind of know which direction you're you kind of know which direction you're you kind of know which direction you're going, but you're still like you're kind going, but you're still like you're kind going, but you're still like you're kind of just like exploring a little bit as of just like exploring a little bit as of just like exploring a little bit as well. you're just like kind of random well. you're just like kind of random

  51. L5 @ 2984s

    well. you're just like kind of random random walking a little bit. Whereas for random walking a little bit. Whereas for random walking a little bit. Whereas for flow matching, you don't really do flow matching, you don't really do flow matching, you don't really do random walking anymore. You just like random walking anymore. You just like random walking anymore. You just like sit on the boat and then go go down the sit on the boat and then go go down the sit on the boat and then go go down the stream uh on the river. So this is kind stream uh on the river. So this is kind stream uh on the river. So this is kind of what like to conceptually what is of what like to conceptually what is of what like to conceptually what is happening. Yeah. happening. Yeah. happening. Yeah. >> Like is this used now? Like if it's so >> Like is this used now? Like if it's so >> Like is this used now? Like if it's so good this good this good this >> great question. >> great question. >> great question. >> Do you know flux? Do you know the the >> Do you know flux? Do you know the the >> Do you know flux? Do you know the the black for that's full matching? Yeah. black for that's full matching? Yeah. black for that's full matching? Yeah. This is like the state of the art This is like the state of the art This is like the state of the art basically. It's actually crazy good and basically. It's actually crazy good and basically. It's actually crazy good and it's so simple, right? Okay. So, so far it's so simple, right? Okay. So, so far it's so simple, right? Okay. So, so far we have seen so many gen models and we we have seen so many gen models and we we have seen so many gen models and we know that the diffusion model and know that the diffusion model and know that the diffusion model and scorebased model are same thing and scorebased model are same thing and scorebased model are same thing and today we learned flow matching. Now the today we learned flow matching. Now the today we learned flow matching. Now the question is is flow matching also the

  52. L5 @ 3027s

    question is is flow matching also the question is is flow matching also the same thing because if you think about it same thing because if you think about it same thing because if you think about it you're also like adding a lot of noise you're also like adding a lot of noise you're also like adding a lot of noise like and then d noiseis is kind of like and then d noiseis is kind of like and then d noiseis is kind of dinoising right you're just like a lot dinoising right you're just like a lot dinoising right you're just like a lot of noisy samples in the middle is are of noisy samples in the middle is are of noisy samples in the middle is are they the same thing? they the same thing? they the same thing? What do we think kind of actually? So basically what you kind of actually? So basically what you can do so this is what we had before can do so this is what we had before can do so this is what we had before right uh we have this like very straight right uh we have this like very straight right uh we have this like very straight line but what if you define something line but what if you define something line but what if you define something because the flow matching theory can be because the flow matching theory can be because the flow matching theory can be applied to any path right like what applied to any path right like what applied to any path right like what Dwanch was asking so we can also define Dwanch was asking so we can also define Dwanch was asking so we can also define a path from what we just learned uh on a path from what we just learned uh on a path from what we just learned uh on Tuesday right so you can define a a v Tuesday right so you can define a a v Tuesday right so you can define a a v path a v uh like diffusion path and you path a v uh like diffusion path and you path a v uh like diffusion path and you can write everything as if it is flow

  53. L5 @ 3072s

    can write everything as if it is flow can write everything as if it is flow matching as well and then you can matching as well and then you can matching as well and then you can actually just recover for V SD with actually just recover for V SD with actually just recover for V SD with obviously obviously obviously with with the noise adding to it. with with the noise adding to it. with with the noise adding to it. And what about if we do what can can And what about if we do what can can And what about if we do what can can what about the other way around, right? what about the other way around, right? what about the other way around, right? Can we do score SDE Can we do score SDE Can we do score SDE transform into an OD a flow OD? Can we transform into an OD a flow OD? Can we transform into an OD a flow OD? Can we do that? What do we think by looking at do that? What do we think by looking at do that? What do we think by looking at this this this formula? Okay, just get rid of the the G part, Okay, just get rid of the the G part, right? Is that what you said? All right. right? Is that what you said? All right. right? Is that what you said? All right. So, basically at this part look like a So, basically at this part look like a So, basically at this part look like a velocity, right? velocity, right? velocity, right? But the problem is like the G part, the But the problem is like the G part, the But the problem is like the G part, the the brown in motion part, this part the brown in motion part, this part the brown in motion part, this part actually induce some probability because actually induce some probability because actually induce some probability because it's a gausian, right? So, we still it's a gausian, right? So, we still it's a gausian, right? So, we still induce some probability there. So, we induce some probability there. So, we induce some probability there. So, we still need to take care of that part. still need to take care of that part.

  54. L5 @ 3123s

    still need to take care of that part. But it turns out that we actually do But it turns out that we actually do But it turns out that we actually do know how to take care of that part. And know how to take care of that part. And know how to take care of that part. And this thing is called focal plank PDE. this thing is called focal plank PDE. this thing is called focal plank PDE. All right? uh the the name is really All right? uh the the name is really All right? uh the the name is really fancy but basically what it means is fancy but basically what it means is fancy but basically what it means is that like if you if you have some that like if you if you have some that like if you if you have some probability in it the focal point PTE um probability in it the focal point PTE um probability in it the focal point PTE um it's going to tell you it's going to tell you it's going to tell you how your probability changes through how your probability changes through how your probability changes through this through this PTE that you can this through this PTE that you can this through this PTE that you can derive uh and if you're interested in derive uh and if you're interested in derive uh and if you're interested in the derivation of this PD for the the derivation of this PD for the the derivation of this PD for the interest of time we're not going to go interest of time we're not going to go interest of time we're not going to go over it but it's in the in the the over it but it's in the in the the over it but it's in the in the the principle of diffusion model book so you principle of diffusion model book so you principle of diffusion model book so you can take a look at it um but basically can take a look at it um but basically can take a look at it um but basically Okay. And now we have probability flow Okay. And now we have probability flow Okay. And now we have probability flow OD, right? So this is the just um say we

  55. L5 @ 3164s

    OD, right? So this is the just um say we OD, right? So this is the just um say we have this particular OD and it turns out have this particular OD and it turns out have this particular OD and it turns out that it's contin continuity equation that it's contin continuity equation that it's contin continuity equation which is like the the PD of the change which is like the the PD of the change which is like the the PD of the change of um probability which is what we of um probability which is what we of um probability which is what we learned today by using the divergence is learned today by using the divergence is learned today by using the divergence is actually so if you just like plug actually so if you just like plug actually so if you just like plug everything in and then you do a lot of everything in and then you do a lot of everything in and then you do a lot of algebra and and calculus uh you are algebra and and calculus uh you are algebra and and calculus uh you are actually going to get the same thing actually going to get the same thing actually going to get the same thing literally the same thing. So this is literally the same thing. So this is literally the same thing. So this is just saying that the rate of like the just saying that the rate of like the just saying that the rate of like the rate of change in probability rate of change in probability rate of change in probability from this score SD or from this 4 SD from this score SD or from this 4 SD from this score SD or from this 4 SD they share the same uh which is the same they share the same uh which is the same they share the same uh which is the same as this probability flow OD as this probability flow OD as this probability flow OD this and the same PD for the rate of this and the same PD for the rate of this and the same PD for the rate of change for the probability means that change for the probability means that

  56. L5 @ 3210s

    change for the probability means that the marginalss are all the same. So this the marginalss are all the same. So this the marginalss are all the same. So this just means that now if you have your just means that now if you have your just means that now if you have your score function instead of sampling from score function instead of sampling from score function instead of sampling from an SD you can do OD samples now. Yay. Yay. Yay. Yay. Well, no excitement. It's okay. Um what Well, no excitement. It's okay. Um what Well, no excitement. It's okay. Um what else can you else can you else can you with OD? Right. I think someone someone with OD? Right. I think someone someone with OD? Right. I think someone someone said before that like we can do density said before that like we can do density said before that like we can do density estimation with this, right? So estimation with this, right? So estimation with this, right? So literally this is true, right? So you literally this is true, right? So you literally this is true, right? So you can literally just um um swap in the the can literally just um um swap in the the can literally just um um swap in the the velocity here and then you just like go velocity here and then you just like go velocity here and then you just like go back in time through your OD and then back in time through your OD and then back in time through your OD and then you're able to calculate the log density you're able to calculate the log density you're able to calculate the log density of your data point. of your data point. of your data point. Cool. And Cool. And Cool. And what about density estimation for what about density estimation for what about density estimation for diffusion, right? diffusion, right? diffusion, right? Oh, density estimation means that like Oh, density estimation means that like Oh, density estimation means that like just like to calculate uh the likelihood just like to calculate uh the likelihood just like to calculate uh the likelihood of the the the the day. of the the the the day. of the the the the day. >> That's a great question. Um so for

  57. L5 @ 3276s

    >> That's a great question. Um so for >> That's a great question. Um so for example, I don't know if you guys example, I don't know if you guys example, I don't know if you guys familiar with the RL. familiar with the RL. familiar with the RL. >> All right, cool. Anyway, >> All right, cool. Anyway, >> All right, cool. Anyway, and so for density estimation for and so for density estimation for and so for density estimation for diffusion, uh the easiest thing is that diffusion, uh the easiest thing is that diffusion, uh the easiest thing is that because the loss function that we because the loss function that we because the loss function that we derived from two lectures ago is a derived from two lectures ago is a derived from two lectures ago is a elbow, right? So we can just directly elbow, right? So we can just directly elbow, right? So we can just directly use this elbow as an estimate of the use this elbow as an estimate of the use this elbow as an estimate of the likelihood. That's fine. But this is not likelihood. That's fine. But this is not likelihood. That's fine. But this is not accurate because it's a elbow. It's a accurate because it's a elbow. It's a accurate because it's a elbow. It's a lower bound. It's it's not really the lower bound. It's it's not really the lower bound. It's it's not really the the real log log likelihood, right? So the real log log likelihood, right? So the real log log likelihood, right? So is there another way to do it? I feel like someone someone here knows I feel like someone someone here knows already. Okay. What do we think? What do we Okay. What do we think? What do we think? We know we know how to calculate the We know we know how to calculate the exact log likelihood from OD. What do we exact log likelihood from OD. What do we exact log likelihood from OD. What do we do if we have a score SDE? What do we do if we have a score SDE? What do we do if we have a score SDE? What do we do? do? do? Use the use the corresponding PF OD of

  58. L5 @ 3340s

    Use the use the corresponding PF OD of Use the use the corresponding PF OD of the SDE. Right? So you just plug it in the SDE. Right? So you just plug it in the SDE. Right? So you just plug it in and then it's the same and then it's the same and then it's the same as for matching. Yes, Yes, >> it's like that. >> it's like that. >> it's like that. >> Um, >> Um, >> Um, one one one one one >> Okay. >> Okay. >> Okay. >> Yeah, >> Yeah, >> Yeah, you can sample with an OD. What is like you can sample with an OD. What is like you can sample with an OD. What is like sampling? sampling? sampling? >> Oh, yes. Yes. So you can sample with >> Oh, yes. Yes. So you can sample with >> Oh, yes. Yes. So you can sample with this OD meaning that like you can you this OD meaning that like you can you this OD meaning that like you can you can literally do uh let's see you can can literally do uh let's see you can can literally do uh let's see you can you can literally just do numerical you can literally just do numerical you can literally just do numerical integration from x0 and then you do x0 integration from x0 and then you do x0 integration from x0 and then you do x0 plus um integrate from zero to one plus um integrate from zero to one plus um integrate from zero to one velocity and this velocity is just this velocity and this velocity is just this velocity and this velocity is just this odz >> uh what do you mean the sampling is only >> uh what do you mean the sampling is only x0 oh you mean like to sample from a x0 oh you mean like to sample from a x0 oh you mean like to sample from a distribution yes that's right Yeah. distribution yes that's right Yeah. distribution yes that's right Yeah. Cool. Cool. Oh, yeah. Yeah. Yeah. Oh, yeah. Yeah. Yeah. >> Yeah. >> Yeah. >> A lot of uh uh lot of PD getting >> A lot of uh uh lot of PD getting

  59. L5 @ 3427s

    >> A lot of uh uh lot of PD getting converted to OD. Is it are we just doing converted to OD. Is it are we just doing converted to OD. Is it are we just doing it because it's simpler or do we you it because it's simpler or do we you it because it's simpler or do we you want a better velocity? want a better velocity? want a better velocity? >> Uh no you mean here here it's because >> Uh no you mean here here it's because >> Uh no you mean here here it's because it's exact. So in in uh in comparison to it's exact. So in in uh in comparison to it's exact. So in in uh in comparison to the elbow right if you compare to the the elbow right if you compare to the the elbow right if you compare to the elbow the elbow is a lower bound of your elbow the elbow is a lower bound of your elbow the elbow is a lower bound of your um of your lo likelihood which is not um of your lo likelihood which is not um of your lo likelihood which is not exact but here we can actually get an exact but here we can actually get an exact but here we can actually get an exact log likelihood. exact log likelihood. exact log likelihood. Yeah. Yeah. >> Intrinsically Asian suffer from >> Intrinsically Asian suffer from >> Intrinsically Asian suffer from diversity, right? Because diversity, right? Because diversity, right? Because >> it suffers from uh discretization error. >> it suffers from uh discretization error. >> it suffers from uh discretization error. Is that what you mean? Like basically if Is that what you mean? Like basically if Is that what you mean? Like basically if you take the step and then you take the step and then you take the step and then because uh because uh because uh >> ah uh well well here you're calculating >> ah uh well well here you're calculating >> ah uh well well here you're calculating the the the log likelihood. So it's not the the the log likelihood. So it's not the the the log likelihood. So it's not diversity but but yes so like diversity but but yes so like

  60. L5 @ 3473s

    diversity but but yes so like essentially um if you're sampling from a essentially um if you're sampling from a essentially um if you're sampling from a SDE then it's going to be yeah because SDE then it's going to be yeah because SDE then it's going to be yeah because you have exploration right so it's more you have exploration right so it's more you have exploration right so it's more diverse than sampling from OD which is diverse than sampling from OD which is diverse than sampling from OD which is completely deterministic completely deterministic completely deterministic >> yeah that's right Cool. Cool. Not cool. Okay. Not cool. Okay. Not cool. Okay. >> So, if if I want to calculate the >> So, if if I want to calculate the >> So, if if I want to calculate the likelihood of X1, likelihood of X1, likelihood of X1, >> I first need to find the corresponding >> I first need to find the corresponding >> I first need to find the corresponding X0 for me to start integrating. And X0 for me to start integrating. And X0 for me to start integrating. And >> hey, you have everyone you should know >> hey, you have everyone you should know >> hey, you have everyone you should know this. But anyway, continue. Yeah. Yeah. this. But anyway, continue. Yeah. Yeah. this. But anyway, continue. Yeah. Yeah. Yeah. Yeah. Oh yeah. So yeah. So essentially what Oh yeah. So yeah. So essentially what Oh yeah. So yeah. So essentially what you do is you you remember how you you you do is you you remember how you you you do is you you remember how you you you reverse the chain, right? >> Yeah, that is right. >> Yeah, that is right. >> That is right. And which is why this x0 >> That is right. And which is why this x0 >> That is right. And which is why this x0 k has a hat on it. So this is a reverse k has a hat on it. So this is a reverse k has a hat on it. So this is a reverse the x0. So you first reverse your the x0. So you first reverse your the x0. So you first reverse your sampling chain through your by by by

  61. L5 @ 3546s

    sampling chain through your by by by sampling chain through your by by by backward integrating your velocity and backward integrating your velocity and backward integrating your velocity and then during the backward integration you then during the backward integration you then during the backward integration you can alo calculate the divergence at the can alo calculate the divergence at the can alo calculate the divergence at the same time. Yeah. So you you need to find the Yeah. So you you need to find the corresponding chain which is corresponding chain which is corresponding chain which is deterministic. But uh But uh >> p of x0 should be constant. >> p of x0 should be constant. >> p of x0 should be constant. >> Oh, you mean p of x? Yeah, p 0 is >> Oh, you mean p of x? Yeah, p 0 is >> Oh, you mean p of x? Yeah, p 0 is constant. Yeah, that's right. Yeah. constant. Yeah, that's right. Yeah. constant. Yeah, that's right. Yeah. Yeah. If if if it's a uniform. Yeah. If if if it's a uniform. Yeah. If if if it's a uniform. All right. All right. All right. More question. More question. More question. All good. All right. All good. All right. All good. All right. Okay. So Okay. So Okay. So to summarize everything and we have seen to summarize everything and we have seen to summarize everything and we have seen three models so far. Uh diffusion model, three models so far. Uh diffusion model, three models so far. Uh diffusion model, squarebased models, flow matching all squarebased models, flow matching all squarebased models, flow matching all model. They are more or less all the model. They are more or less all the model. They are more or less all the same thing. Same same different. same thing. Same same different. same thing. Same same different. I feel like saying this I'm going to I feel like saying this I'm going to I feel like saying this I'm going to offend a lot of people but but yeah know offend a lot of people but but yeah know offend a lot of people but but yeah know you know just like you know you just add you know just like you know you just add

  62. L5 @ 3612s

    you know just like you know you just add something divide something you it's the something divide something you it's the something divide something you it's the same thing. Okay. same thing. Okay. same thing. Okay. >> Yeah everything is the same thing. Uh >> Yeah everything is the same thing. Uh >> Yeah everything is the same thing. Uh but yeah um so yeah congrat oh okay we but yeah um so yeah congrat oh okay we but yeah um so yeah congrat oh okay we haven't congrat Okay. haven't congrat Okay. haven't congrat Okay. >> Uh so if we can still compute the >> Uh so if we can still compute the >> Uh so if we can still compute the likelihood using flow matching why does likelihood using flow matching why does likelihood using flow matching why does it come under likelihood free like it come under likelihood free like it come under likelihood free like what's the classification? Oh here what's the classification? Oh here what's the classification? Oh here likelihood of free means that the likelihood of free means that the likelihood of free means that the training loss doesn't have likelihood training loss doesn't have likelihood training loss doesn't have likelihood it's not maximizing the likelihood but it's not maximizing the likelihood but it's not maximizing the likelihood but you can still compute likelihood right you can still compute likelihood right you can still compute likelihood right score same for scorebased model you can score same for scorebased model you can score same for scorebased model you can compute the likelihood compute the likelihood compute the likelihood through the same o pfod thing right yeah through the same o pfod thing right yeah through the same o pfod thing right yeah all right all right so congratulations all right all right so congratulations all right all right so congratulations we gone through the basics now yay now we gone through the basics now yay now we gone through the basics now yay now we know what is diffusion one is full we know what is diffusion one is full we know what is diffusion one is full matching yay um yeah so starting from matching yay um yeah so starting from matching yay um yeah so starting from next week we're gonna be able to see next week we're gonna be able to see

  63. L5 @ 3661s

    next week we're gonna be able to see like like like uh different options to further uh different options to further uh different options to further improve what we have learned. Um so for improve what we have learned. Um so for improve what we have learned. Um so for example like I I guess you guys have if example like I I guess you guys have if example like I I guess you guys have if you guys have done the homework you you guys have done the homework you you guys have done the homework you should know this by now there's so many should know this by now there's so many should know this by now there's so many knobs that you can tune right for knobs that you can tune right for knobs that you can tune right for diffusion models like what are you use diffusion models like what are you use diffusion models like what are you use how many number of steps and stuff like how many number of steps and stuff like how many number of steps and stuff like that right so you know we're gonna we're that right so you know we're gonna we're that right so you know we're gonna we're going to learn about that like basically going to learn about that like basically going to learn about that like basically uh there's a paper that like just uh there's a paper that like just uh there's a paper that like just systematically study this uh and we are systematically study this uh and we are systematically study this uh and we are like for those of you who did uh the like for those of you who did uh the like for those of you who did uh the homework you should also notice that homework you should also notice that homework you should also notice that diffusion models sample really really diffusion models sample really really diffusion models sample really really slowly and it's like just like slowly and it's like just like slowly and it's like just like It takes so much time. So how do we make It takes so much time. So how do we make It takes so much time. So how do we make the generation faster? And we're going the generation faster? And we're going

  64. L5 @ 3701s

    the generation faster? And we're going to learn uh to do it with or without to learn uh to do it with or without to learn uh to do it with or without training retraining of the model. And training retraining of the model. And training retraining of the model. And lastly, right now we all the thing that lastly, right now we all the thing that lastly, right now we all the thing that we learn are unconditional generations. we learn are unconditional generations. we learn are unconditional generations. So how can we make the generation more So how can we make the generation more So how can we make the generation more controllable or conditional, right? And controllable or conditional, right? And controllable or conditional, right? And we are actually going to learn it uh to we are actually going to learn it uh to we are actually going to learn it uh to do it with training and also without do it with training and also without do it with training and also without training which is very interesting. All training which is very interesting. All training which is very interesting. All right. Uh that's it for today. I'll see right. Uh that's it for today. I'll see right. Uh that's it for today. I'll see you next week. Uh stay warm this uh this you next week. Uh stay warm this uh this you next week. Uh stay warm this uh this uh this this uh weekend. There's a lot uh this this uh weekend. There's a lot uh this this uh weekend. There's a lot of flow going on.

diffusion-principles-vizuara

  1. L2 @ 3s

    Good morning everyone. Good morning everyone. Today we are starting with the next Today we are starting with the next Today we are starting with the next lecture of our course which is lecture of our course which is lecture of our course which is introduction to flow models. introduction to flow models. introduction to flow models. I have decided to divide this topic into I have decided to divide this topic into I have decided to divide this topic into a series of three to four lectures a series of three to four lectures a series of three to four lectures so that I can cover the entire depth in so that I can cover the entire depth in so that I can cover the entire depth in a very comprehensive manner. a very comprehensive manner. a very comprehensive manner. One interesting thing about the theory One interesting thing about the theory One interesting thing about the theory behind flow models is that behind flow models is that behind flow models is that you can teach the same topic in two you can teach the same topic in two you can teach the same topic in two different ways. different ways. different ways. You can teach this entire topic within a You can teach this entire topic within a You can teach this entire topic within a span of one lecture span of one lecture span of one lecture or you can teach it within a span of or you can teach it within a span of or you can teach it within a span of four lectures. Now that is quite paradoxical that how Now that is quite paradoxical that how you can teach the same concept you can teach the same concept you can teach the same concept with these two format types. The main with these two format types. The main with these two format types. The main reason is that if you go through the reason is that if you go through the reason is that if you go through the difficult route of the four lectures, difficult route of the four lectures, difficult route of the four lectures, you end up at a place which is quite you end up at a place which is quite you end up at a place which is quite simple which you could have got in the

  2. L2 @ 70s

    simple which you could have got in the simple which you could have got in the first lecture itself. first lecture itself. first lecture itself. But I don't want to show you that route But I don't want to show you that route But I don't want to show you that route that easily because you won't appreciate that easily because you won't appreciate that easily because you won't appreciate the the the work which has been put behind work which has been put behind work which has been put behind the easy route. So in essence flow model the easy route. So in essence flow model the easy route. So in essence flow model is a very very easy concept and it can is a very very easy concept and it can is a very very easy concept and it can be explained in a very simple and be explained in a very simple and be explained in a very simple and intuitive manner. intuitive manner. intuitive manner. But because But because But because reaching there is not that reaching there is not that reaching there is not that straightforward, we might have to spend straightforward, we might have to spend straightforward, we might have to spend some more time to understand the simple some more time to understand the simple some more time to understand the simple concept. concept. concept. In other words, imagine that we are In other words, imagine that we are In other words, imagine that we are going we want to go from point A to going we want to go from point A to going we want to go from point A to point B, but instead of going through point B, but instead of going through point B, but instead of going through the straight line, we are going to take the straight line, we are going to take the straight line, we are going to take a curve and reach point B. a curve and reach point B. a curve and reach point B. It will be slightly mathematical uh It will be slightly mathematical uh It will be slightly mathematical uh today's lecture but uh I have tried to today's lecture but uh I have tried to today's lecture but uh I have tried to keep the mathematics as simple as keep the mathematics as simple as

  3. L2 @ 126s

    keep the mathematics as simple as possible [snorts] and I have taken possible [snorts] and I have taken possible [snorts] and I have taken inspiration from an uh MIT course to inspiration from an uh MIT course to inspiration from an uh MIT course to structure these lectures. I will put the structure these lectures. I will put the structure these lectures. I will put the link to that course in the description link to that course in the description link to that course in the description and their lecture notes as well. and their lecture notes as well. and their lecture notes as well. So before we get started for those of So before we get started for those of So before we get started for those of you who haven't heard of flow models and you who haven't heard of flow models and you who haven't heard of flow models and uh what these models represent uh what these models represent uh what these models represent first of all let us divide this two first of all let us divide this two first of all let us divide this two words into two distinct words. The first words into two distinct words. The first words into two distinct words. The first word is flow. Now what does flow mean? word is flow. Now what does flow mean? word is flow. Now what does flow mean? When I hear flow the first thing that When I hear flow the first thing that When I hear flow the first thing that comes to my mind is probably a fluid. comes to my mind is probably a fluid. comes to my mind is probably a fluid. Right? So let's click on this and see Right? So let's click on this and see Right? So let's click on this and see something something moving in in space something something moving in in space something something moving in in space something like this something something something like this something something something like this something something like a fluid motion like a fluid motion like a fluid motion and model is of course we are aware a and model is of course we are aware a and model is of course we are aware a model is a representation of a phenomena model is a representation of a phenomena

  4. L2 @ 181s

    model is a representation of a phenomena a neural network is an example of a a neural network is an example of a a neural network is an example of a model or any mathematical function which model or any mathematical function which model or any mathematical function which maps an input to an output is a model. maps an input to an output is a model. maps an input to an output is a model. [snorts] [snorts] [snorts] Now the question is how we are going to Now the question is how we are going to Now the question is how we are going to use these physical notions of the use these physical notions of the use these physical notions of the concept of flow and concept of flow and concept of flow and u solve the problem that we have set out u solve the problem that we have set out u solve the problem that we have set out to solve at the beginning of this to solve at the beginning of this to solve at the beginning of this series. If you remember the problem that series. If you remember the problem that series. If you remember the problem that we have is we have is we have is called as deep generative modeling. >> Now what does this mean? >> Now what does this mean? We are given a bunch of data. Let's say we are given these images of Let's say we are given these images of cats cats cats and we want to find a distribution using and we want to find a distribution using and we want to find a distribution using which we can sample images which look which we can sample images which look which we can sample images which look exactly like these images of cats. So exactly like these images of cats. So exactly like these images of cats. So from data from data from data We want to move to distribution and that is the broad problem that we and that is the broad problem that we want to solve want to solve want to solve in in this whole whole series and so far in in this whole whole series and so far in in this whole whole series and so far we have looked at different techniques we have looked at different techniques

  5. L2 @ 262s

    we have looked at different techniques to solve this broader problem. It to solve this broader problem. It to solve this broader problem. It started out with variational started out with variational started out with variational autoenccoders. Then we move to diffusion autoenccoders. Then we move to diffusion autoenccoders. Then we move to diffusion models and now we are discussing another models and now we are discussing another models and now we are discussing another type of deep generative models which are type of deep generative models which are type of deep generative models which are called as flow models. So we discussed VAS, So we discussed VAS, we discussed diffusion models we discussed diffusion models we discussed diffusion models [clears throat] [clears throat] [clears throat] and now we are going to discuss flow and now we are going to discuss flow and now we are going to discuss flow models. So in our mind we know that okay models So in our mind we know that okay models probably represent deep generative probably represent deep generative probably represent deep generative models and models and models and flow is something like a fluid which flow is something like a fluid which flow is something like a fluid which transports particles from one initial transports particles from one initial transports particles from one initial location to a final location. So so location to a final location. So so location to a final location. So so there is a movement of particles in there is a movement of particles in there is a movement of particles in space space space and something is changing with time and something is changing with time and something is changing with time also. So we want to combine these also. So we want to combine these also. So we want to combine these physical intuitions into a deep physical intuitions into a deep physical intuitions into a deep generative model that helps us map a generative model that helps us map a generative model that helps us map a data to a probability distribution. So let's understand uh before we dive So let's understand uh before we dive deep into the theory. Flow models are deep into the theory. Flow models are deep into the theory. Flow models are one of the most consequential techniques one of the most consequential techniques

  6. L2 @ 334s

    one of the most consequential techniques in modern [snorts] in modern [snorts] in modern [snorts] generative modeling. And the reason it generative modeling. And the reason it generative modeling. And the reason it is very consequential it is it is used is very consequential it is it is used is very consequential it is it is used in a lot of image generation models. Lot in a lot of image generation models. Lot in a lot of image generation models. Lot of state-of-the-art image generation of state-of-the-art image generation of state-of-the-art image generation tools use flow models. It is also used tools use flow models. It is also used tools use flow models. It is also used in robotics. There is a policy which is in robotics. There is a policy which is in robotics. There is a policy which is called as pi zero foundational model. Now this is a policy which is a robotics Now this is a policy which is a robotics foundational model which means that foundational model which means that foundational model which means that similar to how LLMs are trained on similar to how LLMs are trained on similar to how LLMs are trained on massive amounts of web corpus data these massive amounts of web corpus data these massive amounts of web corpus data these models are trained on massive amounts of models are trained on massive amounts of models are trained on massive amounts of videos. So the robot learns from these videos. So the robot learns from these videos. So the robot learns from these static or static videos and learns to static or static videos and learns to static or static videos and learns to perform actions in the real world by perform actions in the real world by perform actions in the real world by learning from these trajectories which learning from these trajectories which learning from these trajectories which have been recorded. have been recorded. have been recorded. Now this policy uses something called as Now this policy uses something called as Now this policy uses something called as a flow model u in in the architecture a flow model u in in the architecture a flow model u in in the architecture and it works exceptionally well. You can and it works exceptionally well. You can

  7. L2 @ 393s

    and it works exceptionally well. You can see here they have this action expert see here they have this action expert see here they have this action expert and this action expert is built [snorts] and this action expert is built [snorts] and this action expert is built [snorts] on top of a flow model. So understanding on top of a flow model. So understanding on top of a flow model. So understanding flow model is extremely critical for us flow model is extremely critical for us flow model is extremely critical for us to understand all these applications of to understand all these applications of to understand all these applications of uh these models in different domains uh these models in different domains uh these models in different domains including robotics including uh image including robotics including uh image including robotics including uh image generation and video generation as well. generation and video generation as well. generation and video generation as well. [snorts] In fact a lot of protein [snorts] In fact a lot of protein [snorts] In fact a lot of protein synthesis synthesis of protein molecules synthesis synthesis of protein molecules synthesis synthesis of protein molecules uh or drug discovery is now being done uh or drug discovery is now being done uh or drug discovery is now being done using flow models. So there are using flow models. So there are using flow models. So there are innumerable number of applications and innumerable number of applications and innumerable number of applications and as I said in the beginning the final as I said in the beginning the final as I said in the beginning the final formulation is so simple that we'll formulation is so simple that we'll formulation is so simple that we'll think back and say why did we spend think back and say why did we spend think back and say why did we spend three lectures on this this could have three lectures on this this could have three lectures on this this could have been maybe done in half an hour but we been maybe done in half an hour but we been maybe done in half an hour but we really want to appreciate how this really want to appreciate how this really want to appreciate how this formula comes about which is very formula comes about which is very

  8. L2 @ 446s

    formula comes about which is very simple. So flow modeling at the heart of simple. So flow modeling at the heart of simple. So flow modeling at the heart of it is very simple and it has huge number it is very simple and it has huge number it is very simple and it has huge number of applications. of applications. of applications. But surprisingly this is the model which But surprisingly this is the model which But surprisingly this is the model which came out the came out the came out the uh came out late compared to other uh came out late compared to other uh came out late compared to other models. For example score based models, models. For example score based models, models. For example score based models, diffusion models all these models came diffusion models all these models came diffusion models all these models came early and then flow models came after early and then flow models came after early and then flow models came after that. [snorts] Okay. So first let's start with a simple Okay. So first let's start with a simple Okay. So first let's start with a simple mathematical description mathematical description mathematical description [snorts] [snorts] where where where we are looking at something called as we are looking at something called as we are looking at something called as time dependent vector fields. [snorts] First let us understand what is a vector First let us understand what is a vector First let us understand what is a vector field. Okay. So imagine that you have field. Okay. So imagine that you have field. Okay. So imagine that you have this 2D two-dimensional grid x-axis and y-axis. two-dimensional grid x-axis and y-axis. [snorts] A vector field is something [snorts] A vector field is something [snorts] A vector field is something where given any point where given any point where given any point it gives a vector as an output. it gives a vector as an output. it gives a vector as an output. So for example, this is a vector field. So for example, this is a vector field. So for example, this is a vector field. [snorts] [snorts] So let's take an example. Imagine that So let's take an example. Imagine that So let's take an example. Imagine that maybe you're sitting on a table right

  9. L2 @ 525s

    maybe you're sitting on a table right maybe you're sitting on a table right now or you have a table in front of you. now or you have a table in front of you. now or you have a table in front of you. Uh you place a bunch of magnets on the Uh you place a bunch of magnets on the Uh you place a bunch of magnets on the table and the magnets table and the magnets table and the magnets create a vector field all around them. create a vector field all around them. create a vector field all around them. Uh or the magnets create a magnetic Uh or the magnets create a magnetic Uh or the magnets create a magnetic field all around them and you take any field all around them and you take any field all around them and you take any point on the table and you ask what is point on the table and you ask what is point on the table and you ask what is the magnetic field at that location. the magnetic field at that location. the magnetic field at that location. That [snorts] is an example of a vector That [snorts] is an example of a vector That [snorts] is an example of a vector field. Given any point in space, it field. Given any point in space, it field. Given any point in space, it gives you a magnitude and a direction. Now here what we are considering is a Now here what we are considering is a timed dependent vector field. timed dependent vector field. timed dependent vector field. So what does a timed dependent vector So what does a timed dependent vector So what does a timed dependent vector field do? Instead of looking at one field do? Instead of looking at one field do? Instead of looking at one single snapshot, so this is maybe a single snapshot, so this is maybe a single snapshot, so this is maybe a snapshot at time t equal to0. snapshot at time t equal to0. snapshot at time t equal to0. This vector field is going to change This vector field is going to change This vector field is going to change with time. So the magnets that we have with time. So the magnets that we have

  10. L2 @ 591s

    with time. So the magnets that we have placed on the table, what if these placed on the table, what if these placed on the table, what if these magnets are constantly moving? magnets are constantly moving? magnets are constantly moving? Then the magnetic field created by them Then the magnetic field created by them Then the magnetic field created by them will also change with time, right? And will also change with time, right? And will also change with time, right? And because of that you will have these because of that you will have these because of that you will have these vector fields which are also now vector fields which are also now vector fields which are also now changing with time. depends on two things. depends on two things. [snorts] [snorts] It depends on first space It depends on first space It depends on first space which is at at what point in this which is at at what point in this which is at at what point in this cartian grid you are looking at and cartian grid you are looking at and cartian grid you are looking at and second it depends on time as well. So [snorts] whatever I have shown here So [snorts] whatever I have shown here it's this is like a single snapshot. it's this is like a single snapshot. it's this is like a single snapshot. So imagine the magnetic field on your So imagine the magnetic field on your So imagine the magnetic field on your table is varying with time. You take a table is varying with time. You take a table is varying with time. You take a camera and you take a photo of the camera and you take a photo of the camera and you take a photo of the magnetic field. So you freeze time and magnetic field. So you freeze time and magnetic field. So you freeze time and then the magnetic field will only vary then the magnetic field will only vary then the magnetic field will only vary with space. Now these photos you can with space. Now these photos you can with space. Now these photos you can take at multiple time steps and hence take at multiple time steps and hence

  11. L2 @ 659s

    take at multiple time steps and hence you will get a vector field which is you will get a vector field which is you will get a vector field which is changing with time. So now we are discussing time dependent So now we are discussing time dependent vector fields. vector fields. vector fields. A time dependent vector field maps a A time dependent vector field maps a A time dependent vector field maps a point x in space point x in space point x in space and time t to a velocity vector. and time t to a velocity vector. and time t to a velocity vector. And mathematically we define such a And mathematically we define such a And mathematically we define such a field of as u of x t. For example, if I field of as u of x t. For example, if I field of as u of x t. For example, if I say u of say u of say u of um um um let's say this x is a vector. So I point let's say this x is a vector. So I point let's say this x is a vector. So I point it as 5 it as 5 it as 5 e ex let's say it's a vector only in the e ex let's say it's a vector only in the e ex let's say it's a vector only in the x direction, x direction, x direction, 0. 0. 0. This means that I'm taking a snapshot at This means that I'm taking a snapshot at This means that I'm taking a snapshot at time t=0 time t=0 time t=0 and I am asking what is the velocity and I am asking what is the velocity and I am asking what is the velocity field at this point field at this point field at this point and and and this is given by a vector which can this is given by a vector which can this is given by a vector which can point in any direction. point in any direction. point in any direction. So the notation u u is a vector which So the notation u u is a vector which So the notation u u is a vector which depends on not just the position in depends on not just the position in

  12. L2 @ 732s

    depends on not just the position in space but also it depends on the time space but also it depends on the time space but also it depends on the time snapshot. So it is a timed dependent snapshot. So it is a timed dependent snapshot. So it is a timed dependent vector field. This is something which is vector field. This is something which is vector field. This is something which is new for some people and it takes a bit new for some people and it takes a bit new for some people and it takes a bit of a time to understand it. But you can of a time to understand it. But you can of a time to understand it. But you can imagine something like imagine something like imagine something like let's say you have a u let's say you have a u let's say you have a u you have a bank or let's say a container you have a bank or let's say a container you have a bank or let's say a container in which you pour water in which you pour water in which you pour water and this water is and you are moving the and this water is and you are moving the and this water is and you are moving the container left and right. So the water container left and right. So the water container left and right. So the water is also moving left and right. Now you is also moving left and right. Now you is also moving left and right. Now you take any point in the container, the take any point in the container, the take any point in the container, the velocity at that point is going to velocity at that point is going to velocity at that point is going to change with time because the container change with time because the container change with time because the container itself is moving and every point in the itself is moving and every point in the itself is moving and every point in the container is going to have different container is going to have different container is going to have different velocities. So this is an example of a velocities. So this is an example of a velocities. So this is an example of a timed dependent vector field.

  13. L2 @ 787s

    timed dependent vector field. So another way of defining this is So another way of defining this is So another way of defining this is people usually write this t as a people usually write this t as a people usually write this t as a subscript and x is written in the subscript and x is written in the subscript and x is written in the bracket. So this means that the velocity bracket. So this means that the velocity bracket. So this means that the velocity field at time snapshot t and at a space field at time snapshot t and at a space field at time snapshot t and at a space position which is denoted as x. position which is denoted as x. position which is denoted as x. Okay. So now the Okay. So now the Okay. So now the interesting part about this velocity interesting part about this velocity interesting part about this velocity field or the vector field is that the field or the vector field is that the field or the vector field is that the trajectories of the points are changing trajectories of the points are changing trajectories of the points are changing according to it. according to it. according to it. So up until now what I have said is that So up until now what I have said is that So up until now what I have said is that all the points in space are having some all the points in space are having some all the points in space are having some vector. vector. vector. Okay. Now I'm saying something else. I'm Okay. Now I'm saying something else. I'm Okay. Now I'm saying something else. I'm saying that these vectors represent the saying that these vectors represent the saying that these vectors represent the velocities of these particles which velocities of these particles which velocities of these particles which means that the means that the means that the position of this particle is going to position of this particle is going to position of this particle is going to change because of these velocities. change because of these velocities. change because of these velocities. So for example, if we take this example So for example, if we take this example

  14. L2 @ 851s

    So for example, if we take this example at at the top at time t equal to0, this at at the top at time t equal to0, this at at the top at time t equal to0, this is what my velocity field looks like. is what my velocity field looks like. is what my velocity field looks like. And because of these this velocity field And because of these this velocity field And because of these this velocity field my particles which is let's say I focus my particles which is let's say I focus my particles which is let's say I focus on this green particle right now the on this green particle right now the on this green particle right now the position of this particle is going to position of this particle is going to change because this velocity vector is change because this velocity vector is change because this velocity vector is acts upon it. So maybe at the next time acts upon it. So maybe at the next time acts upon it. So maybe at the next time step this particle moves here this step this particle moves here this step this particle moves here this particle moves here. This particle moves particle moves here. This particle moves particle moves here. This particle moves here. here. here. So the location of all the particles is So the location of all the particles is So the location of all the particles is going to change because of the velocity going to change because of the velocity going to change because of the velocity which acts upon them and then we move to the next time and then we move to the next time snapshot. Now here again there might be snapshot. Now here again there might be snapshot. Now here again there might be some other velocity field vector which some other velocity field vector which some other velocity field vector which is imposed on these points. So again the is imposed on these points. So again the is imposed on these points. So again the position of these points will change. position of these points will change. position of these points will change. So now we are asking the question how do So now we are asking the question how do

  15. L2 @ 916s

    So now we are asking the question how do the position of particles which are the position of particles which are the position of particles which are inside the space change according to inside the space change according to inside the space change according to this velocity vector field this velocity vector field this velocity vector field and that is given by an equation which and that is given by an equation which and that is given by an equation which looks like this. looks like this. looks like this. [snorts] Now this is very similar to the velocity Now this is very similar to the velocity formula. All of us know that velocity is formula. All of us know that velocity is formula. All of us know that velocity is given by given by given by distance upon time which can also be written as change in which can also be written as change in the position of the particle the position of the particle the position of the particle divided by change in time which is delta divided by change in time which is delta divided by change in time which is delta x divided by delta t. And if these x divided by delta t. And if these x divided by delta t. And if these changes are small enough this can be changes are small enough this can be changes are small enough this can be approximated as dx by dt. approximated as dx by dt. approximated as dx by dt. So given a velocity vector field at any So given a velocity vector field at any So given a velocity vector field at any point if I know a small incremental point if I know a small incremental point if I know a small incremental change in the time I can predict by how change in the time I can predict by how change in the time I can predict by how much the particle is going to move in much the particle is going to move in much the particle is going to move in space. space. space. So I can predict the displacement of the So I can predict the displacement of the So I can predict the displacement of the particle if I know the velocity vector particle if I know the velocity vector

  16. L2 @ 984s

    particle if I know the velocity vector field. This is exactly what this formula field. This is exactly what this formula field. This is exactly what this formula says. says. says. So if I have a particle located at So if I have a particle located at So if I have a particle located at position x at time t, position x at time t, position x at time t, then the change in the location of that then the change in the location of that then the change in the location of that particle is given by is given by the velocity vector field at that same the velocity vector field at that same the velocity vector field at that same point and at time t. This is what this point and at time t. This is what this point and at time t. This is what this formula represents. and we have schematically drawn it in in and we have schematically drawn it in in this figure itself. If we want to this figure itself. If we want to this figure itself. If we want to exactly calculate this displacement, exactly calculate this displacement, exactly calculate this displacement, let [snorts] us do that using this let [snorts] us do that using this let [snorts] us do that using this formula. Let's let's focus on this formula. Let's let's focus on this formula. Let's let's focus on this point. So let's say this velocity is u point. So let's say this velocity is u point. So let's say this velocity is u and now the velocity u of x comma 0. and now the velocity u of x comma 0. and now the velocity u of x comma 0. Okay. So let's say this velocity is Okay. So let's say this velocity is Okay. So let's say this velocity is given by given by 3. Okay. Now I'm interested to find the Okay. Now I'm interested to find the [clears throat] dx by dt [clears throat] dx by dt [clears throat] dx by dt the change in velo the change in the the change in velo the change in the the change in velo the change in the position of this point with time and I know this is equal to u with time and I know this is equal to u of x0 of x0

  17. L2 @ 1079s

    of x0 okay so now this if I write as delta x okay so now this if I write as delta x okay so now this if I write as delta x by delta t by delta t by delta t is equal to mu of u of x0. is equal to mu of u of x0. is equal to mu of u of x0. Now if let's say delta t is equal to 1 Now if let's say delta t is equal to 1 Now if let's say delta t is equal to 1 then delta x is equal to u which is 3 e then delta x is equal to u which is 3 e then delta x is equal to u which is 3 e x + 4 ey. Now this is of course an approximate Now this is of course an approximate usually delta t will be very small in usually delta t will be very small in usually delta t will be very small in magnitude. So what that means is that magnitude. So what that means is that magnitude. So what that means is that this particle will move three places in the x direction and four three places in the x direction and four places in the y direction. places in the y direction. places in the y direction. So if we know the velocity field at all So if we know the velocity field at all So if we know the velocity field at all points at all times snapshots points at all times snapshots points at all times snapshots theoretically we can calculate the exact theoretically we can calculate the exact theoretically we can calculate the exact trajectories of each and every particle trajectories of each and every particle trajectories of each and every particle in the fluid. in the fluid. in the fluid. So imagine that you have a river and you So imagine that you have a river and you So imagine that you have a river and you place a small box in the river place a small box in the river place a small box in the river and you ask the question where will this and you ask the question where will this and you ask the question where will this box end up at time t equal to 10 let's

  18. L2 @ 1148s

    box end up at time t equal to 10 let's box end up at time t equal to 10 let's say. Now if you know the exact velocity say. Now if you know the exact velocity say. Now if you know the exact velocity at all points in the river at all times at all points in the river at all times at all points in the river at all times you can map the trajectory of this box you can map the trajectory of this box you can map the trajectory of this box and how it moves with time and where it and how it moves with time and where it and how it moves with time and where it ends up at time t equal to 10. That is ends up at time t equal to 10. That is ends up at time t equal to 10. That is exactly what we are saying in this exactly what we are saying in this exactly what we are saying in this mathematical description. mathematical description. mathematical description. and uh this is represented by this and uh this is represented by this and uh this is represented by this simple formula which is also a simple formula which is also a simple formula which is also a differential equation. Now the central question that we want to Now the central question that we want to ask is if a point starts at x0 ask is if a point starts at x0 ask is if a point starts at x0 at time t =0 at time t =0 at time t =0 and follows the vector field then where and follows the vector field then where and follows the vector field then where does the particle end up at a later does the particle end up at a later does the particle end up at a later time? So time? So time? So if if [snorts] I have a box and I place if if [snorts] I have a box and I place if if [snorts] I have a box and I place it in the river at time t equal to0, the it in the river at time t equal to0, the it in the river at time t equal to0, the box follows the vector field and the

  19. L2 @ 1202s

    box follows the vector field and the box follows the vector field and the question I'm trying to address is where question I'm trying to address is where question I'm trying to address is where does the box end up later in time does the box end up later in time does the box end up later in time because of the flow of the river. Now this is nicely illustrated in this Now this is nicely illustrated in this uh image. uh image. uh image. So we start at x x0 and you can see how So we start at x x0 and you can see how So we start at x x0 and you can see how the particle moves. the particle moves. the particle moves. Now you can see two things here. The Now you can see two things here. The Now you can see two things here. The green dot is the particle and the white green dot is the particle and the white green dot is the particle and the white line is the trajectory followed by that line is the trajectory followed by that line is the trajectory followed by that particle. particle. particle. But in the background what do we see? We But in the background what do we see? We But in the background what do we see? We see these arrows are constantly changing see these arrows are constantly changing see these arrows are constantly changing with time. This is because with time. This is because with time. This is because as the particle moves the time is also as the particle moves the time is also as the particle moves the time is also changing and the velocity vector field changing and the velocity vector field changing and the velocity vector field at every time is different. So these at every time is different. So these at every time is different. So these arrows represent the vector field. arrows represent the vector field. arrows represent the vector field. Maybe I can uh denote it using a label Maybe I can uh denote it using a label Maybe I can uh denote it using a label over here. So these arrows over here. So these arrows over here. So these arrows they represent the vector field which is they represent the vector field which is

  20. L2 @ 1266s

    they represent the vector field which is time dependent. That is why the arrows time dependent. That is why the arrows time dependent. That is why the arrows are changing with time. And if we know are changing with time. And if we know are changing with time. And if we know the time dependent vector field by using the time dependent vector field by using the time dependent vector field by using the above differential equation, we can the above differential equation, we can the above differential equation, we can exactly pinpoint how each particle will exactly pinpoint how each particle will exactly pinpoint how each particle will move according to these time dependent move according to these time dependent move according to these time dependent vector fields. So to answer this question, we need to So to answer this question, we need to solve the above differential equation. solve the above differential equation. solve the above differential equation. Now the idea is simple. We use velocity Now the idea is simple. We use velocity Now the idea is simple. We use velocity vectors to advance the point in small vectors to advance the point in small vectors to advance the point in small time increments. This is something we time increments. This is something we time increments. This is something we saw in the previous example also that if saw in the previous example also that if saw in the previous example also that if we have the velocity vector [snorts] uh we have the velocity vector [snorts] uh we have the velocity vector [snorts] uh we use we we use we we use we convert dx by dt as delta x by delta t convert dx by dt as delta x by delta t convert dx by dt as delta x by delta t and then we can calculate the individual and then we can calculate the individual and then we can calculate the individual displacements of each point because of displacements of each point because of displacements of each point because of these vector fields. the white line is something that we have the white line is something that we have got as the trajectory of the particle got as the trajectory of the particle got as the trajectory of the particle but it is not quite correct because at

  21. L2 @ 1341s

    but it is not quite correct because at but it is not quite correct because at the end of it we are doing an the end of it we are doing an the end of it we are doing an approximation. approximation. approximation. We are not calculating the exact vector We are not calculating the exact vector We are not calculating the exact vector field but we are replacing dx by dt by field but we are replacing dx by dt by field but we are replacing dx by dt by delta x by delta t. delta x by delta t. delta x by delta t. And that is why we are calculating an And that is why we are calculating an And that is why we are calculating an approximate approximate approximate trajectory of the particle. So you can trajectory of the particle. So you can trajectory of the particle. So you can see the difference between these white see the difference between these white see the difference between these white lines and the gray lines. lines and the gray lines. lines and the gray lines. And this discrepancy appears because of And this discrepancy appears because of And this discrepancy appears because of the way we have discretized the the way we have discretized the the way we have discretized the differential equation. Okay. So this is the first part which is Okay. So this is the first part which is understanding time dependent vector understanding time dependent vector understanding time dependent vector fields. fields. fields. Understanding how these vector fields Understanding how these vector fields Understanding how these vector fields can be used to find the trajectories of can be used to find the trajectories of can be used to find the trajectories of all the particles in space using a all the particles in space using a all the particles in space using a simple ordinary differential equation. Okay, let's move ahead now and we will Okay, let's move ahead now and we will define [snorts] what is flow. define [snorts] what is flow. define [snorts] what is flow. In the literature, you will see flow In the literature, you will see flow In the literature, you will see flow models as the terminology used models as the terminology used models as the terminology used everywhere in a lot of papers. But flow

  22. L2 @ 1418s

    everywhere in a lot of papers. But flow everywhere in a lot of papers. But flow means something very specific. In our means something very specific. In our means something very specific. In our mind, we know flow means something is mind, we know flow means something is mind, we know flow means something is moving. But we will moving. But we will moving. But we will u try to understand it in a bit more u try to understand it in a bit more u try to understand it in a bit more rigorous manner to understand what rigorous manner to understand what rigorous manner to understand what exactly is flow. it will turn out that exactly is flow. it will turn out that exactly is flow. it will turn out that it is exactly the same. Uh it it really it is exactly the same. Uh it it really it is exactly the same. Uh it it really matches our physical intuition. matches our physical intuition. matches our physical intuition. So what my physical intuition tells me So what my physical intuition tells me So what my physical intuition tells me is that let's say is that let's say is that let's say um you have um you have um you have a collection of thousand particles and a collection of thousand particles and a collection of thousand particles and these thousand particles are moving in these thousand particles are moving in these thousand particles are moving in space. So the flow is something which space. So the flow is something which space. So the flow is something which tells you the trajectories of these tells you the trajectories of these tells you the trajectories of these thousand particles and it it's it's thousand particles and it it's it's thousand particles and it it's it's basically a collection of the basically a collection of the basically a collection of the trajectories of all of these particles trajectories of all of these particles trajectories of all of these particles in in space and that is exactly true in in space and that is exactly true in in space and that is exactly true according to the formal definition the according to the formal definition the according to the formal definition the flow is a collection of trajectories flow is a collection of trajectories

  23. L2 @ 1474s

    flow is a collection of trajectories which evolve according to the velocity which evolve according to the velocity which evolve according to the velocity field. We have just seen that if we know field. We have just seen that if we know field. We have just seen that if we know the velocity vector field a time the velocity vector field a time the velocity vector field a time dependent velocity vector field we can dependent velocity vector field we can dependent velocity vector field we can calculate the trajectories of every calculate the trajectories of every calculate the trajectories of every individual particle. Now if we do this individual particle. Now if we do this individual particle. Now if we do this for all the particles in the space for all the particles in the space for all the particles in the space we will get the collective motion of the we will get the collective motion of the we will get the collective motion of the object which is called as flow. object which is called as flow. object which is called as flow. Let us understand this using a Let us understand this using a Let us understand this using a visualization. So visualization. So visualization. So here at the background in in gray what here at the background in in gray what here at the background in in gray what you see is the vector field which is you see is the vector field which is you see is the vector field which is marked in gray. Now I have taken a marked in gray. Now I have taken a marked in gray. Now I have taken a snapshot of one single time. Let's say snapshot of one single time. Let's say snapshot of one single time. Let's say this is at time t equal to0. I have this is at time t equal to0. I have this is at time t equal to0. I have taken a snapshot. taken a snapshot. taken a snapshot. As time changes the gray velocity vector As time changes the gray velocity vector As time changes the gray velocity vector field is also going to change. But for field is also going to change. But for field is also going to change. But for now let us look at one single snapshot now let us look at one single snapshot

  24. L2 @ 1530s

    now let us look at one single snapshot because we have we have restricted with because we have we have restricted with because we have we have restricted with the screen in in front of us. Now you the screen in in front of us. Now you the screen in in front of us. Now you can see there are these blue dots. can see there are these blue dots. can see there are these blue dots. These blue dots are particles. These blue dots are particles. These blue dots are particles. Now I have for simplicity denoted a Now I have for simplicity denoted a Now I have for simplicity denoted a collection of 20 particles over here. collection of 20 particles over here. collection of 20 particles over here. But you can imagine these particles to But you can imagine these particles to But you can imagine these particles to be much much more. be much much more. be much much more. Now every particle starts at the blue Now every particle starts at the blue Now every particle starts at the blue point and it it evolves. This is the point and it it evolves. This is the point and it it evolves. This is the trajectory the way the particle moves trajectory the way the particle moves trajectory the way the particle moves with time. Now this trajectory is given with time. Now this trajectory is given with time. Now this trajectory is given or we can calculate this trajectory or we can calculate this trajectory or we can calculate this trajectory because we know the vector field. So for because we know the vector field. So for because we know the vector field. So for every single particle we know how the every single particle we know how the every single particle we know how the trajectory changes with time because we trajectory changes with time because we trajectory changes with time because we know the velocity vector field. And now know the velocity vector field. And now know the velocity vector field. And now you see we have collected all these you see we have collected all these you see we have collected all these trajectories and we get a overall sense trajectories and we get a overall sense

  25. L2 @ 1581s

    trajectories and we get a overall sense of the flow. You can see that this this of the flow. You can see that this this of the flow. You can see that this this is probably something which is swirling. is probably something which is swirling. is probably something which is swirling. It's like a vortex. Um let's say you It's like a vortex. Um let's say you It's like a vortex. Um let's say you take a bucket of water and you put your take a bucket of water and you put your take a bucket of water and you put your hand inside and you make a swirl. Then hand inside and you make a swirl. Then hand inside and you make a swirl. Then you will see something like this. Now you will see something like this. Now you will see something like this. Now it's it it's very simple to visualize it's it it's very simple to visualize it's it it's very simple to visualize what flow is. But if someone asks you what flow is. But if someone asks you what flow is. But if someone asks you how do you define flow more rigorously, how do you define flow more rigorously, how do you define flow more rigorously, you can say that flow is a collection of you can say that flow is a collection of you can say that flow is a collection of all trajectories for all particles which all trajectories for all particles which all trajectories for all particles which are there in that bucket of water. are there in that bucket of water. are there in that bucket of water. Okay. So uh another interesting way to Okay. So uh another interesting way to Okay. So uh another interesting way to visualize how flow is is to place a visualize how flow is is to place a visualize how flow is is to place a small grid on space and see how it small grid on space and see how it small grid on space and see how it deforms. So for this same above example, deforms. So for this same above example, deforms. So for this same above example, let's let's see how the grid deforms. let's let's see how the grid deforms.

  26. L2 @ 1629s

    let's let's see how the grid deforms. Now note that the grid is going to Now note that the grid is going to Now note that the grid is going to deform in a way that the vector field deform in a way that the vector field deform in a way that the vector field exists in the space. So because we have exists in the space. So because we have exists in the space. So because we have a swirling flow, every single point in a swirling flow, every single point in a swirling flow, every single point in the grid is going to move according to a the grid is going to move according to a the grid is going to move according to a swirling trajectory. swirling trajectory. swirling trajectory. And uh we see a cumulative And uh we see a cumulative And uh we see a cumulative deformation of the grid. It looks kind deformation of the grid. It looks kind deformation of the grid. It looks kind of weird here right now, but that is of weird here right now, but that is of weird here right now, but that is what flow is. A flow is something which what flow is. A flow is something which what flow is. A flow is something which deforms a uniformly placed grid into deforms a uniformly placed grid into deforms a uniformly placed grid into something else because of which is something else because of which is something else because of which is governed by the velocity vector field. So this is a technique which uh is So this is a technique which uh is generally used to visualize the flows. generally used to visualize the flows. generally used to visualize the flows. If if we place a square grid, so square If if we place a square grid, so square If if we place a square grid, so square grid is nothing but just uniformly grid is nothing but just uniformly grid is nothing but just uniformly placed points in in the space and every placed points in in the space and every placed points in in the space and every single point is going to evolve single point is going to evolve single point is going to evolve according to a trajectory which is according to a trajectory which is

  27. L2 @ 1689s

    according to a trajectory which is governed by an ordinary differential governed by an ordinary differential governed by an ordinary differential equation and if we collect all these equation and if we collect all these equation and if we collect all these trajectories together we get the flow. trajectories together we get the flow. trajectories together we get the flow. Now uh Now uh Now uh let's look at an example to visualize let's look at an example to visualize let's look at an example to visualize this. So in this example we are going to So in this example we are going to visualize the flow using code. visualize the flow using code. visualize the flow using code. So I'll just run this piece of cell and So I'll just run this piece of cell and So I'll just run this piece of cell and u this is exactly the same animation which I had shown me. So this animation which I had shown me. So this is the flow field and uh here is the is the flow field and uh here is the is the flow field and uh here is the code where I have defined the velocity code where I have defined the velocity code where I have defined the velocity vector field and the particles are vector field and the particles are vector field and the particles are evolving according to that velocity evolving according to that velocity evolving according to that velocity vector field over here. vector field over here. vector field over here. And then uh if you run the second piece And then uh if you run the second piece And then uh if you run the second piece of cell you'll be able to see the of cell you'll be able to see the of cell you'll be able to see the deformation of the square grid as well. deformation of the square grid as well. deformation of the square grid as well. Probably this might not work. So I will Probably this might not work. So I will Probably this might not work. So I will u run the third piece of cell to u run the third piece of cell to u run the third piece of cell to understand how the grid deforms. Yeah. understand how the grid deforms. Yeah.

  28. L2 @ 1766s

    understand how the grid deforms. Yeah. So this this does work. So this this does work. So this this does work. Okay. So you can actually change these Okay. So you can actually change these Okay. So you can actually change these velocity vector fields uh in the velocity vector fields uh in the velocity vector fields uh in the description and uh you can you know play description and uh you can you know play description and uh you can you know play around with it and see how the flow around with it and see how the flow around with it and see how the flow changes according to the velocity vector changes according to the velocity vector changes according to the velocity vector fields which is rightly so because the fields which is rightly so because the fields which is rightly so because the velocity vector fields are governing the velocity vector fields are governing the velocity vector fields are governing the trajectories which are in turn governing trajectories which are in turn governing trajectories which are in turn governing the flow. So if you were to you know uh the flow. So if you were to you know uh the flow. So if you were to you know uh draw a sequence you can say that the draw a sequence you can say that the draw a sequence you can say that the sequence looks as follows. sequence looks as follows. sequence looks as follows. First is a velocity field. From the velocity field velocity field. From the velocity field you get trajectories and from the trajectories you get the and from the trajectories you get the flow. flow. flow. All these three things are related to All these three things are related to All these three things are related to each other. Okay. So this is the first step. Okay. So this is the first step. All of us at this point should All of us at this point should All of us at this point should understand three things. First is the understand three things. First is the understand three things. First is the velocity field which I'm interchangeably velocity field which I'm interchangeably velocity field which I'm interchangeably using the word vector field also. Second using the word vector field also. Second

  29. L2 @ 1834s

    using the word vector field also. Second is the trajectories which are calculated is the trajectories which are calculated is the trajectories which are calculated from the velocity field using the simple from the velocity field using the simple from the velocity field using the simple ordinary differential equation. And the ordinary differential equation. And the ordinary differential equation. And the third is flow which is the collection of third is flow which is the collection of third is flow which is the collection of all the trajectories in space. all the trajectories in space. all the trajectories in space. These three concepts are going to be These three concepts are going to be These three concepts are going to be very fundamental for us as we understand very fundamental for us as we understand very fundamental for us as we understand uh uh uh flow models in more depth. So once we flow models in more depth. So once we flow models in more depth. So once we understand this everything is built on understand this everything is built on understand this everything is built on top of these three concepts. Okay. Now let us take an example Okay. Now let us take an example and uh try to understand the flow in and uh try to understand the flow in and uh try to understand the flow in more depth using some practical more depth using some practical more depth using some practical examples. Okay. Okay. So the example that we will Okay. Okay. So the example that we will take is suppose we have a velocity field take is suppose we have a velocity field take is suppose we have a velocity field which is given by So first thing we can see is this So first thing we can see is this velocity field is dependent on velocity field is dependent on velocity field is dependent on um it's it's just a one-dimensional um it's it's just a one-dimensional um it's it's just a one-dimensional velocity field. In the above example we velocity field. In the above example we velocity field. In the above example we were considering two dimensions X and Y. were considering two dimensions X and Y. were considering two dimensions X and Y. It is only one dimensional minus theta x It is only one dimensional minus theta x

  30. L2 @ 1906s

    It is only one dimensional minus theta x and it is independent of time. So at and it is independent of time. So at and it is independent of time. So at every time snapshot you take a camera every time snapshot you take a camera every time snapshot you take a camera you take a photo of the space the you take a photo of the space the you take a photo of the space the velocity field will be exactly the same velocity field will be exactly the same velocity field will be exactly the same in all the photos which you have taken. in all the photos which you have taken. in all the photos which you have taken. Now uh [snorts] Now uh [snorts] Now uh [snorts] okay so okay so okay so we have to check whether the flow field we have to check whether the flow field we have to check whether the flow field is given by this formula. is given by this formula. is given by this formula. So given the velocity field then the So given the velocity field then the So given the velocity field then the flow field is given by this formula. But let us see whether that is actually But let us see whether that is actually true or not. true or not. true or not. Okay. So what do we know? Because the Okay. So what do we know? Because the Okay. So what do we know? Because the velocity field is given by this formula, velocity field is given by this formula, velocity field is given by this formula, we can use the differential equation we can use the differential equation we can use the differential equation that we have been talking about in this that we have been talking about in this that we have been talking about in this lecture multiple times. So that lecture multiple times. So that lecture multiple times. So that differential equation simply says dx by differential equation simply says dx by differential equation simply says dx by dt is equal to the velocity field which dt is equal to the velocity field which dt is equal to the velocity field which is dxt by dt is equal to minus theta of is dxt by dt is equal to minus theta of

  31. L2 @ 1972s

    is dxt by dt is equal to minus theta of xt. xt. xt. Now let's see if this is indeed the Now let's see if this is indeed the Now let's see if this is indeed the flow. flow. If we substitute this in xt, it should If we substitute this in xt, it should If we substitute this in xt, it should satisfy this equation. So let us see if satisfy this equation. So let us see if satisfy this equation. So let us see if that actually happens or not. If xt is that actually happens or not. If xt is that actually happens or not. If xt is equal to equal to equal to e^ minus theta tx0, then dxt by dt then dxt by dt [clears throat] [clears throat] is equal to minus theta e raus theta t is equal to minus theta e raus theta t is equal to minus theta e raus theta t x0 x0 x0 which is equal to minus theta xt. which is equal to minus theta xt. which is equal to minus theta xt. So it does satisfy the differential So it does satisfy the differential So it does satisfy the differential equation. equation. equation. Now remember we said that the flow is a Now remember we said that the flow is a Now remember we said that the flow is a collection of trajectories. This is the collection of trajectories. This is the collection of trajectories. This is the collection of trajectories. Depending on the initial position Depending on the initial position you exactly get how the particle changes you exactly get how the particle changes you exactly get how the particle changes or moves with time. Let's see how to or moves with time. Let's see how to or moves with time. Let's see how to visualize this. It's it's very visualize this. It's it's very visualize this. It's it's very interesting. So if x0 is 0, interesting. So if x0 is 0, interesting. So if x0 is 0, you simply get a straight line because you simply get a straight line because you simply get a straight line because theta is equal to zero or this this this theta is equal to zero or this this this theta is equal to zero or this this this is itself completely zero. So the is itself completely zero. So the

  32. L2 @ 2046s

    is itself completely zero. So the position stays zero. But position stays zero. But position stays zero. But let's look at yeah if x0 is any any any let's look at yeah if x0 is any any any let's look at yeah if x0 is any any any positive number then it will simply look positive number then it will simply look positive number then it will simply look like an exponential curve. For example, like an exponential curve. For example, like an exponential curve. For example, if x0 is 3, then it will be 3 e to minus if x0 is 3, then it will be 3 e to minus if x0 is 3, then it will be 3 e to minus theta t. theta t. theta t. So it will decrease like this. And So it will decrease like this. And So it will decrease like this. And similarly for positive and negative x we similarly for positive and negative x we similarly for positive and negative x we can check. So this is how the flow looks can check. So this is how the flow looks can check. So this is how the flow looks like for a simple linear ordinary like for a simple linear ordinary like for a simple linear ordinary differential equation. differential equation. differential equation. Now why is this a flow? Because this is Now why is this a flow? Because this is Now why is this a flow? Because this is a collection of trajectories. Given the a collection of trajectories. Given the a collection of trajectories. Given the initial position of my particle, I can initial position of my particle, I can initial position of my particle, I can exactly pinpoint how the particle moves exactly pinpoint how the particle moves exactly pinpoint how the particle moves with time. and a collection of all these with time. and a collection of all these with time. and a collection of all these trajectories is going to help me trajectories is going to help me trajectories is going to help me identify what my flow is. So we can uh visualize this using the So we can uh visualize this using the same Google collab notebook and um you can see here I have taken the same you can see here I have taken the same example example

  33. L2 @ 2112s

    example x0 into e to minus theta t and we get the exact same plot. and we get the exact same plot. So we can actually play this and see how So we can actually play this and see how So we can actually play this and see how the uh particles move with time. So the the uh particles move with time. So the the uh particles move with time. So the particles start at so all these particles start at so all these particles start at so all these particles are let's say attached on a particles are let's say attached on a particles are let's say attached on a string they start at a different string they start at a different string they start at a different location and all of them are moving and location and all of them are moving and location and all of them are moving and um going towards [clears throat] zero um going towards [clears throat] zero um going towards [clears throat] zero finally all of them are going to end up finally all of them are going to end up finally all of them are going to end up at zero. So this is almost like a sink. at zero. So this is almost like a sink. at zero. So this is almost like a sink. Imagine you have a sink and you start Imagine you have a sink and you start Imagine you have a sink and you start the particles anywhere. Every particle the particles anywhere. Every particle the particles anywhere. Every particle moves towards the sink and ends up in moves towards the sink and ends up in moves towards the sink and ends up in the sink. So that is the flow. the sink. So that is the flow. the sink. So that is the flow. Okay. So now with this example uh I hope Okay. So now with this example uh I hope Okay. So now with this example uh I hope we have understood what flow is how the we have understood what flow is how the we have understood what flow is how the flow is related to the velocity vector flow is related to the velocity vector flow is related to the velocity vector field and how the flow is a collection field and how the flow is a collection

  34. L2 @ 2163s

    field and how the flow is a collection of the trajectories. of the trajectories. of the trajectories. [snorts] Okay. Now uh once we are getting Okay. Now uh once we are getting confident with the definitions of confident with the definitions of confident with the definitions of velocity fields and flow let us move on. velocity fields and flow let us move on. velocity fields and flow let us move on. So the question is and we have already So the question is and we have already So the question is and we have already looked at this briefly. If we have a OD looked at this briefly. If we have a OD looked at this briefly. If we have a OD which is an ordinary differential which is an ordinary differential which is an ordinary differential equation, how do we simulate it and find equation, how do we simulate it and find equation, how do we simulate it and find the trajectories? I have said before that um once you have I have said before that um once you have the vector field you'll be able to find the vector field you'll be able to find the vector field you'll be able to find the trajectories and I have briefly the trajectories and I have briefly the trajectories and I have briefly explained it here as well how that can explained it here as well how that can explained it here as well how that can be done but maybe we can uh explain it a be done but maybe we can uh explain it a be done but maybe we can uh explain it a bit more formally. bit more formally. bit more formally. So the method to do that is called as So the method to do that is called as So the method to do that is called as the oiler method. the oiler method. the oiler method. And what I oiler method says is we And what I oiler method says is we And what I oiler method says is we simply take steps in the direction of simply take steps in the direction of simply take steps in the direction of the velocity field. the velocity field. the velocity field. We take small steps in the direction of We take small steps in the direction of

  35. L2 @ 2226s

    We take small steps in the direction of the velocity field and we see how the the velocity field and we see how the the velocity field and we see how the particle trajectory changes with time. particle trajectory changes with time. particle trajectory changes with time. So for example, if we start here, we So for example, if we start here, we So for example, if we start here, we take a tiny step, we go here. We take take a tiny step, we go here. We take take a tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. We take another tiny step, we go here. So the another tiny step, we go here. So the another tiny step, we go here. So the velocity field will help us to take velocity field will help us to take velocity field will help us to take these tiny steps. these tiny steps. these tiny steps. And this is the same thing which we have And this is the same thing which we have And this is the same thing which we have outlined here. It looks slightly outlined here. It looks slightly outlined here. It looks slightly detailed, but let's understand this. detailed, but let's understand this. detailed, but let's understand this. Okay. So as an input, we have a vector Okay. So as an input, we have a vector Okay. So as an input, we have a vector field which changes with time. We have field which changes with time. We have field which changes with time. We have initial condition x0 initial condition x0 initial condition x0 and uh we have the number of steps n and uh we have the number of steps n and uh we have the number of steps n which is how many times you are going to which is how many times you are going to which is how many times you are going to update the particle trajectory for how update the particle trajectory for how update the particle trajectory for how long is your simulation carried on. long is your simulation carried on.

  36. L2 @ 2276s

    long is your simulation carried on. Okay. So we initialize time as t equal Okay. So we initialize time as t equal Okay. So we initialize time as t equal to0. The step size which is delta t is 1 to0. The step size which is delta t is 1 to0. The step size which is delta t is 1 by n. by n. by n. And uh the basic idea is which is given And uh the basic idea is which is given And uh the basic idea is which is given here. dx by dt is equal to mu here. dx by dt is equal to mu here. dx by dt is equal to mu u rather. So delta x by delta t is equal u rather. So delta x by delta t is equal u rather. So delta x by delta t is equal to d u. So we can write delta x as u * to d u. So we can write delta x as u * to d u. So we can write delta x as u * delta t. delta t. delta t. Okay. And delta x is x t + delta t minus Okay. And delta x is x t + delta t minus Okay. And delta x is x t + delta t minus xt is equal to u * delta t. xt is equal to u * delta t. xt is equal to u * delta t. Now here delta t is replaced by h. Now here delta t is replaced by h. Now here delta t is replaced by h. So x of t + h is equal to xt + h into u So x of t + h is equal to xt + h into u So x of t + h is equal to xt + h into u which is what is mentioned over here. which is what is mentioned over here. which is what is mentioned over here. This is how we update the position of This is how we update the position of This is how we update the position of the particle to calculate the next the particle to calculate the next the particle to calculate the next position. That is all is mentioned in position. That is all is mentioned in

  37. L2 @ 2336s

    position. That is all is mentioned in the flowchart. So basically the flowchart. So basically the flowchart. So basically if we have a particle and if we know the if we have a particle and if we know the if we have a particle and if we know the velocity vector field we take small velocity vector field we take small velocity vector field we take small increments in time we change this delta increments in time we change this delta increments in time we change this delta x this moves here now now we again x this moves here now now we again x this moves here now now we again change it we again change it we again change it we again change it we again change it we again change it we again change it and we do it till the time we change it and we do it till the time we change it and we do it till the time we are ending the simulation are ending the simulation are ending the simulation this is also called as the oiler method this is also called as the oiler method this is also called as the oiler method so the reason why we are describing this so the reason why we are describing this so the reason why we are describing this is if we know the velocity vector field is if we know the velocity vector field is if we know the velocity vector field at every time we can calculate the at every time we can calculate the at every time we can calculate the trajectories of the particles very trajectories of the particles very trajectories of the particles very precisely precisely precisely and this is something we should keep in and this is something we should keep in and this is something we should keep in mind. We are going to utilize this mind. We are going to utilize this mind. We are going to utilize this later. This is what is being used in the later. This is what is being used in the later. This is what is being used in the inference for the flow models. inference for the flow models. inference for the flow models. For example, when we are using flow For example, when we are using flow

  38. L2 @ 2385s

    For example, when we are using flow models for robotics or protein models for robotics or protein models for robotics or protein synthesis, synthesis, synthesis, people calculate the flow which is the people calculate the flow which is the people calculate the flow which is the trajectories of the particles but or the trajectories of the particles but or the trajectories of the particles but or the velocity vector field which governs the velocity vector field which governs the velocity vector field which governs the trajectories of the particles. But from trajectories of the particles. But from trajectories of the particles. But from there, how do you calculate the there, how do you calculate the there, how do you calculate the trajectories itself? trajectories itself? trajectories itself? Then you have to solve that differential Then you have to solve that differential Then you have to solve that differential equation which gives you the equation which gives you the equation which gives you the trajectories of these individual trajectories of these individual trajectories of these individual particles. particles. particles. So the inference happens by solving an So the inference happens by solving an So the inference happens by solving an ordinary differential equation which is ordinary differential equation which is ordinary differential equation which is a very unique property of the flow a very unique property of the flow a very unique property of the flow models. models. models. Remember this never happened in Remember this never happened in Remember this never happened in diffusion. This never happened in VA. diffusion. This never happened in VA. diffusion. This never happened in VA. In diffusion the inference happened by In diffusion the inference happened by In diffusion the inference happened by repeated sampling from noise to data. We repeated sampling from noise to data. We repeated sampling from noise to data. We did not sample a differential equation. did not sample a differential equation. did not sample a differential equation. In uh variational autoenccoders we In uh variational autoenccoders we In uh variational autoenccoders we trained a neural network to predict trained a neural network to predict trained a neural network to predict images. There also we did not have a images. There also we did not have a images. There also we did not have a differential equation. This is what differential equation. This is what differential equation. This is what makes flow models unique.

  39. L2 @ 2449s

    Now we have defined the word flow in the Now we have defined the word flow in the initial part of the lecture. We will initial part of the lecture. We will initial part of the lecture. We will understand what is the meaning of flow understand what is the meaning of flow understand what is the meaning of flow models models models which is the whole title of today's which is the whole title of today's which is the whole title of today's lecture is flow models. Right? We have lecture is flow models. Right? We have lecture is flow models. Right? We have just looked at the meaning of the word just looked at the meaning of the word just looked at the meaning of the word flow. flow. So let's start with our original goal of So let's start with our original goal of So let's start with our original goal of deep generative models. deep generative models. deep generative models. Given we have let's say images of cats. Given we have let's say images of cats. Given we have let's say images of cats. We want to find the distribution which We want to find the distribution which We want to find the distribution which governs these images. governs these images. governs these images. So our goal is to find a complex So our goal is to find a complex So our goal is to find a complex distribution which is P data. But we do distribution which is P data. But we do distribution which is P data. But we do not have P data. All we have is a simple not have P data. All we have is a simple not have P data. All we have is a simple distribution which is P init which maybe distribution which is P init which maybe distribution which is P init which maybe it is a Gshian distribution. it is a Gshian distribution. it is a Gshian distribution. Remember what we did in diffusion Remember what we did in diffusion Remember what we did in diffusion models. We started with noise and we models. We started with noise and we models. We started with noise and we ended with the data distribution. ended with the data distribution. ended with the data distribution. So this is exactly what we are going to

  40. L2 @ 2501s

    So this is exactly what we are going to So this is exactly what we are going to do in flow models. Also our goal is to do in flow models. Also our goal is to do in flow models. Also our goal is to convert a simple distribution pinet to a convert a simple distribution pinet to a convert a simple distribution pinet to a complex distribution which is P data. complex distribution which is P data. complex distribution which is P data. But we do not know what is P data. Okay. So now let us understand how we Okay. So now let us understand how we are going to convert the simple are going to convert the simple are going to convert the simple distribution to a complex distribution distribution to a complex distribution distribution to a complex distribution and the simulation of a ordinary and the simulation of a ordinary and the simulation of a ordinary differential equation is a natural differential equation is a natural differential equation is a natural choice for this transformation. A flow model is described by the A flow model is described by the following ordinary differential following ordinary differential following ordinary differential equation. Okay. So Okay. So how the ordinary differential is how the ordinary differential is how the ordinary differential is equation is given by is equation is given by is equation is given by is x0 is the initial distribution that we x0 is the initial distribution that we x0 is the initial distribution that we start out with. It can be complete start out with. It can be complete start out with. It can be complete noise. It can be a gshian distribution. noise. It can be a gshian distribution. noise. It can be a gshian distribution. And And And now instead of making the particles now instead of making the particles now instead of making the particles evolve with time according to evolve with time according to evolve with time according to differential equation. differential equation. Now the main difference here is that we Now the main difference here is that we Now the main difference here is that we make the probability distribution itself make the probability distribution itself make the probability distribution itself change with time. change with time. change with time. So we use the same ordinary differential

  41. L2 @ 2585s

    So we use the same ordinary differential So we use the same ordinary differential equation dx by dt is equal to u. But equation dx by dt is equal to u. But equation dx by dt is equal to u. But this time the main difference is that so this time the main difference is that so this time the main difference is that so far we have been looking at individual far we have been looking at individual far we have been looking at individual particles right. We have been looking at particles right. We have been looking at particles right. We have been looking at the movement of those particles. the movement of those particles. the movement of those particles. Now what we are doing is Now what we are doing is Now what we are doing is we still have particles but these we still have particles but these we still have particles but these particles are a part of a distribution particles are a part of a distribution particles are a part of a distribution of a probability distribution. of a probability distribution. of a probability distribution. So we are still evolving the So we are still evolving the So we are still evolving the trajectories of the particles but now trajectories of the particles but now trajectories of the particles but now these particles as a unit are these particles as a unit are these particles as a unit are representing something. representing something. representing something. For example let's say we have 100 For example let's say we have 100 For example let's say we have 100 particles in a space in a box. Okay. And particles in a space in a box. Okay. And particles in a space in a box. Okay. And these particles are evolving with time. these particles are evolving with time. these particles are evolving with time. they are going in space according to the they are going in space according to the they are going in space according to the velocity field. We have just understood velocity field. We have just understood velocity field. We have just understood how that works. Now the only thing which how that works. Now the only thing which how that works. Now the only thing which we have added in flow model is that the

  42. L2 @ 2638s

    we have added in flow model is that the we have added in flow model is that the collection of these particles represent collection of these particles represent collection of these particles represent a distribution. a distribution. a distribution. So initially we start with a normal So initially we start with a normal So initially we start with a normal distribution which is a gshian distribution which is a gshian distribution which is a gshian distribution and then we have to make distribution and then we have to make distribution and then we have to make the particles flow such that from the the particles flow such that from the the particles flow such that from the normal distribution you move to the normal distribution you move to the normal distribution you move to the distribution that you want to calculate. distribution that you want to calculate. distribution that you want to calculate. That is why it is called a flow model. A That is why it is called a flow model. A That is why it is called a flow model. A flow is a collection of particles flow is a collection of particles flow is a collection of particles evolving according to trajectories. A evolving according to trajectories. A evolving according to trajectories. A flow model is a collection of particles flow model is a collection of particles flow model is a collection of particles which take the initial distribution and which take the initial distribution and which take the initial distribution and convert it into the actual probability convert it into the actual probability convert it into the actual probability distribution which we want to predict. distribution which we want to predict. distribution which we want to predict. So this is what I have mentioned here. So this is what I have mentioned here. So this is what I have mentioned here. Our goal is to make the end point x1 of Our goal is to make the end point x1 of Our goal is to make the end point x1 of the trajectory have the distribution p the trajectory have the distribution p the trajectory have the distribution p data. data. data. In all of these models, the time always In all of these models, the time always In all of these models, the time always evolves from t =0 to t = 1.

  43. L2 @ 2694s

    evolves from t =0 to t = 1. evolves from t =0 to t = 1. So this is sort of what we want. We want So this is sort of what we want. We want So this is sort of what we want. We want to start from the initial distribution to start from the initial distribution to start from the initial distribution and we want to move at x1. We want to and we want to move at x1. We want to and we want to move at x1. We want to move to the data distribution. move to the data distribution. move to the data distribution. Now all of you might have some ideas in Now all of you might have some ideas in Now all of you might have some ideas in your mind related to flow models and you your mind related to flow models and you your mind related to flow models and you might have visualized this in a certain might have visualized this in a certain might have visualized this in a certain way in your mind but we cannot truly way in your mind but we cannot truly way in your mind but we cannot truly understand this unless we look at some understand this unless we look at some understand this unless we look at some practical examples. practical examples. practical examples. So there is an extremely interesting So there is an extremely interesting So there is an extremely interesting Google collab notebook which uh Google collab notebook which uh Google collab notebook which uh I have opened up I have opened up I have opened up here. Now here the interesting thing is here. Now here the interesting thing is here. Now here the interesting thing is first let us run the first cell. So I'm going to show you how to convert So I'm going to show you how to convert probability distributions from an probability distributions from an probability distributions from an initial distribution to a final initial distribution to a final initial distribution to a final distribution which you want to predict. Okay. So now here you can see this is Okay. So now here you can see this is the start distribution which is a simple the start distribution which is a simple

  44. L2 @ 2757s

    the start distribution which is a simple gshian. It is uniformly distributed. gshian. It is uniformly distributed. gshian. It is uniformly distributed. There are maybe around 2,000 particles There are maybe around 2,000 particles There are maybe around 2,000 particles here. So there are 2,000 particles in here. So there are 2,000 particles in here. So there are 2,000 particles in the space. And these particles represent the space. And these particles represent the space. And these particles represent a distribution. They represent a uniform a distribution. They represent a uniform a distribution. They represent a uniform distribution. distribution. distribution. Okay. And now Okay. And now Okay. And now let us say the distribution that we want let us say the distribution that we want let us say the distribution that we want to predict which is X1. It looks like to predict which is X1. It looks like to predict which is X1. It looks like this. It is the shape of a heart. Now this. It is the shape of a heart. Now this. It is the shape of a heart. Now this also has 200 points. But it has a this also has 200 points. But it has a this also has 200 points. But it has a very specific distribution, right? It is very specific distribution, right? It is very specific distribution, right? It is it it is not like uniform distribution it it is not like uniform distribution it it is not like uniform distribution but it is our data distribution. but it is our data distribution. but it is our data distribution. In reality we will not be knowing this In reality we will not be knowing this In reality we will not be knowing this also but now I'm assuming that we know also but now I'm assuming that we know also but now I'm assuming that we know the start distribution and we know the the start distribution and we know the the start distribution and we know the end distribution and we are going to end distribution and we are going to end distribution and we are going to understand how a flow model takes you understand how a flow model takes you understand how a flow model takes you from the start distribution to the end from the start distribution to the end

  45. L2 @ 2807s

    from the start distribution to the end distribution. Now here there are a lot of options. We Now here there are a lot of options. We can convert can convert can convert we can change the end distribution to we can change the end distribution to we can change the end distribution to maybe a spiral data also. We can change maybe a spiral data also. We can change maybe a spiral data also. We can change it to uh goshian data. But again gshian it to uh goshian data. But again gshian it to uh goshian data. But again gshian data will not make sense. So we have two data will not make sense. So we have two data will not make sense. So we have two options maybe spiral data and hard data options maybe spiral data and hard data options maybe spiral data and hard data over here. over here. over here. Now uh there are some parts here which Now uh there are some parts here which Now uh there are some parts here which will become clear as we go along in the will become clear as we go along in the will become clear as we go along in the next two lectures. next two lectures. next two lectures. For now here I have defined the flow For now here I have defined the flow For now here I have defined the flow model by a neural network which takes a model by a neural network which takes a model by a neural network which takes a position x and time t and outputs a position x and time t and outputs a position x and time t and outputs a velocity vector. This is the velocity velocity vector. This is the velocity velocity vector. This is the velocity vector field which we have been talking vector field which we have been talking vector field which we have been talking about since the start of the lecture. about since the start of the lecture. about since the start of the lecture. Now one crucial thing we have done here Now one crucial thing we have done here Now one crucial thing we have done here is the velocity vector field has been is the velocity vector field has been is the velocity vector field has been replaced by a neural network.

  46. L2 @ 2862s

    replaced by a neural network. replaced by a neural network. So there is another word for this in So there is another word for this in So there is another word for this in literature which is also called as a literature which is also called as a literature which is also called as a neural OD neural OD neural OD which is a neural ordinary differential which is a neural ordinary differential which is a neural ordinary differential equation. equation. It is very similar to what we discussed It is very similar to what we discussed It is very similar to what we discussed before. The only thing is that the input before. The only thing is that the input before. The only thing is that the input is the same X and T. The output is still is the same X and T. The output is still is the same X and T. The output is still the velocity vector but inside you have the velocity vector but inside you have the velocity vector but inside you have a bunch of neurons. Now this velocity field is defined here. Now this velocity field is defined here. So let me run this. Um and then we carry So let me run this. Um and then we carry So let me run this. Um and then we carry the training process. The training the training process. The training the training process. The training process is very interesting and it's process is very interesting and it's process is very interesting and it's extremely simple but as I said before to extremely simple but as I said before to extremely simple but as I said before to reach this simple point is going to take reach this simple point is going to take reach this simple point is going to take us some time. So right now I don't us some time. So right now I don't us some time. So right now I don't expect you to understand the training expect you to understand the training expect you to understand the training process but here you can see the flow process but here you can see the flow process but here you can see the flow model is being trained and the loss is model is being trained and the loss is

  47. L2 @ 2909s

    model is being trained and the loss is going down. What is the meaning of this going down. What is the meaning of this going down. What is the meaning of this statement? The flow model is going to be statement? The flow model is going to be statement? The flow model is going to be trained. The meaning is that we are trained. The meaning is that we are trained. The meaning is that we are training the neural network given by the training the neural network given by the training the neural network given by the velocity vector field which are going to velocity vector field which are going to velocity vector field which are going to take these 2,000 particles in the take these 2,000 particles in the take these 2,000 particles in the initial uniform shape. They are going to initial uniform shape. They are going to initial uniform shape. They are going to evolve the trajectories according to our evolve the trajectories according to our evolve the trajectories according to our differential equation such that the differential equation such that the differential equation such that the final shape will be the heart shape. Next step is to simulate the OD. Now Next step is to simulate the OD. Now this is where it gets very interesting this is where it gets very interesting this is where it gets very interesting and we use the uler step here. You see and we use the uler step here. You see and we use the uler step here. You see here how we are changing the here how we are changing the here how we are changing the uh trajectories of the particles uh trajectories of the particles uh trajectories of the particles according to the oiler step according to the oiler step according to the oiler step and we can see the real data and the and we can see the real data and the and we can see the real data and the generated data. A great way to visualize generated data. A great way to visualize generated data. A great way to visualize this is to see the animation and we can this is to see the animation and we can this is to see the animation and we can see the animation over here once this see the animation over here once this

  48. L2 @ 2965s

    see the animation over here once this code runs. Yeah. So let's see uh as expected we Yeah. So let's see uh as expected we start with the initial gshian start with the initial gshian start with the initial gshian distribution and then slowly the distribution and then slowly the distribution and then slowly the particles evolve and we start to she see particles evolve and we start to she see particles evolve and we start to she see the heart-shaped the heart-shaped the heart-shaped structure. So structure. So structure. So you can see how the particles are slowly you can see how the particles are slowly you can see how the particles are slowly compressing inside moving in and then compressing inside moving in and then compressing inside moving in and then also changing their shape or or moving also changing their shape or or moving also changing their shape or or moving according to lines where they start to according to lines where they start to according to lines where they start to resemble a heart shape. resemble a heart shape. resemble a heart shape. So this is this is very interesting and So this is this is very interesting and So this is this is very interesting and it's almost like what we have done here it's almost like what we have done here it's almost like what we have done here is we have predicted a velocity field is we have predicted a velocity field is we have predicted a velocity field which can take a uniform distribution to which can take a uniform distribution to which can take a uniform distribution to a predicted distribution which we want a predicted distribution which we want a predicted distribution which we want and we can imagine the flow. The and we can imagine the flow. The and we can imagine the flow. The particles are flowing in in time. particles are flowing in in time. particles are flowing in in time. In variation autoenccoders we had an In variation autoenccoders we had an In variation autoenccoders we had an encoder we had a decoder. In diffusion encoder we had a decoder. In diffusion encoder we had a decoder. In diffusion models we had a number of encoders and a models we had a number of encoders and a

  49. L2 @ 3030s

    models we had a number of encoders and a number of decoders. number of decoders. number of decoders. So they were discrete. Here something a So they were discrete. Here something a So they were discrete. Here something a process happens which is continuous. So process happens which is continuous. So process happens which is continuous. So I find this transformation from the I find this transformation from the I find this transformation from the discrete process to a continuous process discrete process to a continuous process discrete process to a continuous process very very interesting in these flow very very interesting in these flow very very interesting in these flow models. models. And uh now what we'll do is let's change And uh now what we'll do is let's change And uh now what we'll do is let's change this target distribution and we'll see this target distribution and we'll see this target distribution and we'll see what happens with the spiral data. Yeah. So now the end distribution is a Yeah. So now the end distribution is a spiral data. We'll again define the spiral data. We'll again define the spiral data. We'll again define the neural vector field. We'll train this neural vector field. We'll train this neural vector field. We'll train this velocity field now. velocity field now. velocity field now. And uh we will see the loss decreases as And uh we will see the loss decreases as And uh we will see the loss decreases as the number of epochs increase. The training is complete. Now we will The training is complete. Now we will simulate the OD simulate the OD simulate the OD using the oiler method. And let's see using the oiler method. And let's see using the oiler method. And let's see the animation. [snorts] Let's see. Let's see. Let's see. So now you see the particles are So now you see the particles are So now you see the particles are revolving and slowly moving into a shape revolving and slowly moving into a shape revolving and slowly moving into a shape which represents a spiral. which represents a spiral. which represents a spiral. Instead of the heart-shaped uh there are Instead of the heart-shaped uh there are Instead of the heart-shaped uh there are the particles who are compressing inside the particles who are compressing inside

  50. L2 @ 3123s

    the particles who are compressing inside here they are moving out and here they are moving out and here they are moving out and slowly resembling that spiral shape. So slowly resembling that spiral shape. So slowly resembling that spiral shape. So this is exactly what we mean by flow this is exactly what we mean by flow this is exactly what we mean by flow models. models. Okay. Let us recap what we did in Okay. Let us recap what we did in Okay. Let us recap what we did in today's lecture. today's lecture. today's lecture. In today's lecture, we first started In today's lecture, we first started In today's lecture, we first started with the concept of time dependent with the concept of time dependent with the concept of time dependent velocity fields. Why we started with the velocity fields. Why we started with the velocity fields. Why we started with the concept of that? Because once we have a concept of that? Because once we have a concept of that? Because once we have a timed dependent velocity field, we can timed dependent velocity field, we can timed dependent velocity field, we can calculate the trajectories of all the calculate the trajectories of all the calculate the trajectories of all the particles in space according to that particles in space according to that particles in space according to that velocity field. How do we do that? We velocity field. How do we do that? We velocity field. How do we do that? We use the the oiler method to do that use the the oiler method to do that use the the oiler method to do that where we discretize the differential where we discretize the differential where we discretize the differential equation and we see how the particles equation and we see how the particles equation and we see how the particles move in time. move in time. move in time. Now the question is okay this is fine we Now the question is okay this is fine we Now the question is okay this is fine we have defined the velocity vector field have defined the velocity vector field have defined the velocity vector field but how how do we solve our problem but how how do we solve our problem but how how do we solve our problem initial problem of converting the uh

  51. L2 @ 3176s

    initial problem of converting the uh initial problem of converting the uh gshian distribution to our actual gshian distribution to our actual gshian distribution to our actual distribution. What we say is that we use distribution. What we say is that we use distribution. What we say is that we use the same concept of a velocity field the same concept of a velocity field the same concept of a velocity field and instead of changing the trajectories and instead of changing the trajectories and instead of changing the trajectories of the particles here we will change the of the particles here we will change the of the particles here we will change the trajectories of particles but we will trajectories of particles but we will trajectories of particles but we will transform the original distribution to a transform the original distribution to a transform the original distribution to a final distribution final distribution final distribution according to the trajectories of the according to the trajectories of the according to the trajectories of the particles which govern that particles which govern that particles which govern that distribution. distribution. And we have a technical word for this And we have a technical word for this And we have a technical word for this which governs the trajectories of all which governs the trajectories of all which governs the trajectories of all these particles which is called as a these particles which is called as a these particles which is called as a flow. flow. And what is a flow model? A flow model And what is a flow model? A flow model And what is a flow model? A flow model is a model which gives us the velocity is a model which gives us the velocity is a model which gives us the velocity vector field vector field vector field that will be used to calculate the that will be used to calculate the that will be used to calculate the trajectories of all the particles trajectories of all the particles trajectories of all the particles and this velocity vector field is often and this velocity vector field is often and this velocity vector field is often times a neural network. times a neural network. times a neural network. So the differential equation is also So the differential equation is also

  52. L2 @ 3230s

    So the differential equation is also termed as a neural ordinary differential termed as a neural ordinary differential termed as a neural ordinary differential equation. So the terms that I have equation. So the terms that I have equation. So the terms that I have introduced in this lecture are first is introduced in this lecture are first is introduced in this lecture are first is ordinary differential equation, neural ordinary differential equation, neural ordinary differential equation, neural ordinary differential equation, vector ordinary differential equation, vector ordinary differential equation, vector field, velocity vector field, timed field, velocity vector field, timed field, velocity vector field, timed dependent velocity vector field, flow dependent velocity vector field, flow dependent velocity vector field, flow and flow models. All of these terms mean and flow models. All of these terms mean and flow models. All of these terms mean something very specific and I want you something very specific and I want you something very specific and I want you to understand these terminologies in to understand these terminologies in to understand these terminologies in detail. detail. detail. Also to give you a vision of where we Also to give you a vision of where we Also to give you a vision of where we are heading something happened in this are heading something happened in this are heading something happened in this training step which allowed us to and training step which allowed us to and training step which allowed us to and this is so simple you can see it's this is so simple you can see it's this is so simple you can see it's hardly 20 lines hardly 20 lines hardly 20 lines and that is exactly the simplicity of and that is exactly the simplicity of and that is exactly the simplicity of the flow modeling approach which we will the flow modeling approach which we will the flow modeling approach which we will get at slowly but we will do it in a get at slowly but we will do it in a get at slowly but we will do it in a very step-by-step manner. very step-by-step manner. very step-by-step manner. Thank you very much everyone and uh this Thank you very much everyone and uh this Thank you very much everyone and uh this ends our first lecture and in the next

  53. L2 @ 3283s

    ends our first lecture and in the next ends our first lecture and in the next lecture we will understand what is the lecture we will understand what is the lecture we will understand what is the target of our flow model. How do we target of our flow model. How do we target of our flow model. How do we define the target and how do we train define the target and how do we train define the target and how do we train the flow models as well.