Velocity Field — across courses
cmu-10799-diffusion-flow
- L10 @ 1924s
and G GMMN can also be seen as a special and G GMMN can also be seen as a special case of Imm. Uh case of Imm. Uh case of Imm. Uh you can easily see that when if you set you can easily see that when if you set you can easily see that when if you set TB uh to be identically equal to one and TB uh to be identically equal to one and TB uh to be identically equal to one and S and R to be identically equal to zero S and R to be identically equal to zero S and R to be identically equal to zero and you will exactly recover GMN's and you will exactly recover GMN's and you will exactly recover GMN's training objective. training objective. training objective. But of course it's it's difficult to get But of course it's it's difficult to get But of course it's it's difficult to get it work uh to get it to work because it work uh to get it to work because it work uh to get it to work because well you have the same pro problem with well you have the same pro problem with well you have the same pro problem with MMD on in high emission uh where you MMD on in high emission uh where you MMD on in high emission uh where you have very little training signal. So the have very little training signal. So the have very little training signal. So the inductive learning uh component is is is inductive learning uh component is is is inductive learning uh component is is is a very necessary component a very necessary component a very necessary component and uh if you are following the and uh if you are following the and uh if you are following the literature closely uh you also may have literature closely uh you also may have literature closely uh you also may have seen this work uh from from Kimhood's seen this work uh from from Kimhood's seen this work uh from from Kimhood's group uh called drifting model uh and group uh called drifting model uh and group uh called drifting model uh and essentially in this paper the drift essentially in this paper the drift
- L10 @ 1973s
essentially in this paper the drift field for their uh for their model is field for their uh for their model is field for their uh for their model is also parameterized via uh via MMD also parameterized via uh via MMD also parameterized via uh via MMD attraction plus MMD repulsion. So it's attraction plus MMD repulsion. So it's attraction plus MMD repulsion. So it's it's quite interesting that you know MMD it's quite interesting that you know MMD it's quite interesting that you know MMD has now uh is now getting some some has now uh is now getting some some has now uh is now getting some some attention from from a more modern attention from from a more modern attention from from a more modern perspective. Yeah. And and if you have perspective. Yeah. And and if you have perspective. Yeah. And and if you have not uh read this paper I highly not uh read this paper I highly not uh read this paper I highly recommend it. recommend it. recommend it. And And and before moving on uh I I also want to and before moving on uh I I also want to and before moving on uh I I also want to note several limitations of IMM. Um note several limitations of IMM. Um note several limitations of IMM. Um and one one is that uh MMD by design and one one is that uh MMD by design and one one is that uh MMD by design needs multiple samples to calculate the needs multiple samples to calculate the needs multiple samples to calculate the objective. So this adds some engineering objective. So this adds some engineering objective. So this adds some engineering burden at at large scale. Uh because say burden at at large scale. Uh because say burden at at large scale. Uh because say you are training a a large video model you are training a a large video model you are training a a large video model uh that that has larger of uh you know uh that that has larger of uh you know uh that that has larger of uh you know number of parameters the the batch size number of parameters the the batch size number of parameters the the batch size per GPU can can usually be as low as
- L5 @ 2s
So last time we learned about how So last time we learned about how diffusion turn noise into data by diffusion turn noise into data by diffusion turn noise into data by gradually adding noise and gradually gradually adding noise and gradually gradually adding noise and gradually denoising. denoising. denoising. H and then we learned another family of H and then we learned another family of H and then we learned another family of model called a scorebased models which model called a scorebased models which model called a scorebased models which is uh doing very very similar things and is uh doing very very similar things and is uh doing very very similar things and it turns out that they're they're pretty it turns out that they're they're pretty it turns out that they're they're pretty much the same. Um if you especially uh much the same. Um if you especially uh much the same. Um if you especially uh if you scale the number of time steps to if you scale the number of time steps to if you scale the number of time steps to infinite and then they all they both infinite and then they all they both infinite and then they all they both becomes this thing called stochastic becomes this thing called stochastic becomes this thing called stochastic process a continuous time stoastic process a continuous time stoastic process a continuous time stoastic process which can be sort of formulated process which can be sort of formulated process which can be sort of formulated into stoastic differential equations. into stoastic differential equations. into stoastic differential equations. All right. So and if you have a forward All right. So and if you have a forward All right. So and if you have a forward SDE uh then you actually directly have a SDE uh then you actually directly have a SDE uh then you actually directly have a formula to uh tell you what your reverse formula to uh tell you what your reverse formula to uh tell you what your reverse process should be which is also defined process should be which is also defined process should be which is also defined by an SDE with this score function and by an SDE with this score function and
- L5 @ 52s
by an SDE with this score function and then we can just do score matching to then we can just do score matching to then we can just do score matching to learn uh how to solve the uh the learn uh how to solve the uh the learn uh how to solve the uh the reversed SDE. reversed SDE. reversed SDE. All right. So, that was like the last All right. So, that was like the last All right. So, that was like the last question question question that we had in class. It's like, is that we had in class. It's like, is that we had in class. It's like, is there a simpler way to do this? And I there a simpler way to do this? And I there a simpler way to do this? And I think someone after they took the quiz, think someone after they took the quiz, think someone after they took the quiz, they were already be like, "Oh, I I feel they were already be like, "Oh, I I feel they were already be like, "Oh, I I feel like I feel like there should be an like I feel like there should be an like I feel like there should be an easier way." Um, yeah. So, this is what easier way." Um, yeah. So, this is what easier way." Um, yeah. So, this is what we're going to learn today. Okay. So, uh we're going to learn today. Okay. So, uh we're going to learn today. Okay. So, uh let's think about this question. Let's let's think about this question. Let's let's think about this question. Let's say we're given a data point. So, we we say we're given a data point. So, we we say we're given a data point. So, we we only have like one image, right? What only have like one image, right? What only have like one image, right? What would be the simplest way to construct a would be the simplest way to construct a would be the simplest way to construct a trajectory from noise to this data trajectory from noise to this data trajectory from noise to this data point? Yeah, point? Yeah, point? Yeah, linear combination. linear combination. linear combination. That's exactly correct. Okay. Why don't That's exactly correct. Okay. Why don't
- L5 @ 101s
That's exactly correct. Okay. Why don't we just do linear interpolation of the we just do linear interpolation of the we just do linear interpolation of the two things, right? Um so exact how two things, right? Um so exact how two things, right? Um so exact how exactly you can do it is basically um exactly you can do it is basically um exactly you can do it is basically um say like bas you just like at the say like bas you just like at the say like bas you just like at the beginning of time you'll have like it's beginning of time you'll have like it's beginning of time you'll have like it's so this noise is fixed and then you have so this noise is fixed and then you have so this noise is fixed and then you have like a large portion of noise mixed with like a large portion of noise mixed with like a large portion of noise mixed with some portion of data and then you get a some portion of data and then you get a some portion of data and then you get a very noisy data and then as you go on very noisy data and then as you go on very noisy data and then as you go on the portion of the noise become fewer the portion of the noise become fewer the portion of the noise become fewer like less and less and then the portion like less and less and then the portion like less and less and then the portion of data become more and more uh and then of data become more and more uh and then of data become more and more uh and then you just construct a chain of you just construct a chain of you just construct a chain of the very noisy to slightly no to the very noisy to slightly no to the very noisy to slightly no to slightly less noisy to very very clean slightly less noisy to very very clean slightly less noisy to very very clean data. Right? So you could just construct data. Right? So you could just construct data. Right? So you could just construct this trajectory in this way. Um yeah so this trajectory in this way. Um yeah so
- L5 @ 146s
this trajectory in this way. Um yeah so remember the same noise same data. remember the same noise same data. remember the same noise same data. Then learning the transformation along Then learning the transformation along Then learning the transformation along this linear trajectory is super easy, this linear trajectory is super easy, this linear trajectory is super easy, right? Because literally what we can do right? Because literally what we can do right? Because literally what we can do is say now we want to uh now we want to is say now we want to uh now we want to is say now we want to uh now we want to um transfer transform from uh time 25% um transfer transform from uh time 25% um transfer transform from uh time 25% to time 50%. uh then what can we do is to time 50%. uh then what can we do is to time 50%. uh then what can we do is we can we literally just have you just we can we literally just have you just we can we literally just have you just need to add 25% more of data and then need to add 25% more of data and then need to add 25% more of data and then minus 25% minus 25% minus 25% uh of of the noise right um so basically uh of of the noise right um so basically uh of of the noise right um so basically and because the delta t is the same so and because the delta t is the same so and because the delta t is the same so you can write it in this way basically you can write it in this way basically you can write it in this way basically just like the just like the just like the uh the the noisy sample at time uh 0.5 uh the the noisy sample at time uh 0.5 uh the the noisy sample at time uh 0.5 is equal to uh the noisy sample at time is equal to uh the noisy sample at time is equal to uh the noisy sample at time 0.25 plus how much time you want to 0.25 plus how much time you want to 0.25 plus how much time you want to progress. So like 0.25 time um times uh
- L5 @ 208s
progress. So like 0.25 time um times uh progress. So like 0.25 time um times uh data minus noise, right? Uh so basically data minus noise, right? Uh so basically data minus noise, right? Uh so basically uh generalizing it into t and delta t. uh generalizing it into t and delta t. uh generalizing it into t and delta t. This is like what you should get. And uh This is like what you should get. And uh This is like what you should get. And uh literally when the delta t goes to zero literally when the delta t goes to zero literally when the delta t goes to zero then you will have this velocity thing then you will have this velocity thing then you will have this velocity thing right. Basically this is just like tells right. Basically this is just like tells right. Basically this is just like tells the velocity thing. just tells you the velocity thing. just tells you the velocity thing. just tells you infinite testimony infinite testimony infinite testimony uh if you take infinite testimony small uh if you take infinite testimony small uh if you take infinite testimony small steps like what is the direction that steps like what is the direction that steps like what is the direction that you should go and no matter what t you should go and no matter what t you should go and no matter what t you're at no matter which time step you're at no matter which time step you're at no matter which time step you're at you're always going like the you're at you're always going like the you're at you're always going like the same speed same direction right so it same speed same direction right so it same speed same direction right so it just like always going to be the just like always going to be the just like always going to be the difference between the data and noise difference between the data and noise difference between the data and noise which is x1 minus minus x0 which is x1 minus minus x0 which is x1 minus minus x0 okay any questions okay any questions okay any questions pretty straightforward right okay So pretty straightforward right okay So pretty straightforward right okay So this is literally what we need to learn,
- L5 @ 260s
this is literally what we need to learn, this is literally what we need to learn, right? This is like literally the only right? This is like literally the only right? This is like literally the only thing that that's stopping us from thing that that's stopping us from thing that that's stopping us from directly getting from noise to data. So directly getting from noise to data. So directly getting from noise to data. So basically the simplest way essentially basically the simplest way essentially basically the simplest way essentially to transform noise straight into data to transform noise straight into data to transform noise straight into data will be at training time. will be at training time. will be at training time. You first sample a noise and then you You first sample a noise and then you You first sample a noise and then you sample a data and then you uniformly sample a data and then you uniformly sample a data and then you uniformly sample a time step and then you compute sample a time step and then you compute sample a time step and then you compute a noisy sample by doing this linear a noisy sample by doing this linear a noisy sample by doing this linear interpolation interpolation interpolation and then you compute the velocity which and then you compute the velocity which and then you compute the velocity which is x1 - x0 just data minus noise and is x1 - x0 just data minus noise and is x1 - x0 just data minus noise and then you learn a then you learn a then you learn a to match this velocity to match this velocity to match this velocity and at sampling time after you learn and at sampling time after you learn and at sampling time after you learn this literally you can just start from t this literally you can just start from t this literally you can just start from t equals z and then sample a noise at time equals z and then sample a noise at time equals z and then sample a noise at time zero zero zero and then progress through the chain. and then progress through the chain. and then progress through the chain. Right? How do you progress through the Right? How do you progress through the
- L5 @ 317s
Right? How do you progress through the chain? You just compute the displacement chain? You just compute the displacement chain? You just compute the displacement for x and then directly apply the for x and then directly apply the for x and then directly apply the displacement to your x to progress on displacement to your x to progress on displacement to your x to progress on the chain and then also progress in time the chain and then also progress in time the chain and then also progress in time steps and at the end just output x1. steps and at the end just output x1. steps and at the end just output x1. Okay. Question. Okay. Question. Okay. Question. somewhere and I'm concerned that this noise has no and I'm concerned that this noise has no semantic meaning. So would it get like semantic meaning. So would it get like semantic meaning. So would it get like like there's no reason why a particular like there's no reason why a particular like there's no reason why a particular gausian noise shouldn't amount to the gausian noise shouldn't amount to the gausian noise shouldn't amount to the rather than the house? rather than the house? rather than the house? >> Cool question. We're going to we're >> Cool question. We're going to we're >> Cool question. We're going to we're going to learn that today. Uh first of going to learn that today. Uh first of going to learn that today. Uh first of all, we're not trying to get any all, we're not trying to get any all, we're not trying to get any semantic meaning here. We're just trying semantic meaning here. We're just trying semantic meaning here. We're just trying to transform noise into data. But second to transform noise into data. But second to transform noise into data. But second of all, uh I think we're going to like of all, uh I think we're going to like of all, uh I think we're going to like answer your question um later today. answer your question um later today. answer your question um later today. Okay, any other questions? Ah, now you have flow matching. This is Ah, now you have flow matching. This is full matching. Class is over. Hey, full matching. Class is over. Hey,
- L5 @ 386s
full matching. Class is over. Hey, no, not really. Okay, fine. Uh but no, not really. Okay, fine. Uh but no, not really. Okay, fine. Uh but basically the the question that we're basically the the question that we're basically the the question that we're gonna ask is that like this is so gonna ask is that like this is so gonna ask is that like this is so simple, right? Like no math, no nothing. simple, right? Like no math, no nothing. simple, right? Like no math, no nothing. Like why do we like why do we need all Like why do we like why do we need all Like why do we like why do we need all those previous diffusion math like those previous diffusion math like those previous diffusion math like stoastic differential equations? Why do stoastic differential equations? Why do stoastic differential equations? Why do we need this? Like also why is this even we need this? Like also why is this even we need this? Like also why is this even valid? Why why why is this the same? Why valid? Why why why is this the same? Why valid? Why why why is this the same? Why can we even do this? Right? If we can can we even do this? Right? If we can can we even do this? Right? If we can just do this, why do we need to do all just do this, why do we need to do all just do this, why do we need to do all the math? Uh well, turns out that thing the math? Uh well, turns out that thing the math? Uh well, turns out that thing also come from math. So we're going to also come from math. So we're going to also come from math. So we're going to learn about the math now. And to understand why this is a proper And to understand why this is a proper um probabilistic gener model we need to um probabilistic gener model we need to um probabilistic gener model we need to go back in time into the time when go back in time into the time when go back in time into the time when people start to de develop this thing. people start to de develop this thing.
- L5 @ 430s
people start to de develop this thing. Okay. So basically back in 2018 there Okay. So basically back in 2018 there Okay. So basically back in 2018 there this like pretty seminal paper um uh this like pretty seminal paper um uh this like pretty seminal paper um uh called continuous normalizing or or like called continuous normalizing or or like called continuous normalizing or or like neuro neuroordinary differential neuro neuroordinary differential neuro neuroordinary differential equation and then basically um this guy equation and then basically um this guy equation and then basically um this guy Ricky who was my uh meta intermentor Ricky who was my uh meta intermentor Ricky who was my uh meta intermentor actually super legendary guy but actually super legendary guy but actually super legendary guy but basically he was like um proposed this basically he was like um proposed this basically he was like um proposed this thing called continuous normalizing thing called continuous normalizing thing called continuous normalizing flows and it's basically just like uh flows and it's basically just like uh flows and it's basically just like uh formulated a general model such that formulated a general model such that formulated a general model such that transform uh the samples from an initial transform uh the samples from an initial transform uh the samples from an initial distribution such as like a gausian to a distribution such as like a gausian to a distribution such as like a gausian to a target distribution such as the image target distribution such as the image target distribution such as the image distribution that we want by integrating distribution that we want by integrating distribution that we want by integrating through an OD through an OD through an OD and this OD is literally just the you and this OD is literally just the you and this OD is literally just the you know the infinite decimal uh change of know the infinite decimal uh change of know the infinite decimal uh change of your data should follow the velocity and your data should follow the velocity and your data should follow the velocity and at sampling time what you can do is you at sampling time what you can do is you at sampling time what you can do is you can just solve this OD by integrating
- L5 @ 485s
can just solve this OD by integrating can just solve this OD by integrating through uh the velocity and then you through uh the velocity and then you through uh the velocity and then you you'll be able to get a sample or like a you'll be able to get a sample or like a you'll be able to get a sample or like a trajectory of samples. Uh trajectory of samples. Uh trajectory of samples. Uh so basically this is like literally what so basically this is like literally what so basically this is like literally what we just did, right? So you have some we just did, right? So you have some we just did, right? So you have some velocity and then you just like velocity and then you just like velocity and then you just like numerically is solve the OD at sampling numerically is solve the OD at sampling numerically is solve the OD at sampling time. time. time. Okay. So like how to interpret this, Okay. So like how to interpret this, Okay. So like how to interpret this, right? So basically just like think of right? So basically just like think of right? So basically just like think of it as like the wind flow transport some it as like the wind flow transport some it as like the wind flow transport some like water some amount of water from the like water some amount of water from the like water some amount of water from the west coast to the east coast or from west coast to the east coast or from west coast to the east coast or from like the Pacific Ocean to the uh to the like the Pacific Ocean to the uh to the like the Pacific Ocean to the uh to the east coast let's just say and um the east coast let's just say and um the east coast let's just say and um the question here is that why this is a question here is that why this is a question here is that why this is a normalizing flow normalizing flow normalizing flow first of all like what is a normalizing first of all like what is a normalizing first of all like what is a normalizing flow? Anyone else I haven't talked to today?
- L5 @ 535s
Anyone else I haven't talked to today? Nobody remember what is normalizing Nobody remember what is normalizing Nobody remember what is normalizing flow? >> You can reconstruct the process. >> You can reconstruct the process. >> Yes. So it's a invertible uh generate >> Yes. So it's a invertible uh generate >> Yes. So it's a invertible uh generate model, right? Yes. And the reason why model, right? Yes. And the reason why model, right? Yes. And the reason why this is a normalizing flow is because this is a normalizing flow is because this is a normalizing flow is because the streams never cross. So imagine how the streams never cross. So imagine how the streams never cross. So imagine how like you're following a stream of river like you're following a stream of river like you're following a stream of river or like water or like the stream of the or like water or like the stream of the or like water or like the stream of the wind, right? Like if you just follow the wind, right? Like if you just follow the wind, right? Like if you just follow the stream, the stream will never like it stream, the stream will never like it stream, the stream will never like it will always point to one direction. So will always point to one direction. So will always point to one direction. So it will never like just get get it will never like just get get it will never like just get get if you just like follow the wind, you're if you just like follow the wind, you're if you just like follow the wind, you're able to basically get from one point to able to basically get from one point to able to basically get from one point to the other point guaranteed. the other point guaranteed. the other point guaranteed. So like deterministically and because So like deterministically and because So like deterministically and because this process is deterministic, if you this process is deterministic, if you this process is deterministic, if you follow this wind, if you follow this follow this wind, if you follow this follow this wind, if you follow this flow, you're actually going to get an flow, you're actually going to get an flow, you're actually going to get an invertible transformation because
- L5 @ 590s
invertible transformation because invertible transformation because everything is deterministic. everything is deterministic. everything is deterministic. Okay, any question? Okay, any question? Okay, any question? Cool. You have a question. Okay. Are we Cool. You have a question. Okay. Are we Cool. You have a question. Okay. Are we saying that it's actually saying that it's actually saying that it's actually >> uh the way that we construct it is is it the way that we construct it is is it yeah it is invertible. Yeah the way that yeah it is invertible. Yeah the way that yeah it is invertible. Yeah the way that we construct it. Okay. we construct it. Okay. we construct it. Okay. Any more question? Cool. All right. Cool. All right. Uh okay. Okay. So how does this connect Uh okay. Okay. So how does this connect Uh okay. Okay. So how does this connect to probability? to probability? to probability? Well, basically in order to for this Well, basically in order to for this Well, basically in order to for this kind of like continuous flow thing model kind of like continuous flow thing model kind of like continuous flow thing model to transport between one probability to transport between one probability to transport between one probability distribution to another we need to have distribution to another we need to have distribution to another we need to have the following assumptions. The first the following assumptions. The first the following assumptions. The first thing is what we call the conservation thing is what we call the conservation thing is what we call the conservation of mass. Basically just imagine that the of mass. Basically just imagine that the of mass. Basically just imagine that the the wind carried a certain amount of the wind carried a certain amount of the wind carried a certain amount of water and this certain amount of water water and this certain amount of water water and this certain amount of water never change like the amount of water never change like the amount of water never change like the amount of water that you transporting never change. So that you transporting never change. So that you transporting never change. So no new mass uh is is going to get added no new mass uh is is going to get added
- L5 @ 661s
no new mass uh is is going to get added and no mass is going to disappear and no mass is going to disappear and no mass is going to disappear getting dropped. So like translate it getting dropped. So like translate it getting dropped. So like translate it into probabilistic statement just means into probabilistic statement just means into probabilistic statement just means that the probability should always add that the probability should always add that the probability should always add up to one because otherwise it won't be up to one because otherwise it won't be up to one because otherwise it won't be a valid probability distribution. And a valid probability distribution. And a valid probability distribution. And then second thing is what we call the then second thing is what we call the then second thing is what we call the continuity equation which is like not continuity equation which is like not continuity equation which is like not only that the mass is conserved it's only that the mass is conserved it's only that the mass is conserved it's also not you cannot teleport them uh also not you cannot teleport them uh also not you cannot teleport them uh which means that everything need to move which means that everything need to move which means that everything need to move smoothly everything should move like smoothly everything should move like smoothly everything should move like continuously continuously continuously okay so basically just means that the okay so basically just means that the okay so basically just means that the probability can only get changed probability can only get changed probability can only get changed continuously you cannot just like jump continuously you cannot just like jump continuously you cannot just like jump from from point to point because it's from from point to point because it's from from point to point because it's it's a river okay any question it's a river okay any question it's a river okay any question cool all right Um so because of this we cool all right Um so because of this we cool all right Um so because of this we get to define something called a get to define something called a get to define something called a probability flux. So imagine a flux is probability flux. So imagine a flux is
- L5 @ 710s
probability flux. So imagine a flux is like the amount of flow the amount of like the amount of flow the amount of like the amount of flow the amount of water uh per unit per time that's going water uh per unit per time that's going water uh per unit per time that's going through a certain a certain uh unit through a certain a certain uh unit through a certain a certain uh unit space. So if you're thinking about like space. So if you're thinking about like space. So if you're thinking about like how the probability flows is a how the probability flows is a how the probability flows is a probability flux is equal to the probability flux is equal to the probability flux is equal to the velocity times the the the density sort velocity times the the the density sort velocity times the the the density sort of. So the velocity is giving you where of. So the velocity is giving you where of. So the velocity is giving you where and how fast the probability flows and and how fast the probability flows and and how fast the probability flows and then the density is telling you how much then the density is telling you how much then the density is telling you how much probability there was so that probability there was so that probability there was so that it flows out right um okay so this is it flows out right um okay so this is it flows out right um okay so this is like a like a like a um visualization generated by vo I um visualization generated by vo I um visualization generated by vo I already tried but let's just uh look at already tried but let's just uh look at already tried but let's just uh look at it okay okay maybe I should put on okay do that maybe I should put on okay do that maybe I should put on okay do that All right. Okay. So, as you can see, as the thing Okay. So, as you can see, as the thing as the water flows, first of all, it's as the water flows, first of all, it's as the water flows, first of all, it's in a container. It's in like a contained
- L5 @ 775s
in a container. It's in like a contained in a container. It's in like a contained tank. So, the probability mass never tank. So, the probability mass never tank. So, the probability mass never change. And second, like basically based change. And second, like basically based change. And second, like basically based on how fast the particle flows um like on how fast the particle flows um like on how fast the particle flows um like the probability density, which is like the probability density, which is like the probability density, which is like the water amount at this particular the water amount at this particular the water amount at this particular location, also changes. location, also changes. location, also changes. Okay, any questions? >> Flux in this case like similar to like >> Flux in this case like similar to like magnetic flu magnetic flu magnetic flu like it's like the amount of like like it's like the amount of like like it's like the amount of like >> Nope. >> Nope. >> Nope. >> It's a it just I think water is a better >> It's a it just I think water is a better >> It's a it just I think water is a better um um um I I mean I I mean I I mean >> like >> like >> like >> Yeah. Yeah. Kind of >> Yeah. Yeah. Kind of >> Yeah. Yeah. Kind of >> kind of. Yeah. It's like amount of flows >> kind of. Yeah. It's like amount of flows >> kind of. Yeah. It's like amount of flows that goes through a certain place within that goes through a certain place within that goes through a certain place within a within a unit unit amount of time. a within a unit unit amount of time. a within a unit unit amount of time. >> A sphere >> A sphere >> A sphere 0. 0. 0. >> Yeah. Yeah. Kind of kind. Okay. But >> Yeah. Yeah. Kind of kind. Okay. But >> Yeah. Yeah. Kind of kind. Okay. But anyway basically uh because of this we anyway basically uh because of this we anyway basically uh because of this we can actually write the probability flux can actually write the probability flux can actually write the probability flux in math in this way. So this is like
- L5 @ 840s
in math in this way. So this is like in math in this way. So this is like basically like how much probability gets basically like how much probability gets basically like how much probability gets changed um infinite uh through time and changed um infinite uh through time and changed um infinite uh through time and then this is what we call the divergence then this is what we call the divergence then this is what we call the divergence of velocity times density and what is of velocity times density and what is of velocity times density and what is divergence divergence is what we have divergence divergence is what we have divergence divergence is what we have seen actually uh last class it's the seen actually uh last class it's the seen actually uh last class it's the trace of the jacobian what what it means trace of the jacobian what what it means trace of the jacobian what what it means is basically like how much um how like is basically like how much um how like is basically like how much um how like how much do you change in every how much do you change in every how much do you change in every direction direction direction um of the of the flow o of the of the um of the of the flow o of the of the um of the of the flow o of the of the data space. data space. data space. Okay, any questions? Okay, any questions? Okay, any questions? Everyone understand this? Yay. Yeah, this is like very physics Yay. Yeah, this is like very physics coded. Okay, any any more questions? Direction. Direction. >> Oh, D is dimension. Sorry. Yeah. So, in >> Oh, D is dimension. Sorry. Yeah. So, in >> Oh, D is dimension. Sorry. Yeah. So, in every dimension, how does this thing every dimension, how does this thing every dimension, how does this thing change? basically. Oh, uh I think uh there's a P here that Oh, uh I think uh there's a P here that is got getting uh um missed. is got getting uh um missed. is got getting uh um missed. >> Yeah. >> Yeah. >> Yeah. >> What's the second assumption? There's >> What's the second assumption? There's >> What's the second assumption? There's two assumptions. two assumptions.
- L5 @ 912s
two assumptions. >> Uh two assumptions. One is that >> Uh two assumptions. One is that >> Uh two assumptions. One is that add up to one. The second one is that add up to one. The second one is that add up to one. The second one is that the the thing changes uh continuously. the the thing changes uh continuously. the the thing changes uh continuously. But how is the first one encoded in But how is the first one encoded in But how is the first one encoded in this? this? this? >> Uh it >> Uh it >> Uh it I I guess just the first one is not I I guess just the first one is not I I guess just the first one is not really encoded in this but like you just really encoded in this but like you just really encoded in this but like you just need to have a probability that add up need to have a probability that add up need to have a probability that add up to one. Yeah. >> Flux is a weird word for me. Would it be >> Flux is a weird word for me. Would it be weird to think of it as like viscosity weird to think of it as like viscosity weird to think of it as like viscosity like the amount of flow period of time like the amount of flow period of time like the amount of flow period of time and you're counting for density to like and you're counting for density to like and you're counting for density to like >> I mean the changes in space right and >> I mean the changes in space right and >> I mean the changes in space right and the viscosity should be uniform. Sure. the viscosity should be uniform. Sure. the viscosity should be uniform. Sure. But like viscosity with the thing moving But like viscosity with the thing moving But like viscosity with the thing moving over time. >> Um now flux is actually also a physics >> Um now flux is actually also a physics term and it means like the amount of term and it means like the amount of term and it means like the amount of thing that like flow out thing that like flow out
- L5 @ 964s
thing that like flow out >> as physically inclined. >> as physically inclined. >> as physically inclined. >> It is okay. >> It is okay. >> It is okay. >> I'm just a civil engineer. >> I'm just a civil engineer. >> I'm just a civil engineer. >> I'm also not a physics major but uh you >> I'm also not a physics major but uh you >> I'm also not a physics major but uh you know this is like it's not really >> um I mean just a term. It's not really >> um I mean just a term. It's not really like um Yeah. Yeah. But but but like you like um Yeah. Yeah. But but but like you like um Yeah. Yeah. But but but like you can you can imagine it however way you can you can imagine it however way you can you can imagine it however way you want but that's what it means basically want but that's what it means basically want but that's what it means basically just the amount of things that it just the amount of things that it just the amount of things that it outflows in a unit time basically. outflows in a unit time basically. outflows in a unit time basically. >> Yeah, >> why is it noisy? system external noise >> why is it noisy? system external noise >> uh depending on what kind of things that >> uh depending on what kind of things that >> uh depending on what kind of things that you are. So basically if you're thinking you are. So basically if you're thinking you are. So basically if you're thinking about the path that we just constructed about the path that we just constructed about the path that we just constructed then yes each XT is noisy but this is a then yes each XT is noisy but this is a then yes each XT is noisy but this is a generalized um um like notion it it's generalized um um like notion it it's generalized um um like notion it it's like just any flow this is the this is like just any flow this is the this is like just any flow this is the this is the the definition any other question any other question yes yes yes the divergence connection again
- L5 @ 1028s
the divergence connection again the divergence connection again >> yes I I think I missed a P here actually >> yes I I think I missed a P here actually >> yes I I think I missed a P here actually by the way uh but basically by the way uh but basically by the way uh but basically The divergence means that like how much The divergence means that like how much The divergence means that like how much uh probability do you change in every uh probability do you change in every uh probability do you change in every direction of the data space. Yeah. So direction of the data space. Yeah. So direction of the data space. Yeah. So that's why it's the trace of the that's why it's the trace of the that's why it's the trace of the Jacobian in every direction, every Jacobian in every direction, every Jacobian in every direction, every dimension here. Yeah. I I miss a P here. because like this case, right? because like this case, right? >> We don't we haven't talked about flow >> We don't we haven't talked about flow >> We don't we haven't talked about flow matching yet. >> All right. Any more question? >> All right. Any more question? Cool. All right. Yeah. So, this is Cool. All right. Yeah. So, this is Cool. All right. Yeah. So, this is divergence. How much probability that divergence. How much probability that divergence. How much probability that outflows from a given point per unit outflows from a given point per unit outflows from a given point per unit time in every direction? time in every direction? time in every direction? Okay. Okay. Okay. Cool. All right. All right. So uh like a Cool. All right. All right. So uh like a Cool. All right. All right. So uh like a good thing about um this thing is that good thing about um this thing is that good thing about um this thing is that we can actually transform this PDE into we can actually transform this PDE into we can actually transform this PDE into an OD. All right. So what it means is an OD. All right. So what it means is an OD. All right. So what it means is that like so this is the correct
- L5 @ 1101s
that like so this is the correct that like so this is the correct divergence. Sorry that that that one I divergence. Sorry that that that one I divergence. Sorry that that that one I missed P. Um but basically we can go missed P. Um but basically we can go missed P. Um but basically we can go from here uh and basically you do the from here uh and basically you do the from here uh and basically you do the greatest uh mathematical trick of all greatest uh mathematical trick of all greatest uh mathematical trick of all time. You multiply time. You multiply time. You multiply two things at the same time from the two things at the same time from the two things at the same time from the left hand side to the right hand side left hand side to the right hand side left hand side to the right hand side and uh basically this thing because the and uh basically this thing because the and uh basically this thing because the because the the the the derivative of because the the the the derivative of because the the the the derivative of log is equal to 1 /x. So it becomes the log is equal to 1 /x. So it becomes the log is equal to 1 /x. So it becomes the derivative of the uh the log and then um derivative of the uh the log and then um derivative of the uh the log and then um basically you just apply the same thing basically you just apply the same thing basically you just apply the same thing applying the same trick here and then it applying the same trick here and then it applying the same trick here and then it becomes something like this. Um and then like essentially you basically and then like essentially you basically just do a lot of like calculus and then just do a lot of like calculus and then just do a lot of like calculus and then you're actually able to get uh an OD of you're actually able to get uh an OD of you're actually able to get uh an OD of the log probability of of the log the log probability of of the log
- L5 @ 1153s
the log probability of of the log density. Okay. And this OD is actually density. Okay. And this OD is actually density. Okay. And this OD is actually equal to so the derivative of the total equal to so the derivative of the total equal to so the derivative of the total derivative of log probability is equal derivative of log probability is equal derivative of log probability is equal to the divergence of the velocity to the divergence of the velocity to the divergence of the velocity period. Okay. Any questions here? Okay. Any questions here? >> Yes. >> Yes. >> Yes. >> PFS to be zero. >> PFS to be zero. >> PFS to be zero. >> Yes. Yes. That's a that's a that's a >> Yes. Yes. That's a that's a that's a >> Yes. Yes. That's a that's a that's a great uh question. Yes. PFS is greater great uh question. Yes. PFS is greater great uh question. Yes. PFS is greater than zero everywhere. Yeah. Basically, you just do a lot of Yeah. Basically, you just do a lot of calculus and then you get this. >> No question. >> No question. >> Yes. >> Yes. >> Because it's still depending on the >> Because it's still depending on the >> Because it's still depending on the variable. variable. variable. No, basically from this line, I don't No, basically from this line, I don't No, basically from this line, I don't know if you can see my cursor. This line know if you can see my cursor. This line know if you can see my cursor. This line is literally the formula for the total is literally the formula for the total is literally the formula for the total derivative, right? derivative, right? derivative, right? Yeah. No more question. No more question. Okay, cool. Uh so basically then you can Okay, cool. Uh so basically then you can Okay, cool. Uh so basically then you can calculus then you can calculate your calculus then you can calculate your calculus then you can calculate your likelihood, right? because now you have likelihood, right? because now you have likelihood, right? because now you have a another OD that you can directly solve a another OD that you can directly solve a another OD that you can directly solve by doing numerical integration. So exactly so so
- L5 @ 1252s
So exactly so so how do you train your continuous how do you train your continuous how do you train your continuous normalizing flow from here? What do we normalizing flow from here? What do we normalizing flow from here? What do we think jumping ahead. Jumping ahead. What is jumping ahead. Jumping ahead. What is the Now we have a formula for the the Now we have a formula for the the Now we have a formula for the likelihood log likelihood you can do for the the most naive thing you can do for the the most naive thing to think right is the maximum log to think right is the maximum log to think right is the maximum log likelihood. So basically uh yeah so likelihood. So basically uh yeah so likelihood. So basically uh yeah so because you have a formula to calculate because you have a formula to calculate because you have a formula to calculate the log likelihood at any point of the the log likelihood at any point of the the log likelihood at any point of the trajectory obviously you can also trajectory obviously you can also trajectory obviously you can also calculate the log likelihood of the end calculate the log likelihood of the end calculate the log likelihood of the end point right so you just do arg max of point right so you just do arg max of point right so you just do arg max of the likelihood what could be a problem the likelihood what could be a problem the likelihood what could be a problem here >> yeah yeah >> yeah yeah >> don't have access to p 0 >> don't have access to p 0 >> don't have access to p 0 >> you we can define that we define p 0 to >> you we can define that we define p 0 to >> you we can define that we define p 0 to be a gausian for example that's fine be a gausian for example that's fine be a gausian for example that's fine Very very good. So basically in order to very good. So basically in order to calculate the log likelihood of the data calculate the log likelihood of the data calculate the log likelihood of the data we actually need to do numerical we actually need to do numerical
- L5 @ 1324s
we actually need to do numerical integration right. But this is at integration right. But this is at integration right. But this is at training time. So that just means that training time. So that just means that training time. So that just means that we need to do many many many many steps we need to do many many many many steps we need to do many many many many steps of integration at training time and that of integration at training time and that of integration at training time and that thing is really really expensive. thing is really really expensive. thing is really really expensive. Right. Right. Right. Okay. What what would be a better way to Okay. What what would be a better way to Okay. What what would be a better way to do this? do this? do this? >> Yeah. >> If we take the if we take the gradient >> If we take the if we take the gradient of the left hand side, we get a score of the left hand side, we get a score of the left hand side, we get a score and the left hand side removes, and the left hand side removes, and the left hand side removes, >> right? >> right? >> right? >> Uh >> Uh >> Uh even if you do that right, the even if you do that right, the even if you do that right, the divergence is still pretty difficult to divergence is still pretty difficult to divergence is still pretty difficult to calculate. calculate. calculate. >> Okay. But what else? Yes, >> we can we can use this numerator or we >> we can we can use this numerator or we can just directly sample can just directly sample can just directly sample so that it'll be uh we we'll end up like so that it'll be uh we we'll end up like so that it'll be uh we we'll end up like uh we'll end up with the summation. >> You you mean you change the integration >> You you mean you change the integration into summation? into summation? into summation? >> Yeah. Yeah. Yeah. But that's just >> Yeah. Yeah. Yeah. But that's just >> Yeah. Yeah. Yeah. But that's just numerical integration, right? So it numerical integration, right? So it
- L5 @ 1387s
numerical integration, right? So it still has a lot of steps. Can we do still has a lot of steps. Can we do still has a lot of steps. Can we do something that doesn't require a lot of something that doesn't require a lot of something that doesn't require a lot of steps? Like we just have one four pass steps? Like we just have one four pass steps? Like we just have one four pass during the training time. I think during the training time. I think during the training time. I think someone said it before. Who said velocity before? Who did it? Who said velocity before? Who did it? Oh, you did? Oh, you did? Oh, you did? >> Oh, someone who said velocity. >> Oh, okay. Why? Why did you say velocity? >> Oh, okay. Why? Why did you say velocity? >> Because >> Because >> Because the model's learning velocity. Okay, it actually Okay, it actually basically if you think about it right basically if you think about it right basically if you think about it right both sampling like in order to sample or both sampling like in order to sample or both sampling like in order to sample or calculate the log likelihood you you calculate the log likelihood you you calculate the log likelihood you you pretty much only need to parameterize pretty much only need to parameterize pretty much only need to parameterize the velocity right so if you have a the velocity right so if you have a the velocity right so if you have a model for the velocity then you can model for the velocity then you can model for the velocity then you can calculate then you can do both sampling calculate then you can do both sampling calculate then you can do both sampling and calculate the log log likelihood so and calculate the log log likelihood so and calculate the log log likelihood so why don't we we just need to make sure why don't we we just need to make sure why don't we we just need to make sure that the velocity is correct right so we that the velocity is correct right so we that the velocity is correct right so we just need to make sure that the velocity just need to make sure that the velocity
- L5 @ 1451s
just need to make sure that the velocity actually match with the ground truth actually match with the ground truth actually match with the ground truth velocity. velocity. velocity. So you just do L2 to match with the So you just do L2 to match with the So you just do L2 to match with the ground truth velocity. ground truth velocity. ground truth velocity. Easy, right? Easy, right? Easy, right? What is the problem? Yeah. In order to do either sampling or like In order to do either sampling or like critical estimation, we just need a critical estimation, we just need a critical estimation, we just need a velocity. So if we have a model for velocity. So if we have a model for velocity. So if we have a model for velocity, then we could can do velocity, then we could can do velocity, then we could can do integration at inference time. We don't integration at inference time. We don't integration at inference time. We don't need to do integration at training time need to do integration at training time need to do integration at training time anymore, right? But what what is the anymore, right? But what what is the anymore, right? But what what is the problem here? The ground truth, right? We don't have The ground truth, right? We don't have the ground truth. If we have the ground the ground truth. If we have the ground the ground truth. If we have the ground truth, we would know that a winter storm truth, we would know that a winter storm truth, we would know that a winter storm is coming. Apparently, it's a Saturday. is coming. Apparently, it's a Saturday. is coming. Apparently, it's a Saturday. It's a very cold day. So be prepared. It's a very cold day. So be prepared. It's a very cold day. So be prepared. Okay, winter's coming. Uh anyway, so Okay, winter's coming. Uh anyway, so Okay, winter's coming. Uh anyway, so yeah, but we do we do not know. So yeah, but we do we do not know. So yeah, but we do we do not know. So that's why the meteorologists are that's why the meteorologists are that's why the meteorologists are changing every day be like actually I changing every day be like actually I
- L5 @ 1512s
changing every day be like actually I think that the the storm is moving north think that the the storm is moving north think that the the storm is moving north or maybe not. We don't know, right? So or maybe not. We don't know, right? So or maybe not. We don't know, right? So we do not know the ground velocity. How should we solve this? Well, How should we solve this? Well, basically say we don't care about basically say we don't care about basically say we don't care about everywhere else. We just care about how everywhere else. We just care about how everywhere else. We just care about how the storm is going to move to the storm is going to move to the storm is going to move to Pittsburgh. Let's just say and say now Pittsburgh. Let's just say and say now Pittsburgh. Let's just say and say now you just want to like say you now you're you just want to like say you now you're you just want to like say you now you're Poseidon. Okay, whatever. And then you Poseidon. Okay, whatever. And then you Poseidon. Okay, whatever. And then you you want to transform a bunch of water you want to transform a bunch of water you want to transform a bunch of water or a bunch of like humidity to or a bunch of like humidity to or a bunch of like humidity to Pittsburgh so that Pittsburgh would Pittsburgh so that Pittsburgh would Pittsburgh so that Pittsburgh would snow. snow. snow. What would you do? What is the easiest What would you do? What is the easiest What would you do? What is the easiest way? What is the fastest way for you to way? What is the fastest way for you to way? What is the fastest way for you to do it? How how would you do that? do it? How how would you do that? do it? How how would you do that? >> You just Yeah. >> You just Yeah. >> You just Yeah. >> Just go straight there. >> Just go straight there. >> Just go straight there. >> Just go straight there. Yeah. Yeah. >> Just go straight there. Yeah. Yeah. >> Just go straight there. Yeah. Yeah. Yeah. That's right. That's right. You Yeah. That's right. That's right. You
- L5 @ 1558s
Yeah. That's right. That's right. You just go straight. just go straight. just go straight. Yeah, exactly. Uh so basically this is Yeah, exactly. Uh so basically this is Yeah, exactly. Uh so basically this is what we're going to do as well. Uh so what we're going to do as well. Uh so what we're going to do as well. Uh so let's say we fix a data point. We don't let's say we fix a data point. We don't let's say we fix a data point. We don't care about everywhere else. We just care care about everywhere else. We just care care about everywhere else. We just care about one data point. And then it's about one data point. And then it's about one data point. And then it's usually uh very easy to define a what we usually uh very easy to define a what we usually uh very easy to define a what we call continuous velocity field which uh call continuous velocity field which uh call continuous velocity field which uh sorry a conditional velocity field which sorry a conditional velocity field which sorry a conditional velocity field which means that means that means that basically it just basically it just basically it just velocity is depending on the end point velocity is depending on the end point velocity is depending on the end point right and then we call the trajectory of right and then we call the trajectory of right and then we call the trajectory of the probability distribution that gen the probability distribution that gen the probability distribution that gen get generated along the get generated along the get generated along the way of that uh velocity field the way of that uh velocity field the way of that uh velocity field the conditional probability path. So the conditional probability path. So the conditional probability path. So the conditional probability path is consists conditional probability path is consists conditional probability path is consists of like all the uh all the pts along the of like all the uh all the pts along the of like all the uh all the pts along the way. Okay. And here the conditional way. Okay. And here the conditional way. Okay. And here the conditional probability path will probability path will probability path will just start from some prior which we can
- L5 @ 1617s
just start from some prior which we can just start from some prior which we can choose. So it doesn't matter we can choose. So it doesn't matter we can choose. So it doesn't matter we can choose the prior and then we will always choose the prior and then we will always choose the prior and then we will always end up either directly at Pittsburgh or end up either directly at Pittsburgh or end up either directly at Pittsburgh or somewhere around Pittsburgh. So like we somewhere around Pittsburgh. So like we somewhere around Pittsburgh. So like we just like always end up at x1 or a small just like always end up at x1 or a small just like always end up at x1 or a small small gausian that concentrate around small gausian that concentrate around small gausian that concentrate around x1. Okay. And uh by the way so this x1. Okay. And uh by the way so this x1. Okay. And uh by the way so this thing is called direct delta. So it just thing is called direct delta. So it just thing is called direct delta. So it just means that it's a point mass at at x1. means that it's a point mass at at x1. means that it's a point mass at at x1. Okay. Then uh the marginal probability Okay. Then uh the marginal probability Okay. Then uh the marginal probability path can be written in this way, right? path can be written in this way, right? path can be written in this way, right? Basically just like sum up all the Basically just like sum up all the Basically just like sum up all the possible uh sum up all the possible uh possible uh sum up all the possible uh possible uh sum up all the possible uh uh like data points and then basically uh like data points and then basically uh like data points and then basically just like yeah like and and times the just like yeah like and and times the just like yeah like and and times the the the conditional probability given the the conditional probability given the the conditional probability given the the data point then you can get a the the data point then you can get a
- L5 @ 1663s
the the data point then you can get a marginal marginal marginal uh yeah and also uh just the boundary uh yeah and also uh just the boundary uh yeah and also uh just the boundary condition essentially uh the probability condition essentially uh the probability condition essentially uh the probability uh at time uh at time uh at time should recover the data probability. should recover the data probability. should recover the data probability. This is basically what we're trying to This is basically what we're trying to This is basically what we're trying to construct. construct. construct. Okay. >> All right. Cool. Um so say now we have a >> All right. Cool. Um so say now we have a conditional velocity here. uh then we conditional velocity here. uh then we conditional velocity here. uh then we can also define a marginal velocity. can also define a marginal velocity. can also define a marginal velocity. What does what does this mean? So What does what does this mean? So What does what does this mean? So basically this just means that uh we are basically this just means that uh we are basically this just means that uh we are sort of like we're like sort of like sort of like we're like sort of like sort of like we're like sort of like apply a bay theorem on on your uh apply a bay theorem on on your uh apply a bay theorem on on your uh conditional velocity. So that basically conditional velocity. So that basically conditional velocity. So that basically it's like reweighted it's like reweighted it's like reweighted uh like um taking account into like uh like um taking account into like uh like um taking account into like basically um how likely is your current basically um how likely is your current basically um how likely is your current uh noisy point like your intermediate uh noisy point like your intermediate uh noisy point like your intermediate data point along the the the conditional data point along the the the conditional data point along the the the conditional probability path. How likely is the data probability path. How likely is the data probability path. How likely is the data point that defined this conditional point that defined this conditional point that defined this conditional probability path and also just like how
- L5 @ 1743s
probability path and also just like how probability path and also just like how like and reweed by how likely in general like and reweed by how likely in general like and reweed by how likely in general of this current intermediate point. Does of this current intermediate point. Does of this current intermediate point. Does this make sense? All right. So basically you can sort of All right. So basically you can sort of like intuitively understand this as like like intuitively understand this as like like intuitively understand this as like this is the average conditional velocity this is the average conditional velocity this is the average conditional velocity at this current location in this current at this current location in this current at this current location in this current time weighted by how likely uh is the time weighted by how likely uh is the time weighted by how likely uh is the data point for this current location in data point for this current location in data point for this current location in time. part. Then in the next slide when we're part. Then in the next slide when we're doing it for the velocity, is it doing it for the velocity, is it doing it for the velocity, is it something we're doing from that step or something we're doing from that step or something we're doing from that step or is it just like separate entirely? is it just like separate entirely? is it just like separate entirely? >> Separate. I'll go level resurrected. Uh yeah. So, I'll go level resurrected. Uh yeah. So, so, so this is kind of like what we so, so this is kind of like what we so, so this is kind of like what we defined. It's not like it's two defined. It's not like it's two defined. It's not like it's two different notions different notions different notions and like you can intuitively understand and like you can intuitively understand and like you can intuitively understand it as like sort of like the um like the it as like sort of like the um like the it as like sort of like the um like the average uh conditional velocity and then average uh conditional velocity and then average uh conditional velocity and then this average is like averaging by like
- L5 @ 1838s
this average is like averaging by like this average is like averaging by like how likely is the data point given your how likely is the data point given your how likely is the data point given your current position and the data points are available. and the data points are available. All right, does this make sense? we'll get to it. We'll get to it. Um all we'll get to it. We'll get to it. Um all right. Uh and a very interesting thing right. Uh and a very interesting thing right. Uh and a very interesting thing here is that the marginal velocity we here is that the marginal velocity we here is that the marginal velocity we just saw actually generates the marginal just saw actually generates the marginal just saw actually generates the marginal probability path that we want. Okay. So what it means is what it means by So what it means is what it means by generate is basically just means that generate is basically just means that generate is basically just means that like the the continuity equation is like the the continuity equation is like the the continuity equation is satisfies the the PTE is satisfied and satisfies the the PTE is satisfied and satisfies the the PTE is satisfied and why does it satisfy is because if you why does it satisfy is because if you why does it satisfy is because if you just break everything down uh and then just break everything down uh and then just break everything down uh and then you are able to get basically the the you are able to get basically the the you are able to get basically the the divergence back. Uh so this is just divergence back. Uh so this is just divergence back. Uh so this is just like step by step how you will get a like step by step how you will get a like step by step how you will get a divergence back. Uh so you first um swap divergence back. Uh so you first um swap divergence back. Uh so you first um swap in the the the derivative with some in the the the derivative with some in the the the derivative with some regularization condition. This can can regularization condition. This can can
- L5 @ 1904s
regularization condition. This can can can be done. Uh and then uh basically can be done. Uh and then uh basically can be done. Uh and then uh basically you uh input your um because we know you uh input your um because we know you uh input your um because we know that the marginal sorry the conditional that the marginal sorry the conditional that the marginal sorry the conditional velocity generates the conditional path. velocity generates the conditional path. velocity generates the conditional path. So we can uh input the continuity So we can uh input the continuity So we can uh input the continuity equation there and then it becomes this equation there and then it becomes this equation there and then it becomes this whole thing with divergence. Uh and then whole thing with divergence. Uh and then whole thing with divergence. Uh and then plug in the definition that we just saw plug in the definition that we just saw plug in the definition that we just saw before we can actually get something before we can actually get something before we can actually get something like this and again pulling out the the like this and again pulling out the the like this and again pulling out the the the the derivative here because because the the derivative here because because the the derivative here because because of some regularization condition that we of some regularization condition that we of some regularization condition that we can satisfy it. uh and this thing is can satisfy it. uh and this thing is can satisfy it. uh and this thing is literally just the divergence of the literally just the divergence of the literally just the divergence of the marginal uh velocity times the marginal marginal uh velocity times the marginal marginal uh velocity times the marginal density. density. density. So all in all this is to say that the So all in all this is to say that the So all in all this is to say that the marginal velocity that we defined before marginal velocity that we defined before marginal velocity that we defined before from the conditional velocity that we from the conditional velocity that we from the conditional velocity that we can construct can generate the marginal can construct can generate the marginal
- L5 @ 1966s
can construct can generate the marginal property path which is what we want. property path which is what we want. property path which is what we want. Oh, I saw so many confusion. Oh, I saw so many confusion. Oh, I saw so many confusion. All good. All good. We're getting there. All good. All good. We're getting there. All good. All good. We're getting there. We're getting there. I promise. Ah, We're getting there. I promise. Ah, We're getting there. I promise. Ah, okay. The other thing which is the most okay. The other thing which is the most okay. The other thing which is the most important thing and the most magical important thing and the most magical important thing and the most magical part is that actually part is that actually part is that actually so this is the the the the equation that so this is the the the the equation that so this is the the the the equation that we want, right? So this is the equation we want, right? So this is the equation we want, right? So this is the equation that we that that that we came up with. that we that that that we came up with. that we that that that we came up with. You just need to match the ground truth You just need to match the ground truth You just need to match the ground truth marginal velocity. But we do not have marginal velocity. But we do not have marginal velocity. But we do not have the groundous marginal velocity, right? the groundous marginal velocity, right? the groundous marginal velocity, right? And it actually turns out that if you And it actually turns out that if you And it actually turns out that if you just match the conditional velocity, just match the conditional velocity, just match the conditional velocity, you're actually going to end up with the you're actually going to end up with the you're actually going to end up with the same uh global optimal. You're like same uh global optimal. You're like same uh global optimal. You're like basically just like matching the ground basically just like matching the ground basically just like matching the ground truth marginal velocity is equivalent to truth marginal velocity is equivalent to truth marginal velocity is equivalent to ma matching the conditional velocity.
- L5 @ 2030s
Think about that. Isn't that pretty Think about that. Isn't that pretty cool? Okay. Why? Why? Why is that? Okay. cool? Okay. Why? Why? Why is that? Okay. cool? Okay. Why? Why? Why is that? Okay. So, the reason why is because we can So, the reason why is because we can So, the reason why is because we can break the two L2 down into this thing, break the two L2 down into this thing, break the two L2 down into this thing, right? We we saw this yesterday, not right? We we saw this yesterday, not right? We we saw this yesterday, not yesterday, on Tuesday already, right? yesterday, on Tuesday already, right? yesterday, on Tuesday already, right? And the first term for both equation are And the first term for both equation are And the first term for both equation are constant with respect to our learn constant with respect to our learn constant with respect to our learn parameter. So, we're just going to parameter. So, we're just going to parameter. So, we're just going to cancel it out, right? cancel it out, right? cancel it out, right? Uh and then now we get to two equations Uh and then now we get to two equations Uh and then now we get to two equations each with two components. Right? So each with two components. Right? So each with two components. Right? So we're going to look at the first we're going to look at the first we're going to look at the first components together first. All right. So components together first. All right. So components together first. All right. So the first component if you if you look the first component if you if you look the first component if you if you look at the conditional flow matching uh at the conditional flow matching uh at the conditional flow matching uh equation here uh basically this thing equation here uh basically this thing equation here uh basically this thing you can um decompose it into two you can um decompose it into two you can um decompose it into two integrals and then the these two integrals and then the these two integrals and then the these two integrals integrals integrals um can be um can be um can be uh uh uh Like basically you if you swap uh the
- L5 @ 2086s
Like basically you if you swap uh the Like basically you if you swap uh the dx1 term inside so you swap the order of dx1 term inside so you swap the order of dx1 term inside so you swap the order of the integration you're actually going to the integration you're actually going to the integration you're actually going to get the first term of the first equation get the first term of the first equation get the first term of the first equation back. All right. This part is clear. Cool. All right. All right. Now look at Cool. All right. All right. Now look at the second part. The second part is the second part. The second part is the second part. The second part is pretty The second part is pretty pretty The second part is pretty pretty The second part is pretty similar. Basically what you do is you similar. Basically what you do is you similar. Basically what you do is you just uh starting from the second part of just uh starting from the second part of just uh starting from the second part of the first the first the first uh equation and then you just um plug in uh equation and then you just um plug in uh equation and then you just um plug in what we have as definition and then you what we have as definition and then you what we have as definition and then you do a lot of integrals in between and do a lot of integrals in between and do a lot of integrals in between and then you're actually going to wind up then you're actually going to wind up then you're actually going to wind up at the second part of the second at the second part of the second at the second part of the second equation. equation. equation. Any question? Yes. No. No. Okay. understand everything Yes. No. No. Okay. understand everything everyone. everyone. everyone. >> It's not so it's not even that we're >> It's not so it's not even that we're >> It's not so it's not even that we're sort of having a elbow type thing where sort of having a elbow type thing where sort of having a elbow type thing where we're compressing the bullet. we're compressing the bullet.
- L5 @ 2150s
we're compressing the bullet. >> It's literally the same. It's literally >> It's literally the same. It's literally >> It's literally the same. It's literally the same. It's equivalent. It's not even the same. It's equivalent. It's not even the same. It's equivalent. It's not even elbow. No question. All right. No question. All right. So, this just means that like we don't So, this just means that like we don't So, this just means that like we don't need to match to the thing that we don't need to match to the thing that we don't need to match to the thing that we don't know. we actually can match something know. we actually can match something know. we actually can match something that we do know and do know how to that we do know and do know how to that we do know and do know how to construct which is the conditional construct which is the conditional construct which is the conditional velocity. velocity. Okay. >> Yes, pretty much right. And then you >> Yes, pretty much right. And then you average out and then it is actually the average out and then it is actually the average out and then it is actually the same. Um so uh now let's just look at same. Um so uh now let's just look at same. Um so uh now let's just look at the example that we have right. So now the example that we have right. So now the example that we have right. So now let's suppose our conditional let's suppose our conditional let's suppose our conditional probability path is to transport uh probability path is to transport uh probability path is to transport uh trans transform a gausian a standard trans transform a gausian a standard trans transform a gausian a standard normal gausian straight to a single normal gausian straight to a single normal gausian straight to a single point with constant speed. So this is a point with constant speed. So this is a point with constant speed. So this is a conditional right. So that's why we can conditional right. So that's why we can conditional right. So that's why we can have a single point here. What do we do? Well we first have the the the endpoint
- L5 @ 2213s
Well we first have the the the endpoint distribution and because we're distribution and because we're distribution and because we're transforming it straight to the point transforming it straight to the point transforming it straight to the point with constant speed right? So just with constant speed right? So just with constant speed right? So just linearly very straight. Um so we can linearly very straight. Um so we can linearly very straight. Um so we can literally have a formula for each literally have a formula for each literally have a formula for each intermediate point as a linear intermediate point as a linear intermediate point as a linear combination of uh of the data and the combination of uh of the data and the combination of uh of the data and the noise and this linear combination will noise and this linear combination will noise and this linear combination will add up to one right and uh the add up to one right and uh the add up to one right and uh the probability path can be written in this probability path can be written in this probability path can be written in this way. And most importantly, way. And most importantly, way. And most importantly, the velocity is literally just xt taking the velocity is literally just xt taking the velocity is literally just xt taking derivative of t, which is equal to theta derivative of t, which is equal to theta derivative of t, which is equal to theta minus noise. That's it. And this is what we're That's it. And this is what we're matching to at training time, which we matching to at training time, which we matching to at training time, which we have already seen. >> Yeah. >> First question. So this us >> First question. So this us >> First question. So this us parameterizing it as this linear parameterizing it as this linear parameterizing it as this linear combination that fact is not used really combination that fact is not used really combination that fact is not used really in any of the other. in any of the other. in any of the other. >> Yeah. No no this is what we chose. This >> Yeah. No no this is what we chose. This
- L5 @ 2272s
>> Yeah. No no this is what we chose. This is a design choice. So that so this is a design choice. So that so this is a design choice. So that so this thing is generalizable to any kind of thing is generalizable to any kind of thing is generalizable to any kind of path. Yeah. path. Yeah. path. Yeah. >> How is the ptxt given x1 that >> How is the ptxt given x1 that >> How is the ptxt given x1 that >> how is what >> how is what >> how is what >> the third length? How is the third? >> the third length? How is the third? >> the third length? How is the third? Oh, here. Oh, here. Oh, here. >> Well, because x0 is zero mean gausian. >> Well, because x0 is zero mean gausian. >> Well, because x0 is zero mean gausian. >> Yeah. >> Yeah. All right. Any other? Yeah. All right. Any other? Yeah. All right. Any other? Yeah. >> I don't see any like >> I don't see any like >> I don't see any like gausian assumption. gausian assumption. gausian assumption. >> Oh, there's no gausian assumption. You >> Oh, there's no gausian assumption. You >> Oh, there's no gausian assumption. You don't need gian assumption. don't need gian assumption. don't need gian assumption. >> Just assume that the noise is any >> Just assume that the noise is any >> Just assume that the noise is any distribution. It still works. distribution. It still works. distribution. It still works. >> Yes. Well, um, well, Well, um, well, yeah, pretty much. Yeah, you you don't yeah, pretty much. Yeah, you you don't yeah, pretty much. Yeah, you you don't you don't really need Gausian here. >> Any reason other than simplicity for >> Any reason other than simplicity for choosing? choosing? choosing? Um Um Um here here here not really like but like what what is not really like but like what what is not really like but like what what is the other uh uh what is the other the other uh uh what is the other the other uh uh what is the other distribution that you can think of that distribution that you can think of that distribution that you can think of that we can directly sample from we can directly sample from
- L5 @ 2349s
we can directly sample from >> therefor you >> like previous lectures like gausian >> like previous lectures like gausian assumption is used like this closure on assumption is used like this closure on assumption is used like this closure on the gian used in the derivation but in the gian used in the derivation but in the gian used in the derivation but in flow matching nowhere is flow matching nowhere is flow matching nowhere is >> yeah >> yeah >> yeah it's just here it's just that basically >> like we have previously seen that we can >> like we have previously seen that we can get some kind of stoastic differential get some kind of stoastic differential get some kind of stoastic differential equation for the equation for the equation for the for our scores. So here, don't we just for our scores. So here, don't we just for our scores. So here, don't we just have like a parameterization of that have like a parameterization of that have like a parameterization of that same equation that will get us the same same equation that will get us the same same equation that will get us the same result? result? result? >> Great question. We're going to talk >> Great question. We're going to talk >> Great question. We're going to talk about it later. about it later. about it later. >> Yeah. >> Yeah. >> Assuming that there's an assumption that >> Assuming that there's an assumption that >> Assuming that there's an assumption that I think we assumed earlier to the come I think we assumed earlier to the come I think we assumed earlier to the come earlier that the you said the velocity earlier that the you said the velocity earlier that the you said the velocity is constant. So is constant. So is constant. So >> yeah, the velocity yes is constant >> yeah, the velocity yes is constant >> yeah, the velocity yes is constant speed. Yeah. But is it actually speed. Yeah. But is it actually speed. Yeah. But is it actually constant? Because last lecture I think constant? Because last lecture I think constant? Because last lecture I think we talked about stuff when we u when we we talked about stuff when we u when we
- L5 @ 2422s
we talked about stuff when we u when we actually uh decrease the the velocity at actually uh decrease the the velocity at actually uh decrease the the velocity at which we adding noise. which we adding noise. which we adding noise. >> Here the velocity is different from >> Here the velocity is different from >> Here the velocity is different from previous class. previous class. previous class. So this is two different models. Um and So this is two different models. Um and So this is two different models. Um and the reason why this the reason why this the reason why this constant speed is because literally you constant speed is because literally you constant speed is because literally you have the t there, right? And then like have the t there, right? And then like have the t there, right? And then like literally if you derive the velocity is literally if you derive the velocity is literally if you derive the velocity is literally a constant literally a constant literally a constant right you have question. you have question. >> Oh, okay. Cool. Uh yes. Can >> Oh, okay. Cool. Uh yes. Can >> Oh, okay. Cool. Uh yes. Can >> we use this to do likelihood estimation >> we use this to do likelihood estimation >> we use this to do likelihood estimation as well or? as well or? as well or? >> Yes. We're going to talk about it later. >> Yes. We're going to talk about it later. >> Yes. We're going to talk about it later. Yeah. Yeah. Yeah. Anyone else? Anyone else? Anyone else? Cool. Yeah. And this is the this is the Cool. Yeah. And this is the this is the Cool. Yeah. And this is the this is the uh the loss function. Literally uh the loss function. Literally uh the loss function. Literally literally just you learn the model literally just you learn the model literally just you learn the model should predict data minus noise. Yeah. should predict data minus noise. Yeah. should predict data minus noise. Yeah. >> The velocity stays constant. >> The velocity stays constant. >> The velocity stays constant. >> Um the the velocity is constant only on >> Um the the velocity is constant only on >> Um the the velocity is constant only on this particular path. Only on this
- L5 @ 2486s
this particular path. Only on this this particular path. Only on this particular x0 to x1 path. But it's not particular x0 to x1 path. But it's not particular x0 to x1 path. But it's not in general. in general it's not actually in general. in general it's not actually in general. in general it's not actually I think this may answer your question I think this may answer your question I think this may answer your question yeah so basically in like so the left yeah so basically in like so the left yeah so basically in like so the left hand side is like the the path that you hand side is like the the path that you hand side is like the the path that you construct at training time so everything construct at training time so everything construct at training time so everything is straight right everything is straight is straight right everything is straight is straight right everything is straight but the learn the function will you have but the learn the function will you have but the learn the function will you have this averaging effect right so you learn this averaging effect right so you learn this averaging effect right so you learn the the the the entire field so you the the the the entire field so you the the the the entire field so you don't don't don't have this straight line anymore actually have this straight line anymore actually have this straight line anymore actually but you have this like curve curve the but you have this like curve curve the but you have this like curve curve the slightly curved average field still normalizing flow right yeah still normalizing flow right yeah because because you have the one to one because because you have the one to one because because you have the one to one you have the the the invertible you have the the the invertible you have the the the invertible invertability invertability invertability okay any question okay any question okay any question >> yes >> yes >> yes >> why is it slightly collapsing in the >> why is it slightly collapsing in the >> why is it slightly collapsing in the middle middle middle >> because if you look at the if you look
- L5 @ 2544s
>> because if you look at the if you look >> because if you look at the if you look at the this the the left hand side right at the this the the left hand side right at the this the the left hand side right so like you can see how the field is not so like you can see how the field is not so like you can see how the field is not like the field has a collapsing. So like like the field has a collapsing. So like like the field has a collapsing. So like the the the margin so we're later on the the the margin so we're later on the the the margin so we're later on when when when sampling we're using it as if it's the sampling we're using it as if it's the sampling we're using it as if it's the marginal and the marginal has this like marginal and the marginal has this like marginal and the marginal has this like slight slight concentration in the in slight slight concentration in the in slight slight concentration in the in the middle. Yep. the middle. Yep. the middle. Yep. >> So in this case if I were to pass the >> So in this case if I were to pass the >> So in this case if I were to pass the same gshian input twice to the model I same gshian input twice to the model I same gshian input twice to the model I get the same image out. get the same image out. get the same image out. >> Yes. It's deterministic completely which >> Yes. It's deterministic completely which >> Yes. It's deterministic completely which is different from diffusion right? is different from diffusion right? is different from diffusion right? >> Yes. >> Yes. Okay, any more question? Cool. Actually, this thing is like super Cool. Actually, this thing is like super cool because the same algorithm was cool because the same algorithm was cool because the same algorithm was developed by three different group of developed by three different group of developed by three different group of people at literally the same time like people at literally the same time like people at literally the same time like literally the same conference and they literally the same conference and they
- L5 @ 2596s
literally the same conference and they all got into the same conference. all got into the same conference. all got into the same conference. So yeah, so the first group is a group So yeah, so the first group is a group So yeah, so the first group is a group in Metafair and in Metafair and in Metafair and this is the group that I interned at this is the group that I interned at this is the group that I interned at actually. Um and then the second group I actually. Um and then the second group I actually. Um and then the second group I believe is a group from UT Austin and believe is a group from UT Austin and believe is a group from UT Austin and then the third group is a group from uh then the third group is a group from uh then the third group is a group from uh NYU and they all derive the same NYU and they all derive the same NYU and they all derive the same algorithm from different uh theoretical algorithm from different uh theoretical algorithm from different uh theoretical perspective and all submit to the same perspective and all submit to the same perspective and all submit to the same conference and all got in. conference and all got in. conference and all got in. So this is like sort of like a beautiful So this is like sort of like a beautiful So this is like sort of like a beautiful moment of science where you just kind of moment of science where you just kind of moment of science where you just kind of have like a collapsing ideas but like have like a collapsing ideas but like have like a collapsing ideas but like they're from different points you know they're from different points you know they're from different points you know it's just like our you know flow it's just like our you know flow it's just like our you know flow matching everyone goes to the same point matching everyone goes to the same point matching everyone goes to the same point anyway. Yeah, pretty pretty magical. Um, anyway. Yeah, pretty pretty magical. Um, anyway. Yeah, pretty pretty magical. Um, but you may have this question, right? I but you may have this question, right? I
- L5 @ 2640s
but you may have this question, right? I feel like some people are having this feel like some people are having this feel like some people are having this question but they didn't ask. So like question but they didn't ask. So like question but they didn't ask. So like the marginal flow has some dips to it, the marginal flow has some dips to it, the marginal flow has some dips to it, right? So it doesn't it's not completely right? So it doesn't it's not completely right? So it doesn't it's not completely straight. It's not the straightest. So straight. It's not the straightest. So straight. It's not the straightest. So is there a way that we can like let's is there a way that we can like let's is there a way that we can like let's say now we have trained a flow matching say now we have trained a flow matching say now we have trained a flow matching model now. So now we have this this this model now. So now we have this this this model now. So now we have this this this um like this invertability this coupling um like this invertability this coupling um like this invertability this coupling already. Is there a way to already. Is there a way to already. Is there a way to make this even straighter? Is it make this even straighter? Is it make this even straighter? Is it possible directly move from your P 0 to directly move from your P 0 to >> Yeah, pretty much. Pretty much. Um, so >> Yeah, pretty much. Pretty much. Um, so >> Yeah, pretty much. Pretty much. Um, so this thing is what we call reflow. So this thing is what we call reflow. So this thing is what we call reflow. So this is a technique that developed by by this is a technique that developed by by this is a technique that developed by by the second group of people who proposed the second group of people who proposed the second group of people who proposed the same thing. Uh so basically what you the same thing. Uh so basically what you the same thing. Uh so basically what you do is now let's say you you after do is now let's say you you after
- L5 @ 2698s
do is now let's say you you after training with the linear interpolation training with the linear interpolation training with the linear interpolation you already got this coupling. So the you already got this coupling. So the you already got this coupling. So the blue coupling and the the green coupling blue coupling and the the green coupling blue coupling and the the green coupling and then what you can do is you can just and then what you can do is you can just and then what you can do is you can just like use this coupling as your new like use this coupling as your new like use this coupling as your new ground truth and then instead of like ground truth and then instead of like ground truth and then instead of like randomly sampling from a gausian you randomly sampling from a gausian you randomly sampling from a gausian you sample from the coupling and then you sample from the coupling and then you sample from the coupling and then you can get the straight points can get the straight points can get the straight points and then it become more straight. Let me explain again. Let me explain again. >> Oh, okay. One more time. One more time. >> Oh, okay. One more time. One more time. >> Oh, okay. One more time. One more time. So, basically this is like your first So, basically this is like your first So, basically this is like your first time uh flow matching. So, basically time uh flow matching. So, basically time uh flow matching. So, basically after computing all the the the the in after computing all the the the the in after computing all the the the the in the uh linear interpolations, you get the uh linear interpolations, you get the uh linear interpolations, you get this like slightly dipped um marginals, this like slightly dipped um marginals, this like slightly dipped um marginals, right? You get this slightly dipped right? You get this slightly dipped right? You get this slightly dipped version, but now you have a a coupling version, but now you have a a coupling version, but now you have a a coupling because the thing is deterministic, because the thing is deterministic, because the thing is deterministic, right? So you can map from one point in
- L5 @ 2750s
right? So you can map from one point in right? So you can map from one point in P 0 to another point in P1 and then you P 0 to another point in P1 and then you P 0 to another point in P1 and then you just use that coupling as your new just use that coupling as your new just use that coupling as your new ground truth. So instead of like ground truth. So instead of like ground truth. So instead of like randomly sampling your your your randomly sampling your your your randomly sampling your your your coupling like what we do what we did in coupling like what we do what we did in coupling like what we do what we did in the original flow matching you just use the original flow matching you just use the original flow matching you just use that coupling as your ground truth and that coupling as your ground truth and that coupling as your ground truth and train for matching again and then it's train for matching again and then it's train for matching again and then it's going to get straightened up. >> Yeah. Go ahead. Because when you sample >> Yeah. Go ahead. Because when you sample um the more the more linear your clothes um the more the more linear your clothes um the more the more linear your clothes are, the less test. are, the less test. are, the less test. >> Yes, exactly. So, think about it, right? >> Yes, exactly. So, think about it, right? >> Yes, exactly. So, think about it, right? Um like if you have say if you have this Um like if you have say if you have this Um like if you have say if you have this dip, right? You have you have this dip dip, right? You have you have this dip dip, right? You have you have this dip then like like the safer bet for you to then like like the safer bet for you to then like like the safer bet for you to sample is to take two steps, right? sample is to take two steps, right? sample is to take two steps, right? because you don't know like basically if because you don't know like basically if
- L5 @ 2794s
because you don't know like basically if you if you only do one step there can be you if you only do one step there can be you if you only do one step there can be like a lot of um uh uh like like like a lot of um uh uh like like like a lot of um uh uh like like discretization error that is happening discretization error that is happening discretization error that is happening because of this dip. But if you have a because of this dip. But if you have a because of this dip. But if you have a very straight line then you can just do very straight line then you can just do very straight line then you can just do one step and it's straight. So one step one step and it's straight. So one step one step and it's straight. So one step is fine right? So straighter you are uh is fine right? So straighter you are uh is fine right? So straighter you are uh the better you are at this like few step the better you are at this like few step the better you are at this like few step sampling. How do you decide what to cover? How do you decide what to cover? >> By your previous model, right? By the >> By your previous model, right? By the >> By your previous model, right? By the model that you learned. So, model that you learned. So, model that you learned. So, so this blue this this this blue and so this blue this this this blue and so this blue this this this blue and green are the coupling induced by your green are the coupling induced by your green are the coupling induced by your learn the flow matching model and then learn the flow matching model and then learn the flow matching model and then you just use this this coupling and then you just use this this coupling and then you just use this this coupling and then do flow matching again on the new model. do flow matching again on the new model. do flow matching again on the new model. Yeah. >> It won't collide because it's >> It won't collide because it's It's a flow. It never cross. Yeah.
- L5 @ 2857s
>> The third image is the linear >> The third image is the linear interpolation of the new coupling. interpolation of the new coupling. interpolation of the new coupling. >> That what is that doing? >> That what is that doing? >> That what is that doing? >> Oh, this thing is just like show you the >> Oh, this thing is just like show you the >> Oh, this thing is just like show you the marginal kind of. marginal kind of. marginal kind of. >> Yeah. >> Yeah. >> The green is going up. But so on the >> The green is going up. But so on the >> The green is going up. But so on the >> Yeah. Yeah. Yeah. No. Remember how this >> Yeah. Yeah. Yeah. No. Remember how this >> Yeah. Yeah. Yeah. No. Remember how this you have a you have a concentration in you have a you have a concentration in you have a you have a concentration in the middle. you have a dip, right? So, the middle. you have a dip, right? So, the middle. you have a dip, right? So, it's sort of the same thing here. it's sort of the same thing here. it's sort of the same thing here. >> Is it when you're doing the sampling? >> Is it when you're doing the sampling? >> Is it when you're doing the sampling? >> No, no. Linear interpolation is just >> No, no. Linear interpolation is just >> No, no. Linear interpolation is just coupling. So, like you you you choose coupling. So, like you you you choose coupling. So, like you you you choose you you choose a you you choose a sample you you choose a you you choose a sample you you choose a you you choose a sample here, you choose a sample here and then here, you choose a sample here and then here, you choose a sample here and then you straight line and then you connect you straight line and then you connect you straight line and then you connect them with a straight line. them with a straight line. them with a straight line. >> Sample from both side. We mostly >> Sample from both side. We mostly >> Sample from both side. We mostly connected this side to that side and
- L5 @ 2897s
connected this side to that side and connected this side to that side and that side to this side. There's no that side to this side. There's no that side to this side. There's no there's no this anymore because because there's no this anymore because because there's no this anymore because because it won't be basically the the the it won't be basically the the the it won't be basically the the the coupling tells you that you will not go coupling tells you that you will not go coupling tells you that you will not go to the other side anymore because to the other side anymore because to the other side anymore because otherwise it will cross and then your otherwise it will cross and then your otherwise it will cross and then your stream will never cross. stream will never cross. stream will never cross. >> Yeah. >> Yeah. >> You have a data set that you know has >> You have a data set that you know has >> You have a data set that you know has multiple classes. Could you use this to multiple classes. Could you use this to multiple classes. Could you use this to disentangle them? disentangle them? disentangle them? >> Yes. And also actually um there's >> Yes. And also actually um there's >> Yes. And also actually um there's another paper I didn't put it here that another paper I didn't put it here that another paper I didn't put it here that if you already have a coupling in your if you already have a coupling in your if you already have a coupling in your data set you can just directly use it to data set you can just directly use it to data set you can just directly use it to train flow matching. You don't have to train flow matching. You don't have to train flow matching. You don't have to do random sampling. Yeah, I think do random sampling. Yeah, I think do random sampling. Yeah, I think someone has Yeah. someone has Yeah. someone has Yeah. >> If your flow matching model is not >> If your flow matching model is not >> If your flow matching model is not perfectly trained into this process, perfectly trained into this process,
- L5 @ 2936s
perfectly trained into this process, >> then you do it again. Yeah. Yeah. So, >> then you do it again. Yeah. Yeah. So, >> then you do it again. Yeah. Yeah. So, this is just you you get it this is just you you get it this is just you you get it straightened. It's not guaranteed that straightened. It's not guaranteed that straightened. It's not guaranteed that you're going to get straight uh like you're going to get straight uh like you're going to get straight uh like path on on on the on one go. Just like path on on on the on one go. Just like path on on on the on one go. Just like you're just going to get straighter and you're just going to get straighter and you're just going to get straighter and straighter kind of. Yeah, it it's straighter kind of. Yeah, it it's straighter kind of. Yeah, it it's definitely possible. Yeah. Oh. No. Okay. No, nobody. No. Okay. No, nobody. Cool. All right. So, just to summarize Cool. All right. So, just to summarize Cool. All right. So, just to summarize everything. So, just what's the everything. So, just what's the everything. So, just what's the difference between diffusion and flow difference between diffusion and flow difference between diffusion and flow matching in this context, right? So, matching in this context, right? So, matching in this context, right? So, diffusion is sort of like you are diffusion is sort of like you are diffusion is sort of like you are wandering in the wood and with a compass wandering in the wood and with a compass wandering in the wood and with a compass uh like in your hand. So, you know like uh like in your hand. So, you know like uh like in your hand. So, you know like you kind of know which direction you're you kind of know which direction you're you kind of know which direction you're going, but you're still like you're kind going, but you're still like you're kind going, but you're still like you're kind of just like exploring a little bit as of just like exploring a little bit as of just like exploring a little bit as well. you're just like kind of random well. you're just like kind of random
- L5 @ 2984s
well. you're just like kind of random random walking a little bit. Whereas for random walking a little bit. Whereas for random walking a little bit. Whereas for flow matching, you don't really do flow matching, you don't really do flow matching, you don't really do random walking anymore. You just like random walking anymore. You just like random walking anymore. You just like sit on the boat and then go go down the sit on the boat and then go go down the sit on the boat and then go go down the stream uh on the river. So this is kind stream uh on the river. So this is kind stream uh on the river. So this is kind of what like to conceptually what is of what like to conceptually what is of what like to conceptually what is happening. Yeah. happening. Yeah. happening. Yeah. >> Like is this used now? Like if it's so >> Like is this used now? Like if it's so >> Like is this used now? Like if it's so good this good this good this >> great question. >> great question. >> great question. >> Do you know flux? Do you know the the >> Do you know flux? Do you know the the >> Do you know flux? Do you know the the black for that's full matching? Yeah. black for that's full matching? Yeah. black for that's full matching? Yeah. This is like the state of the art This is like the state of the art This is like the state of the art basically. It's actually crazy good and basically. It's actually crazy good and basically. It's actually crazy good and it's so simple, right? Okay. So, so far it's so simple, right? Okay. So, so far it's so simple, right? Okay. So, so far we have seen so many gen models and we we have seen so many gen models and we we have seen so many gen models and we know that the diffusion model and know that the diffusion model and know that the diffusion model and scorebased model are same thing and scorebased model are same thing and scorebased model are same thing and today we learned flow matching. Now the today we learned flow matching. Now the today we learned flow matching. Now the question is is flow matching also the
- L5 @ 3027s
question is is flow matching also the question is is flow matching also the same thing because if you think about it same thing because if you think about it same thing because if you think about it you're also like adding a lot of noise you're also like adding a lot of noise you're also like adding a lot of noise like and then d noiseis is kind of like and then d noiseis is kind of like and then d noiseis is kind of dinoising right you're just like a lot dinoising right you're just like a lot dinoising right you're just like a lot of noisy samples in the middle is are of noisy samples in the middle is are of noisy samples in the middle is are they the same thing? they the same thing? they the same thing? What do we think kind of actually? So basically what you kind of actually? So basically what you can do so this is what we had before can do so this is what we had before can do so this is what we had before right uh we have this like very straight right uh we have this like very straight right uh we have this like very straight line but what if you define something line but what if you define something line but what if you define something because the flow matching theory can be because the flow matching theory can be because the flow matching theory can be applied to any path right like what applied to any path right like what applied to any path right like what Dwanch was asking so we can also define Dwanch was asking so we can also define Dwanch was asking so we can also define a path from what we just learned uh on a path from what we just learned uh on a path from what we just learned uh on Tuesday right so you can define a a v Tuesday right so you can define a a v Tuesday right so you can define a a v path a v uh like diffusion path and you path a v uh like diffusion path and you path a v uh like diffusion path and you can write everything as if it is flow
- L5 @ 3072s
can write everything as if it is flow can write everything as if it is flow matching as well and then you can matching as well and then you can matching as well and then you can actually just recover for V SD with actually just recover for V SD with actually just recover for V SD with obviously obviously obviously with with the noise adding to it. with with the noise adding to it. with with the noise adding to it. And what about if we do what can can And what about if we do what can can And what about if we do what can can what about the other way around, right? what about the other way around, right? what about the other way around, right? Can we do score SDE Can we do score SDE Can we do score SDE transform into an OD a flow OD? Can we transform into an OD a flow OD? Can we transform into an OD a flow OD? Can we do that? What do we think by looking at do that? What do we think by looking at do that? What do we think by looking at this this this formula? Okay, just get rid of the the G part, Okay, just get rid of the the G part, right? Is that what you said? All right. right? Is that what you said? All right. right? Is that what you said? All right. So, basically at this part look like a So, basically at this part look like a So, basically at this part look like a velocity, right? velocity, right? velocity, right? But the problem is like the G part, the But the problem is like the G part, the But the problem is like the G part, the the brown in motion part, this part the brown in motion part, this part the brown in motion part, this part actually induce some probability because actually induce some probability because actually induce some probability because it's a gausian, right? So, we still it's a gausian, right? So, we still it's a gausian, right? So, we still induce some probability there. So, we induce some probability there. So, we induce some probability there. So, we still need to take care of that part. still need to take care of that part.
- L5 @ 3123s
still need to take care of that part. But it turns out that we actually do But it turns out that we actually do But it turns out that we actually do know how to take care of that part. And know how to take care of that part. And know how to take care of that part. And this thing is called focal plank PDE. this thing is called focal plank PDE. this thing is called focal plank PDE. All right? uh the the name is really All right? uh the the name is really All right? uh the the name is really fancy but basically what it means is fancy but basically what it means is fancy but basically what it means is that like if you if you have some that like if you if you have some that like if you if you have some probability in it the focal point PTE um probability in it the focal point PTE um probability in it the focal point PTE um it's going to tell you it's going to tell you it's going to tell you how your probability changes through how your probability changes through how your probability changes through this through this PTE that you can this through this PTE that you can this through this PTE that you can derive uh and if you're interested in derive uh and if you're interested in derive uh and if you're interested in the derivation of this PD for the the derivation of this PD for the the derivation of this PD for the interest of time we're not going to go interest of time we're not going to go interest of time we're not going to go over it but it's in the in the the over it but it's in the in the the over it but it's in the in the the principle of diffusion model book so you principle of diffusion model book so you principle of diffusion model book so you can take a look at it um but basically can take a look at it um but basically can take a look at it um but basically Okay. And now we have probability flow Okay. And now we have probability flow Okay. And now we have probability flow OD, right? So this is the just um say we
- L5 @ 3164s
OD, right? So this is the just um say we OD, right? So this is the just um say we have this particular OD and it turns out have this particular OD and it turns out have this particular OD and it turns out that it's contin continuity equation that it's contin continuity equation that it's contin continuity equation which is like the the PD of the change which is like the the PD of the change which is like the the PD of the change of um probability which is what we of um probability which is what we of um probability which is what we learned today by using the divergence is learned today by using the divergence is learned today by using the divergence is actually so if you just like plug actually so if you just like plug actually so if you just like plug everything in and then you do a lot of everything in and then you do a lot of everything in and then you do a lot of algebra and and calculus uh you are algebra and and calculus uh you are algebra and and calculus uh you are actually going to get the same thing actually going to get the same thing actually going to get the same thing literally the same thing. So this is literally the same thing. So this is literally the same thing. So this is just saying that the rate of like the just saying that the rate of like the just saying that the rate of like the rate of change in probability rate of change in probability rate of change in probability from this score SD or from this 4 SD from this score SD or from this 4 SD from this score SD or from this 4 SD they share the same uh which is the same they share the same uh which is the same they share the same uh which is the same as this probability flow OD as this probability flow OD as this probability flow OD this and the same PD for the rate of this and the same PD for the rate of this and the same PD for the rate of change for the probability means that change for the probability means that
- L5 @ 3210s
change for the probability means that the marginalss are all the same. So this the marginalss are all the same. So this the marginalss are all the same. So this just means that now if you have your just means that now if you have your just means that now if you have your score function instead of sampling from score function instead of sampling from score function instead of sampling from an SD you can do OD samples now. Yay. Yay. Yay. Yay. Well, no excitement. It's okay. Um what Well, no excitement. It's okay. Um what Well, no excitement. It's okay. Um what else can you else can you else can you with OD? Right. I think someone someone with OD? Right. I think someone someone with OD? Right. I think someone someone said before that like we can do density said before that like we can do density said before that like we can do density estimation with this, right? So estimation with this, right? So estimation with this, right? So literally this is true, right? So you literally this is true, right? So you literally this is true, right? So you can literally just um um swap in the the can literally just um um swap in the the can literally just um um swap in the the velocity here and then you just like go velocity here and then you just like go velocity here and then you just like go back in time through your OD and then back in time through your OD and then back in time through your OD and then you're able to calculate the log density you're able to calculate the log density you're able to calculate the log density of your data point. of your data point. of your data point. Cool. And Cool. And Cool. And what about density estimation for what about density estimation for what about density estimation for diffusion, right? diffusion, right? diffusion, right? Oh, density estimation means that like Oh, density estimation means that like Oh, density estimation means that like just like to calculate uh the likelihood just like to calculate uh the likelihood just like to calculate uh the likelihood of the the the the day. of the the the the day. of the the the the day. >> That's a great question. Um so for
- L5 @ 3276s
>> That's a great question. Um so for >> That's a great question. Um so for example, I don't know if you guys example, I don't know if you guys example, I don't know if you guys familiar with the RL. familiar with the RL. familiar with the RL. >> All right, cool. Anyway, >> All right, cool. Anyway, >> All right, cool. Anyway, and so for density estimation for and so for density estimation for and so for density estimation for diffusion, uh the easiest thing is that diffusion, uh the easiest thing is that diffusion, uh the easiest thing is that because the loss function that we because the loss function that we because the loss function that we derived from two lectures ago is a derived from two lectures ago is a derived from two lectures ago is a elbow, right? So we can just directly elbow, right? So we can just directly elbow, right? So we can just directly use this elbow as an estimate of the use this elbow as an estimate of the use this elbow as an estimate of the likelihood. That's fine. But this is not likelihood. That's fine. But this is not likelihood. That's fine. But this is not accurate because it's a elbow. It's a accurate because it's a elbow. It's a accurate because it's a elbow. It's a lower bound. It's it's not really the lower bound. It's it's not really the lower bound. It's it's not really the the real log log likelihood, right? So the real log log likelihood, right? So the real log log likelihood, right? So is there another way to do it? I feel like someone someone here knows I feel like someone someone here knows already. Okay. What do we think? What do we Okay. What do we think? What do we think? We know we know how to calculate the We know we know how to calculate the exact log likelihood from OD. What do we exact log likelihood from OD. What do we exact log likelihood from OD. What do we do if we have a score SDE? What do we do if we have a score SDE? What do we do if we have a score SDE? What do we do? do? do? Use the use the corresponding PF OD of
- L5 @ 3340s
Use the use the corresponding PF OD of Use the use the corresponding PF OD of the SDE. Right? So you just plug it in the SDE. Right? So you just plug it in the SDE. Right? So you just plug it in and then it's the same and then it's the same and then it's the same as for matching. Yes, Yes, >> it's like that. >> it's like that. >> it's like that. >> Um, >> Um, >> Um, one one one one one >> Okay. >> Okay. >> Okay. >> Yeah, >> Yeah, >> Yeah, you can sample with an OD. What is like you can sample with an OD. What is like you can sample with an OD. What is like sampling? sampling? sampling? >> Oh, yes. Yes. So you can sample with >> Oh, yes. Yes. So you can sample with >> Oh, yes. Yes. So you can sample with this OD meaning that like you can you this OD meaning that like you can you this OD meaning that like you can you can literally do uh let's see you can can literally do uh let's see you can can literally do uh let's see you can you can literally just do numerical you can literally just do numerical you can literally just do numerical integration from x0 and then you do x0 integration from x0 and then you do x0 integration from x0 and then you do x0 plus um integrate from zero to one plus um integrate from zero to one plus um integrate from zero to one velocity and this velocity is just this velocity and this velocity is just this velocity and this velocity is just this odz >> uh what do you mean the sampling is only >> uh what do you mean the sampling is only x0 oh you mean like to sample from a x0 oh you mean like to sample from a x0 oh you mean like to sample from a distribution yes that's right Yeah. distribution yes that's right Yeah. distribution yes that's right Yeah. Cool. Cool. Oh, yeah. Yeah. Yeah. Oh, yeah. Yeah. Yeah. >> Yeah. >> Yeah. >> A lot of uh uh lot of PD getting >> A lot of uh uh lot of PD getting
- L5 @ 3427s
>> A lot of uh uh lot of PD getting converted to OD. Is it are we just doing converted to OD. Is it are we just doing converted to OD. Is it are we just doing it because it's simpler or do we you it because it's simpler or do we you it because it's simpler or do we you want a better velocity? want a better velocity? want a better velocity? >> Uh no you mean here here it's because >> Uh no you mean here here it's because >> Uh no you mean here here it's because it's exact. So in in uh in comparison to it's exact. So in in uh in comparison to it's exact. So in in uh in comparison to the elbow right if you compare to the the elbow right if you compare to the the elbow right if you compare to the elbow the elbow is a lower bound of your elbow the elbow is a lower bound of your elbow the elbow is a lower bound of your um of your lo likelihood which is not um of your lo likelihood which is not um of your lo likelihood which is not exact but here we can actually get an exact but here we can actually get an exact but here we can actually get an exact log likelihood. exact log likelihood. exact log likelihood. Yeah. Yeah. >> Intrinsically Asian suffer from >> Intrinsically Asian suffer from >> Intrinsically Asian suffer from diversity, right? Because diversity, right? Because diversity, right? Because >> it suffers from uh discretization error. >> it suffers from uh discretization error. >> it suffers from uh discretization error. Is that what you mean? Like basically if Is that what you mean? Like basically if Is that what you mean? Like basically if you take the step and then you take the step and then you take the step and then because uh because uh because uh >> ah uh well well here you're calculating >> ah uh well well here you're calculating >> ah uh well well here you're calculating the the the log likelihood. So it's not the the the log likelihood. So it's not the the the log likelihood. So it's not diversity but but yes so like diversity but but yes so like
- L5 @ 3473s
diversity but but yes so like essentially um if you're sampling from a essentially um if you're sampling from a essentially um if you're sampling from a SDE then it's going to be yeah because SDE then it's going to be yeah because SDE then it's going to be yeah because you have exploration right so it's more you have exploration right so it's more you have exploration right so it's more diverse than sampling from OD which is diverse than sampling from OD which is diverse than sampling from OD which is completely deterministic completely deterministic completely deterministic >> yeah that's right Cool. Cool. Not cool. Okay. Not cool. Okay. Not cool. Okay. >> So, if if I want to calculate the >> So, if if I want to calculate the >> So, if if I want to calculate the likelihood of X1, likelihood of X1, likelihood of X1, >> I first need to find the corresponding >> I first need to find the corresponding >> I first need to find the corresponding X0 for me to start integrating. And X0 for me to start integrating. And X0 for me to start integrating. And >> hey, you have everyone you should know >> hey, you have everyone you should know >> hey, you have everyone you should know this. But anyway, continue. Yeah. Yeah. this. But anyway, continue. Yeah. Yeah. this. But anyway, continue. Yeah. Yeah. Yeah. Yeah. Oh yeah. So yeah. So essentially what Oh yeah. So yeah. So essentially what Oh yeah. So yeah. So essentially what you do is you you remember how you you you do is you you remember how you you you do is you you remember how you you you reverse the chain, right? >> Yeah, that is right. >> Yeah, that is right. >> That is right. And which is why this x0 >> That is right. And which is why this x0 >> That is right. And which is why this x0 k has a hat on it. So this is a reverse k has a hat on it. So this is a reverse k has a hat on it. So this is a reverse the x0. So you first reverse your the x0. So you first reverse your the x0. So you first reverse your sampling chain through your by by by
- L5 @ 3546s
sampling chain through your by by by sampling chain through your by by by backward integrating your velocity and backward integrating your velocity and backward integrating your velocity and then during the backward integration you then during the backward integration you then during the backward integration you can alo calculate the divergence at the can alo calculate the divergence at the can alo calculate the divergence at the same time. Yeah. So you you need to find the Yeah. So you you need to find the corresponding chain which is corresponding chain which is corresponding chain which is deterministic. But uh But uh >> p of x0 should be constant. >> p of x0 should be constant. >> p of x0 should be constant. >> Oh, you mean p of x? Yeah, p 0 is >> Oh, you mean p of x? Yeah, p 0 is >> Oh, you mean p of x? Yeah, p 0 is constant. Yeah, that's right. Yeah. constant. Yeah, that's right. Yeah. constant. Yeah, that's right. Yeah. Yeah. If if if it's a uniform. Yeah. If if if it's a uniform. Yeah. If if if it's a uniform. All right. All right. All right. More question. More question. More question. All good. All right. All good. All right. All good. All right. Okay. So Okay. So Okay. So to summarize everything and we have seen to summarize everything and we have seen to summarize everything and we have seen three models so far. Uh diffusion model, three models so far. Uh diffusion model, three models so far. Uh diffusion model, squarebased models, flow matching all squarebased models, flow matching all squarebased models, flow matching all model. They are more or less all the model. They are more or less all the model. They are more or less all the same thing. Same same different. same thing. Same same different. same thing. Same same different. I feel like saying this I'm going to I feel like saying this I'm going to I feel like saying this I'm going to offend a lot of people but but yeah know offend a lot of people but but yeah know offend a lot of people but but yeah know you know just like you know you just add you know just like you know you just add
- L5 @ 3612s
you know just like you know you just add something divide something you it's the something divide something you it's the something divide something you it's the same thing. Okay. same thing. Okay. same thing. Okay. >> Yeah everything is the same thing. Uh >> Yeah everything is the same thing. Uh >> Yeah everything is the same thing. Uh but yeah um so yeah congrat oh okay we but yeah um so yeah congrat oh okay we but yeah um so yeah congrat oh okay we haven't congrat Okay. haven't congrat Okay. haven't congrat Okay. >> Uh so if we can still compute the >> Uh so if we can still compute the >> Uh so if we can still compute the likelihood using flow matching why does likelihood using flow matching why does likelihood using flow matching why does it come under likelihood free like it come under likelihood free like it come under likelihood free like what's the classification? Oh here what's the classification? Oh here what's the classification? Oh here likelihood of free means that the likelihood of free means that the likelihood of free means that the training loss doesn't have likelihood training loss doesn't have likelihood training loss doesn't have likelihood it's not maximizing the likelihood but it's not maximizing the likelihood but it's not maximizing the likelihood but you can still compute likelihood right you can still compute likelihood right you can still compute likelihood right score same for scorebased model you can score same for scorebased model you can score same for scorebased model you can compute the likelihood compute the likelihood compute the likelihood through the same o pfod thing right yeah through the same o pfod thing right yeah through the same o pfod thing right yeah all right all right so congratulations all right all right so congratulations all right all right so congratulations we gone through the basics now yay now we gone through the basics now yay now we gone through the basics now yay now we know what is diffusion one is full we know what is diffusion one is full we know what is diffusion one is full matching yay um yeah so starting from matching yay um yeah so starting from matching yay um yeah so starting from next week we're gonna be able to see next week we're gonna be able to see
- L5 @ 3661s
next week we're gonna be able to see like like like uh different options to further uh different options to further uh different options to further improve what we have learned. Um so for improve what we have learned. Um so for improve what we have learned. Um so for example like I I guess you guys have if example like I I guess you guys have if example like I I guess you guys have if you guys have done the homework you you guys have done the homework you you guys have done the homework you should know this by now there's so many should know this by now there's so many should know this by now there's so many knobs that you can tune right for knobs that you can tune right for knobs that you can tune right for diffusion models like what are you use diffusion models like what are you use diffusion models like what are you use how many number of steps and stuff like how many number of steps and stuff like how many number of steps and stuff like that right so you know we're gonna we're that right so you know we're gonna we're that right so you know we're gonna we're going to learn about that like basically going to learn about that like basically going to learn about that like basically uh there's a paper that like just uh there's a paper that like just uh there's a paper that like just systematically study this uh and we are systematically study this uh and we are systematically study this uh and we are like for those of you who did uh the like for those of you who did uh the like for those of you who did uh the homework you should also notice that homework you should also notice that homework you should also notice that diffusion models sample really really diffusion models sample really really diffusion models sample really really slowly and it's like just like slowly and it's like just like slowly and it's like just like It takes so much time. So how do we make It takes so much time. So how do we make It takes so much time. So how do we make the generation faster? And we're going the generation faster? And we're going
- L5 @ 3701s
the generation faster? And we're going to learn uh to do it with or without to learn uh to do it with or without to learn uh to do it with or without training retraining of the model. And training retraining of the model. And training retraining of the model. And lastly, right now we all the thing that lastly, right now we all the thing that lastly, right now we all the thing that we learn are unconditional generations. we learn are unconditional generations. we learn are unconditional generations. So how can we make the generation more So how can we make the generation more So how can we make the generation more controllable or conditional, right? And controllable or conditional, right? And controllable or conditional, right? And we are actually going to learn it uh to we are actually going to learn it uh to we are actually going to learn it uh to do it with training and also without do it with training and also without do it with training and also without training which is very interesting. All training which is very interesting. All training which is very interesting. All right. Uh that's it for today. I'll see right. Uh that's it for today. I'll see right. Uh that's it for today. I'll see you next week. Uh stay warm this uh this you next week. Uh stay warm this uh this you next week. Uh stay warm this uh this uh this this uh weekend. There's a lot uh this this uh weekend. There's a lot uh this this uh weekend. There's a lot of flow going on.
diffusion-principles-vizuara
- L1 @ 4s
Hello everyone and welcome to the next Hello everyone and welcome to the next lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion models. diffusion models. diffusion models. In the last lecture we started by In the last lecture we started by In the last lecture we started by introducing flow models. introducing flow models. introducing flow models. In particular we first understood the In particular we first understood the In particular we first understood the meaning of flow itself and later we meaning of flow itself and later we meaning of flow itself and later we moved on to understanding what is the moved on to understanding what is the moved on to understanding what is the meaning of flow modeling. meaning of flow modeling. meaning of flow modeling. Let us do a brief recap. Let us do a brief recap. Let us do a brief recap. of what we covered in the last lecture. of what we covered in the last lecture. of what we covered in the last lecture. Before we move ahead to this lecture, and this box includes a lot of tiny and this box includes a lot of tiny particles. particles. particles. So this is particle P1. Let's focus on So this is particle P1. Let's focus on So this is particle P1. Let's focus on that particle. that particle. that particle. Now imagine that for some reason these Now imagine that for some reason these Now imagine that for some reason these particles are moving inside the box. particles are moving inside the box. particles are moving inside the box. So the particles are following specific So the particles are following specific So the particles are following specific trajectories. trajectories. trajectories. For example, this is the trajectory For example, this is the trajectory For example, this is the trajectory which is followed by P1. Then we take another point P2 Then we take another point P2 and we find that this is the trajectory and we find that this is the trajectory and we find that this is the trajectory which is followed by the point P2. which is followed by the point P2. which is followed by the point P2. Similarly, every single point in the box Similarly, every single point in the box
- L1 @ 87s
Similarly, every single point in the box is following a specific trajectory is following a specific trajectory is following a specific trajectory and the particles are moving as and the particles are moving as and the particles are moving as [clears throat] time is progressing [clears throat] time is progressing [clears throat] time is progressing ahead. ahead. ahead. You can imagine this to be like a fluid You can imagine this to be like a fluid You can imagine this to be like a fluid which is moving with time. which is moving with time. which is moving with time. [snorts] Okay. Now [snorts] Okay. Now [snorts] Okay. Now we ask a simple question. What is making these particles move What is making these particles move according to the specific trajectories? For example, For example, at every point, at every point, at every point, [snorts] we know that there is a [snorts] we know that there is a [snorts] we know that there is a velocity vector which is causing the velocity vector which is causing the velocity vector which is causing the particle to move in that direction. >> [snorts] >> [snorts] >> So we try to think about >> So we try to think about >> So we try to think about what is what is responsible for the what is what is responsible for the what is what is responsible for the trajectories of the particles to behave trajectories of the particles to behave trajectories of the particles to behave in a certain way. in a certain way. in a certain way. It's almost like a detective who is It's almost like a detective who is It's almost like a detective who is trying to find out the reason behind a trying to find out the reason behind a trying to find out the reason behind a mysterious behavior. mysterious behavior. mysterious behavior. Now our detective tells us that the Now our detective tells us that the Now our detective tells us that the reason the particles move in this way is reason the particles move in this way is reason the particles move in this way is because because because there is a velocity field which is there is a velocity field which is there is a velocity field which is changing changing changing with not just space but it is changing
- L1 @ 169s
with not just space but it is changing with not just space but it is changing with time also. [snorts] Why it is with time also. [snorts] Why it is with time also. [snorts] Why it is changing with space? It's because the changing with space? It's because the changing with space? It's because the trajectories for all all the particles trajectories for all all the particles trajectories for all all the particles in the box are different. So it has to in the box are different. So it has to in the box are different. So it has to change with space. change with space. change with space. Well, it is not completely necessary Well, it is not completely necessary Well, it is not completely necessary that it changes with time as well. But that it changes with time as well. But that it changes with time as well. But we'll consider a more general case where we'll consider a more general case where we'll consider a more general case where the velocity field [snorts] which is the the velocity field [snorts] which is the the velocity field [snorts] which is the culprit which the the detective has culprit which the the detective has culprit which the the detective has found out is changing with both space found out is changing with both space found out is changing with both space and with time as well. [snorts] and with time as well. [snorts] and with time as well. [snorts] Let us denote this velocity field as u Let us denote this velocity field as u Let us denote this velocity field as u of of of x t. x t. x t. [snorts] The detective has done its job. The The detective has done its job. The detective has found that uh there is a detective has found that uh there is a detective has found that uh there is a velocity field which is um responsible velocity field which is um responsible velocity field which is um responsible for the movement of the trajectories or for the movement of the trajectories or for the movement of the trajectories or the movement of the particles and the the movement of the particles and the the movement of the particles and the generation of these trajectories. generation of these trajectories. generation of these trajectories. Now we approach another person. Let's
- L1 @ 224s
Now we approach another person. Let's Now we approach another person. Let's call that person uh detective number call that person uh detective number call that person uh detective number two. And we ask the question well uh two. And we ask the question well uh two. And we ask the question well uh let's say I give you a velocity field. let's say I give you a velocity field. let's say I give you a velocity field. [snorts] [snorts] [snorts] Can you tell me the trajectories of all Can you tell me the trajectories of all Can you tell me the trajectories of all the particles in the box and how exactly the particles in the box and how exactly the particles in the box and how exactly they will move? they will move? they will move? The detective thinks and says um okay so The detective thinks and says um okay so The detective thinks and says um okay so this looks like a different problem now this looks like a different problem now this looks like a different problem now that you have used the services of that you have used the services of that you have used the services of detective one and understood that there detective one and understood that there detective one and understood that there is an underlying velocity field which is is an underlying velocity field which is is an underlying velocity field which is responsible for the movement of these responsible for the movement of these responsible for the movement of these particles now you're trying to find out particles now you're trying to find out particles now you're trying to find out [snorts] given the velocity field can I [snorts] given the velocity field can I [snorts] given the velocity field can I generate the trajectory so essentially generate the trajectory so essentially generate the trajectory so essentially you're trying to solve the reverse you're trying to solve the reverse you're trying to solve the reverse problem problem problem [snorts] uh so detective one first [snorts] uh so detective one first [snorts] uh so detective one first detective two thinks that okay initially detective two thinks that okay initially detective two thinks that okay initially These people are thinking about going These people are thinking about going These people are thinking about going from trajectories and then calculating
- L1 @ 276s
from trajectories and then calculating from trajectories and then calculating the velocity field. [snorts] Now they the velocity field. [snorts] Now they the velocity field. [snorts] Now they are thinking in the reverse way from are thinking in the reverse way from are thinking in the reverse way from given the velocity field can I calculate given the velocity field can I calculate given the velocity field can I calculate the trajectories of these particles. [snorts] The detective too remembers uh [snorts] The detective too remembers uh some of the some of the some of the theory he had learned in mathematics in theory he had learned in mathematics in theory he had learned in mathematics in college. and comes up with an ordinary college. and comes up with an ordinary college. and comes up with an ordinary differential equation and thinks that differential equation and thinks that differential equation and thinks that okay this is straightforward. We can okay this is straightforward. We can okay this is straightforward. We can simply write down dx of t of t dt dt dt is equal to u of x t. is equal to u of x t. is equal to u of x t. [snorts] [snorts] A simple differential equation which is A simple differential equation which is A simple differential equation which is an ordinary differential equation is an ordinary differential equation is an ordinary differential equation is written to written to written to [snorts] calculate the trajectories. [snorts] calculate the trajectories. [snorts] calculate the trajectories. But then how do we go from here to But then how do we go from here to But then how do we go from here to calculating the trajectories in the calculating the trajectories in the calculating the trajectories in the first place? Well, what you do is you first place? Well, what you do is you first place? Well, what you do is you discretise this discretise this discretise this by saying delta x by delta t is equal to by saying delta x by delta t is equal to by saying delta x by delta t is equal to mu of u of x t >> [snorts] >> So let's say you start at >> So let's say you start at >> So let's say you start at x0 x0 x0 which is a specific point in space.
- L1 @ 364s
which is a specific point in space. which is a specific point in space. [snorts] Let's say this is x0. Now what is x1? X1 is equal to Now what is x1? X1 is equal to X0 + delta X X0 + delta X X0 + delta X and delta X can be written as and delta X can be written as and delta X can be written as delta T * delta T * delta T * U of the velocity at that point X0 and U of the velocity at that point X0 and U of the velocity at that point X0 and the initial time stamp. So let's call it the initial time stamp. So let's call it the initial time stamp. So let's call it zero. zero. zero. So now you get X1. So now you get X1. So now you get X1. So the particle has moved from X0 to X1. So the particle has moved from X0 to X1. So the particle has moved from X0 to X1. Similarly, you can calculate x2 Similarly, you can calculate x2 Similarly, you can calculate x2 x3 and you can get the entire trajectory x3 and you can get the entire trajectory x3 and you can get the entire trajectory of the particle. [snorts] of the particle. [snorts] of the particle. [snorts] Why are we doing this? Well, this means Why are we doing this? Well, this means Why are we doing this? Well, this means that if I know the velocity field and that if I know the velocity field and that if I know the velocity field and how it changes with space and time, I how it changes with space and time, I how it changes with space and time, I can exactly pinpoint the trajectories of can exactly pinpoint the trajectories of can exactly pinpoint the trajectories of all the single part all the particles in all the single part all the particles in all the single part all the particles in the box. the box. the box. So, okay fine. So, that is great. Now so So, okay fine. So, that is great. Now so So, okay fine. So, that is great. Now so far we have looked at individual far we have looked at individual far we have looked at individual particles. We have looked at their
- L1 @ 431s
particles. We have looked at their particles. We have looked at their trajectories. We have looked at the trajectories. We have looked at the trajectories. We have looked at the mysterious force that is causing these mysterious force that is causing these mysterious force that is causing these trajectories. Then we looked at the trajectories. Then we looked at the trajectories. Then we looked at the reverse problem. If we know this reverse problem. If we know this reverse problem. If we know this mysterious force, can we calculate the mysterious force, can we calculate the mysterious force, can we calculate the trajectories? And the answer is yes. trajectories? And the answer is yes. trajectories? And the answer is yes. Now let's come to the main part of uh Now let's come to the main part of uh Now let's come to the main part of uh the discussion which is flow. And now the discussion which is flow. And now the discussion which is flow. And now once we understand this understanding once we understand this understanding once we understand this understanding flow is fairly straightforward. Flow is flow is fairly straightforward. Flow is flow is fairly straightforward. Flow is defined as a collection of the defined as a collection of the defined as a collection of the trajectories of all these particles. trajectories of all these particles. trajectories of all these particles. Every single particle moves in a certain Every single particle moves in a certain Every single particle moves in a certain way, right? So you aggregate the way, right? So you aggregate the way, right? So you aggregate the trajectories of all the particles and trajectories of all the particles and trajectories of all the particles and you call that to be flow. you call that to be flow. you call that to be flow. So it's fairly intuitive what flow So it's fairly intuitive what flow So it's fairly intuitive what flow means. means. means. So a visual description of the flow for So a visual description of the flow for So a visual description of the flow for this same box would look as something this same box would look as something this same box would look as something like follows. like follows. like follows. you have uh all the particles following you have uh all the particles following you have uh all the particles following their trajectories their trajectories
- L1 @ 488s
their trajectories and you collect all these trajectories and you collect all these trajectories and you collect all these trajectories and you call them as a flow map or just and you call them as a flow map or just and you call them as a flow map or just flow. flow. flow. A very [snorts] intuitive way to A very [snorts] intuitive way to A very [snorts] intuitive way to visualize this is to um take a square visualize this is to um take a square visualize this is to um take a square grid, put the square grid on the box and grid, put the square grid on the box and grid, put the square grid on the box and then see how the square grid deforms then see how the square grid deforms then see how the square grid deforms with time. We actually saw this in the with time. We actually saw this in the with time. We actually saw this in the uh last lecture uh last lecture uh last lecture where uh the square grid is nothing but where uh the square grid is nothing but where uh the square grid is nothing but a representation of let's say it's a 4x4 a representation of let's say it's a 4x4 a representation of let's say it's a 4x4 grid 16 points in space and then the grid 16 points in space and then the grid 16 points in space and then the points deform but then you exactly see points deform but then you exactly see points deform but then you exactly see how the grid deforms with time. So it it how the grid deforms with time. So it it how the grid deforms with time. So it it helps you to visually understand what helps you to visually understand what helps you to visually understand what the flow is doing. Is it a spiral flow? the flow is doing. Is it a spiral flow? the flow is doing. Is it a spiral flow? Is it a flow which is um you know Is it a flow which is um you know Is it a flow which is um you know converging towards a sink etc. So you converging towards a sink etc. So you converging towards a sink etc. So you can get a fair idea about what the flow
- L1 @ 541s
can get a fair idea about what the flow can get a fair idea about what the flow is representing is representing is representing by by by having this square grid and having this square grid and having this square grid and understanding how individual points in understanding how individual points in understanding how individual points in this square grid are deforming with this square grid are deforming with this square grid are deforming with time. We saw a very nice example of this time. We saw a very nice example of this time. We saw a very nice example of this in the last lecture. Let me just show in the last lecture. Let me just show in the last lecture. Let me just show that to you. that to you. that to you. Yeah, we saw this example where this is Yeah, we saw this example where this is Yeah, we saw this example where this is the square grid. Imagine you immerse the the square grid. Imagine you immerse the the square grid. Imagine you immerse the square grid in the box and then see how square grid in the box and then see how square grid in the box and then see how it deforms with time. So [snorts] this it deforms with time. So [snorts] this it deforms with time. So [snorts] this is exactly how you can visualize the is exactly how you can visualize the is exactly how you can visualize the flow. flow. Okay. Now this is great. Uh we have Okay. Now this is great. Uh we have Okay. Now this is great. Uh we have understood what flow is. Now the understood what flow is. Now the understood what flow is. Now the question is what is the meaning of a question is what is the meaning of a question is what is the meaning of a flow model? [snorts] flow model? [snorts] flow model? [snorts] Uh and this this brings us to the whole Uh and this this brings us to the whole Uh and this this brings us to the whole objective of why we are focusing on flow objective of why we are focusing on flow objective of why we are focusing on flow flow models in the first place. So far flow models in the first place. So far
- L1 @ 590s
flow models in the first place. So far whatever we have discussed can be whatever we have discussed can be whatever we have discussed can be understood by a physics student easily understood by a physics student easily understood by a physics student easily with absolutely no background in AI or with absolutely no background in AI or with absolutely no background in AI or ML right because this is fairly ML right because this is fairly ML right because this is fairly intuitive stuff in the field of fluid intuitive stuff in the field of fluid intuitive stuff in the field of fluid mechanics or physics or [snorts] even mechanics or physics or [snorts] even mechanics or physics or [snorts] even basic for that matter. Now we relate basic for that matter. Now we relate basic for that matter. Now we relate this to the broader problem which we are this to the broader problem which we are this to the broader problem which we are trying to solve. Remember the broader trying to solve. Remember the broader trying to solve. Remember the broader problem that we are trying to solve is problem that we are trying to solve is problem that we are trying to solve is we want to calculate the probability we want to calculate the probability we want to calculate the probability distribution of the underlying data so distribution of the underlying data so distribution of the underlying data so that we can sample from it. So that we can sample from it. So that we can sample from it. So essentially we are in the domain of deep essentially we are in the domain of deep essentially we are in the domain of deep generative modeling. So [snorts] what we are trying to do So [snorts] what we are trying to do let's say you are given images of cats let's say you are given images of cats let's say you are given images of cats you are given one to 100 images of cats you are given one to 100 images of cats you are given one to 100 images of cats and someone tells you and someone tells you and someone tells you okay generate a new image of a cat. okay generate a new image of a cat.
- L1 @ 647s
okay generate a new image of a cat. Now the way you solve this problem is Now the way you solve this problem is Now the way you solve this problem is you try to find a distribution you try to find a distribution you try to find a distribution which which which governs the behavior or the properties governs the behavior or the properties governs the behavior or the properties of all these images of cats and later of all these images of cats and later of all these images of cats and later you just sample from that distribution. So [snorts] the biggest challenge that So [snorts] the biggest challenge that we are trying to solve is we want to go we are trying to solve is we want to go we are trying to solve is we want to go from data to distribution which is not from data to distribution which is not from data to distribution which is not easy because we have absolutely no idea easy because we have absolutely no idea easy because we have absolutely no idea what the underlying distribution is what the underlying distribution is what the underlying distribution is going to look like. [snorts] Okay. Now uh we have seen how to solve Okay. Now uh we have seen how to solve Okay. Now uh we have seen how to solve this classes of problems. We have the this classes of problems. We have the this classes of problems. We have the first model which we used to solve this first model which we used to solve this first model which we used to solve this was variational autoenccoders. was variational autoenccoders. was variational autoenccoders. Then we looked at diffusion models which Then we looked at diffusion models which Then we looked at diffusion models which is another class of models to solve this is another class of models to solve this is another class of models to solve this problem. What typically happens in problem. What typically happens in problem. What typically happens in diffusion models is diffusion models is diffusion models is to find this distribution you start with to find this distribution you start with to find this distribution you start with a gshian distribution that is noise and a gshian distribution that is noise and
- L1 @ 710s
a gshian distribution that is noise and you end up with the actual distribution. you end up with the actual distribution. you end up with the actual distribution. So the path that we are going to follow So the path that we are going to follow So the path that we are going to follow here is that in in flow models is here is that in in flow models is here is that in in flow models is [snorts] [snorts] um we are going to assume that we have um we are going to assume that we have um we are going to assume that we have an initial distribution which is denoted an initial distribution which is denoted an initial distribution which is denoted by x of0 by x of0 by x of0 which is given by pinet. which is given by pinet. which is given by pinet. This is the distribution that we are This is the distribution that we are This is the distribution that we are aware about. It is probably a gshian aware about. It is probably a gshian aware about. It is probably a gshian distribution which is uniformly distribution which is uniformly distribution which is uniformly distributed in space distributed in space distributed in space [snorts] and we want to end up [snorts] and we want to end up [snorts] and we want to end up at the distribution of the data that we at the distribution of the data that we at the distribution of the data that we want to predict. want to predict. want to predict. Here 0 represents the time step t=0 and Here 0 represents the time step t=0 and Here 0 represents the time step t=0 and 1 represents the time t= 1. [snorts] So 1 represents the time t= 1. [snorts] So 1 represents the time t= 1. [snorts] So we want to do something which changes we want to do something which changes we want to do something which changes the initial distribution to the final the initial distribution to the final the initial distribution to the final distribution. distribution. distribution. >> [snorts] >> [snorts] >> Now >> Now >> Now people thought of casting this problem people thought of casting this problem people thought of casting this problem in a way which is similar to how the
- L1 @ 775s
in a way which is similar to how the in a way which is similar to how the particles are moving in space and time. particles are moving in space and time. particles are moving in space and time. What do I mean by that? Well, imagine What do I mean by that? Well, imagine What do I mean by that? Well, imagine that that that your pinet which is the initial your pinet which is the initial your pinet which is the initial distribution distribution distribution is just given by a bunch of particles is just given by a bunch of particles is just given by a bunch of particles which are arranged uniformly which are arranged uniformly which are arranged uniformly symmetrically in space like this. symmetrically in space like this. symmetrically in space like this. uniform, completely uniform. uniform, completely uniform. uniform, completely uniform. And then you make these particles, let's And then you make these particles, let's And then you make these particles, let's say there are thousand of these say there are thousand of these say there are thousand of these particles, you make these particles move particles, you make these particles move particles, you make these particles move in such a way that you change it to the distribution that you change it to the distribution that you want to predict. So this is a spiral you want to predict. So this is a spiral you want to predict. So this is a spiral distribution, let's say. distribution, let's say. distribution, let's say. So let's let's look at a figure to So let's let's look at a figure to So let's let's look at a figure to understand this or a animation. So you understand this or a animation. So you understand this or a animation. So you see here we are initially starting with see here we are initially starting with see here we are initially starting with a gshian distribution and we slowly move a gshian distribution and we slowly move a gshian distribution and we slowly move to the distribution of a heart shape to the distribution of a heart shape to the distribution of a heart shape that we want to predict. that we want to predict. that we want to predict. So we are casting the deep generative So we are casting the deep generative
- L1 @ 840s
So we are casting the deep generative modeling problem modeling problem modeling problem as a problem where we are trying to as a problem where we are trying to as a problem where we are trying to predict the flow map from the initial predict the flow map from the initial predict the flow map from the initial distribution of the particles to the distribution of the particles to the distribution of the particles to the final distribution of the particles. final distribution of the particles. final distribution of the particles. [snorts] Now from the initial discussion [snorts] Now from the initial discussion [snorts] Now from the initial discussion we had your question might be okay how we had your question might be okay how we had your question might be okay how how do I generate this map in the first how do I generate this map in the first how do I generate this map in the first place? How do I generate this flow? And place? How do I generate this flow? And place? How do I generate this flow? And we just discussed that the trajectories we just discussed that the trajectories we just discussed that the trajectories of the particles of the particles of the particles completely depend on the velocity field. completely depend on the velocity field. completely depend on the velocity field. Right? So what if you write down the Right? So what if you write down the Right? So what if you write down the equation as dx by dt. equation as dx by dt. equation as dx by dt. Now here x represents the evolution of Now here x represents the evolution of Now here x represents the evolution of the probability field the probability field the probability field which is similar to the trajecolution of which is similar to the trajecolution of which is similar to the trajecolution of the trajectories followed by every the trajectories followed by every the trajectories followed by every single particle which are representing single particle which are representing single particle which are representing that field according to a that field according to a that field according to a velocity field which depends on x velocity field which depends on x velocity field which depends on x [snorts] and it depends on time as well. [snorts] and it depends on time as well.
- L1 @ 905s
[snorts] and it depends on time as well. So this is exactly the same equation So this is exactly the same equation So this is exactly the same equation that we looked at before. that we looked at before. that we looked at before. Now this is something if we know this Now this is something if we know this Now this is something if we know this velocity field we can calculate exactly velocity field we can calculate exactly velocity field we can calculate exactly how the trajectories will move and we how the trajectories will move and we how the trajectories will move and we want to find the field which can take us want to find the field which can take us want to find the field which can take us from the initial distribution to the from the initial distribution to the from the initial distribution to the final data distribution. final data distribution. final data distribution. So [snorts] we have cast the problem of So [snorts] we have cast the problem of So [snorts] we have cast the problem of deep generative modeling into a problem deep generative modeling into a problem deep generative modeling into a problem where we want to predict the velocity where we want to predict the velocity where we want to predict the velocity field which governs the trajectories of field which governs the trajectories of field which governs the trajectories of these particles. these particles. these particles. Now this is exactly what is called as a Now this is exactly what is called as a Now this is exactly what is called as a flow model. The name is slightly misleading because The name is slightly misleading because I think this should be called as a I think this should be called as a I think this should be called as a velocity field model because in the end velocity field model because in the end velocity field model because in the end what people do is that they approximate what people do is that they approximate what people do is that they approximate this with a neural network and they try this with a neural network and they try this with a neural network and they try to predict this. So this is approximated to predict this. So this is approximated
- L1 @ 954s
to predict this. So this is approximated by a neural network. by a neural network. by a neural network. Okay. And these are the conditions we Okay. And these are the conditions we Okay. And these are the conditions we want to satisfy. X of0 is the initial want to satisfy. X of0 is the initial want to satisfy. X of0 is the initial gshian distribution and we want to move gshian distribution and we want to move gshian distribution and we want to move to our actual data distribution. to our actual data distribution. to our actual data distribution. If you want to revise this properly, If you want to revise this properly, If you want to revise this properly, please go ahead and refer to the please go ahead and refer to the please go ahead and refer to the previous lecture where I have discussed previous lecture where I have discussed previous lecture where I have discussed this in complete detail. this in complete detail. this in complete detail. [snorts] [snorts] Okay. So, uh now the question is Okay. So, uh now the question is Okay. So, uh now the question is [snorts] [snorts] we want to simulate an OD which looks as we want to simulate an OD which looks as we want to simulate an OD which looks as follows. follows. follows. dxt by dt is equal to dxt by dt is equal to dxt by dt is equal to the velocity field the velocity field the velocity field >> [snorts] >> [snorts] >> uh at at that time and at the location >> uh at at that time and at the location >> uh at at that time and at the location in space. in space. in space. [snorts] Conceptually this looks [snorts] Conceptually this looks [snorts] Conceptually this looks excellent but the main question is that excellent but the main question is that excellent but the main question is that how do we go ahead and find this ideal how do we go ahead and find this ideal how do we go ahead and find this ideal velocity field that transports our velocity field that transports our velocity field that transports our initial probability distribution to the initial probability distribution to the initial probability distribution to the distribution that we want to predict in distribution that we want to predict in
- L1 @ 1016s
distribution that we want to predict in the first place. the first place. the first place. And it turns out that this is at the end And it turns out that this is at the end And it turns out that this is at the end you will see this is very very easy and you will see this is very very easy and you will see this is very very easy and straightforward. straightforward. straightforward. But as I said in the first lecture we But as I said in the first lecture we But as I said in the first lecture we are going to take a path which is going are going to take a path which is going are going to take a path which is going to help us understand and appreciate the to help us understand and appreciate the to help us understand and appreciate the final easy formulation of this in a much final easy formulation of this in a much final easy formulation of this in a much better way. Okay. So the first thing that we think Okay. So the first thing that we think about is let's say we u about is let's say we u about is let's say we u use a neural network to approximate this use a neural network to approximate this use a neural network to approximate this velocity field and we randomly velocity field and we randomly velocity field and we randomly initialize the parameters theta of the initialize the parameters theta of the initialize the parameters theta of the neural network. neural network. neural network. Well then the OD will just produce Well then the OD will just produce Well then the OD will just produce nonsense. You will start with a washian nonsense. You will start with a washian nonsense. You will start with a washian distribution but you might end up with distribution but you might end up with distribution but you might end up with something which is completely random. something which is completely random. something which is completely random. So you're trying to flow the So you're trying to flow the So you're trying to flow the distribution in a specific way, right? distribution in a specific way, right? distribution in a specific way, right? But what if you start with a gshian
- L1 @ 1072s
But what if you start with a gshian But what if you start with a gshian distribution and you end up with distribution and you end up with distribution and you end up with something that does not represent your something that does not represent your something that does not represent your data at all. So you can't init randomly data at all. So you can't init randomly data at all. So you can't init randomly initialize the weights. initialize the weights. initialize the weights. This is where we come to the main This is where we come to the main This is where we come to the main concept in machine learning that we need concept in machine learning that we need concept in machine learning that we need a t we need a target to train our neural a t we need a target to train our neural a t we need a target to train our neural network. network. network. Now usually the target in let's say Now usually the target in let's say Now usually the target in let's say supervised learning task is known to us. supervised learning task is known to us. supervised learning task is known to us. For example, if we are trying to uh For example, if we are trying to uh For example, if we are trying to uh predict predict predict the house prices which are going to the house prices which are going to the house prices which are going to change in the coming years. We generally change in the coming years. We generally change in the coming years. We generally use a regression model to solve this use a regression model to solve this use a regression model to solve this problem where we have a previous set of problem where we have a previous set of problem where we have a previous set of data and we extrapolate to understand data and we extrapolate to understand data and we extrapolate to understand what happens in the next 5 years. what happens in the next 5 years. what happens in the next 5 years. So the target is very clear. The target So the target is very clear. The target So the target is very clear. The target is the price of the house. In this case,
- L1 @ 1125s
is the price of the house. In this case, is the price of the house. In this case, we want to minimize the loss function so we want to minimize the loss function so we want to minimize the loss function so that that the predicted velocity vector field the predicted velocity vector field the predicted velocity vector field matches the target velocity vector matches the target velocity vector matches the target velocity vector field. field. field. [snorts] All of us intuitively [snorts] All of us intuitively [snorts] All of us intuitively understand this. This is very similar to understand this. This is very similar to understand this. This is very similar to what we do in all types of machine what we do in all types of machine what we do in all types of machine learning problems where we are trying to learning problems where we are trying to learning problems where we are trying to predict the parameters of a network and predict the parameters of a network and predict the parameters of a network and there is a target which we are trying to there is a target which we are trying to there is a target which we are trying to match. So in most of the cases in match. So in most of the cases in match. So in most of the cases in supervised learning tasks this target is supervised learning tasks this target is supervised learning tasks this target is absolutely known to us. So it becomes absolutely known to us. So it becomes absolutely known to us. So it becomes easy to find the parameter so that the easy to find the parameter so that the easy to find the parameter so that the loss between our model and the target is loss between our model and the target is loss between our model and the target is minimized. minimized. minimized. Now as a mathematical formulation this Now as a mathematical formulation this Now as a mathematical formulation this looks very clean but let us come to the looks very clean but let us come to the looks very clean but let us come to the core problem. The core problem here is that predicting The core problem here is that predicting this target is not very easy. In fact we this target is not very easy. In fact we
- L1 @ 1182s
this target is not very easy. In fact we have absolutely no idea what this target have absolutely no idea what this target have absolutely no idea what this target is. is. is. And that is going to be the focus of And that is going to be the focus of And that is going to be the focus of this lecture is to make it somewhat easy this lecture is to make it somewhat easy this lecture is to make it somewhat easy for us to identify this target. Now Now at the end you will see that the final at the end you will see that the final at the end you will see that the final training target is just a linear training target is just a linear training target is just a linear equation. It is very easy but we are equation. It is very easy but we are equation. It is very easy but we are going to take some time to come to that going to take some time to come to that going to take some time to come to that training target. training target. training target. So in this lecture our goal is to find So in this lecture our goal is to find So in this lecture our goal is to find an equation for the training target. an equation for the training target. an equation for the training target. Maybe we'll not be able to complete it Maybe we'll not be able to complete it Maybe we'll not be able to complete it in this specific lecture but by the next in this specific lecture but by the next in this specific lecture but by the next lecture we will have the complete flow lecture we will have the complete flow lecture we will have the complete flow modeling pipeline ready. Before we get to the training target and Before we get to the training target and uh how we predict the training target, uh how we predict the training target, uh how we predict the training target, we need to understand some terminologies we need to understand some terminologies we need to understand some terminologies which are very critical to understand which are very critical to understand which are very critical to understand this this this uh how the training target is predicted
- L1 @ 1246s
uh how the training target is predicted uh how the training target is predicted in the first place. in the first place. in the first place. And whatever I'm going to discuss next And whatever I'm going to discuss next And whatever I'm going to discuss next is going to form the critical piece of is going to form the critical piece of is going to form the critical piece of training flow models which is used in training flow models which is used in training flow models which is used in most of the modern uh diffusion models most of the modern uh diffusion models most of the modern uh diffusion models or flow-based models which are either or flow-based models which are either or flow-based models which are either used for image generation, video used for image generation, video used for image generation, video generation or even in robotics for that generation or even in robotics for that generation or even in robotics for that matter. matter. matter. [snorts] [snorts] Okay. So remember our goal is to convert Okay. So remember our goal is to convert Okay. So remember our goal is to convert the initial probability distribution to a final data distribution. So here to a final data distribution. So here [snorts] [snorts] this is the initial distribution which this is the initial distribution which this is the initial distribution which is uh noise. This is simply noise. It is uh noise. This is simply noise. It is uh noise. This is simply noise. It does not have any structure to it. does not have any structure to it. does not have any structure to it. But uh for the example of cats that we But uh for the example of cats that we But uh for the example of cats that we have referred to several times in this have referred to several times in this have referred to several times in this series, series, series, we want to get to a distribution which we want to get to a distribution which we want to get to a distribution which looks like this where you pick any point and you get a where you pick any point and you get a cat which is maybe specific to that cat which is maybe specific to that cat which is maybe specific to that breed. breed.
- L1 @ 1311s
breed. So we have different So we have different So we have different we have a probability distribution that we have a probability distribution that we have a probability distribution that you know that is grounded in data which you know that is grounded in data which you know that is grounded in data which actually makes sense which you can actually makes sense which you can actually makes sense which you can sample from and generate images of cats. sample from and generate images of cats. sample from and generate images of cats. Now a naive thinking would tell you that Now a naive thinking would tell you that Now a naive thinking would tell you that well to go from noise to data somewhere well to go from noise to data somewhere well to go from noise to data somewhere we need we need to use the data right we need we need to use the data right we need we need to use the data right otherwise and that has to be a big part otherwise and that has to be a big part otherwise and that has to be a big part of our training target otherwise how of our training target otherwise how of our training target otherwise how will our model know that the final will our model know that the final will our model know that the final distribution that we want to morph the distribution that we want to morph the distribution that we want to morph the initial noisy distribution to represents initial noisy distribution to represents initial noisy distribution to represents the distribution of cats in this case. the distribution of cats in this case. the distribution of cats in this case. So uh okay so here we will use a So uh okay so here we will use a So uh okay so here we will use a [snorts] most general and most simple [snorts] most general and most simple [snorts] most general and most simple technique which is in line with most of technique which is in line with most of technique which is in line with most of the state-of-the-art models what is the state-of-the-art models what is the state-of-the-art models what is going to follow next and uh we are going going to follow next and uh we are going
- L1 @ 1366s
going to follow next and uh we are going to go through a little bit of to go through a little bit of to go through a little bit of mathematics right now so please don't be mathematics right now so please don't be mathematics right now so please don't be repelled by the maths which you are repelled by the maths which you are repelled by the maths which you are about to see it is very intuitive I'm about to see it is very intuitive I'm about to see it is very intuitive I'm not going to introduce any equations not going to introduce any equations not going to introduce any equations it's just going to be some terminologies it's just going to be some terminologies it's just going to be some terminologies which I'm going to introduce which I'm going to introduce which I'm going to introduce but once we understand these but once we understand these but once we understand these terminologies it will help us understand terminologies it will help us understand terminologies it will help us understand flow models in a much better way and in flow models in a much better way and in flow models in a much better way and in fact we'll also be drawing some fact we'll also be drawing some fact we'll also be drawing some parallels with the diffusion models parallels with the diffusion models parallels with the diffusion models which we have looked at before in the which we have looked at before in the which we have looked at before in the previous lectures. previous lectures. previous lectures. [snorts] So if you are new to the field [snorts] So if you are new to the field [snorts] So if you are new to the field of deep generative modeling um don't of deep generative modeling um don't of deep generative modeling um don't worry about diffusion models for now. We worry about diffusion models for now. We worry about diffusion models for now. We will make sure that the following will make sure that the following will make sure that the following terminologies can be understood by terminologies can be understood by terminologies can be understood by everyone who is new to the field as everyone who is new to the field as everyone who is new to the field as well.
- L1 @ 1418s
Okay. So, first concept that we will Okay. So, first concept that we will discuss is called as So let's let's try to understand what is So let's let's try to understand what is the meaning of [snorts] conditional the meaning of [snorts] conditional the meaning of [snorts] conditional probability path. probability path. probability path. Conditional probability path simply Conditional probability path simply Conditional probability path simply means that given a point zed. means that given a point zed. means that given a point zed. We define the conditional probability We define the conditional probability We define the conditional probability path as a transformation from the path as a transformation from the path as a transformation from the initial distri the initial distribution initial distri the initial distribution initial distri the initial distribution pinet pinet pinet to the point zed. to the point zed. to the point zed. So we are not trying here to morph the So we are not trying here to morph the So we are not trying here to morph the initial distribution to the final initial distribution to the final initial distribution to the final unknown distribution that we have no unknown distribution that we have no unknown distribution that we have no idea about. What we have idea about is idea about. What we have idea about is idea about. What we have idea about is the data. I know individual images of the data. I know individual images of the data. I know individual images of the cats, right? the cats, right? the cats, right? So there is some data which is given to So there is some data which is given to So there is some data which is given to me and the first thought that comes to me and the first thought that comes to me and the first thought that comes to my mind is instead of taking the initial my mind is instead of taking the initial my mind is instead of taking the initial gshian distribution to a distribution gshian distribution to a distribution gshian distribution to a distribution which I do not know. What if I take it which I do not know. What if I take it which I do not know. What if I take it to something I do know even if it is to something I do know even if it is
- L1 @ 1491s
to something I do know even if it is like just one simple single data point like just one simple single data point like just one simple single data point [snorts] and that is exactly what [snorts] and that is exactly what [snorts] and that is exactly what conditional probability path means. Why conditional probability path means. Why conditional probability path means. Why conditional? because it is conditional. conditional? because it is conditional. conditional? because it is conditional. It is conditioned on that data sample. It is conditioned on that data sample. It is conditioned on that data sample. We are not predicting the We are not predicting the We are not predicting the complete probability distribution or the complete probability distribution or the complete probability distribution or the complete path but we are pro we are complete path but we are pro we are complete path but we are pro we are predicting the path which is conditioned predicting the path which is conditioned predicting the path which is conditioned on the point Z. on the point Z. on the point Z. [snorts] Let's try to visualize this to [snorts] Let's try to visualize this to [snorts] Let's try to visualize this to make sure we understand this properly. make sure we understand this properly. make sure we understand this properly. Now in the figure you can see that Now in the figure you can see that Now in the figure you can see that the big circle is the initial the big circle is the initial the big circle is the initial distribution and the green is the point distribution and the green is the point distribution and the green is the point where we want to move the initial where we want to move the initial where we want to move the initial distribution towards. distribution towards. distribution towards. Okay. So this is how our transformation is going this is how our transformation is going to look like. [snorts] And the good part to look like. [snorts] And the good part to look like. [snorts] And the good part about this is that we know point Z. We about this is that we know point Z. We about this is that we know point Z. We know the data which is given to us. We know the data which is given to us. We
- L1 @ 1551s
know the data which is given to us. We know this initial distribution also. know this initial distribution also. know this initial distribution also. So it is very rational to think that we So it is very rational to think that we So it is very rational to think that we can probably predict this we can probably predict the velocity we can probably predict the velocity field that field that field that transforms this initial distribution and transforms this initial distribution and transforms this initial distribution and takes it to the final point Z. So how how do we construct this path? So how how do we construct this path? Let's first talk about the probability Let's first talk about the probability Let's first talk about the probability path. What do I mean by a probability path. What do I mean by a probability path. What do I mean by a probability path? Well, let's see. path? Well, let's see. path? Well, let's see. The probability at time t =0, the The probability at time t =0, the The probability at time t =0, the probability distribution is let's say probability distribution is let's say probability distribution is let's say uniform. uniform. uniform. And at time t = 1, there is no distribution. It is just one there is no distribution. It is just one single point zed. Right? single point zed. Right? single point zed. Right? Now what probability path tells us is Now what probability path tells us is Now what probability path tells us is that at every time step in this that at every time step in this that at every time step in this transformation transformation transformation can you generate a probability can you generate a probability can you generate a probability distribution for me. That is the meaning distribution for me. That is the meaning distribution for me. That is the meaning of probability path. And why conditional of probability path. And why conditional of probability path. And why conditional probability path? Because we have probability path? Because we have probability path? Because we have already know that the finally it should already know that the finally it should already know that the finally it should go to the point Z. So it is conditioned go to the point Z. So it is conditioned
- L1 @ 1625s
go to the point Z. So it is conditioned on the point Z. on the point Z. So what can I do so that each individual So what can I do so that each individual So what can I do so that each individual distribution I know and I can take from distribution I know and I can take from distribution I know and I can take from the initial distribution to the final the initial distribution to the final distribution and what we do is very simple in in and what we do is very simple in in reality what is done is let's say the initial distribution is let's say the initial distribution is such that the mean is zero such that the mean is zero such that the mean is zero and the standard deviation is And now the final distribution is such that now the final distribution is such that the mean is zed the mean is zed the mean is zed and the deviation is zero because every and the deviation is zero because every and the deviation is zero because every time you sample you should get zed. It's time you sample you should get zed. It's time you sample you should get zed. It's a fixed point. Right? [snorts] a fixed point. Right? [snorts] a fixed point. Right? [snorts] Now what if I define Now what if I define Now what if I define every single every single every single time step to be a gshian such that the time step to be a gshian such that the time step to be a gshian such that the mu starts from zero and it ends up with mu starts from zero and it ends up with mu starts from zero and it ends up with zed zed zed at time t = 1. at time t = 1. at time t = 1. This is time t equal to0 and uh this is z and uh this is z and the variance starts with one one and the variance ends at zero. What this will do is that it will What this will do is that it will provide me with the probability path provide me with the probability path provide me with the probability path such that at every time step I will have
- L1 @ 1726s
such that at every time step I will have such that at every time step I will have a gshian and a gshian and a gshian and the probability distribution will change the probability distribution will change the probability distribution will change in such a way that the gshian will in such a way that the gshian will in such a way that the gshian will slowly contract and be displaced towards slowly contract and be displaced towards slowly contract and be displaced towards the point z. the point z. the point z. I can visualize this very nicely here. I can visualize this very nicely here. I can visualize this very nicely here. Initially it starts like this with a Initially it starts like this with a Initially it starts like this with a mean zero and a deviation of one. mean zero and a deviation of one. mean zero and a deviation of one. [snorts] Now this is the point which I [snorts] Now this is the point which I [snorts] Now this is the point which I have conditioned on. I have to move the have conditioned on. I have to move the have conditioned on. I have to move the distribution finally to this point. So distribution finally to this point. So distribution finally to this point. So you can see the mean is shifting as we you can see the mean is shifting as we you can see the mean is shifting as we go from time step t=0 to time step t= 1. go from time step t=0 to time step t= 1. go from time step t=0 to time step t= 1. The mean is slowly shifting towards the The mean is slowly shifting towards the The mean is slowly shifting towards the right and it's becoming narrow and right and it's becoming narrow and right and it's becoming narrow and narrower and narrower. So the deviation narrower and narrower. So the deviation narrower and narrower. So the deviation is reducing and finally I get a single is reducing and finally I get a single is reducing and finally I get a single point which is zed. point which is zed. point which is zed. So I have defined a conditional So I have defined a conditional So I have defined a conditional probability path which is linear in
- L1 @ 1779s
probability path which is linear in probability path which is linear in time. time. time. So the mean increases from zero to the So the mean increases from zero to the So the mean increases from zero to the value of zed and the deviation decreases value of zed and the deviation decreases value of zed and the deviation decreases from 1 to 0. So [snorts] this can be visualized as So [snorts] this can be visualized as follows. Here also we have taken the follows. Here also we have taken the follows. Here also we have taken the example of Z= 3. The mean increases from example of Z= 3. The mean increases from example of Z= 3. The mean increases from 0 to 3 and the variance decreases from 1 0 to 3 and the variance decreases from 1 0 to 3 and the variance decreases from 1 to 0. to 0. to 0. So now you know I am I know the So now you know I am I know the So now you know I am I know the trajectories which are followed by every trajectories which are followed by every trajectories which are followed by every single particle. single particle. single particle. And uh And uh And uh because I know the trajectories which because I know the trajectories which because I know the trajectories which are followed by every single particle, are followed by every single particle, are followed by every single particle, you might refer to our discussion about you might refer to our discussion about you might refer to our discussion about the detectives and say that well then the detectives and say that well then the detectives and say that well then maybe I can predict the velocity field maybe I can predict the velocity field maybe I can predict the velocity field here, right? here, right? here, right? [snorts] Maybe I know the velocity field [snorts] Maybe I know the velocity field [snorts] Maybe I know the velocity field because since I know the trajectories because since I know the trajectories because since I know the trajectories very clearly, very clearly, very clearly, I can find the velocity field which is I can find the velocity field which is I can find the velocity field which is going to generate those trajectories. going to generate those trajectories.
- L1 @ 1839s
going to generate those trajectories. uh and this is exactly what we are going uh and this is exactly what we are going uh and this is exactly what we are going to do next. But before we move, I want to do next. But before we move, I want to do next. But before we move, I want you to clearly understand what we have you to clearly understand what we have you to clearly understand what we have done so far. This does not mean we have done so far. This does not mean we have done so far. This does not mean we have solved the problem. We have simply let's solved the problem. We have simply let's solved the problem. We have simply let's say there are thousand images of cats, say there are thousand images of cats, say there are thousand images of cats, right? That are given to us. Now for right? That are given to us. Now for right? That are given to us. Now for every single image, we can construct a every single image, we can construct a every single image, we can construct a probability path which transforms the probability path which transforms the probability path which transforms the initial gshian distribution and takes it initial gshian distribution and takes it initial gshian distribution and takes it to that single image. We do this for all to that single image. We do this for all to that single image. We do this for all the thousand images. So far we have only the thousand images. So far we have only the thousand images. So far we have only discussed this much. We have nowhere discussed this much. We have nowhere discussed this much. We have nowhere discussed about discussed about discussed about uh how to construct a training target. uh how to construct a training target. uh how to construct a training target. But it turns out that this is exactly But it turns out that this is exactly But it turns out that this is exactly what we use for the training target. But what we use for the training target. But what we use for the training target. But anyways uh for now just think about this anyways uh for now just think about this
- L1 @ 1883s
anyways uh for now just think about this as being a conditional probability path as being a conditional probability path as being a conditional probability path which is conditioned on the point Z which is conditioned on the point Z which is conditioned on the point Z head. head. head. Now this technique of conditioning is at Now this technique of conditioning is at Now this technique of conditioning is at the heart of deep generative modeling the heart of deep generative modeling the heart of deep generative modeling which I'm also realizing as I'm which I'm also realizing as I'm which I'm also realizing as I'm researching this topic. It is coming up researching this topic. It is coming up researching this topic. It is coming up everywhere to design training targets everywhere to design training targets everywhere to design training targets even for score based models diffusion even for score based models diffusion even for score based models diffusion models. Wherever we are not able to models. Wherever we are not able to models. Wherever we are not able to construct a training target people construct a training target people construct a training target people condition it on the data sample and then condition it on the data sample and then condition it on the data sample and then magically we can construct the training magically we can construct the training magically we can construct the training target. target. target. We will come to that how the We will come to that how the We will come to that how the construction appears but for now just construction appears but for now just construction appears but for now just make sure you understand the conditional make sure you understand the conditional make sure you understand the conditional probability path and understand the the probability path and understand the the probability path and understand the the terms probability path why because terms probability path why because terms probability path why because probability path is nothing but a probability path is nothing but a probability path is nothing but a trajectory. We are transforming the trajectory. We are transforming the trajectory. We are transforming the initial gshian distribution to the final initial gshian distribution to the final initial gshian distribution to the final single point that is the probability single point that is the probability
- L1 @ 1935s
single point that is the probability path and we want to know at every single path and we want to know at every single path and we want to know at every single time step in this transition how does time step in this transition how does time step in this transition how does the probability distribution look like. the probability distribution look like. the probability distribution look like. Why conditional is because this is not Why conditional is because this is not Why conditional is because this is not the actual probability path. It is the actual probability path. It is the actual probability path. It is conditioned on the point's head. conditioned on the point's head. conditioned on the point's head. [snorts] [snorts] Okay. Now as we discussed before because Okay. Now as we discussed before because Okay. Now as we discussed before because now we know the trajectories of each now we know the trajectories of each now we know the trajectories of each particle we we know that there should be particle we we know that there should be particle we we know that there should be a velocity field associated with it a velocity field associated with it a velocity field associated with it right. right. right. So for any conditional probability path So for any conditional probability path So for any conditional probability path there exists an equivalent vector field. [snorts] So instead of [snorts] So instead of sampling from probabilities at each time sampling from probabilities at each time sampling from probabilities at each time step, we can simply sample a point from step, we can simply sample a point from step, we can simply sample a point from P in it and we can just follow the P in it and we can just follow the P in it and we can just follow the vector field. This is exactly what we vector field. This is exactly what we vector field. This is exactly what we discussed at the beginning of the discussed at the beginning of the discussed at the beginning of the lecture. Remember if we want to go from lecture. Remember if we want to go from lecture. Remember if we want to go from the velocity vector field to the the velocity vector field to the the velocity vector field to the trajectories the reverse problem
- L1 @ 1997s
trajectories the reverse problem trajectories the reverse problem we simply sample the initial point which we simply sample the initial point which we simply sample the initial point which was x0 in our case and we use the oiler was x0 in our case and we use the oiler was x0 in our case and we use the oiler discretization to discretization to discretization to find the next point x1 then we go to x2 find the next point x1 then we go to x2 find the next point x1 then we go to x2 and similarly we can we know the entire and similarly we can we know the entire and similarly we can we know the entire trajectory of the particle. trajectory of the particle. trajectory of the particle. [snorts] So that's what we are trying to [snorts] So that's what we are trying to [snorts] So that's what we are trying to say in this piece of text that once you say in this piece of text that once you say in this piece of text that once you know the velocity field you can use that know the velocity field you can use that know the velocity field you can use that velocity field to get the entire velocity field to get the entire velocity field to get the entire trajectory of a particle trajectory of a particle trajectory of a particle and uh here you can actually see the and uh here you can actually see the and uh here you can actually see the conditional probability path and the conditional probability path and the conditional probability path and the conditional vector field. [snorts] So conditional vector field. [snorts] So conditional vector field. [snorts] So first let's look at the conditional first let's look at the conditional first let's look at the conditional probability path. Every single particle probability path. Every single particle probability path. Every single particle is being [snorts] transformed from the is being [snorts] transformed from the is being [snorts] transformed from the initial distribution uh to the final initial distribution uh to the final initial distribution uh to the final point zed. Now who is responsible for point zed. Now who is responsible for point zed. Now who is responsible for this transformation? There is a vector this transformation? There is a vector
- L1 @ 2046s
this transformation? There is a vector field which is responsible for this field which is responsible for this field which is responsible for this transformation transformation and [snorts] and [snorts] and [snorts] rightfully so the vector field is rightfully so the vector field is rightfully so the vector field is pointing you can see if I zoom in here pointing you can see if I zoom in here pointing you can see if I zoom in here this is the vector field all the vector this is the vector field all the vector this is the vector field all the vector points all the vectors are pointing points all the vectors are pointing points all the vectors are pointing towards the direction of zed because towards the direction of zed because towards the direction of zed because this is what we want to move the points this is what we want to move the points this is what we want to move the points towards. towards. towards. [snorts] So these are the vector field [snorts] So these are the vector field [snorts] So these are the vector field lines and the white lines are the lines and the white lines are the lines and the white lines are the trajectories which are followed by the trajectories which are followed by the trajectories which are followed by the individual green particles. Now here is another example where this Now here is another example where this is the target. So we want to move our is the target. So we want to move our is the target. So we want to move our distribution which is uh shown over here distribution which is uh shown over here distribution which is uh shown over here to this target. to this target. to this target. And you can see as time proceeds the And you can see as time proceeds the And you can see as time proceeds the distribution slowly moves towards the distribution slowly moves towards the distribution slowly moves towards the target. But I want you to focus on these target. But I want you to focus on these target. But I want you to focus on these arrows which are the velocity vectors. [snorts] First of all note that how [snorts] First of all note that how every single point the velocity vector every single point the velocity vector
- L1 @ 2106s
every single point the velocity vector is pointed towards them. And And for points which are far away the for points which are far away the for points which are far away the vectors are longer in magnitude. As you vectors are longer in magnitude. As you vectors are longer in magnitude. As you move closer the vectors are reducing in move closer the vectors are reducing in move closer the vectors are reducing in in in magnitude. in in magnitude. in in magnitude. And the formula for this is actually And the formula for this is actually And the formula for this is actually given very simply by this. given very simply by this. given very simply by this. So zed is our point which is our target. So zed is our point which is our target. So zed is our point which is our target. If x is very far from zed the magnitude If x is very far from zed the magnitude If x is very far from zed the magnitude is very high. If x is very close to zed is very high. If x is very close to zed is very high. If x is very close to zed it's almost zero. You can see here. it's almost zero. You can see here. it's almost zero. You can see here. But not just that there is a variation But not just that there is a variation But not just that there is a variation with time also. As as time goes close to with time also. As as time goes close to with time also. As as time goes close to one as you go towards the final one as you go towards the final one as you go towards the final distribution you can see the magnitude distribution you can see the magnitude distribution you can see the magnitude increases. So [snorts] increases. So [snorts] increases. So [snorts] from here to here you can see the arrows from here to here you can see the arrows from here to here you can see the arrows increasing in the magnitude increasing in the magnitude increasing in the magnitude from left to right from left to right from left to right and [snorts] uh and [snorts] uh and [snorts] uh this is because as as as time proceeds
- L1 @ 2165s
this is because as as as time proceeds this is because as as as time proceeds we probably want to uh move the velocity we probably want to uh move the velocity we probably want to uh move the velocity field in a more aggressive manner field in a more aggressive manner field in a more aggressive manner towards our target. So first of all the towards our target. So first of all the towards our target. So first of all the first observation is that wherever your first observation is that wherever your first observation is that wherever your point is the velocity field is targeted point is the velocity field is targeted point is the velocity field is targeted towards the location's head and the towards the location's head and the towards the location's head and the magnitude is proportional to how far magnitude is proportional to how far magnitude is proportional to how far your point is from the target your point is from the target your point is from the target which probably makes sense because the which probably makes sense because the which probably makes sense because the further your point is from the target further your point is from the target further your point is from the target you want to push the particle more with you want to push the particle more with you want to push the particle more with with more force. So in reality what with more force. So in reality what with more force. So in reality what happens is that every single particle is happens is that every single particle is happens is that every single particle is is pushed towards your target. So that's is pushed towards your target. So that's is pushed towards your target. So that's why you get this conditional probability why you get this conditional probability why you get this conditional probability path [snorts] and this this formula for path [snorts] and this this formula for path [snorts] and this this formula for the velocity field is is uh once we know the velocity field is is uh once we know the velocity field is is uh once we know this velocity field formula this velocity field formula this velocity field formula we don't need anything else using this we don't need anything else using this
- L1 @ 2216s
we don't need anything else using this formula we can simply calculate the formula we can simply calculate the formula we can simply calculate the trajectories of each and every point so trajectories of each and every point so trajectories of each and every point so [snorts] that every single point goes [snorts] that every single point goes [snorts] that every single point goes from the initial gshian distribution to from the initial gshian distribution to from the initial gshian distribution to your point Z. your point Z. your point Z. So the velocity field is given by the So the velocity field is given by the So the velocity field is given by the ratio of two quantities. The numerator ratio of two quantities. The numerator ratio of two quantities. The numerator is simply a vector from the point to the is simply a vector from the point to the is simply a vector from the point to the target and the denominator captures the target and the denominator captures the target and the denominator captures the evolution of time. evolution of time. evolution of time. So you observe something very critical So you observe something very critical So you observe something very critical here. Uh as we move forward in time the here. Uh as we move forward in time the here. Uh as we move forward in time the value of denominator decreases which value of denominator decreases which value of denominator decreases which means the magnitude of the velocity means the magnitude of the velocity means the magnitude of the velocity field lines increase as as we go forward field lines increase as as we go forward field lines increase as as we go forward in time. This can be seen from the above in time. This can be seen from the above in time. This can be seen from the above figures as the arrows get bigger and figures as the arrows get bigger and figures as the arrows get bigger and bigger as time proceeds. So this this bigger as time proceeds. So this this bigger as time proceeds. So this this this overall intuition will be critical this overall intuition will be critical this overall intuition will be critical for us as we move ahead. for us as we move ahead. for us as we move ahead. Now this u of t is is called as
- L1 @ 2266s
Now this u of t is is called as Now this u of t is is called as conditional So you you might look at this and think So you you might look at this and think uh well if you look at the training uh well if you look at the training uh well if you look at the training target target target wasn't our training target also a wasn't our training target also a wasn't our training target also a velocity field so can we directly use velocity field so can we directly use velocity field so can we directly use this as a training target [snorts] well this as a training target [snorts] well this as a training target [snorts] well my first answer to that is you can't my first answer to that is you can't my first answer to that is you can't directly use it because this is not our directly use it because this is not our directly use it because this is not our training target remember it is training target remember it is training target remember it is conditioned on the data z conditioned on the data z conditioned on the data z uh so at a first glance it looks like uh so at a first glance it looks like uh so at a first glance it looks like okay this is very different from the okay this is very different from the okay this is very different from the actual velocity field because there the actual velocity field because there the actual velocity field because there the the target which I looked at before that the target which I looked at before that the target which I looked at before that is not conditioned on the point Z head is not conditioned on the point Z head is not conditioned on the point Z head but here we know the exact formula for but here we know the exact formula for but here we know the exact formula for this condition velocity field. So this condition velocity field. So this condition velocity field. So ideally we can we can replace this in in ideally we can we can replace this in in ideally we can we can replace this in in our target and train our neural network our target and train our neural network
- L1 @ 2320s
our target and train our neural network to predict it for every single point to predict it for every single point to predict it for every single point zed. zed. zed. It turns out that what we will do is It turns out that what we will do is It turns out that what we will do is exactly that. But then let me not exactly that. But then let me not exactly that. But then let me not explain to you right now why that explain to you right now why that explain to you right now why that happens. And that's why this formulation happens. And that's why this formulation happens. And that's why this formulation is very easy. But for now for our is very easy. But for now for our is very easy. But for now for our where we are right now in terms of where we are right now in terms of where we are right now in terms of accessing this material is that we know accessing this material is that we know accessing this material is that we know that this is the conditional vector that this is the conditional vector that this is the conditional vector field and we can completely predict this field and we can completely predict this field and we can completely predict this conditional vector field. Okay, so now so far we have looked at Okay, so now so far we have looked at the conditional probability path and the the conditional probability path and the the conditional probability path and the conditional vector field and [snorts] we conditional vector field and [snorts] we conditional vector field and [snorts] we can predict the conditional probability can predict the conditional probability can predict the conditional probability path as a sequence of gshians which path as a sequence of gshians which path as a sequence of gshians which takes us from the initial gshian takes us from the initial gshian takes us from the initial gshian distribution to a final point and what distribution to a final point and what distribution to a final point and what is responsible for this path is the is responsible for this path is the is responsible for this path is the velocity vector field which is pointed velocity vector field which is pointed
- L1 @ 2377s
velocity vector field which is pointed from our data to the data point where we from our data to the data point where we from our data to the data point where we want to go. So it's it's a it's arrow want to go. So it's it's a it's arrow want to go. So it's it's a it's arrow which directly goes from where we are which directly goes from where we are which directly goes from where we are right now x and where we want to go that right now x and where we want to go that right now x and where we want to go that is zed. Okay. So now we will go to marginal Okay. So now we will go to marginal probability path. probability path. So now marginal is exactly the opposite So now marginal is exactly the opposite So now marginal is exactly the opposite of conditional. Imagine that instead of of conditional. Imagine that instead of of conditional. Imagine that instead of a single point zed we have a complex a single point zed we have a complex a single point zed we have a complex data distribution p data. And for every data distribution p data. And for every data distribution p data. And for every point in this distribution which is point in this distribution which is point in this distribution which is given to us, we can build a conditional given to us, we can build a conditional given to us, we can build a conditional probability path. And a collection of probability path. And a collection of probability path. And a collection of all the conditional probability path all the conditional probability path all the conditional probability path gives us the marginal probability path. gives us the marginal probability path. gives us the marginal probability path. So it's like once we understand So it's like once we understand So it's like once we understand conditional probability path we can conditional probability path we can conditional probability path we can think in this direction that if you think in this direction that if you think in this direction that if you aggregate all these paths together the aggregate all these paths together the aggregate all these paths together the final flow that you get the the final final flow that you get the the final
- L1 @ 2432s
final flow that you get the the final trajectories that you get those are the trajectories that you get those are the trajectories that you get those are the marginal probability paths. marginal probability paths. marginal probability paths. We have in fact defined this as flow in We have in fact defined this as flow in We have in fact defined this as flow in the last lecture but now we are giving the last lecture but now we are giving the last lecture but now we are giving it another name which is a marginal it another name which is a marginal it another name which is a marginal probability path. probability path. So you can imagine the distribution to So you can imagine the distribution to So you can imagine the distribution to be made up of a collection of zeds and a be made up of a collection of zeds and a be made up of a collection of zeds and a marginal path is a set of distributions marginal path is a set of distributions marginal path is a set of distributions which transforms the initial which transforms the initial which transforms the initial distribution pinet to P data. distribution pinet to P data. distribution pinet to P data. And if you recall this is exactly what And if you recall this is exactly what And if you recall this is exactly what we want to predict. we want to predict. we want to predict. [snorts] But here the interesting thing [snorts] But here the interesting thing [snorts] But here the interesting thing is that P data is not known to us. So we is that P data is not known to us. So we is that P data is not known to us. So we are just discussing this for a are just discussing this for a are just discussing this for a conceptual understanding of the conceptual understanding of the conceptual understanding of the difference between uh conditional difference between uh conditional difference between uh conditional probability path and marginal probability path and marginal probability path and marginal probability path probability path probability path but in reality the p data is not known but in reality the p data is not known but in reality the p data is not known to us. So here if you see
- L1 @ 2484s
to us. So here if you see to us. So here if you see the final distribution that we get which the final distribution that we get which the final distribution that we get which is which are these um is which are these um is which are these um eight clusters right that is the eight clusters right that is the eight clusters right that is the distribution that we want to predict. If distribution that we want to predict. If distribution that we want to predict. If that was known to us, we could directly that was known to us, we could directly that was known to us, we could directly construct this marginal probability construct this marginal probability construct this marginal probability path. path. path. But all we know is probably just But all we know is probably just But all we know is probably just thousand points within that thousand points within that thousand points within that distribution. We don't know the final distribution. We don't know the final distribution. We don't know the final overall distribution yet. overall distribution yet. overall distribution yet. And similarly, uh we can actually And similarly, uh we can actually And similarly, uh we can actually understand this. Let's let's understand this. Let's let's understand this. Let's let's go to this Google Collab notebook. >> Okay. So >> Okay. So I'm going to run this piece of code so I'm going to run this piece of code so I'm going to run this piece of code so that we understand what is the meaning that we understand what is the meaning that we understand what is the meaning of a marginal probability path. of a marginal probability path. of a marginal probability path. So we'll run this first piece of code So we'll run this first piece of code So we'll run this first piece of code over here. So this is very similar to the final So this is very similar to the final animation which I showed you. Uh animation which I showed you. Uh animation which I showed you. Uh so here this is the target data where we so here this is the target data where we so here this is the target data where we want to reach which is a sample of eight want to reach which is a sample of eight
- L1 @ 2570s
want to reach which is a sample of eight washians washians washians and initially we start with a random and initially we start with a random and initially we start with a random noise. noise. noise. So here you see the animation is now So here you see the animation is now So here you see the animation is now available to us. available to us. available to us. So we go from the initial distribution So we go from the initial distribution So we go from the initial distribution of random noise to a final distribution of random noise to a final distribution of random noise to a final distribution where these eight gshians are where these eight gshians are where these eight gshians are distributed in space. distributed in space. distributed in space. Now that is this is exactly what we want Now that is this is exactly what we want Now that is this is exactly what we want to predict the marginal probability path to predict the marginal probability path to predict the marginal probability path in in reality the distribution of cats in in reality the distribution of cats in in reality the distribution of cats will also be very complex right will also be very complex right will also be very complex right and uh along with the marginal and uh along with the marginal and uh along with the marginal probability path now you know that there probability path now you know that there probability path now you know that there will be a field also associated with it will be a field also associated with it will be a field also associated with it similar to how there was a vector field similar to how there was a vector field similar to how there was a vector field associated with the conditional associated with the conditional associated with the conditional probability path. Similarly for marginal probability path. Similarly for marginal probability path. Similarly for marginal probability path there will be a vector probability path there will be a vector probability path there will be a vector field associated and ideally this vector field associated and ideally this vector field associated and ideally this vector field is our training target. field is our training target. field is our training target. Now uh here let's play this Now uh here let's play this
- L1 @ 2625s
Now uh here let's play this piece of code to understand this piece of code to understand this piece of code to understand this u and you'll probably get something like and you'll probably get something like this at the end. So here you see the this at the end. So here you see the this at the end. So here you see the velocity vector field. How they change velocity vector field. How they change velocity vector field. How they change with time, how they evolve with time. So with time, how they evolve with time. So with time, how they evolve with time. So it's a vector field which is dependent it's a vector field which is dependent it's a vector field which is dependent on space and time. But it is marginal on space and time. But it is marginal on space and time. But it is marginal vector field because we are not just vector field because we are not just vector field because we are not just finding a vector field for one single finding a vector field for one single finding a vector field for one single point which takes from a distribution to point which takes from a distribution to point which takes from a distribution to that point but rather we have a that point but rather we have a that point but rather we have a collection of points in space. So and we collection of points in space. So and we collection of points in space. So and we are we are calculating the vector field are we are calculating the vector field are we are calculating the vector field for all those points which make them for all those points which make them for all those points which make them evolve according to their probability evolve according to their probability evolve according to their probability paths. [snorts] paths. [snorts] paths. [snorts] So if we had access to this there would So if we had access to this there would So if we had access to this there would not be any question that we replace this not be any question that we replace this not be any question that we replace this in our target. But uh the the issue is in our target. But uh the the issue is
- L1 @ 2679s
in our target. But uh the the issue is that we do not have access to this that we do not have access to this that we do not have access to this because we only have access to our because we only have access to our because we only have access to our individual data points. Okay. Okay. So here is another example Okay. Okay. So here is another example to understand the difference between to understand the difference between to understand the difference between conditional and marginal paths. So here conditional and marginal paths. So here conditional and marginal paths. So here you see you see you see u u u our actual distribution is are these our actual distribution is are these our actual distribution is are these five spheres that I want to go to right five spheres that I want to go to right five spheres that I want to go to right which which are shown in these circles. which which are shown in these circles. which which are shown in these circles. This is the distribution which is a This is the distribution which is a This is the distribution which is a collection of these five spheres. But collection of these five spheres. But collection of these five spheres. But I'm not aware of these distributions. I I'm not aware of these distributions. I I'm not aware of these distributions. I I I just know a single point zed. I I just know a single point zed. I I just know a single point zed. So I take my original gshian and I move So I take my original gshian and I move So I take my original gshian and I move it towards this single point zed it towards this single point zed it towards this single point zed according to these trajectories according to these trajectories according to these trajectories which I get by solving this conditional which I get by solving this conditional which I get by solving this conditional distribution uh this this conditional od distribution uh this this conditional od distribution uh this this conditional od where we have seen that this mu is given where we have seen that this mu is given where we have seen that this mu is given by z minus x divided by 1 minus t.
- L1 @ 2739s
by z minus x divided by 1 minus t. by z minus x divided by 1 minus t. [snorts] So these these these field [snorts] So these these these field [snorts] So these these these field lines are almost linear from the source lines are almost linear from the source lines are almost linear from the source to the target over here. to the target over here. to the target over here. So we are calculating the trajectories So we are calculating the trajectories So we are calculating the trajectories from the original gshian to the single from the original gshian to the single from the original gshian to the single point Z which has been sampled from our point Z which has been sampled from our point Z which has been sampled from our distribution. distribution. This is what is the meaning of a This is what is the meaning of a This is what is the meaning of a conditional probability path. conditional probability path. conditional probability path. Whereas [snorts] Whereas [snorts] Whereas [snorts] marginal probability path is something marginal probability path is something marginal probability path is something very different. Marginal probability very different. Marginal probability very different. Marginal probability path is uh something which takes us from path is uh something which takes us from path is uh something which takes us from our original uh our original uh our original uh distribution which is the gshian distribution which is the gshian distribution which is the gshian distribution over here and then you see distribution over here and then you see distribution over here and then you see how we are learning all these how we are learning all these how we are learning all these trajectories. So basically you sample a trajectories. So basically you sample a trajectories. So basically you sample a lot of zed from the uh and and if you lot of zed from the uh and and if you lot of zed from the uh and and if you have access to infinite samples then have access to infinite samples then have access to infinite samples then obviously you will go down this path but obviously you will go down this path but obviously you will go down this path but we do not have access to infinite we do not have access to infinite
- L1 @ 2790s
we do not have access to infinite samples. we have access to finite samples. we have access to finite samples. we have access to finite samples samples samples and uh this this uh credit goes to uh and uh this this uh credit goes to uh and uh this this uh credit goes to uh the MIT course by Peter Bodarit. I have the MIT course by Peter Bodarit. I have the MIT course by Peter Bodarit. I have link this in in the original flow model link this in in the original flow model link this in in the original flow model lecture also. So this this makes the lecture also. So this this makes the lecture also. So this this makes the difference between the conditional paths difference between the conditional paths difference between the conditional paths and the marginal paths very clear. and the marginal paths very clear. and the marginal paths very clear. Now uh the reason why we went into Now uh the reason why we went into Now uh the reason why we went into conditional and marginal paths is conditional and marginal paths is conditional and marginal paths is because it will help us understand the because it will help us understand the because it will help us understand the training target and what compromise we training target and what compromise we training target and what compromise we have done to to get to the training have done to to get to the training have done to to get to the training target. The ideal training target should target. The ideal training target should target. The ideal training target should be the marginal vector field but uh we be the marginal vector field but uh we be the marginal vector field but uh we will understand that we make a will understand that we make a will understand that we make a compromise to it and we use a very compromise to it and we use a very compromise to it and we use a very different type of training target. different type of training target. different type of training target. So now uh let's let's try to broadly So now uh let's let's try to broadly So now uh let's let's try to broadly understand what we are doing here. understand what we are doing here.
- L1 @ 2838s
understand what we are doing here. Essentially we are treating data Essentially we are treating data Essentially we are treating data generation as a flow problem. We know generation as a flow problem. We know generation as a flow problem. We know mathematically that mathematically that mathematically that for any complex data set whether it's for any complex data set whether it's for any complex data set whether it's pictures of cats, robotics policies or pictures of cats, robotics policies or pictures of cats, robotics policies or Giblly scenes there exists a vector Giblly scenes there exists a vector Giblly scenes there exists a vector field that connects a cloud of random field that connects a cloud of random field that connects a cloud of random noise directly to that data and this is noise directly to that data and this is noise directly to that data and this is the marginal vector field. Okay. Now if we had a map of this Okay. Now if we had a map of this current uh generating a new image would current uh generating a new image would current uh generating a new image would be simple and deterministic. So we be simple and deterministic. So we be simple and deterministic. So we already know this if we have access to already know this if we have access to already know this if we have access to this vector field uh we we simply use this vector field uh we we simply use this vector field uh we we simply use this vector field and uh we calculate this vector field and uh we calculate this vector field and uh we calculate the trajectory using this vector field the trajectory using this vector field the trajectory using this vector field using the oiler method. This is what using the oiler method. This is what using the oiler method. This is what detective 2 said at the start of today's detective 2 said at the start of today's detective 2 said at the start of today's lecture. lecture. lecture. But the biggest problem is that we do But the biggest problem is that we do But the biggest problem is that we do not have the formula for this vector not have the formula for this vector not have the formula for this vector field is impossibly complex. field is impossibly complex.
- L1 @ 2897s
field is impossibly complex. You cannot write down an equation that You cannot write down an equation that You cannot write down an equation that defines what a Giblly movie looks like. defines what a Giblly movie looks like. defines what a Giblly movie looks like. So we we cannot calculate the vector So we we cannot calculate the vector So we we cannot calculate the vector field but we can approximate it. And uh field but we can approximate it. And uh field but we can approximate it. And uh in today's lecture we are not going to in today's lecture we are not going to in today's lecture we are not going to look at how exactly the target of the look at how exactly the target of the look at how exactly the target of the velocity field is constructed. But from velocity field is constructed. But from velocity field is constructed. But from a mathematical point of view if you look a mathematical point of view if you look a mathematical point of view if you look at it today's lecture was a bit involved at it today's lecture was a bit involved at it today's lecture was a bit involved into condition and marginal path. The into condition and marginal path. The into condition and marginal path. The next lecture is going to be very easy next lecture is going to be very easy next lecture is going to be very easy for us because it is simply going to for us because it is simply going to for us because it is simply going to build upon today's lecture and in fact build upon today's lecture and in fact build upon today's lecture and in fact we will understand how easy it is to we will understand how easy it is to we will understand how easy it is to construct the training target. Uh and construct the training target. Uh and construct the training target. Uh and and you might think okay so does this and you might think okay so does this and you might think okay so does this actually work? It it sounds too good to actually work? It it sounds too good to actually work? It it sounds too good to be true and it does actually work. We we
- L1 @ 2946s
be true and it does actually work. We we be true and it does actually work. We we have seen this in other uh aspects of have seen this in other uh aspects of have seen this in other uh aspects of deep generative models like diffusion deep generative models like diffusion deep generative models like diffusion models and uh uh score based models and uh uh score based models and uh uh score based generative models that conditional generative models that conditional generative models that conditional models or if you condition it on the models or if you condition it on the models or if you condition it on the data you you get a training target which data you you get a training target which data you you get a training target which is feasible and magically it just it is feasible and magically it just it is feasible and magically it just it just works. So we are going to just works. So we are going to just works. So we are going to understand how to construct this understand how to construct this understand how to construct this training target which is going to be training target which is going to be training target which is going to be exceedingly simple and then we are going exceedingly simple and then we are going exceedingly simple and then we are going to understand how to set up our whole to understand how to set up our whole to understand how to set up our whole flow pipeline and uh we will also look flow pipeline and uh we will also look flow pipeline and uh we will also look at a practical example of how to at a practical example of how to at a practical example of how to generate a flow model uh how to train a generate a flow model uh how to train a generate a flow model uh how to train a flow model using uh the training target flow model using uh the training target flow model using uh the training target and how to sample from it. [snorts] So and how to sample from it. [snorts] So and how to sample from it. [snorts] So uh I I hope this lecture was interesting uh I I hope this lecture was interesting
- L1 @ 2994s
uh I I hope this lecture was interesting for all of you. I haven't yet given the for all of you. I haven't yet given the for all of you. I haven't yet given the final reveal. Uh I'm I'm delaying it final reveal. Uh I'm I'm delaying it final reveal. Uh I'm I'm delaying it purposefully. As I said at the start of purposefully. As I said at the start of purposefully. As I said at the start of the first lecture in the flow series, I the first lecture in the flow series, I the first lecture in the flow series, I could have wrapped this entire series in could have wrapped this entire series in could have wrapped this entire series in just maybe 40 minutes. But then you will just maybe 40 minutes. But then you will just maybe 40 minutes. But then you will not appreciate the kind of thinking not appreciate the kind of thinking not appreciate the kind of thinking which has gone into constructing this which has gone into constructing this which has gone into constructing this training target. And uh today's lecture training target. And uh today's lecture training target. And uh today's lecture I wanted to specifically devote to that I wanted to specifically devote to that I wanted to specifically devote to that so that all of you understand uh how so that all of you understand uh how so that all of you understand uh how these targets are constructed. Before we these targets are constructed. Before we these targets are constructed. Before we go ahead, I just want to uh motivate all go ahead, I just want to uh motivate all go ahead, I just want to uh motivate all of you by showing this paper pi 0.5 of you by showing this paper pi 0.5 of you by showing this paper pi 0.5 or let's look at pi 0. or let's look at pi 0. or let's look at pi 0. So this is a robotics foundational model So this is a robotics foundational model So this is a robotics foundational model and uh the reason I'm showing this to and uh the reason I'm showing this to and uh the reason I'm showing this to you right now is you right now is you right now is this this came out last year
- L1 @ 3045s
this this came out last year this this came out last year and this is based on uh flow matching. and this is based on uh flow matching. and this is based on uh flow matching. So here essentially what we are doing is So here essentially what we are doing is So here essentially what we are doing is uh when you see flow matching now all of uh when you see flow matching now all of uh when you see flow matching now all of you can understand that we are trying to you can understand that we are trying to you can understand that we are trying to find the trajectories from the initial find the trajectories from the initial find the trajectories from the initial gshian distribution to a final gshian distribution to a final gshian distribution to a final distribution and we [snorts] are trying distribution and we [snorts] are trying distribution and we [snorts] are trying to find a velocity field that takes us to find a velocity field that takes us to find a velocity field that takes us from the initial to the final from the initial to the final from the initial to the final distribution. The only thing to think distribution. The only thing to think distribution. The only thing to think about is what is this distribution about is what is this distribution about is what is this distribution representing. So, so far we have seen representing. So, so far we have seen representing. So, so far we have seen distributions representing images. But distributions representing images. But distributions representing images. But distributions can also represent robotic distributions can also represent robotic distributions can also represent robotic actions which is exactly what is actions which is exactly what is actions which is exactly what is happening in this in this paper which is happening in this in this paper which is happening in this in this paper which is amazing. If you see the applications of amazing. If you see the applications of amazing. If you see the applications of flow they they have a flow expert which flow they they have a flow expert which flow they they have a flow expert which is which is the action expert here which is which is the action expert here which
- L1 @ 3092s
is which is the action expert here which is a it's based on flow matching and uh is a it's based on flow matching and uh is a it's based on flow matching and uh the training target that you see here is the training target that you see here is the training target that you see here is exceedingly simple. Maybe I can show exceedingly simple. Maybe I can show exceedingly simple. Maybe I can show that to you uh which is written in in in that to you uh which is written in in in that to you uh which is written in in in in this section. We will we will look at in this section. We will we will look at in this section. We will we will look at this in detail in the next lecture. But this in detail in the next lecture. But this in detail in the next lecture. But once you go through the next lecture, once you go through the next lecture, once you go through the next lecture, you will be able to understand these you will be able to understand these you will be able to understand these sections which are coming in any paper. sections which are coming in any paper. sections which are coming in any paper. So that whenever you see flow matching, So that whenever you see flow matching, So that whenever you see flow matching, it will suddenly not be very it will suddenly not be very it will suddenly not be very inaccessible field to you, but you can inaccessible field to you, but you can inaccessible field to you, but you can relate to what we have seen in the relate to what we have seen in the relate to what we have seen in the lecture and build a very strong lecture and build a very strong lecture and build a very strong intuition for it. Thank you very much intuition for it. Thank you very much intuition for it. Thank you very much everyone and I will see you in the next everyone and I will see you in the next everyone and I will see you in the next lecture which is going to be like a lecture which is going to be like a
- L1 @ 3129s
lecture which is going to be like a magician's reveal. Thank you.