← back to concept

Langevin Dynamics — across courses

cmu-10799-diffusion-flow

  1. L1 @ 4s

    So let's begin. Previously last class uh So let's begin. Previously last class uh we have talked about what is diffusion we have talked about what is diffusion we have talked about what is diffusion model and essentially it's just like um model and essentially it's just like um model and essentially it's just like um very nice way to turn noise into data. very nice way to turn noise into data. very nice way to turn noise into data. And how do we do that? Well, basically And how do we do that? Well, basically And how do we do that? Well, basically uh we in order to construct um the uh we in order to construct um the uh we in order to construct um the training data or the training signal, training data or the training signal, training data or the training signal, the supervision signal, we first start the supervision signal, we first start the supervision signal, we first start from our training data and then we from our training data and then we from our training data and then we gradually add noise until it becomes gradually add noise until it becomes gradually add noise until it becomes full Gausian noise. uh and then at uh full Gausian noise. uh and then at uh full Gausian noise. uh and then at uh you know at at testing time or at you know at at testing time or at you know at at testing time or at inference time we used to learn the inference time we used to learn the inference time we used to learn the model uh to gradually den noiseise uh model uh to gradually den noiseise uh model uh to gradually den noiseise uh and the and the process of adding noise and the and the process of adding noise and the and the process of adding noise is what we call a for process and then is what we call a for process and then is what we call a for process and then the process of dnoising is what we call the process of dnoising is what we call the process of dnoising is what we call a reversed process a reversed process a reversed process um reverse not um reverse not um reverse not reserve sorry um all right so just a

  2. L1 @ 57s

    reserve sorry um all right so just a reserve sorry um all right so just a reminder what is the forward process reminder what is the forward process reminder what is the forward process basically ally between each step uh basically ally between each step uh basically ally between each step uh we're adding some amount of gausian we're adding some amount of gausian we're adding some amount of gausian noise to the scale version of your noise to the scale version of your noise to the scale version of your previous uh data sample. Um so basically previous uh data sample. Um so basically previous uh data sample. Um so basically uh what's going to h what's happening in uh what's going to h what's happening in uh what's going to h what's happening in math is just uh you know defined as math is just uh you know defined as math is just uh you know defined as above. So like the your next time your above. So like the your next time your above. So like the your next time your next time step is equal to basically the next time step is equal to basically the next time step is equal to basically the scaled version of your previous time scaled version of your previous time scaled version of your previous time step plus some gian plus some scaled step plus some gian plus some scaled step plus some gian plus some scaled version of the gausian noise. version of the gausian noise. version of the gausian noise. All right. So, this is the this is a do All right. So, this is the this is a do All right. So, this is the this is a do that that you need to remember for the that that you need to remember for the that that you need to remember for the rest of your life. Um, all right. And uh rest of your life. Um, all right. And uh rest of your life. Um, all right. And uh what's nice about it being Gausian is what's nice about it being Gausian is what's nice about it being Gausian is that basically because uh two gausians that basically because uh two gausians that basically because uh two gausians sums up together is another gausian and sums up together is another gausian and

  3. L1 @ 104s

    sums up together is another gausian and you can just sum up a bunch of gausians. you can just sum up a bunch of gausians. you can just sum up a bunch of gausians. So that's why in the forward process you So that's why in the forward process you So that's why in the forward process you can actually you don't actually need to can actually you don't actually need to can actually you don't actually need to have this cascading chain. uh what you have this cascading chain. uh what you have this cascading chain. uh what you can do is you can literally just like can do is you can literally just like can do is you can literally just like you can go from x0 and then you can you can go from x0 and then you can you can go from x0 and then you can directly get the distribution at any directly get the distribution at any directly get the distribution at any time step. Um so basically what's time step. Um so basically what's time step. Um so basically what's happening is that um like we can we can happening is that um like we can we can happening is that um like we can we can just like calculate some coefficients just like calculate some coefficients just like calculate some coefficients and then we can directly get the and then we can directly get the and then we can directly get the distribution of each time step um and given any data each time step um and given any data sample right and basically XT which is sample right and basically XT which is sample right and basically XT which is the noisy sample at at time step t is the noisy sample at at time step t is the noisy sample at at time step t is equal to your the scale version of your equal to your the scale version of your equal to your the scale version of your training data plus some scale version of training data plus some scale version of training data plus some scale version of the gausian noise again. Okay, so this the gausian noise again. Okay, so this the gausian noise again. Okay, so this is the diffusion for process. All right. is the diffusion for process. All right.

  4. L1 @ 155s

    is the diffusion for process. All right. Now, uh how now that we uh we have the Now, uh how now that we uh we have the Now, uh how now that we uh we have the forward process to train to construct forward process to train to construct forward process to train to construct our supervision signals, how do we learn our supervision signals, how do we learn our supervision signals, how do we learn a model to D noiseis and once we have a model to D noiseis and once we have a model to D noiseis and once we have the model, how do we actually sample the model, how do we actually sample the model, how do we actually sample from that model? All right. So basic from that model? All right. So basic from that model? All right. So basic basically uh naively uh you can define basically uh naively uh you can define basically uh naively uh you can define another gausian uh distribution uh to go another gausian uh distribution uh to go another gausian uh distribution uh to go backwards backwards backwards essentially. Um but uh basically DDPM uh essentially. Um but uh basically DDPM uh essentially. Um but uh basically DDPM uh which is the main model that we learned which is the main model that we learned which is the main model that we learned the last time tells us that if we just the last time tells us that if we just the last time tells us that if we just like simplify everything we if we just like simplify everything we if we just like simplify everything we if we just like simplify all the settings for like simplify all the settings for like simplify all the settings for example we just like assume that we we example we just like assume that we we example we just like assume that we we we fix the variance of the backore we fix the variance of the backore we fix the variance of the backore process then we only need to learn the process then we only need to learn the process then we only need to learn the mean of the gausian uh and in addition mean of the gausian uh and in addition

  5. L1 @ 204s

    mean of the gausian uh and in addition we can actually rewrite this mean this we can actually rewrite this mean this we can actually rewrite this mean this prediction of mean into something that prediction of mean into something that prediction of mean into something that is with respect to is with respect to is with respect to with respect to the prediction of the with respect to the prediction of the with respect to the prediction of the noise and that just becomes like you're noise and that just becomes like you're noise and that just becomes like you're learning a noise predictor or you're learning a noise predictor or you're learning a noise predictor or you're learning a dino noiser essentially right learning a dino noiser essentially right learning a dino noiser essentially right so at training time what you do is just so at training time what you do is just so at training time what you do is just literally do regression on the noise literally do regression on the noise literally do regression on the noise that you're adding to your in in your that you're adding to your in in your that you're adding to your in in your forward process uh and then in the at forward process uh and then in the at forward process uh and then in the at sampling time you just basically do okay sampling time you just basically do okay sampling time you just basically do okay my current time stamp minus a scaled my current time stamp minus a scaled my current time stamp minus a scaled version of the noise that I predicted version of the noise that I predicted version of the noise that I predicted plus some you know exploration plus some you know exploration plus some you know exploration randomness and then I'll get the and randomness and then I'll get the and randomness and then I'll get the and then I just like go through the entire then I just like go through the entire then I just like go through the entire chain and I'll get the sampling and uh chain and I'll get the sampling and uh chain and I'll get the sampling and uh last time there's some questions about last time there's some questions about

  6. L1 @ 248s

    last time there's some questions about why should we do this reparameterization why should we do this reparameterization why should we do this reparameterization well because doing this regression is well because doing this regression is well because doing this regression is very is much much easier to train uh very is much much easier to train uh very is much much easier to train uh than if you just learn this um this very than if you just learn this um this very than if you just learn this um this very complicated mean which like you know has complicated mean which like you know has complicated mean which like you know has like very complicated structures at like very complicated structures at like very complicated structures at different time steps. All right, cool. different time steps. All right, cool. different time steps. All right, cool. So this is what we learned the last So this is what we learned the last So this is what we learned the last time. Um so so far we have seen a lot of time. Um so so far we have seen a lot of time. Um so so far we have seen a lot of gener models right so this is uh this is gener models right so this is uh this is gener models right so this is uh this is what we seen the first uh in the first what we seen the first uh in the first what we seen the first uh in the first lecture uh we have talked about uh in lecture uh we have talked about uh in lecture uh we have talked about uh in details auto reggressive models and VAEs details auto reggressive models and VAEs details auto reggressive models and VAEs and uh we haven't talked about what is and uh we haven't talked about what is and uh we haven't talked about what is normalizing flow and what is EVM but normalizing flow and what is EVM but normalizing flow and what is EVM but like we're just going to briefly mention like we're just going to briefly mention like we're just going to briefly mention this uh before I do that does anyone this uh before I do that does anyone

  7. L1 @ 294s

    this uh before I do that does anyone know in the audience what is normalizing know in the audience what is normalizing know in the audience what is normalizing flow by any No way. Nobody know. No way. Nobody know. Wow. Okay. I feel a little bit sad, but Wow. Okay. I feel a little bit sad, but Wow. Okay. I feel a little bit sad, but this is okay. Um, yeah, it it used to be this is okay. Um, yeah, it it used to be this is okay. Um, yeah, it it used to be pretty popular, I guess, but there's a pretty popular, I guess, but there's a pretty popular, I guess, but there's a reason why it's not popular. Uh, but reason why it's not popular. Uh, but reason why it's not popular. Uh, but normalizing flow is basically uh imagine normalizing flow is basically uh imagine normalizing flow is basically uh imagine that you're like, you know how like in that you're like, you know how like in that you're like, you know how like in VA we're learning like a encoder and VA we're learning like a encoder and VA we're learning like a encoder and decoder. Well, now basically you just decoder. Well, now basically you just decoder. Well, now basically you just say that okay, I'm just going to like uh say that okay, I'm just going to like uh say that okay, I'm just going to like uh predefine the structure of my gener predefine the structure of my gener predefine the structure of my gener model so that like basically the mapping model so that like basically the mapping model so that like basically the mapping from data to the latent quote unquote uh from data to the latent quote unquote uh from data to the latent quote unquote uh and and the mapping from the latent to and and the mapping from the latent to and and the mapping from the latent to the data is going to be inversible. So the data is going to be inversible. So the data is going to be inversible. So like basically the the mapping from data like basically the the mapping from data like basically the the mapping from data to uh latent is the inverse of the to uh latent is the inverse of the

  8. L1 @ 345s

    to uh latent is the inverse of the mapping from latent to data. So this is mapping from latent to data. So this is mapping from latent to data. So this is sort of like normalizing flows. Um and sort of like normalizing flows. Um and sort of like normalizing flows. Um and then EBM is basically just like um then EBM is basically just like um then EBM is basically just like um because you know how like we like all because you know how like we like all because you know how like we like all the probability needs to add up to one, the probability needs to add up to one, the probability needs to add up to one, right? So the EBM is basically just right? So the EBM is basically just right? So the EBM is basically just saying that okay, we're just going to saying that okay, we're just going to saying that okay, we're just going to learn like a unnormalized version of the learn like a unnormalized version of the learn like a unnormalized version of the distribution and then we're going to distribution and then we're going to distribution and then we're going to normalize it later on using this thing normalize it later on using this thing normalize it later on using this thing called partition function. And this called partition function. And this called partition function. And this partition function is literally just partition function is literally just partition function is literally just like you know integrate like you know integrate like you know integrate all the probability together the all the probability together the all the probability together the anomorized probability together and this anomorized probability together and this anomorized probability together and this way you can get uh a normalized way you can get uh a normalized way you can get uh a normalized distribution. Okay cool. Um so distribution. Okay cool. Um so distribution. Okay cool. Um so unfortunately as we have learned before unfortunately as we have learned before unfortunately as we have learned before uh likelihood is very difficult to uh likelihood is very difficult to uh likelihood is very difficult to calculate. Why? Because uh for example calculate. Why? Because uh for example calculate. Why? Because uh for example if you have a auto reggressive models

  9. L1 @ 400s

    if you have a auto reggressive models if you have a auto reggressive models then you need to break things up by the then you need to break things up by the then you need to break things up by the chain rule and then you need to chain rule and then you need to chain rule and then you need to calculate each components one by one. So calculate each components one by one. So calculate each components one by one. So if you have a very high dimensional data if you have a very high dimensional data if you have a very high dimensional data like image then you're going to be like image then you're going to be like image then you're going to be calculating you know uh the the the calculating you know uh the the the calculating you know uh the the the chain rule for for each uh pixel and chain rule for for each uh pixel and chain rule for for each uh pixel and that's just like really really painful. that's just like really really painful. that's just like really really painful. And then for VAEEs, while this is nice, And then for VAEEs, while this is nice, And then for VAEEs, while this is nice, but you're actually um you're still but you're actually um you're still but you're actually um you're still using a surrogated loss, meaning that using a surrogated loss, meaning that using a surrogated loss, meaning that you're you're actually not directly uh you're you're actually not directly uh you're you're actually not directly uh maximizing the likelihood. What you do maximizing the likelihood. What you do maximizing the likelihood. What you do is you're maximizing the the like the is you're maximizing the the like the is you're maximizing the the like the lower bound of a likelihood. So, and lower bound of a likelihood. So, and lower bound of a likelihood. So, and that's like after all the hard math that that's like after all the hard math that that's like after all the hard math that we did. So, this is like still kind of we did. So, this is like still kind of we did. So, this is like still kind of suboptimal. suboptimal. suboptimal. And uh for normalizing flow just by And uh for normalizing flow just by

  10. L1 @ 445s

    And uh for normalizing flow just by looking at this can anyone tell me what looking at this can anyone tell me what looking at this can anyone tell me what potential problem it can it can have. Yes. Yes. >> Hard to get function inverse and >> Hard to get function inverse and >> Hard to get function inverse and >> very good. Very good. So for normalizing >> very good. Very good. So for normalizing >> very good. Very good. So for normalizing flow uh you'll need weird architectures flow uh you'll need weird architectures flow uh you'll need weird architectures essentially to make sure that everything essentially to make sure that everything essentially to make sure that everything is invert invertible. And what about is invert invertible. And what about is invert invertible. And what about energy based model? What about EBM? energy based model? What about EBM? energy based model? What about EBM? Yes, partition function. Yeah, so this Yes, partition function. Yeah, so this Yes, partition function. Yeah, so this partition function is generally very partition function is generally very partition function is generally very very expensive to compute or just very expensive to compute or just very expensive to compute or just intractable in general. All right. H so intractable in general. All right. H so intractable in general. All right. H so likelihood is difficult to calculate, likelihood is difficult to calculate, likelihood is difficult to calculate, right? H so maybe let's just go right? H so maybe let's just go right? H so maybe let's just go likelihood free. Would that be easier? likelihood free. Would that be easier? likelihood free. Would that be easier? Uh well for likelihood free model we Uh well for likelihood free model we Uh well for likelihood free model we have learned GANs and that dominated the have learned GANs and that dominated the have learned GANs and that dominated the genom or the image generate model for genom or the image generate model for genom or the image generate model for many many years like five years at many many years like five years at many many years like five years at least. Um but then it as we have learned least. Um but then it as we have learned

  11. L1 @ 503s

    least. Um but then it as we have learned last time it also has a problem of like last time it also has a problem of like last time it also has a problem of like very unstable training and also it very unstable training and also it very unstable training and also it suffers a lot from mo collapse without suffers a lot from mo collapse without suffers a lot from mo collapse without any tricks. All right. any tricks. All right. any tricks. All right. So So So is there a better way to avoid directly is there a better way to avoid directly is there a better way to avoid directly calculating the likelihood? Then is calculating the likelihood? Then is calculating the likelihood? Then is there a better way to do sort of there a better way to do sort of there a better way to do sort of likelihood free training? This is the likelihood free training? This is the likelihood free training? This is the question we're going to answer today. question we're going to answer today. question we're going to answer today. So if we think about it right, if we So if we think about it right, if we So if we think about it right, if we just want to do sampling like just just want to do sampling like just just want to do sampling like just forget about density estimation and forget about density estimation and forget about density estimation and everything like that for a bit. If we everything like that for a bit. If we everything like that for a bit. If we just want to do sampling, uh do we just want to do sampling, uh do we just want to do sampling, uh do we actually need to have a model to actually need to have a model to actually need to have a model to actually predict the likelihood of the actually predict the likelihood of the actually predict the likelihood of the data to sample from? >> Hold on. Many voices. Yeah. >> Hold on. Many voices. Yeah. to learn a transformation from some to learn a transformation from some to learn a transformation from some distribution that we know to the space distribution that we know to the space distribution that we know to the space of images rather than actually having to

  12. L1 @ 564s

    of images rather than actually having to of images rather than actually having to comput the comput the comput the >> yes that's right so you just need to >> yes that's right so you just need to >> yes that's right so you just need to learn a transformation right you don't learn a transformation right you don't learn a transformation right you don't actually need the you don't actually actually need the you don't actually actually need the you don't actually need the likelihood function and I think need the likelihood function and I think need the likelihood function and I think someone else also someone else also someone else also oh yeah oh yeah oh yeah >> you're jumping ahead but yeah so >> you're jumping ahead but yeah so >> you're jumping ahead but yeah so basically the idea is that basically the idea is that basically the idea is that we don't right and like we We just need we don't right and like we We just need we don't right and like we We just need a way to transform the like a data a way to transform the like a data a way to transform the like a data sample from whatever initial sample from whatever initial sample from whatever initial distribution that we have to a data distribution that we have to a data distribution that we have to a data sample in the desired distribution right sample in the desired distribution right sample in the desired distribution right so how do we do that how do we construct so how do we do that how do we construct so how do we do that how do we construct this transformation well the idea is this transformation well the idea is this transformation well the idea is that like I I'm sure you guys all know that like I I'm sure you guys all know that like I I'm sure you guys all know what is uh stoastic gradient descent what is uh stoastic gradient descent what is uh stoastic gradient descent right so this is what you train with right so this is what you train with right so this is what you train with model so what you do is you sort of just model so what you do is you sort of just

  13. L1 @ 613s

    model so what you do is you sort of just like follow the gra so say suppose you like follow the gra so say suppose you like follow the gra so say suppose you have a gradient that gives you like u have a gradient that gives you like u have a gradient that gives you like u useful information about which direction useful information about which direction useful information about which direction and where and how how far uh you should and where and how how far uh you should and where and how how far uh you should go uh to get your desired outcome. So go uh to get your desired outcome. So go uh to get your desired outcome. So you can just like sort of like follow you can just like sort of like follow you can just like sort of like follow this this gradient with some soassity this this gradient with some soassity this this gradient with some soassity right and and then you can you can get right and and then you can you can get right and and then you can you can get the desired outcome. So basically what the desired outcome. So basically what the desired outcome. So basically what we can do here is sort of we we don't we can do here is sort of we we don't we can do here is sort of we we don't really actually need the likelihood all really actually need the likelihood all really actually need the likelihood all we need is sort of like the gradient of we need is sort of like the gradient of we need is sort of like the gradient of the log likelihood so that we can just the log likelihood so that we can just the log likelihood so that we can just sort of do stoastic gradient descent or sort of do stoastic gradient descent or sort of do stoastic gradient descent or ascent I guess in in this case to ascent I guess in in this case to ascent I guess in in this case to maximize the log likelihood in the data maximize the log likelihood in the data maximize the log likelihood in the data space right so basically what you can space right so basically what you can

  14. L1 @ 660s

    space right so basically what you can imagine is that you can start from imagine is that you can start from imagine is that you can start from anywhere on the data space so in this anywhere on the data space so in this anywhere on the data space so in this case it's like this 2D plane right so case it's like this 2D plane right so case it's like this 2D plane right so you just say you Start from here and you just say you Start from here and you just say you Start from here and then you just follow the arrow. So then you just follow the arrow. So then you just follow the arrow. So follow the or like just like go with the follow the or like just like go with the follow the or like just like go with the flow flow flow essentially. So right so this is your essentially. So right so this is your essentially. So right so this is your gradient and then just like go and then gradient and then just like go and then gradient and then just like go and then you're going to eventually wind up at you're going to eventually wind up at you're going to eventually wind up at some somewhere where it's supposed to be some somewhere where it's supposed to be some somewhere where it's supposed to be high density. Um and this gradient is high density. Um and this gradient is high density. Um and this gradient is the gradient of the log likelihood. the gradient of the log likelihood. the gradient of the log likelihood. Okay. Or the the gradient of the real Okay. Or the the gradient of the real Okay. Or the the gradient of the real density the log density. Uh so what we density the log density. Uh so what we density the log density. Uh so what we call this is call this is call this is we call the score function right it's we call the score function right it's we call the score function right it's the gradient with respect to x so with the gradient with respect to x so with the gradient with respect to x so with respect to the data point of the log respect to the data point of the log

  15. L1 @ 705s

    respect to the data point of the log likelihood so what is happening is that likelihood so what is happening is that likelihood so what is happening is that like so and then again like what you can like so and then again like what you can like so and then again like what you can do is you can just sample in the data do is you can just sample in the data do is you can just sample in the data space and then go with the gradient in space and then go with the gradient in space and then go with the gradient in the data space and then you're gonna the data space and then you're gonna the data space and then you're gonna eventually end up at somewhere where eventually end up at somewhere where eventually end up at somewhere where it's supposed to be have high likelihood it's supposed to be have high likelihood it's supposed to be have high likelihood okay okay okay cool any questions Yeah, cool any questions Yeah, cool any questions Yeah, >> here the data space is the 2D plane. >> here the data space is the 2D plane. >> here the data space is the 2D plane. >> So will this 2D plate be like >> So will this 2D plate be like >> So will this 2D plate be like transformed to regular space? transformed to regular space? transformed to regular space? >> No. So in this case the uh I guess it's >> No. So in this case the uh I guess it's >> No. So in this case the uh I guess it's not clear from here but the uh the color not clear from here but the uh the color not clear from here but the uh the color the point are our uh desired the point are our uh desired the point are our uh desired distribution. So this is a so we desire distribution. So this is a so we desire distribution. So this is a so we desire to transform. That's a actually a great to transform. That's a actually a great to transform. That's a actually a great question. So your source distribution question. So your source distribution question. So your source distribution just like in diffusion right your source just like in diffusion right your source

  16. L1 @ 751s

    just like in diffusion right your source distribution and your target distribution and your target distribution and your target distribution should always be in the distribution should always be in the distribution should always be in the same space. So here our uh like desired same space. So here our uh like desired same space. So here our uh like desired distribution distribution distribution are these like colored mixture of are these like colored mixture of are these like colored mixture of gausian here. So the here and here and gausian here. So the here and here and gausian here. So the here and here and we are just uh randomly sample and and we are just uh randomly sample and and we are just uh randomly sample and and and the in other words the target and the in other words the target and the in other words the target distribution is also in the 2D plane and distribution is also in the 2D plane and distribution is also in the 2D plane and then we just like randomly sample a then we just like randomly sample a then we just like randomly sample a point on this the same 2D plane and then point on this the same 2D plane and then point on this the same 2D plane and then we follow the gradient of the log we follow the gradient of the log we follow the gradient of the log density to get to um to get to the the density to get to um to get to the the density to get to um to get to the the high density region of our desired high density region of our desired high density region of our desired distribution. I'm sorry. I'm sorry. >> Be easier. >> Be easier. >> Be easier. >> It will be easier. It could be just It >> It will be easier. It could be just It >> It will be easier. It could be just It could be just uh P like the gradient of could be just uh P like the gradient of could be just uh P like the gradient of P theta. That's fine. Sorry. The P theta. That's fine. Sorry. The P theta. That's fine. Sorry. The gradient of uh of of P gradient of uh of of P

  17. L1 @ 808s

    gradient of uh of of P X uh with respect to X. That's fine. But X uh with respect to X. That's fine. But X uh with respect to X. That's fine. But later on we're going to see why because later on we're going to see why because later on we're going to see why because log is easier to calculate in many log is easier to calculate in many log is easier to calculate in many cases. Yeah. cases. Yeah. cases. Yeah. >> Why do we suppose we know this function? >> Why do we suppose we know this function? >> Why do we suppose we know this function? How do we SAMPLE IT? How do we SAMPLE IT? How do we SAMPLE IT? YEAH, GREAT QUESTION. THIS IS WHAT WE'RE YEAH, GREAT QUESTION. THIS IS WHAT WE'RE YEAH, GREAT QUESTION. THIS IS WHAT WE'RE GOING TO learn today. All right. Cool, GOING TO learn today. All right. Cool, GOING TO learn today. All right. Cool, cool, cool. Okay. Okay. Um, so let's cool, cool. Okay. Okay. Um, so let's cool, cool. Okay. Okay. Um, so let's first define what we're learning here first define what we're learning here first define what we're learning here first. Okay. Um, so uh, so now first. Okay. Um, so uh, so now first. Okay. Um, so uh, so now we are clear that the goal of our we are clear that the goal of our we are clear that the goal of our modeling is to get an estimation of the modeling is to get an estimation of the modeling is to get an estimation of the so-called score function. Right? So so-called score function. Right? So so-called score function. Right? So basically what we can do is literally we basically what we can do is literally we basically what we can do is literally we just need to train a model to estimate just need to train a model to estimate just need to train a model to estimate the score and we can just do that by the score and we can just do that by the score and we can just do that by minimizing an L2 right regression. This minimizing an L2 right regression. This minimizing an L2 right regression. This is uh machine learning 101 right. So is uh machine learning 101 right. So

  18. L1 @ 855s

    is uh machine learning 101 right. So very easy very easy uh uh uh very easy very easy uh uh uh very easy very easy uh uh uh formulation. And this L2 here uh it has formulation. And this L2 here uh it has formulation. And this L2 here uh it has a fancy name specifically because it has a fancy name specifically because it has a fancy name specifically because it has the this um score function thing but the this um score function thing but the this um score function thing but it's called fishial divergence. You it's called fishial divergence. You it's called fishial divergence. You don't really need to remember this this don't really need to remember this this don't really need to remember this this name but just but just like it's name but just but just like it's name but just but just like it's literally just you are minimizing an L2 literally just you are minimizing an L2 literally just you are minimizing an L2 which is L2 which is L2 which is L2 onto the the true score but here comes onto the the true score but here comes onto the the true score but here comes the question right oh actually before we the question right oh actually before we the question right oh actually before we here comes the question this thing is here comes the question this thing is here comes the question this thing is nice now because it doesn't have any nice now because it doesn't have any nice now because it doesn't have any inractable partition function like EBM inractable partition function like EBM inractable partition function like EBM uh it has no adversary training like in uh it has no adversary training like in uh it has no adversary training like in GANs it has no weird architectures to GANs it has no weird architectures to GANs it has no weird architectures to enforce invertability like uh enforce invertability like uh enforce invertability like uh normalizing flows and you also do not normalizing flows and you also do not normalizing flows and you also do not need to break everything up into chain need to break everything up into chain need to break everything up into chain rules rules rules like auto reggressive models. So it can

  19. L1 @ 908s

    like auto reggressive models. So it can like auto reggressive models. So it can sample or generate everything all at sample or generate everything all at sample or generate everything all at once once you have uh you know learned once once you have uh you know learned once once you have uh you know learned the model supposedly right like all the the model supposedly right like all the the model supposedly right like all the dimensions all at once. Uh so that's dimensions all at once. Uh so that's dimensions all at once. Uh so that's nice but the question is how do you nice but the question is how do you nice but the question is how do you train the model right because now this train the model right because now this train the model right because now this even though the formulation is very easy even though the formulation is very easy even though the formulation is very easy it it comes the qu like the first it it comes the qu like the first it it comes the qu like the first question you should ask is how do we question you should ask is how do we question you should ask is how do we even get this ground true score right even get this ground true score right even get this ground true score right because the data set usually doesn't because the data set usually doesn't because the data set usually doesn't come with the score it it certainly do come with the score it it certainly do come with the score it it certainly do not come with the the log density right not come with the the log density right not come with the the log density right because if it does come with log density because if it does come with log density because if it does come with log density then we what what why what are we then we what what why what are we then we what what why what are we learning here right so how do we even do learning here right so how do we even do learning here right so how do we even do this L2 without this ground truth score. this L2 without this ground truth score.

  20. L1 @ 947s

    this L2 without this ground truth score. Turns out you don't even need to use the Turns out you don't even need to use the Turns out you don't even need to use the ground truth score in your loss ground truth score in your loss ground truth score in your loss function. Why? function. Why? function. Why? H because of calculus practice. H because of calculus practice. H because of calculus practice. All right. So basically what you need to All right. So basically what you need to All right. So basically what you need to do here is you need to first break this do here is you need to first break this do here is you need to first break this L2 up this expectation of L2 up uh into L2 up this expectation of L2 up uh into L2 up this expectation of L2 up uh into three parts. Uh and then you may notice three parts. Uh and then you may notice three parts. Uh and then you may notice that because this is going to be a loss that because this is going to be a loss that because this is going to be a loss function, right? It's a loss function function, right? It's a loss function function, right? It's a loss function with respect to theta and the first part with respect to theta and the first part with respect to theta and the first part of the the first part of the equation is of the the first part of the equation is of the the first part of the equation is not even like related to theta. It's a not even like related to theta. It's a not even like related to theta. It's a constant with respect to theta. Right? constant with respect to theta. Right? constant with respect to theta. Right? So the first part of the the three So the first part of the the three So the first part of the the three components we can just cross it out. We components we can just cross it out. We components we can just cross it out. We don't even need it. All right. So now we don't even need it. All right. So now we don't even need it. All right. So now we simplify uh the L2 the the original L2

  21. L1 @ 995s

    simplify uh the L2 the the original L2 simplify uh the L2 the the original L2 into two parts here. And then the first into two parts here. And then the first into two parts here. And then the first part doesn't really have the log density part doesn't really have the log density part doesn't really have the log density or the score function anymore. So this or the score function anymore. So this or the score function anymore. So this is good. So we can just get that. Uh the is good. So we can just get that. Uh the is good. So we can just get that. Uh the second part still have log density. And second part still have log density. And second part still have log density. And how do we simplify that? Uh well how do we simplify that? Uh well how do we simplify that? Uh well basically you it's literally just basically you it's literally just basically you it's literally just calculus practice here. Um you don't calculus practice here. Um you don't calculus practice here. Um you don't really need to like like know the really need to like like know the really need to like like know the details now, but I'm just going to walk details now, but I'm just going to walk details now, but I'm just going to walk through everything. Um but basically through everything. Um but basically through everything. Um but basically what you do is you first uh break up the what you do is you first uh break up the what you do is you first uh break up the expectation and then notice that the expectation and then notice that the expectation and then notice that the gradient of log probability or log any gradient of log probability or log any gradient of log probability or log any the gradient of log actually is the the gradient of log actually is the the gradient of log actually is the derivative of log is one over derivative of log is one over derivative of log is one over the the function right so basically you the the function right so basically you the the function right so basically you can have the gradient of uh log of p can have the gradient of uh log of p

  22. L1 @ 1047s

    can have the gradient of uh log of p with respect to x into you can break it with respect to x into you can break it with respect to x into you can break it into this thing right uh and then these into this thing right uh and then these into this thing right uh and then these two things cancel these 2 px p theta x two things cancel these 2 px p theta x two things cancel these 2 px p theta x cancelled and then you you're going to cancelled and then you you're going to cancelled and then you you're going to get something very nice just a very get something very nice just a very get something very nice just a very short the integral of the inner product short the integral of the inner product short the integral of the inner product of the gradient of px and the learn the of the gradient of px and the learn the of the gradient of px and the learn the function and then basically what you do is this and then basically what you do is this is this part is like just trust me bro is this part is like just trust me bro is this part is like just trust me bro it's going to get there type of thing it's going to get there type of thing it's going to get there type of thing where you just integrate by par and then where you just integrate by par and then where you just integrate by par and then you can actually eliminate the first you can actually eliminate the first you can actually eliminate the first part of the by par part um because the part of the by par part um because the part of the by par part um because the boundary will vanish when the when the boundary will vanish when the when the boundary will vanish when the when the coordinate goes to infinite um or or at coordinate goes to infinite um or or at coordinate goes to infinite um or or at least this is like our assumption. Uh least this is like our assumption. Uh least this is like our assumption. Uh and then you can basically just get and then you can basically just get

  23. L1 @ 1093s

    and then you can basically just get something like this. So um yeah this is something like this. So um yeah this is something like this. So um yeah this is a this is like we we probably wouldn't a this is like we we probably wouldn't a this is like we we probably wouldn't have time to do all the derivation of have time to do all the derivation of have time to do all the derivation of this part but this is a trust me bro this part but this is a trust me bro this part but this is a trust me bro moment and feel free to look at the moment and feel free to look at the moment and feel free to look at the original paper after the class but original paper after the class but original paper after the class but basically what you will get after basically what you will get after basically what you will get after integration by power and with the integration by power and with the integration by power and with the assumption of uh boundary vanishing assumption of uh boundary vanishing assumption of uh boundary vanishing you're going to get uh something like you're going to get uh something like you're going to get uh something like this and this is literally just the this and this is literally just the this and this is literally just the negative of expectation of this this negative of expectation of this this negative of expectation of this this thing and then this thing is a the trace thing and then this thing is a the trace thing and then this thing is a the trace of the Jacobian of the Jacobian of the Jacobian of the score function. All right. So all of the score function. All right. So all of the score function. All right. So all the math doesn't really matter that the math doesn't really matter that the math doesn't really matter that much. I mean does matter but like much. I mean does matter but like much. I mean does matter but like doesn't really matter for now. Um all we doesn't really matter for now. Um all we doesn't really matter for now. Um all we need to remember is that the second part need to remember is that the second part

  24. L1 @ 1138s

    need to remember is that the second part of the two things left. It's going to of the two things left. It's going to of the two things left. It's going to come boils down to the negative of the come boils down to the negative of the come boils down to the negative of the expectation of the trace of the Jacobia expectation of the trace of the Jacobia expectation of the trace of the Jacobia of the score model. All right. Cool. Uh of the score model. All right. Cool. Uh of the score model. All right. Cool. Uh so so this is what we get at the end. Um so so this is what we get at the end. Um so so this is what we get at the end. Um so the first part is sort of like L2 so the first part is sort of like L2 so the first part is sort of like L2 norm of your uh score function and then norm of your uh score function and then norm of your uh score function and then the second part is the trace of the the second part is the trace of the the second part is the trace of the Jacobian of the score function. Uh yeah Jacobian of the score function. Uh yeah Jacobian of the score function. Uh yeah and uh yeah so this is like the the and uh yeah so this is like the the and uh yeah so this is like the the final score matching loss and this is final score matching loss and this is final score matching loss and this is what we call the score matching loss. what we call the score matching loss. what we call the score matching loss. Yeah. All right. So you may notice this Yeah. All right. So you may notice this Yeah. All right. So you may notice this is very nice, right? Because right now is very nice, right? Because right now is very nice, right? Because right now in the score matching loss in the final in the score matching loss in the final in the score matching loss in the final loss that we get, we don't really need a loss that we get, we don't really need a

  25. L1 @ 1181s

    loss that we get, we don't really need a ground true score. It's it's like ground true score. It's it's like ground true score. It's it's like actually pretty pretty sick. So all you actually pretty pretty sick. So all you actually pretty pretty sick. So all you need to do is you just need you can just need to do is you just need you can just need to do is you just need you can just calculate everything with your model and calculate everything with your model and calculate everything with your model and then it can magically work. This is then it can magically work. This is then it can magically work. This is amazing, right? Is this truly amazing? amazing, right? Is this truly amazing? amazing, right? Is this truly amazing? What do we think? What do we think? Oh What do we think? What do we think? Oh What do we think? What do we think? Oh yeah. and and obviously like if you if yeah. and and obviously like if you if yeah. and and obviously like if you if you have a bunch of data then you can you have a bunch of data then you can you have a bunch of data then you can just you know average over your data. just you know average over your data. just you know average over your data. So this is amazing right now we can do So this is amazing right now we can do So this is amazing right now we can do learning learning learning or can we actually or can we actually or can we actually what can be a potential problem if what can be a potential problem if what can be a potential problem if you're if you're gonna implement this you're if you're gonna implement this you're if you're gonna implement this how do you think of the problem how do you think of the problem how do you think of the problem very strange to me very strange to me very strange to me >> how do you well >> how do you well >> how do you well >> like >> like >> like >> okay yeah yeah like answer the question >> yeah exactly right because the score >> yeah exactly right because the score function has the same dimensionality of

  26. L1 @ 1238s

    function has the same dimensionality of function has the same dimensionality of the of the data, right? Yeah. So, you the of the data, right? Yeah. So, you the of the data, right? Yeah. So, you can just Yeah, can just Yeah, can just Yeah, >> you can just take the jacobian there. >> you can just take the jacobian there. >> you can just take the jacobian there. >> Yeah. Yeah. Exactly. Okay. But you were >> Yeah. Yeah. Exactly. Okay. But you were >> Yeah. Yeah. Exactly. Okay. But you were gonna say why is this not practical, gonna say why is this not practical, gonna say why is this not practical, right? right? right? >> Yes, that's right. Uh >> Yes, that's right. Uh >> Yes, that's right. Uh that is correct. that is correct. that is correct. Um yeah. So for any of you who have uh Um yeah. So for any of you who have uh Um yeah. So for any of you who have uh you know tried to calculate Jacobian you know tried to calculate Jacobian you know tried to calculate Jacobian with PyTorch uh you realize that just with PyTorch uh you realize that just with PyTorch uh you realize that just just calculating Jacobian itself is just calculating Jacobian itself is just calculating Jacobian itself is actually not optimized that well at all actually not optimized that well at all actually not optimized that well at all and it's actually really really and it's actually really really and it's actually really really expensive and uh yeah so this this thing expensive and uh yeah so this this thing expensive and uh yeah so this this thing is like super super expensive to is like super super expensive to is like super super expensive to calculate. Uh so is there another way calculate. Uh so is there another way calculate. Uh so is there another way is there another way to do this? is there another way to do this? is there another way to do this? Yes. Estimate. Estimate. Continue. Well, Yes. Estimate. Estimate. Continue. Well, Yes. Estimate. Estimate. Continue. Well, actually, does anyone else? Uh, actually, does anyone else? Uh, actually, does anyone else? Uh, no one else. Do you have a Do you have a no one else. Do you have a Do you have a

  27. L1 @ 1297s

    no one else. Do you have a Do you have a Do you have a Do you have a Do you have a >> Well, yes, but it's kind of related to >> Well, yes, but it's kind of related to >> Well, yes, but it's kind of related to how how how to get transform. to get transform. to get transform. So, maybe just sample those. You're you're on you're on a good track, You're you're on you're on a good track, but this is not not exactly. Basically, but this is not not exactly. Basically, but this is not not exactly. Basically, we're just trying to like avoid this we're just trying to like avoid this we're just trying to like avoid this whole Jacovian thing in in entirely. whole Jacovian thing in in entirely. whole Jacovian thing in in entirely. Does anyone else have any thoughts? Does anyone else have any thoughts? Does anyone else have any thoughts? >> From the distribution >> From the distribution >> From the distribution >> where >> where >> where the distribution the distribution the distribution >> sample distribution directly >> sample distribution directly >> sample distribution directly >> from a distribution directly. What kind >> from a distribution directly. What kind >> from a distribution directly. What kind of distribution? Just give me the first distribution that Just give me the first distribution that you that you can comment. you that you can comment. you that you can comment. Did someone say Gausian? Yes. Yeah. Did someone say Gausian? Yes. Yeah. Did someone say Gausian? Yes. Yeah. That's right. That's right. Exactly. That's right. That's right. Exactly. That's right. That's right. Exactly. This is why I say you need a tattoo bay This is why I say you need a tattoo bay This is why I say you need a tattoo bay rule on your right hand and then the the rule on your right hand and then the the rule on your right hand and then the the portrait of Gausian on the left hand. portrait of Gausian on the left hand. portrait of Gausian on the left hand. This is how important it is. All right. This is how important it is. All right. This is how important it is. All right. Literally Gausian. Okay. So basically Literally Gausian. Okay. So basically

  28. L1 @ 1362s

    Literally Gausian. Okay. So basically what you do is uh and this is what we what you do is uh and this is what we what you do is uh and this is what we call dnoising score matching. Why so get call dnoising score matching. Why so get call dnoising score matching. Why so get den noising? Well, basically uh you know den noising? Well, basically uh you know den noising? Well, basically uh you know the the true score of the data is very the the true score of the data is very the the true score of the data is very very difficult to calculate, right? Uh very difficult to calculate, right? Uh very difficult to calculate, right? Uh but if you just perturb a little bit of but if you just perturb a little bit of but if you just perturb a little bit of your your data, so you from this super your your data, so you from this super your your data, so you from this super clean image to a slightly noisy image. clean image to a slightly noisy image. clean image to a slightly noisy image. But now this say this perturbation But now this say this perturbation But now this say this perturbation kernel is a gausian, right? So you just kernel is a gausian, right? So you just kernel is a gausian, right? So you just add some gausian noise to it. Uh then add some gausian noise to it. Uh then add some gausian noise to it. Uh then now uh this this this uh this now uh this this this uh this now uh this this this uh this distribution became known, right? it distribution became known, right? it distribution became known, right? it became became became it becomes like a distribution that we it becomes like a distribution that we it becomes like a distribution that we know how to calculate the score of know how to calculate the score of know how to calculate the score of right. So basically what you need to do right. So basically what you need to do right. So basically what you need to do is you just need a sample data and then is you just need a sample data and then

  29. L1 @ 1405s

    is you just need a sample data and then perturb it a little bit and now it perturb it a little bit and now it perturb it a little bit and now it becomes this like it's easier to becomes this like it's easier to becomes this like it's easier to calculate distribution. Yeah the the calculate distribution. Yeah the the calculate distribution. Yeah the the dude the the the goat. Okay. So dude the the the goat. Okay. So dude the the the goat. Okay. So basically um yeah to to to just write it basically um yeah to to to just write it basically um yeah to to to just write it down formally uh if your perturbation down formally uh if your perturbation down formally uh if your perturbation kernel is some like zero mean gausian uh kernel is some like zero mean gausian uh kernel is some like zero mean gausian uh so you just add some gausian noise to it so you just add some gausian noise to it so you just add some gausian noise to it um then the perturb the distribution um then the perturb the distribution um then the perturb the distribution like so the distribution that you're like so the distribution that you're like so the distribution that you're actually going to calculate the score actually going to calculate the score actually going to calculate the score from uh it's just have this very easy from uh it's just have this very easy from uh it's just have this very easy and nice uh formula that tells you the and nice uh formula that tells you the and nice uh formula that tells you the density or the conditional density I density or the conditional density I density or the conditional density I guess because you're conditioning on a guess because you're conditioning on a guess because you're conditioning on a data data data and the score is literally literally and the score is literally literally and the score is literally literally just this. You can do your algebra. just this. You can do your algebra. just this. You can do your algebra. Sorry, you can do your calculus. This is Sorry, you can do your calculus. This is Sorry, you can do your calculus. This is literally this. Uh so yeah, for for

  30. L1 @ 1455s

    literally this. Uh so yeah, for for literally this. Uh so yeah, for for those of you who wonder why log density those of you who wonder why log density those of you who wonder why log density easier basically. Um yeah. So so so now easier basically. Um yeah. So so so now easier basically. Um yeah. So so so now the gradient of the log density is the gradient of the log density is the gradient of the log density is literally just one over theta square, literally just one over theta square, literally just one over theta square, which is your variance. one over which is your variance. one over which is your variance. one over variance times the noise essentially. variance times the noise essentially. variance times the noise essentially. Uh or like the negative of the noise I Uh or like the negative of the noise I Uh or like the negative of the noise I guess. guess. guess. All right. All right. All right. Any question here? No. No. Okay. It's interesting. It's curious Okay. It's interesting. It's curious Okay. It's interesting. It's curious that you guys do not have question that you guys do not have question that you guys do not have question because I would have had question. Yeah. >> There's no Jacobian anymore because >> There's no Jacobian anymore because uh because we can ex we can calculate uh because we can ex we can calculate uh because we can ex we can calculate exactly the ground now we have the exactly the ground now we have the exactly the ground now we have the ground truth. So the Jacobian is for the ground truth. So the Jacobian is for the ground truth. So the Jacobian is for the case where we do not have the ground case where we do not have the ground case where we do not have the ground truth score and here we do have the truth score and here we do have the truth score and here we do have the ground truth score. uh it just that the ground truth score. uh it just that the ground truth score. uh it just that the ground truth is not the actual ground ground truth is not the actual ground ground truth is not the actual ground truth it's a perturbed ground truth then

  31. L1 @ 1525s

    truth it's a perturbed ground truth then truth it's a perturbed ground truth then you may have a question but like let's you may have a question but like let's you may have a question but like let's marinate question a little bit um okay marinate question a little bit um okay marinate question a little bit um okay but let's say everything works out right but let's say everything works out right but let's say everything works out right let's say everything works out then once let's say everything works out then once let's say everything works out then once we have a trained model now we have we we have a trained model now we have we we have a trained model now we have we know two ways to train our models once know two ways to train our models once know two ways to train our models once that we have the trained model uh then that we have the trained model uh then that we have the trained model uh then we can do gradient as right just like we can do gradient as right just like we can do gradient as right just like what we did in the data space to to get what we did in the data space to to get what we did in the data space to to get a sample and this sort of gradient thing a sample and this sort of gradient thing a sample and this sort of gradient thing is called long driven dynamic is such a is called long driven dynamic is such a is called long driven dynamic is such a cool name. It's a Yeah, I feel like I chose my uh you know Yeah, I feel like I chose my uh you know my my my name for the for the Twitter my my my name for the for the Twitter my my my name for the for the Twitter too soon. I should have chose long too soon. I should have chose long too soon. I should have chose long dynamics better than electronic kel but dynamics better than electronic kel but dynamics better than electronic kel but anyway it's fine. Um uh so what you do anyway it's fine. Um uh so what you do

  32. L1 @ 1564s

    anyway it's fine. Um uh so what you do is you first draw draw a sample from is you first draw draw a sample from is you first draw draw a sample from your data space using some easy to your data space using some easy to your data space using some easy to sample prior distribution for example sample prior distribution for example sample prior distribution for example like a gausian or something like that. like a gausian or something like that. like a gausian or something like that. And then basically at each step what you And then basically at each step what you And then basically at each step what you do is you uh you you do gradient ascent do is you uh you you do gradient ascent do is you uh you you do gradient ascent with some small step size and then you with some small step size and then you with some small step size and then you add some stoasticity in it. Um and then add some stoasticity in it. Um and then add some stoasticity in it. Um and then if you have a train the model already if you have a train the model already if you have a train the model already you can swap the this this score part you can swap the this this score part you can swap the this this score part into the score model right that you into the score model right that you into the score model right that you just learn to predict the score. So it just learn to predict the score. So it just learn to predict the score. So it become the predicted score. Uh and then become the predicted score. Uh and then become the predicted score. Uh and then uh this noise thing is just to add uh this noise thing is just to add uh this noise thing is just to add exploration so that you don't you know exploration so that you don't you know exploration so that you don't you know just uh overconfident exploit too much. just uh overconfident exploit too much. just uh overconfident exploit too much. Okay. Okay. Okay. So this is what could have happened if So this is what could have happened if

  33. L1 @ 1610s

    So this is what could have happened if you do laundry dynamic essentially. So you do laundry dynamic essentially. So you do laundry dynamic essentially. So at the beginning, right, we just like at the beginning, right, we just like at the beginning, right, we just like just sample all over the place. And then just sample all over the place. And then just sample all over the place. And then as you progress in your gradient ascent, as you progress in your gradient ascent, as you progress in your gradient ascent, you're going to ended up in this nice you're going to ended up in this nice you're going to ended up in this nice location, this nice like cluster where location, this nice like cluster where location, this nice like cluster where it's supposed to have high density. Okay, any question? Okay, any question? All right. Anyway, point being uh so All right. Anyway, point being uh so All right. Anyway, point being uh so you have question. This is great. I you have question. This is great. I you have question. This is great. I actually Actually maybe let me let me uh actually Actually maybe let me let me uh actually Actually maybe let me let me uh let me go through this part and then and let me go through this part and then and let me go through this part and then and then maybe this question will be um so then maybe this question will be um so then maybe this question will be um so uh so this is the uh so this is the uh so this is the scorebased model pipeline. You basically scorebased model pipeline. You basically scorebased model pipeline. You basically train your model using score matching or train your model using score matching or train your model using score matching or the noising score matching or any type the noising score matching or any type the noising score matching or any type of score matching. Uh they have like of score matching. Uh they have like of score matching. Uh they have like variants of score matchings. Uh but variants of score matchings. Uh but variants of score matchings. Uh but anyway you just train your model to anyway you just train your model to

  34. L1 @ 1664s

    anyway you just train your model to predict the score and then you use predict the score and then you use predict the score and then you use dynamic to sample. Uh however does it dynamic to sample. Uh however does it dynamic to sample. Uh however does it actually work? What is your question? Uh actually work? What is your question? Uh actually work? What is your question? Uh why do we need exploration if we are why do we need exploration if we are why do we need exploration if we are already following the gradient? already following the gradient? already following the gradient? >> Great great question. Why? Uh turns out >> Great great question. Why? Uh turns out >> Great great question. Why? Uh turns out it's actually very important. But first it's actually very important. But first it's actually very important. But first let's you can let's think about it. Uh let's you can let's think about it. Uh let's you can let's think about it. Uh does it actually work? And also another does it actually work? And also another does it actually work? And also another question is if it doesn't have question is if it doesn't have question is if it doesn't have exploration, what's going to happen? exploration, what's going to happen? exploration, what's going to happen? Maybe this is like actually a good Maybe this is like actually a good Maybe this is like actually a good All right, talk to your neighbor. Uh All right, talk to your neighbor. Uh All right, talk to your neighbor. Uh yeah, just uh for let's say three yeah, just uh for let's say three yeah, just uh for let's say three minuteish Um, and then we're going to minuteish Um, and then we're going to minuteish Um, and then we're going to come back. come back. come back. It's three to five minute. Talk to your It's three to five minute. Talk to your It's three to five minute. Talk to your neighbor. Think about it. And then I'm neighbor. Think about it. And then I'm neighbor. Think about it. And then I'm going to survey the classroom again. Times up. Times up. All right, let's Times up. Times up. All right, let's survey. Let's survey. Uh we are starting survey. Let's survey. Uh we are starting

  35. L1 @ 1740s

    survey. Let's survey. Uh we are starting from this column again. from this column again. from this column again. Hello friends. Uh Hello friends. Uh Hello friends. Uh does anyone want to share their thoughts does anyone want to share their thoughts does anyone want to share their thoughts whether or not this actually works? All right, we're Oh, someone saved you. All right, we're Oh, someone saved you. But uh you know, next time next time But uh you know, next time next time But uh you know, next time next time we're already make eye contact. we're already make eye contact. we're already make eye contact. >> All right, >> All right, >> All right, >> we believe that it's really difficult to >> we believe that it's really difficult to >> we believe that it's really difficult to actually sample from um sample the actually sample from um sample the actually sample from um sample the scores from the uh from the model scores from the uh from the model scores from the uh from the model because you might end up in the low because you might end up in the low because you might end up in the low density region density region density region >> which is which is kind of difficult to >> which is which is kind of difficult to >> which is which is kind of difficult to uh you know because the model is not uh you know because the model is not uh you know because the model is not trained. trained. trained. >> Very nice. Very nice. Great. Great >> Very nice. Very nice. Great. Great >> Very nice. Very nice. Great. Great answer. All right. Uh what about the answer. All right. Uh what about the answer. All right. Uh what about the middle column? middle column? middle column? Middle column. Do we have another answer of why why it may or may not work? of why why it may or may not work? Oh, did you raise your hand? Oh no, did you raise your hand? Oh no, this is very unfortunate. Someone this is very unfortunate. Someone this is very unfortunate. Someone someone in the middle column that I someone in the middle column that I someone in the middle column that I haven't been that I haven't been talked

  36. L1 @ 1806s

    haven't been that I haven't been talked haven't been that I haven't been talked to today. There's so many people in the to today. There's so many people in the to today. There's so many people in the middle column though. for real. All right. for real. All right. Um, Um, Um, what about the left column first? Some thoughts? Okay. Yes. Some thoughts? Okay. Yes. It's too too difficult to me to know. It's too too difficult to me to know. It's too too difficult to me to know. >> It's just I don't know if this >> It's just I don't know if this >> It's just I don't know if this but I guess maybe the sound tool always but I guess maybe the sound tool always but I guess maybe the sound tool always move moves towards move moves towards move moves towards a very dense distribution rather than a very dense distribution rather than a very dense distribution rather than the right distribution. the right distribution. the right distribution. >> Very nice. Great answer too. Great >> Very nice. Great answer too. Great >> Very nice. Great answer too. Great answer. All right. Um answer. All right. Um answer. All right. Um let's uh let's review the the solution let's uh let's review the the solution let's uh let's review the the solution or let's review the answer. Okay. Um so or let's review the answer. Okay. Um so or let's review the answer. Okay. Um so yeah, no is that it actually very yeah, no is that it actually very yeah, no is that it actually very difficult to make it work in practice. difficult to make it work in practice. difficult to make it work in practice. And why? Uh the first one is actually a And why? Uh the first one is actually a And why? Uh the first one is actually a little bit more difficult to think little bit more difficult to think little bit more difficult to think about. Uh but the other two people about. Uh but the other two people about. Uh but the other two people nailed it basically. Um so the first one nailed it basically. Um so the first one nailed it basically. Um so the first one is what we call the manifold is what we call the manifold

  37. L1 @ 1872s

    is what we call the manifold hypothesis. What does it mean is that hypothesis. What does it mean is that hypothesis. What does it mean is that basically usually the data that we want basically usually the data that we want basically usually the data that we want to train on uh are reside on the like a to train on uh are reside on the like a to train on uh are reside on the like a lower dimensional manifold and what it lower dimensional manifold and what it lower dimensional manifold and what it means is basically like uh even though means is basically like uh even though means is basically like uh even though you have a very very high dimensional you have a very very high dimensional you have a very very high dimensional data like image like 4K image or data like image like 4K image or data like image like 4K image or something um like say you're only the something um like say you're only the something um like say you're only the data is only about human faces right data is only about human faces right data is only about human faces right then you can sort of just describe the then you can sort of just describe the then you can sort of just describe the human faces with like I don't know 128 human faces with like I don't know 128 human faces with like I don't know 128 like facial landmarks and that's the like facial landmarks and that's the like facial landmarks and that's the actual dimensionality quote unquote like actual dimensionality quote unquote like actual dimensionality quote unquote like the meaningful dimensionality of your of the meaningful dimensionality of your of the meaningful dimensionality of your of your data. So rather than 4K time 4K, your data. So rather than 4K time 4K, your data. So rather than 4K time 4K, you actually have a very low dimensional you actually have a very low dimensional you actually have a very low dimensional manifold for the data. And what it also manifold for the data. And what it also manifold for the data. And what it also means is that what means is that what means is that what what here says like some area of the

  38. L1 @ 1919s

    what here says like some area of the what here says like some area of the data space will not have support. What data space will not have support. What data space will not have support. What it means is that basically not all 4K it means is that basically not all 4K it means is that basically not all 4K images are human face images, right? So images are human face images, right? So images are human face images, right? So basically like you could have like basically like you could have like basically like you could have like complete gion noise or you can have like complete gion noise or you can have like complete gion noise or you can have like human face with like five eyes or human face with like five eyes or human face with like five eyes or something that that that may that's um something that that that may that's um something that that that may that's um could be I guess or or like human face could be I guess or or like human face could be I guess or or like human face with um with um with um I don't know with some animal features I don't know with some animal features I don't know with some animal features or something like right like like that or something like right like like that or something like right like like that right so that may that may or may not be right so that may that may or may not be right so that may that may or may not be considered as human right so basically considered as human right so basically considered as human right so basically that just means that not all images are that just means that not all images are that just means that not all images are human faces and what's gonna happen is human faces and what's gonna happen is human faces and what's gonna happen is that just that just that just Like when the score that is going Like when the score that is going Like when the score that is going outside that's going out from the outside that's going out from the outside that's going out from the manifold from some point from on the manifold from some point from on the

  39. L1 @ 1963s

    manifold from some point from on the manifold to some point that's not on the manifold to some point that's not on the manifold to some point that's not on the manifold the score is going to blow up manifold the score is going to blow up manifold the score is going to blow up right this because like basically you're right this because like basically you're right this because like basically you're going from nonzero density or like non going from nonzero density or like non going from nonzero density or like non negative infinity density to negative negative infinity density to negative negative infinity density to negative infinity density right so score is just infinity density right so score is just infinity density right so score is just gonna like the the the magnitude is just gonna like the the the magnitude is just gonna like the the the magnitude is just gonna blow up right and And the other gonna blow up right and And the other gonna blow up right and And the other thing is that as we can see here thing is that as we can see here thing is that as we can see here basically this is like just a score basically this is like just a score basically this is like just a score matching uh experiment that people ran matching uh experiment that people ran matching uh experiment that people ran and if you have this sort of like and if you have this sort of like and if you have this sort of like manifold hypothesis of your data then manifold hypothesis of your data then manifold hypothesis of your data then the the the score matching um the score the the the score matching um the score the the the score matching um the score estimation is actually not very estimation is actually not very estimation is actually not very consistent. So it just like really consistent. So it just like really consistent. So it just like really really unstable. really unstable. really unstable. Okay. Um so this is the first thing and Okay. Um so this is the first thing and Okay. Um so this is the first thing and then the second thing. Yeah. Yeah. Yeah. then the second thing. Yeah. Yeah. Yeah.

  40. L1 @ 2008s

    then the second thing. Yeah. Yeah. Yeah. >> What is this the left plot even showing >> What is this the left plot even showing >> What is this the left plot even showing either? Oh sorry the the the so uh the either? Oh sorry the the the so uh the either? Oh sorry the the the so uh the left plot is showing that basically this left plot is showing that basically this left plot is showing that basically this uh score matching so basically this one uh score matching so basically this one uh score matching so basically this one type of score matching loss when you type of score matching loss when you type of score matching loss when you only when you train on very very clean only when you train on very very clean only when you train on very very clean data and I think this is CR10 or data and I think this is CR10 or data and I think this is CR10 or something but basically this is like if something but basically this is like if something but basically this is like if you just do score matching if you you just do score matching if you you just do score matching if you actually train your model with score actually train your model with score actually train your model with score matching your loss function is going to matching your loss function is going to matching your loss function is going to look like this and it's like completely look like this and it's like completely look like this and it's like completely just like sh just like very bad um but just like sh just like very bad um but just like sh just like very bad um but this is basically ally if you just this is basically ally if you just this is basically ally if you just perturb it with like a very very like perturb it with like a very very like perturb it with like a very very like invisible gausian noise and because it's invisible gausian noise and because it's invisible gausian noise and because it's gausian right now the entire ambient gausian right now the entire ambient gausian right now the entire ambient space or entire data space now has space or entire data space now has

  41. L1 @ 2050s

    space or entire data space now has support uh and then the score matching support uh and then the score matching support uh and then the score matching actually behave really really nicely actually behave really really nicely actually behave really really nicely yeah yeah yeah >> so the loss become sort of converging >> so the loss become sort of converging >> so the loss become sort of converging it's still pretty high right it's like it's still pretty high right it's like it's still pretty high right it's like minus minus minus >> yeah yeah and >> yeah yeah and >> yeah yeah and >> it's like 10^ six uh >> it's like 10^ six uh >> it's like 10^ six uh >> is that that >> is that that >> is that that >> but like but this is not what supposed >> but like but this is not what supposed >> but like but this is not what supposed to do by the way. But this is just to to do by the way. But this is just to to do by the way. But this is just to show that like the score matching loss show that like the score matching loss show that like the score matching loss is like super unstable when you have is like super unstable when you have is like super unstable when you have like data that reside on low dimensional like data that reside on low dimensional like data that reside on low dimensional manifold and if you have some invisible manifold and if you have some invisible manifold and if you have some invisible gausian noise that basically put support gausian noise that basically put support gausian noise that basically put support on even if just a little bit on on even if just a little bit on on even if just a little bit on everywhere else it's going to stabilize everywhere else it's going to stabilize everywhere else it's going to stabilize your training. Uh okay divine you first your training. Uh okay divine you first your training. Uh okay divine you first the the the >> example that you mentioned where the >> example that you mentioned where the >> example that you mentioned where the image is eventually boiled down to say image is eventually boiled down to say

  42. L1 @ 2097s

    image is eventually boiled down to say 10 features or something in that case 10 features or something in that case 10 features or something in that case the manifold is load eventually how is the manifold is load eventually how is the manifold is load eventually how is this making it load eventually how how this making it load eventually how how this making it load eventually how how >> oh this is not making it low dimensional >> oh this is not making it low dimensional >> oh this is not making it low dimensional this is making a a data that basically this is making a a data that basically this is making a a data that basically so the data the data dimension itself is so the data the data dimension itself is so the data the data dimension itself is really high but the actual manifold like really high but the actual manifold like really high but the actual manifold like the actual meaningful dimensions are low the actual meaningful dimensions are low the actual meaningful dimensions are low >> noise makes the for manifold low is that >> noise makes the for manifold low is that >> noise makes the for manifold low is that >> no no no the the the the the basically >> no no no the the the the the basically >> no no no the the the the the basically what I was saying that like the the data what I was saying that like the the data what I was saying that like the the data before before you add gian noise the before before you add gian noise the before before you add gian noise the data only has support on the parts where data only has support on the parts where data only has support on the parts where it can be projected onto this low it can be projected onto this low it can be projected onto this low dimensional manifold. So not everywhere. dimensional manifold. So not everywhere. dimensional manifold. So not everywhere. So basically you you're going to have So basically you you're going to have So basically you you're going to have somewhere that there's like no support somewhere that there's like no support somewhere that there's like no support there's no probability and that's why

  43. L1 @ 2145s

    there's no probability and that's why there's no probability and that's why the score is going to blow up and adding the score is going to blow up and adding the score is going to blow up and adding small amount of gausian noise make small amount of gausian noise make small amount of gausian noise make everywhere has probability everywhere has probability everywhere has probability data that's good because then you have data that's good because then you have data that's good because then you have support everywhere support everywhere support everywhere >> well but your this is not your choice >> well but your this is not your choice >> well but your this is not your choice right right right >> is your data >> is your data >> is your data >> but to understand >> but to understand >> but to understand >> it's not necessarily like good or bad >> it's not necessarily like good or bad >> it's not necessarily like good or bad it's just saying that this loss function it's just saying that this loss function it's just saying that this loss function is not good enough for the data that we is not good enough for the data that we is not good enough for the data that we usually usually usually We assume that when perturbing with some We assume that when perturbing with some We assume that when perturbing with some amount of ging noise our data moves out amount of ging noise our data moves out amount of ging noise our data moves out of this low dimensional manifold and of this low dimensional manifold and of this low dimensional manifold and sort of like a sort of like a sort of like a >> yeah right like if you add gausian noise >> yeah right like if you add gausian noise >> yeah right like if you add gausian noise is no longer clean anymore right yeah is no longer clean anymore right yeah is no longer clean anymore right yeah that's correct yes there's going to be noise to it right there's going to be noise to it right >> uh >> uh >> uh so those are kind of already So this is what already happens. So this So this is what already happens. So this is like they they they train on the

  44. L1 @ 2202s

    is like they they they train on the is like they they they train on the clean data. So it's like not I mean yes clean data. So it's like not I mean yes clean data. So it's like not I mean yes obviously there will be noise but obviously there will be noise but obviously there will be noise but basically even those noisy data you can basically even those noisy data you can basically even those noisy data you can fit like yeah there's still low fit like yeah there's still low fit like yeah there's still low dimensional manifold. dimensional manifold. dimensional manifold. >> Yeah compared to if you actually add >> Yeah compared to if you actually add >> Yeah compared to if you actually add noise to it. Yeah. Okay. Any more noise to it. Yeah. Okay. Any more noise to it. Yeah. Okay. Any more questions? questions? questions? No. Okay cool. All right. Uh the second No. Okay cool. All right. Uh the second No. Okay cool. All right. Uh the second pitfall is actually come from the right pitfall is actually come from the right pitfall is actually come from the right column. Very nice. Uh basically what's column. Very nice. Uh basically what's column. Very nice. Uh basically what's happening is that even if we do have happening is that even if we do have happening is that even if we do have full support data space like even if we full support data space like even if we full support data space like even if we have probability everywhere the score have probability everywhere the score have probability everywhere the score estimation is still inaccurate in the estimation is still inaccurate in the estimation is still inaccurate in the low low density region right because we low low density region right because we low low density region right because we don't even see those data very often. So don't even see those data very often. So don't even see those data very often. So how can we learn a a an accurate model how can we learn a a an accurate model how can we learn a a an accurate model without any data right? So what's going without any data right? So what's going

  45. L1 @ 2248s

    without any data right? So what's going to happen here is that um so say this is to happen here is that um so say this is to happen here is that um so say this is like your actual data score that your like your actual data score that your like your actual data score that your estimated score may look like something estimated score may look like something estimated score may look like something like this right then our initial samples like this right then our initial samples like this right then our initial samples are usually in the low density regions are usually in the low density regions are usually in the low density regions right for example say we sample from a right for example say we sample from a right for example say we sample from a gausian right the gausian a gausian gausian right the gausian a gausian gausian right the gausian a gausian image is very very very low density if image is very very very low density if image is very very very low density if they have any density right it in the they have any density right it in the they have any density right it in the actual real human face image um actual real human face image um actual real human face image um distribution so what you're going to end distribution so what you're going to end distribution so what you're going to end up with you're going to end up with up with you're going to end up with up with you're going to end up with something here basically. And if you if something here basically. And if you if something here basically. And if you if you follow the real score, you could by you follow the real score, you could by you follow the real score, you could by long dynamic go to the high density long dynamic go to the high density long dynamic go to the high density region. But if you use you learn the region. But if you use you learn the region. But if you use you learn the score because the learn score doesn't score because the learn score doesn't score because the learn score doesn't really know where to move in the low really know where to move in the low

  46. L1 @ 2294s

    really know where to move in the low density region. So you're just going to density region. So you're just going to density region. So you're just going to move sort of randomly and then you you move sort of randomly and then you you move sort of randomly and then you you can end up in somewhere like here where can end up in somewhere like here where can end up in somewhere like here where there's still low density. there's still low density. there's still low density. Okay. Okay. Any any questions? Any any questions? Any any questions? No question. No question. No question. All right. Uh, last one is from the All right. Uh, last one is from the All right. Uh, last one is from the Yeah, it's from the left column. Yeah, it's from the left column. Yeah, it's from the left column. Basically, what happen is that if you Basically, what happen is that if you Basically, what happen is that if you just go with a flow, if you just do just go with a flow, if you just do just go with a flow, if you just do gradient or ascent, right? The gradient gradient or ascent, right? The gradient gradient or ascent, right? The gradient doesn't really know like it just knows doesn't really know like it just knows doesn't really know like it just knows that okay uh like from here I should that okay uh like from here I should that okay uh like from here I should flow to here, from here I should flow to flow to here, from here I should flow to flow to here, from here I should flow to here, right? it doesn't really know uh here, right? it doesn't really know uh here, right? it doesn't really know uh like basically it cannot really like basically it cannot really like basically it cannot really distinguish or like a lot of times it's distinguish or like a lot of times it's distinguish or like a lot of times it's not completely it's not 100% you cannot not completely it's not 100% you cannot not completely it's not 100% you cannot but a lot of times it cannot distinguish but a lot of times it cannot distinguish but a lot of times it cannot distinguish high density from higher density right

  47. L1 @ 2342s

    high density from higher density right high density from higher density right so like say this is your like intended so like say this is your like intended so like say this is your like intended data data distribution then your dynamic data data distribution then your dynamic data data distribution then your dynamic can just look like they have equal can just look like they have equal can just look like they have equal density in two in the two mixture density in two in the two mixture density in two in the two mixture whereas because you just follow the whereas because you just follow the whereas because you just follow the gradient you just go with the flow but gradient you just go with the flow but gradient you just go with the flow but in reality ity they could have one has in reality ity they could have one has in reality ity they could have one has higher density and one have lower higher density and one have lower higher density and one have lower density even though they're both like density even though they're both like density even though they're both like high density regions so to speak. high density regions so to speak. high density regions so to speak. >> Yeah. couldn't. So assuming we had the >> Yeah. couldn't. So assuming we had the >> Yeah. couldn't. So assuming we had the real score function in that case this is real score function in that case this is real score function in that case this is not a problem right it's because we not a problem right it's because we not a problem right it's because we could not learn it because in that case could not learn it because in that case could not learn it because in that case even even even >> even if you have it is still a problem >> even if you have it is still a problem >> even if you have it is still a problem right right >> the direction from that low density >> the direction from that low density >> the direction from that low density region the directions would map towards region the directions would map towards region the directions would map towards the the >> it it depends on like what exactly is

  48. L1 @ 2388s

    >> it it depends on like what exactly is >> it it depends on like what exactly is this distribution say if this uh this distribution say if this uh this distribution say if this uh distribution is a mixture of gausian distribution is a mixture of gausian distribution is a mixture of gausian where those the the weight of two where those the the weight of two where those the the weight of two components add up to one so it's like components add up to one so it's like components add up to one so it's like one minus lambda for for one and then one minus lambda for for one and then one minus lambda for for one and then lambda for the other then if you lambda for the other then if you lambda for the other then if you calculate the score the score doesn't calculate the score the score doesn't calculate the score the score doesn't have lambda in it right so then you have have lambda in it right so then you have have lambda in it right so then you have no idea I mean technically technically no idea I mean technically technically no idea I mean technically technically if you move slow enough right if you if you move slow enough right if you if you move slow enough right if you move like if you just like make micro move like if you just like make micro move like if you just like make micro movement every time then technically you movement every time then technically you movement every time then technically you could still sample the correct could still sample the correct could still sample the correct distribution but then you need to just distribution but then you need to just distribution but then you need to just do micro movement every time so it's do micro movement every time so it's do micro movement every time so it's going to be very very slow and in the going to be very very slow and in the going to be very very slow and in the paper it's actually called slow mixing paper it's actually called slow mixing paper it's actually called slow mixing of launch dynamic. Yeah, but good of launch dynamic. Yeah, but good of launch dynamic. Yeah, but good question. question. question. Okay.

  49. L1 @ 2434s

    Okay. All right. Any other question? Cool. Cool. All right. So, we have all these All right. So, we have all these All right. So, we have all these problems. problems. problems. Is it problem? Is it possible to solve Is it problem? Is it possible to solve Is it problem? Is it possible to solve all these problem all at once? all these problem all at once? all these problem all at once? What do we think? Very true. Very true. But but okay say Very true. Very true. But but okay say it is possible. it is possible. it is possible. What would be the thing to fix What would be the thing to fix What would be the thing to fix everything? Just say some words like the the the the Just say some words like the the the the first thing that you think of like the first thing that you think of like the first thing that you think of like the theme of this this whole lecture theory. theme of this this whole lecture theory. theme of this this whole lecture theory. Yeah. Yeah. Yay. Yeah. Yeah. Literally. Yeah. Yeah. Yay. Yeah. Yeah. Literally. Yeah. Yeah. Yay. Yeah. Yeah. Literally. So, yeah, just think about it like every So, yeah, just think about it like every So, yeah, just think about it like every time you don't know what to do, you you time you don't know what to do, you you time you don't know what to do, you you channel your inner gouge and this is channel your inner gouge and this is channel your inner gouge and this is what this is what you do. Okay. Um, what this is what you do. Okay. Um, what this is what you do. Okay. Um, okay. Actually, literally adding gausian okay. Actually, literally adding gausian okay. Actually, literally adding gausian noise just solves everything, right? noise just solves everything, right? noise just solves everything, right? Because basically if you have a perturb Because basically if you have a perturb Because basically if you have a perturb the distribution now first of all it has the distribution now first of all it has the distribution now first of all it has support everywhere. So, the manifold support everywhere. So, the manifold support everywhere. So, the manifold thing doesn't exist anymore, right? And

  50. L1 @ 2508s

    thing doesn't exist anymore, right? And thing doesn't exist anymore, right? And then now the perturbed like say you have then now the perturbed like say you have then now the perturbed like say you have like a high density to higher density like a high density to higher density like a high density to higher density right then like uh like like the higher right then like uh like like the higher right then like uh like like the higher and the high are going to be more and the high are going to be more and the high are going to be more distinguishable and also the low density distinguishable and also the low density distinguishable and also the low density region are going to be easier to get region are going to be easier to get region are going to be easier to get sampled when at during training right so sampled when at during training right so sampled when at during training right so it's just going to be like easier for it's just going to be like easier for it's just going to be like easier for the model to learn all right the model to learn all right the model to learn all right now we should have some questions right now we should have some questions right now we should have some questions right yes That's exactly correct. Right. So the That's exactly correct. Right. So the problem is if we add noise, right? Like problem is if we add noise, right? Like problem is if we add noise, right? Like how much noise should we add? Because if how much noise should we add? Because if how much noise should we add? Because if if if we we add too small amount of if if we we add too small amount of if if we we add too small amount of noise, then it's just going to not going noise, then it's just going to not going noise, then it's just going to not going to have any effect, right? If we add too to have any effect, right? If we add too to have any effect, right? If we add too large of a of a noise, then it you're large of a of a noise, then it you're

  51. L1 @ 2557s

    large of a of a noise, then it you're just learning a very noisy distribution. just learning a very noisy distribution. just learning a very noisy distribution. That's not what we want. So how much That's not what we want. So how much That's not what we want. So how much noise should we add? Right? That's a noise should we add? Right? That's a noise should we add? Right? That's a that's the question. That's the question that's the question. That's the question that's the question. That's the question that you that you asked like a while that you that you asked like a while that you that you asked like a while ago. All right. So, ago. All right. So, ago. All right. So, what do we think? >> Yes. >> Yes. >> Should be smaller than the step size. >> Should be smaller than the step size. >> Should be smaller than the step size. >> But now that you you mentioned the step >> But now that you you mentioned the step >> But now that you you mentioned the step size, size, size, >> what are we what are we thinking? >> what are we what are we thinking? >> what are we what are we thinking? We we have we're taking many many steps, We we have we're taking many many steps, We we have we're taking many many steps, right? So, right? So, right? So, It should decrease as we get closer to It should decrease as we get closer to It should decrease as we get closer to the the >> Yes. >> Yes. >> Yeah. Yeah. Yeah. Exactly. Correct. So, >> Yeah. Yeah. Yeah. Exactly. Correct. So, >> Yeah. Yeah. Yeah. Exactly. Correct. So, what if we just like start big? So, we what if we just like start big? So, we what if we just like start big? So, we just like at the beginning, let's just just like at the beginning, let's just just like at the beginning, let's just add a bunch of noise, right? And then as add a bunch of noise, right? And then as add a bunch of noise, right? And then as we become like more and more confident we become like more and more confident we become like more and more confident about which region that we are uh that

  52. L1 @ 2613s

    about which region that we are uh that about which region that we are uh that we're s we should be sampling at, then we're s we should be sampling at, then we're s we should be sampling at, then we're just going to like decrease the we're just going to like decrease the we're just going to like decrease the noise as we go. noise as we go. noise as we go. Right? So this thing is called a Right? So this thing is called a Right? So this thing is called a kneeling longer dynamic kneeling longer dynamic kneeling longer dynamic because it's a kneeling, right? because it's a kneeling, right? because it's a kneeling, right? Yeah. Also, by the way, a new launch Yeah. Also, by the way, a new launch Yeah. Also, by the way, a new launch dynamic has a really, really cool dynamic has a really, really cool dynamic has a really, really cool Chinese name. Like the Chinese Chinese name. Like the Chinese Chinese name. Like the Chinese translation is so cool. It's like translation is so cool. It's like translation is so cool. It's like anyway, great name for your Twitter anyway, great name for your Twitter anyway, great name for your Twitter handle. Not important. The point being, handle. Not important. The point being, handle. Not important. The point being, so this is how you this is how you do a so this is how you this is how you do a so this is how you this is how you do a neo dynamic. So you start from the right neo dynamic. So you start from the right neo dynamic. So you start from the right actually. So you start from the right, actually. So you start from the right, actually. So you start from the right, you add a lot of noise and then as you you add a lot of noise and then as you you add a lot of noise and then as you get closer to or more and more certain get closer to or more and more certain get closer to or more and more certain about which uh region that you should about which uh region that you should about which uh region that you should sample from, you are going to just sample from, you are going to just

  53. L1 @ 2659s

    sample from, you are going to just gradually decrease the noise or the gradually decrease the noise or the gradually decrease the noise or the exploration term that you're adding at exploration term that you're adding at exploration term that you're adding at each um that at each step. So this also each um that at each step. So this also each um that at each step. So this also answer the question of why we need answer the question of why we need answer the question of why we need exploration. Yeah, exploration. Yeah, exploration. Yeah, >> this analing effect is what we're doing >> this analing effect is what we're doing >> this analing effect is what we're doing sampling during training. is still just sampling during training. is still just sampling during training. is still just part of the distribution with that one part of the distribution with that one part of the distribution with that one decided noise. decided noise. decided noise. >> What do you think? Do you think we >> What do you think? Do you think we >> What do you think? Do you think we should do that or do you think we should do that or do you think we should do that or do you think we shouldn't do that? We should not. Why? shouldn't do that? We should not. Why? shouldn't do that? We should not. Why? >> Because then >> Because then >> Because then we we have that distribution and we need we we have that distribution and we need we we have that distribution and we need messing up that distribution. messing up that distribution. messing up that distribution. >> Yeah. Right. So you don't even know like >> Yeah. Right. So you don't even know like >> Yeah. Right. So you don't even know like basically so at training time you should basically so at training time you should basically so at training time you should be learning a noise level condition be learning a noise level condition be learning a noise level condition model. model. model. >> Very very good. Very very good. Do we >> Very very good. Very very good. Do we >> Very very good. Very very good. Do we have that next? uh we have the pseudo have that next? uh we have the pseudo

  54. L1 @ 2705s

    have that next? uh we have the pseudo code and you know the long dynam dynamic code and you know the long dynam dynamic code and you know the long dynam dynamic left over. Yeah. But basically literally left over. Yeah. But basically literally left over. Yeah. But basically literally just uh you you you do you have a step just uh you you you do you have a step just uh you you you do you have a step size and then you know as as time goes size and then you know as as time goes size and then you know as as time goes you you decrease the noise that you add. you you decrease the noise that you add. you you decrease the noise that you add. Um but yeah Um but yeah Um but yeah this is why we're learning a noise this is why we're learning a noise this is why we're learning a noise conditioned score model because now we conditioned score model because now we conditioned score model because now we do not have one level of noise. We have do not have one level of noise. We have do not have one level of noise. We have multiple levels of noise. Right? So this multiple levels of noise. Right? So this multiple levels of noise. Right? So this is like how you would have sample uh is like how you would have sample uh is like how you would have sample uh from a noise condition score network in from a noise condition score network in from a noise condition score network in CSN. Uh and u yeah so basically how you CSN. Uh and u yeah so basically how you CSN. Uh and u yeah so basically how you should parameterize your network is should parameterize your network is should parameterize your network is going to be uh you should be taking your going to be uh you should be taking your going to be uh you should be taking your current sample noisy and the current current sample noisy and the current current sample noisy and the current noise level that you're dealing with. noise level that you're dealing with. noise level that you're dealing with. And then at training time you do this And then at training time you do this

  55. L1 @ 2750s

    And then at training time you do this multi-level den noising score matching multi-level den noising score matching multi-level den noising score matching because at each time you basically because at each time you basically because at each time you basically you're doing a denoising score matching you're doing a denoising score matching you're doing a denoising score matching at every noise level. It just you just at every noise level. It just you just at every noise level. It just you just need to be able to do it with in need to be able to do it with in need to be able to do it with in multiple levels and at sampling time you multiple levels and at sampling time you multiple levels and at sampling time you can just do a new launch dynamic. >> Uh you mean like density estimation or >> Uh you mean like density estimation or what do you mean by learning what do you mean by learning what do you mean by learning distribution? This is learning distribution? This is learning distribution? This is learning distribution. distribution. distribution. Right. Right. Right. >> Yeah. But if you're asking about density >> Yeah. But if you're asking about density >> Yeah. But if you're asking about density estimation, we're going to talk about estimation, we're going to talk about estimation, we're going to talk about next class. Actually, we don't have time next class. Actually, we don't have time next class. Actually, we don't have time for this class. Actually, we may not for this class. Actually, we may not for this class. Actually, we may not even get through all everything this even get through all everything this even get through all everything this class, which is interesting. I guess I'm class, which is interesting. I guess I'm class, which is interesting. I guess I'm improving. Anyway, uh, any more improving. Anyway, uh, any more improving. Anyway, uh, any more questions? Cool. No more question. Cool. No more question. Hold up. Wait a minute. Hold up. Wait a minute. Hold up. Wait a minute. Doesn't that look familiar? It kind of just look like diffusion, It kind of just look like diffusion, right? What is going on? right? What is going on? right? What is going on? Is this just diffusion? Yeah, just just think about it. Uh,

  56. L1 @ 2843s

    Yeah, just just think about it. Uh, basically the difference between DDPM, basically the difference between DDPM, basically the difference between DDPM, which is what we learned last last time, which is what we learned last last time, which is what we learned last last time, and this new thing noise condition score and this new thing noise condition score and this new thing noise condition score network, network, network, if you look at them, if you look at them, if you look at them, they're practically the same. specifically the only difference sort of specifically the only difference sort of is how you define your noising process is how you define your noising process is how you define your noising process essentially and how you calculate the essentially and how you calculate the essentially and how you calculate the loss, right? loss, right? loss, right? And uh and it turns out that And uh and it turns out that And uh and it turns out that they are kind of connected. So basically they are kind of connected. So basically they are kind of connected. So basically what you can do is you can sort of if what you can do is you can sort of if what you can do is you can sort of if you define your um the noisy sample with you define your um the noisy sample with you define your um the noisy sample with respect to the noise and the clean data respect to the noise and the clean data respect to the noise and the clean data you can sort of like write both of them you can sort of like write both of them you can sort of like write both of them into the same formula. You just have into the same formula. You just have into the same formula. You just have different hyperparameter choices for the different hyperparameter choices for the different hyperparameter choices for the formula formula formula and the loss function are pretty much the the loss function are pretty much the same. same. same. like you're both learning to d noiseise like you're both learning to d noiseise like you're both learning to d noiseise essentially and the only difference is essentially and the only difference is essentially and the only difference is that the the the score is sort of it has

  57. L1 @ 2928s

    that the the the score is sort of it has that the the the score is sort of it has like another factor or it has another like another factor or it has another like another factor or it has another coefficient coefficient coefficient in the front. So basically DDPN and this in the front. So basically DDPN and this in the front. So basically DDPN and this noise noise condition score thing is the noise noise condition score thing is the noise noise condition score thing is the same thing. same thing. same thing. Basically practically the same thing. Basically practically the same thing. Basically practically the same thing. In other words, In other words, In other words, this multi-level score matching thing is this multi-level score matching thing is this multi-level score matching thing is the same as all the elbow and stuff that the same as all the elbow and stuff that the same as all the elbow and stuff that we derived from last class. we derived from last class. we derived from last class. Isn't that cool? I thought that was Isn't that cool? I thought that was Isn't that cool? I thought that was pretty cool. pretty cool. pretty cool. Anyhow, so basically they're the same Anyhow, so basically they're the same Anyhow, so basically they're the same thing. If you need to remember anything thing. If you need to remember anything thing. If you need to remember anything from this class, this is one you need to from this class, this is one you need to from this class, this is one you need to remember. Scorebased model, diffusion remember. Scorebased model, diffusion remember. Scorebased model, diffusion model, pretty much the same thing. model, pretty much the same thing. model, pretty much the same thing. We just have two ways to derive them, We just have two ways to derive them, We just have two ways to derive them, but they're the same. Uh, but they're the same. Uh, but they're the same. Uh, okay, let's see if we can get through okay, let's see if we can get through okay, let's see if we can get through this. Uh, so what's going to happen if this. Uh, so what's going to happen if this. Uh, so what's going to happen if we have infinite numbers of noise level?

  58. L1 @ 2989s

    we have infinite numbers of noise level? we have infinite numbers of noise level? What do you what do we think is going to What do you what do we think is going to What do you what do we think is going to happen? mode collapse. mode collapse. >> It's actually not going to do more >> It's actually not going to do more >> It's actually not going to do more collapse. Good choice. Good. Good. Good collapse. Good choice. Good. Good. Good collapse. Good choice. Good. Good. Good try. But it's not going to try. But it's not going to try. But it's not going to >> Yeah. Yeah. Yeah. What you say? >> Yeah. Yeah. Yeah. What you say? >> Yeah. Yeah. Yeah. What you say? >> Then just have like a continuous. >> Yes. You're just gonna have a continuous >> Yes. You're just gonna have a continuous thing, right? So it becomes what we call thing, right? So it becomes what we call thing, right? So it becomes what we call a continuous time stoastic process which a continuous time stoastic process which a continuous time stoastic process which can pretty much be defined by a stoastic can pretty much be defined by a stoastic can pretty much be defined by a stoastic differential equation or what we just differential equation or what we just differential equation or what we just going to call now SDEES. going to call now SDEES. going to call now SDEES. All right. All right. And uh so when we have infinite amount And uh so when we have infinite amount And uh so when we have infinite amount uh infinite numbers of noise levels uh uh infinite numbers of noise levels uh uh infinite numbers of noise levels uh what it's going to do is that you can what it's going to do is that you can what it's going to do is that you can actually write the forward process into actually write the forward process into actually write the forward process into this um so this is like a first order this um so this is like a first order this um so this is like a first order tailor estimation of of of the real tailor estimation of of of the real

  59. L1 @ 3047s

    tailor estimation of of of the real thing and you can actually write both of thing and you can actually write both of thing and you can actually write both of them into this this format and uh if you them into this this format and uh if you them into this this format and uh if you organize it a little bit both of them organize it a little bit both of them organize it a little bit both of them actually also share the same formulation actually also share the same formulation actually also share the same formulation now. So now it becomes basically now. So now it becomes basically now. So now it becomes basically to move forward for delta time delta t to move forward for delta time delta t to move forward for delta time delta t steps you are actually just adding steps you are actually just adding steps you are actually just adding some function times delta t which is the some function times delta t which is the some function times delta t which is the time interval that you're going to move time interval that you're going to move time interval that you're going to move forward plus forward plus forward plus some scaled uh some some scaled gausian some scaled uh some some scaled gausian some scaled uh some some scaled gausian noise and this is scaling is also noise and this is scaling is also noise and this is scaling is also depending on the delta t that you're depending on the delta t that you're depending on the delta t that you're moving and similarly for ddpm this is moving and similarly for ddpm this is moving and similarly for ddpm this is the same only the only difference is that the only the only difference is that the choice of the function the two functions choice of the function the two functions choice of the function the two functions are going to be different for both. Um are going to be different for both. Um are going to be different for both. Um so essentially what you can do and so essentially what you can do and so essentially what you can do and because you see how like basically if because you see how like basically if

  60. L1 @ 3102s

    because you see how like basically if you do xt plus delta t minus xt is going you do xt plus delta t minus xt is going you do xt plus delta t minus xt is going to be this thing right. So it's going to to be this thing right. So it's going to to be this thing right. So it's going to be uh f * delta t plus g * square root be uh f * delta t plus g * square root be uh f * delta t plus g * square root of delta t plus uh times some gausian of delta t plus uh times some gausian of delta t plus uh times some gausian noise. So basically if you take delta t noise. So basically if you take delta t noise. So basically if you take delta t to infinite decimally small then you are to infinite decimally small then you are to infinite decimally small then you are going to be able to get this sd right. going to be able to get this sd right. going to be able to get this sd right. Why this is a sd because uh this part is Why this is a sd because uh this part is Why this is a sd because uh this part is deterministic right? So this part is deterministic right? So this part is deterministic right? So this part is what we call the drift. So like what we call the drift. So like what we call the drift. So like basically this is just like how you're basically this is just like how you're basically this is just like how you're gonna shift and then this is where you gonna shift and then this is where you gonna shift and then this is where you add a gausian noise. This W thing is add a gausian noise. This W thing is add a gausian noise. This W thing is what they call the wiener process. Uh what they call the wiener process. Uh what they call the wiener process. Uh but it's also just brownian motion just but it's also just brownian motion just but it's also just brownian motion just literally gausian noise basically. So literally gausian noise basically. So literally gausian noise basically. So you just add some stoasticity. So this

  61. L1 @ 3148s

    you just add some stoasticity. So this you just add some stoasticity. So this is why it's called stochastic is why it's called stochastic is why it's called stochastic differential equation. differential equation. differential equation. Okay. Okay, any question? So this is a Okay. Okay, any question? So this is a Okay. Okay, any question? So this is a forward process. All right, now we have a forward All right, now we have a forward process. How do we learn the backward process. How do we learn the backward process. How do we learn the backward process? >> It's literally just a reverse SDE. >> It's literally just a reverse SDE. Actually, there's a dude called Anderson Actually, there's a dude called Anderson Actually, there's a dude called Anderson and this dude in 1982 just give you the and this dude in 1982 just give you the and this dude in 1982 just give you the formula. Basically it just be like if formula. Basically it just be like if formula. Basically it just be like if you have a for SDE you can literally get you have a for SDE you can literally get you have a for SDE you can literally get a backward SD uh like back if you are a backward SD uh like back if you are a backward SD uh like back if you are trying to estimate some distribution and trying to estimate some distribution and trying to estimate some distribution and and and this backore SD or this reverse and and this backore SD or this reverse and and this backore SD or this reverse SDE is going to have a score function in SDE is going to have a score function in SDE is going to have a score function in it. Oh it. Oh it. Oh dang. Uh so now now if you if you look dang. Uh so now now if you if you look dang. Uh so now now if you if you look at it right the f and the g are both at it right the f and the g are both at it right the f and the g are both defined by our foret defined by our foret defined by our foret and we choose the for process right so and we choose the for process right so

  62. L1 @ 3208s

    and we choose the for process right so we know f and g uh and then it's we know f and g uh and then it's we know f and g uh and then it's literally just gausian right so who literally just gausian right so who literally just gausian right so who cares like we you can sample gausian any cares like we you can sample gausian any cares like we you can sample gausian any any from any time right so the only any from any time right so the only any from any time right so the only thing that needs to be learned is thing that needs to be learned is thing that needs to be learned is literally the score function right so literally the score function right so literally the score function right so how you train a score SDE is literally how you train a score SDE is literally how you train a score SDE is literally just score matching in continuous time just score matching in continuous time just score matching in continuous time and that's it. So we just already and that's it. So we just already and that's it. So we just already learned how to you know train this re learned how to you know train this re learned how to you know train this re reverse SDE model already. reverse SDE model already. reverse SDE model already. Okay. Okay. And for sampling uh basically for any And for sampling uh basically for any And for sampling uh basically for any differential equation you can sort try differential equation you can sort try differential equation you can sort try to use a solver to solve it. This is to use a solver to solve it. This is to use a solver to solve it. This is like some fancy name um but it's like some fancy name um but it's like some fancy name um but it's literally just like how you discretise literally just like how you discretise literally just like how you discretise your steps and then try to make your steps and then try to make your steps and then try to make estimations and for example if you try estimations and for example if you try estimations and for example if you try to use a oiler solver which is like

  63. L1 @ 3261s

    to use a oiler solver which is like to use a oiler solver which is like probably simple simplest solver uh that probably simple simplest solver uh that probably simple simplest solver uh that then you can literally just do your then you can literally just do your then you can literally just do your first sample from your source first sample from your source first sample from your source distribution and remember now because we distribution and remember now because we distribution and remember now because we have continuous time so the time goes have continuous time so the time goes have continuous time so the time goes from zero to one continuously so there's from zero to one continuously so there's from zero to one continuously so there's no big t anymore okay so now we sample no big t anymore okay so now we sample no big t anymore okay so now we sample x1 one from the source distribution. x1 one from the source distribution. x1 one from the source distribution. This is basically X big T. And then we This is basically X big T. And then we This is basically X big T. And then we estimate the displacement of the X that estimate the displacement of the X that estimate the displacement of the X that we're going to make for this delta T we're going to make for this delta T we're going to make for this delta T time and we just use the reverse SDE time and we just use the reverse SDE time and we just use the reverse SDE formula, right? And then the only thing formula, right? And then the only thing formula, right? And then the only thing that we do not know from the reverse SD that we do not know from the reverse SD that we do not know from the reverse SD formula is the score. But we already formula is the score. But we already formula is the score. But we already learned a model from score matching. So learned a model from score matching. So learned a model from score matching. So this is nice. And we literally just this is nice. And we literally just this is nice. And we literally just apply the change in x to x and then we

  64. L1 @ 3309s

    apply the change in x to x and then we apply the change in x to x and then we apply delta t to t apply delta t to t apply delta t to t and that's it. and that's it. and that's it. And then we iterate until t becomes And then we iterate until t becomes And then we iterate until t becomes zero. zero. zero. Oh, I guess in this case I guess it Oh, I guess in this case I guess it Oh, I guess in this case I guess it should it should start from zero and should it should start from zero and should it should start from zero and then t becomes one. But you know same then t becomes one. But you know same then t becomes one. But you know same similar or or or I guess the t should be minus. or or or I guess the t should be minus. Yeah. Any question? Yeah. question? Yeah. >> In DDPM in the forward process, we can >> In DDPM in the forward process, we can >> In DDPM in the forward process, we can get the image at any time step get the image at any time step get the image at any time step instantly. Can we do this with the SD as instantly. Can we do this with the SD as instantly. Can we do this with the SD as well? well? well? >> What do you think? >> What do you think? >> What do you think? >> Not sure. >> Not sure. >> Not sure. >> Why not? It's still a bunch of gausian, >> Why not? It's still a bunch of gausian, >> Why not? It's still a bunch of gausian, right? Actually, great question. Now right? Actually, great question. Now right? Actually, great question. Now that you mentioned the DDPM, uh basically uh basically uh the DDPM and NCSN are this uh the DDPM and NCSN are this uh the DDPM and NCSN are this three time version of the of the model, three time version of the of the model, three time version of the of the model, right? And we know that they can be right? And we know that they can be right? And we know that they can be written as this like this function this

  65. L1 @ 3373s

    written as this like this function this written as this like this function this this uh so there are four process can be this uh so there are four process can be this uh so there are four process can be written as this function of f and g written as this function of f and g written as this function of f and g right. So actually if you just make it right. So actually if you just make it right. So actually if you just make it continuous time and then it's just going continuous time and then it's just going continuous time and then it's just going to become the continuous version of DDPM to become the continuous version of DDPM to become the continuous version of DDPM and uh ncsn um and this is just going to and uh ncsn um and this is just going to and uh ncsn um and this is just going to become become become literally literally literally uh yeah it's literally just the same uh yeah it's literally just the same uh yeah it's literally just the same thing basically but with continuous time thing basically but with continuous time thing basically but with continuous time step. Okay, any more questions? Okay, any more questions? >> Yeah. VP. VP. >> Great question. VP means variance >> Great question. VP means variance >> Great question. VP means variance preserving. V means variance exploding. preserving. V means variance exploding. preserving. V means variance exploding. Why? What do you think? >> All right. Yeah. Yeah. >> All right. Yeah. Yeah. >> Because >> Because >> Because like adds variance to your uh to your like adds variance to your uh to your like adds variance to your uh to your process. So the more fine steps you process. So the more fine steps you process. So the more fine steps you take, the more variance. take, the more variance. take, the more variance. >> Yeah, basic. Yeah, pretty much. So >> Yeah, basic. Yeah, pretty much. So >> Yeah, basic. Yeah, pretty much. So basically as you can see here right so basically as you can see here right so basically as you can see here right so the G as variance and in the VE the G as variance and in the VE

  66. L1 @ 3440s

    the G as variance and in the VE formulation we do not have any formulation we do not have any formulation we do not have any adjustment of the X like whatever X you adjustment of the X like whatever X you adjustment of the X like whatever X you have you're going to add more variance have you're going to add more variance have you're going to add more variance to it so that's why it's various to it so that's why it's various to it so that's why it's various exploding and then variance preserving exploding and then variance preserving exploding and then variance preserving is because we're actually going to is because we're actually going to is because we're actually going to adjust the variance the total variance adjust the variance the total variance adjust the variance the total variance of the whole thing so that the variance of the whole thing so that the variance of the whole thing so that the variance is sort of just like quote unquote uni is sort of just like quote unquote uni is sort of just like quote unquote uni variant if you will so you're just like variant if you will so you're just like variant if you will so you're just like variance preserving so you all you're variance preserving so you all you're variance preserving so you all you're going to preserve the same variance as going to preserve the same variance as going to preserve the same variance as you you you Okay, Okay, Okay, perfect timing actually. Okay, so now we perfect timing actually. Okay, so now we perfect timing actually. Okay, so now we can train this score SD model and it can train this score SD model and it can train this score SD model and it works super well. This is like a 1K works super well. This is like a 1K works super well. This is like a 1K image. Absolutely gorgeous. Um, yeah. So image. Absolutely gorgeous. Um, yeah. So image. Absolutely gorgeous. Um, yeah. So now uh we have seen a bunch now uh we have seen a bunch now uh we have seen a bunch of gener models and the last class we of gener models and the last class we of gener models and the last class we did diffusion. This class we did

  67. L1 @ 3489s

    did diffusion. This class we did did diffusion. This class we did scorebased model and we also learned scorebased model and we also learned scorebased model and we also learned that they're basically just the same that they're basically just the same that they're basically just the same thing, right? Okay, they they can be thing, right? Okay, they they can be thing, right? Okay, they they can be unified um under the same score SDE unified um under the same score SDE unified um under the same score SDE formulation. Um the question now that formulation. Um the question now that formulation. Um the question now that Joshman asked was that Joshman asked was that Joshman asked was that now but we derive it everything from the now but we derive it everything from the now but we derive it everything from the sampling perspective. sampling perspective. sampling perspective. How do we do density estimation? Right? How do we do density estimation? Right? How do we do density estimation? Right? How do we actually calculate the How do we actually calculate the How do we actually calculate the distribution or the likelihood? Um we distribution or the likelihood? Um we distribution or the likelihood? Um we haven't talked about that. We're going haven't talked about that. We're going haven't talked about that. We're going to talk about that next class. And also, to talk about that next class. And also, to talk about that next class. And also, is there even simpler way to do the same is there even simpler way to do the same is there even simpler way to do the same thing? I feel like everything just too thing? I feel like everything just too thing? I feel like everything just too complicated. Is what is the simplest way complicated. Is what is the simplest way complicated. Is what is the simplest way to do the same thing that we want to do? to do the same thing that we want to do? to do the same thing that we want to do? This is what we're going to talk about This is what we're going to talk about This is what we're going to talk about next class, which is flow matching. next class, which is flow matching. next class, which is flow matching. Okay, class is over. Thank you guys.

diffusion-principles-vizuara

  1. L3 @ 5s

    Hello everyone and welcome to the next Hello everyone and welcome to the next lecture of the course principles of lecture of the course principles of lecture of the course principles of diffusion models. diffusion models. diffusion models. In this lecture I have adopted a In this lecture I have adopted a In this lecture I have adopted a different style of teaching different style of teaching different style of teaching just because the content is so just because the content is so just because the content is so interesting interesting interesting and there is a lot of connection with and there is a lot of connection with and there is a lot of connection with physics. physics. physics. So I'm going to show you or explain the So I'm going to show you or explain the So I'm going to show you or explain the concepts to you via the means of slides. concepts to you via the means of slides. concepts to you via the means of slides. So what are we going to learn today So what are we going to learn today So what are we going to learn today in the principles of diffusion models in the principles of diffusion models in the principles of diffusion models series? We have so far covered variation series? We have so far covered variation series? We have so far covered variation autoenccoders then DDPMS autoenccoders then DDPMS autoenccoders then DDPMS denoising diffusion probabilistic denoising diffusion probabilistic denoising diffusion probabilistic models. Then we have covered the energy models. Then we have covered the energy models. Then we have covered the energy based approach which includes score based approach which includes score based approach which includes score function and noise condition score function and noise condition score function and noise condition score network. Today we are going to look at a network. Today we are going to look at a network. Today we are going to look at a framework which unifies everything framework which unifies everything framework which unifies everything together. together. together. If you are watching this lecture for the If you are watching this lecture for the If you are watching this lecture for the first time in the series, you have not first time in the series, you have not first time in the series, you have not looked at other lectures. This is

  2. L3 @ 69s

    looked at other lectures. This is looked at other lectures. This is supposed to be a standalone lecture to supposed to be a standalone lecture to supposed to be a standalone lecture to help you get into the diffusion concept help you get into the diffusion concept help you get into the diffusion concept which is at the heart of so many things which is at the heart of so many things which is at the heart of so many things at the moment. There are LLMs being made at the moment. There are LLMs being made at the moment. There are LLMs being made in a diffusive manner as opposed to auto in a diffusive manner as opposed to auto in a diffusive manner as opposed to auto reggressive manner. In robotics, reggressive manner. In robotics, reggressive manner. In robotics, diffusion models are used in audio diffusion models are used in audio diffusion models are used in audio models. The applications are enormous. models. The applications are enormous. models. The applications are enormous. Today we are going to look at a Today we are going to look at a Today we are going to look at a framework which unifies the framework which unifies the framework which unifies the two approaches. One is the DDPM approach two approaches. One is the DDPM approach two approaches. One is the DDPM approach and second is the score-based approach. and second is the score-based approach. and second is the score-based approach. And interestingly, this unification And interestingly, this unification And interestingly, this unification happened later after both these previous happened later after both these previous happened later after both these previous papers came out. papers came out. papers came out. And when you view everything from this And when you view everything from this And when you view everything from this lens, it makes a lot of sense. lens, it makes a lot of sense. lens, it makes a lot of sense. And it turns out that both DDPM and And it turns out that both DDPM and And it turns out that both DDPM and score-based approach are special cases score-based approach are special cases score-based approach are special cases of this overall unified approach that we of this overall unified approach that we of this overall unified approach that we are going to discuss today.

  3. L3 @ 126s

    are going to discuss today. are going to discuss today. The good thing is that this unified The good thing is that this unified The good thing is that this unified approach has a lot of connection with approach has a lot of connection with approach has a lot of connection with physics. So there are there is going to physics. So there are there is going to physics. So there are there is going to be a lot of physical intuition that we be a lot of physical intuition that we be a lot of physical intuition that we will build in the process. Okay. So firstly by diffusion we Okay. So firstly by diffusion we understand that something is spreading understand that something is spreading understand that something is spreading in space right as you can see in these in space right as you can see in these in space right as you can see in these images there is there is something there images there is there is something there images there is there is something there is a spread of material in space. Now in today's lecture we are going to Now in today's lecture we are going to understand how this concept of diffusion understand how this concept of diffusion understand how this concept of diffusion can be connected can be connected can be connected to probabilistic models and essentially to probabilistic models and essentially to probabilistic models and essentially what is the problem that we are even what is the problem that we are even what is the problem that we are even trying to solve trying to solve trying to solve and how is it that physics is helping AI and how is it that physics is helping AI and how is it that physics is helping AI what is the connection between both of what is the connection between both of what is the connection between both of these fields these fields these fields if you have a background in physics or if you have a background in physics or if you have a background in physics or mechanics particles fluid meanan mechanics particles fluid meanan mechanics particles fluid meanan mechanics you'll find this lecture very mechanics you'll find this lecture very mechanics you'll find this lecture very interesting. Okay. So what do we know about

  4. L3 @ 193s

    Okay. So what do we know about diffusion? In this image diffusion? In this image diffusion? In this image I have shown a dye diffusing or I have shown a dye diffusing or I have shown a dye diffusing or spreading in a glass of water. spreading in a glass of water. spreading in a glass of water. I will use the word spreading when I I will use the word spreading when I I will use the word spreading when I intend to say diffusion because to intend to say diffusion because to intend to say diffusion because to explain diffusion I cannot use the same explain diffusion I cannot use the same explain diffusion I cannot use the same word diffusion. word diffusion. word diffusion. So initially the shape of the dye was So initially the shape of the dye was So initially the shape of the dye was maybe spherical. maybe spherical. maybe spherical. When you put it on the surface of the When you put it on the surface of the When you put it on the surface of the liquid liquid liquid then the drop slowly goes down and you then the drop slowly goes down and you then the drop slowly goes down and you can see the liquid gets filled. can see the liquid gets filled. can see the liquid gets filled. Initially the liquid is white in color Initially the liquid is white in color Initially the liquid is white in color and slowly the color changes to red. So the structure of the dye which was So the structure of the dye which was spherical to begin with is completely spherical to begin with is completely spherical to begin with is completely destroyed in the process. destroyed in the process. destroyed in the process. It becomes uniform. It becomes uniform. It becomes uniform. Now this is the first property of Now this is the first property of Now this is the first property of diffusion that we understand. The question that we are trying to ask The question that we are trying to ask in this entire lecture, you will see in this entire lecture, you will see in this entire lecture, you will see this is a common thread throughout this is a common thread throughout this is a common thread throughout is that can we reverse this process?

  5. L3 @ 277s

    Can we or in other words can we recover Can we or in other words can we recover the original structure? the original structure? the original structure? So imagine you're sitting and you see So imagine you're sitting and you see So imagine you're sitting and you see this glass filled with this red liquid this glass filled with this red liquid this glass filled with this red liquid and someone tells you what was the and someone tells you what was the and someone tells you what was the original structure of this dye when it original structure of this dye when it original structure of this dye when it was poured in the liquid. was poured in the liquid. was poured in the liquid. You are thinking like okay this is very You are thinking like okay this is very You are thinking like okay this is very difficult for me because difficult for me because difficult for me because I cannot reverse time. You can see in I cannot reverse time. You can see in I cannot reverse time. You can see in this figure that if I was able to this figure that if I was able to this figure that if I was able to reverse time, I could exactly predict reverse time, I could exactly predict reverse time, I could exactly predict the initial shape of the D when it made the initial shape of the D when it made the initial shape of the D when it made the contact with water. the contact with water. the contact with water. But the problem is that we live in a But the problem is that we live in a But the problem is that we live in a world where we cannot reverse time. world where we cannot reverse time. world where we cannot reverse time. If I had a time machine, I could use the If I had a time machine, I could use the If I had a time machine, I could use the time machine. I can go back in time time machine. I can go back in time time machine. I can go back in time and I can easily look at the original and I can easily look at the original and I can easily look at the original shape of the die. shape of the die.

  6. L3 @ 329s

    shape of the die. But that is not the case. [snorts] But that is not the case. [snorts] But that is not the case. [snorts] So reversing time is something which is So reversing time is something which is So reversing time is something which is even out of the question. So how do you even out of the question. So how do you even out of the question. So how do you recover the structure of the die? This is where it gets more interesting. This is where it gets more interesting. We cannot just reverse time and all of We cannot just reverse time and all of We cannot just reverse time and all of us are aware about it. us are aware about it. us are aware about it. But let us take a lens and zoom in on But let us take a lens and zoom in on But let us take a lens and zoom in on the die and see exactly what we see. So the die and see exactly what we see. So the die and see exactly what we see. So I'm going to take a lens a microscope I'm going to take a lens a microscope I'm going to take a lens a microscope and I'm going to zoom in on the red dye. and I'm going to zoom in on the red dye. and I'm going to zoom in on the red dye. I'm going to zoom in even more till I go I'm going to zoom in even more till I go I'm going to zoom in even more till I go to the particle or the atomic level. to the particle or the atomic level. to the particle or the atomic level. When I zoom in on the die, I see When I zoom in on the die, I see When I zoom in on the die, I see something like this. something like this. something like this. I see tiny particles moving around and I see tiny particles moving around and I see tiny particles moving around and wobbling around here and there. wobbling around here and there. wobbling around here and there. It's almost like the motion is It's almost like the motion is It's almost like the motion is unpredictable. It's it's very random. unpredictable. It's it's very random.

  7. L3 @ 383s

    unpredictable. It's it's very random. It's not moving in a specific direction, It's not moving in a specific direction, It's not moving in a specific direction, but but but they're moving up and down. It's like a they're moving up and down. It's like a they're moving up and down. It's like a zigzag motion. zigzag motion. zigzag motion. All of us have have seen how the price All of us have have seen how the price All of us have have seen how the price of a stock varies. It's it's up and of a stock varies. It's it's up and of a stock varies. It's it's up and down. It's very zigzag, right? This this down. It's very zigzag, right? This this down. It's very zigzag, right? This this is almost very similar to that. A more technical word for this motion is A more technical word for this motion is also called as Brownian motion. So also called as Brownian motion. So also called as Brownian motion. So particles are exhibiting brownian motion particles are exhibiting brownian motion particles are exhibiting brownian motion and uh this dye is made up of a lot of and uh this dye is made up of a lot of and uh this dye is made up of a lot of particles like this which are moving in particles like this which are moving in particles like this which are moving in a fashion like this. a fashion like this. a fashion like this. It's it's a very peculiar movement. They It's it's a very peculiar movement. They It's it's a very peculiar movement. They are not flowing in a specific direction. are not flowing in a specific direction. are not flowing in a specific direction. They are just moving around here and They are just moving around here and They are just moving around here and there. Now the question that Now the question that I want to ask all of you is that just by I want to ask all of you is that just by I want to ask all of you is that just by looking at this motion looking at this motion looking at this motion can you tell whether it's the forward can you tell whether it's the forward can you tell whether it's the forward motion or the backward motion?

  8. L3 @ 450s

    So when the dye was diffusing and when So when the dye was diffusing and when we reversed the whole process, we reversed the whole process, we reversed the whole process, can you tell this movement of particles? can you tell this movement of particles? can you tell this movement of particles? Does it correspond to the forward motion Does it correspond to the forward motion Does it correspond to the forward motion or the reverse motion? That is difficult to predict. In fact, That is difficult to predict. In fact, just by looking at this picture, I have just by looking at this picture, I have just by looking at this picture, I have absolutely no idea whether this absolutely no idea whether this absolutely no idea whether this corresponds to the forward motion or the corresponds to the forward motion or the corresponds to the forward motion or the reverse motion. reverse motion. reverse motion. And this is where we come to a very And this is where we come to a very And this is where we come to a very interesting property. It it turns out interesting property. It it turns out interesting property. It it turns out that even though macroscopic that even though macroscopic that even though macroscopic motions are not time reversible like we motions are not time reversible like we motions are not time reversible like we just saw with the spread of the die, we just saw with the spread of the die, we just saw with the spread of the die, we cannot simply reverse time. But at the cannot simply reverse time. But at the cannot simply reverse time. But at the microscopic level, if you look at these microscopic level, if you look at these microscopic level, if you look at these individual tiny particles, individual tiny particles, individual tiny particles, if you look at the way they are moving, if you look at the way they are moving, if you look at the way they are moving, it turns out that these motions are it turns out that these motions are it turns out that these motions are exactly reversible. exactly reversible. exactly reversible. Which means that if someone if 10 Which means that if someone if 10 Which means that if someone if 10 seconds have passed and someone tells me

  9. L3 @ 509s

    seconds have passed and someone tells me seconds have passed and someone tells me that Rajett this is the particle which that Rajett this is the particle which that Rajett this is the particle which is located here right now show me where is located here right now show me where is located here right now show me where it was located 10 seconds back. I can it was located 10 seconds back. I can it was located 10 seconds back. I can exactly map the reverse trajectory of exactly map the reverse trajectory of exactly map the reverse trajectory of that particle and I can pinpoint the that particle and I can pinpoint the that particle and I can pinpoint the original location and this is because of the property that and this is because of the property that microscopic motions are time reversible microscopic motions are time reversible microscopic motions are time reversible and this is the only theme which is used and this is the only theme which is used and this is the only theme which is used in the entire diffusion literature in in the entire diffusion literature in in the entire diffusion literature in all the different approaches. This is all the different approaches. This is all the different approaches. This is the common arc which you will find in the common arc which you will find in the common arc which you will find in many of these papers. many of these papers. many of these papers. something will be reversed in the paper something will be reversed in the paper something will be reversed in the paper and you'll be wondering how how did this and you'll be wondering how how did this and you'll be wondering how how did this magically work magically work magically work and at the heart of it it turns out that and at the heart of it it turns out that and at the heart of it it turns out that it is because of this property of it is because of this property of it is because of this property of microscopic motions being time microscopic motions being time microscopic motions being time reversible. Now this is where the connection between Now this is where the connection between physics and AI comes into the picture. physics and AI comes into the picture.

  10. L3 @ 569s

    physics and AI comes into the picture. This is a well-known physical property This is a well-known physical property This is a well-known physical property and it makes sense in when you when we and it makes sense in when you when we and it makes sense in when you when we look at diffusion of liquids and we look look at diffusion of liquids and we look look at diffusion of liquids and we look at the microscopic and microscopic lens. at the microscopic and microscopic lens. at the microscopic and microscopic lens. But remember here we are dealing with But remember here we are dealing with But remember here we are dealing with data. We do not have liquids which we data. We do not have liquids which we data. We do not have liquids which we can so readily imagine the movement of can so readily imagine the movement of can so readily imagine the movement of these liquids. these liquids. these liquids. We have data, we have pixels, we have We have data, we have pixels, we have We have data, we have pixels, we have numbers. So how do we even connect it to numbers. So how do we even connect it to numbers. So how do we even connect it to the process of diffusion? All of it the process of diffusion? All of it the process of diffusion? All of it looks very looks very looks very random and vague. random and vague. random and vague. We are going to form this connection in We are going to form this connection in We are going to form this connection in in in this lecture and u these in in this lecture and u these in in this lecture and u these individual particles right which you see individual particles right which you see individual particles right which you see as we move proceed in the lecture you as we move proceed in the lecture you as we move proceed in the lecture you will see that we will view our data as will see that we will view our data as will see that we will view our data as one of these particles. So that is the broader arc which we are So that is the broader arc which we are going to take in today's lecture.

  11. L3 @ 633s

    Okay. Now let us apply this concept Okay. Now let us apply this concept to data. to data. to data. Why are we applying this concept to Why are we applying this concept to Why are we applying this concept to data? The diffusion models were first data? The diffusion models were first data? The diffusion models were first proven to be incredibly successful at proven to be incredibly successful at proven to be incredibly successful at generating data. generating data. generating data. Applications like stable diffusion, Applications like stable diffusion, Applications like stable diffusion, midjourney, midjourney, midjourney, all of them relied on the process of all of them relied on the process of all of them relied on the process of diffusion to generate data. diffusion to generate data. diffusion to generate data. Okay. So now what is the problem that we Okay. So now what is the problem that we Okay. So now what is the problem that we are dealing with? Someone comes up to us are dealing with? Someone comes up to us are dealing with? Someone comes up to us and shows us a bunch of cat images. Here and shows us a bunch of cat images. Here and shows us a bunch of cat images. Here I have shown eight images for reference I have shown eight images for reference I have shown eight images for reference but it can be even 80 or 100 images. but it can be even 80 or 100 images. but it can be even 80 or 100 images. Someone shows us all of these images of Someone shows us all of these images of Someone shows us all of these images of cats which are very diverse. You can see cats which are very diverse. You can see cats which are very diverse. You can see that some of them are whitish, some of that some of them are whitish, some of that some of them are whitish, some of them have a darker color. them have a darker color. them have a darker color. And the eyes are pretty different, some And the eyes are pretty different, some And the eyes are pretty different, some of them have pretty smaller eyes, some of them have pretty smaller eyes, some of them have pretty smaller eyes, some of them have wide eyes. of them have wide eyes.

  12. L3 @ 690s

    of them have wide eyes. And the question they ask us is, And the question they ask us is, And the question they ask us is, can you find a distribution can you find a distribution can you find a distribution using which we can sample images which using which we can sample images which using which we can sample images which look like cats? look like cats? look like cats? [snorts] Now, what is this concept of [snorts] Now, what is this concept of [snorts] Now, what is this concept of distribution? All of a sudden distribution? All of a sudden distribution? All of a sudden distribution simply means that we want distribution simply means that we want distribution simply means that we want to find a common something common between all of common something common between all of these cats and it should be so these cats and it should be so these cats and it should be so deeprooted that deeprooted that deeprooted that once we find those common factors it once we find those common factors it once we find those common factors it should allow us to create images which should allow us to create images which should allow us to create images which look like cats. look like cats. look like cats. Let's take an example of a dice being Let's take an example of a dice being Let's take an example of a dice being rolled. A dice has six faces 1 2 3 4 5 rolled. A dice has six faces 1 2 3 4 5 rolled. A dice has six faces 1 2 3 4 5 6. And what will the distribution say? 6. And what will the distribution say? 6. And what will the distribution say? Whenever I sample from this Whenever I sample from this Whenever I sample from this distribution, I will either get 1 2 3 4 distribution, I will either get 1 2 3 4 distribution, I will either get 1 2 3 4 5 or six. I will never get seven or 5 or six. I will never get seven or 5 or six. I will never get seven or eight. eight. eight. So the meaning of distribution is that So the meaning of distribution is that So the meaning of distribution is that when you sample from it, you should get

  13. L3 @ 755s

    when you sample from it, you should get when you sample from it, you should get something which lies within that something which lies within that something which lies within that distribution. distribution. distribution. For example, if you have a normal For example, if you have a normal For example, if you have a normal distribution which is centered around distribution which is centered around distribution which is centered around some value and it tapers off on left and some value and it tapers off on left and some value and it tapers off on left and right, it means that the probability of right, it means that the probability of right, it means that the probability of the mean is the maximum and it dies down the mean is the maximum and it dies down the mean is the maximum and it dies down as you go away from the mean. as you go away from the mean. as you go away from the mean. So So So our whole objective is always to find a our whole objective is always to find a our whole objective is always to find a distribution using which we can sample distribution using which we can sample distribution using which we can sample more images. more images. more images. If we don't have a distribution then we If we don't have a distribution then we If we don't have a distribution then we cannot sample variety. We can simply cannot sample variety. We can simply cannot sample variety. We can simply learn based on these cats. We can learn based on these cats. We can learn based on these cats. We can understand there are eight cats in the understand there are eight cats in the understand there are eight cats in the figure and I learn about eight cats. But figure and I learn about eight cats. But figure and I learn about eight cats. But if we have a distribution, it suddenly if we have a distribution, it suddenly if we have a distribution, it suddenly gives us the power to sample more images gives us the power to sample more images gives us the power to sample more images which look like cats. which look like cats. which look like cats. And to find such distribution we need an And to find such distribution we need an

  14. L3 @ 808s

    And to find such distribution we need an algorithm which is clever enough which algorithm which is clever enough which algorithm which is clever enough which can understand that okay uh cats have a can understand that okay uh cats have a can understand that okay uh cats have a specific shape they have a specific set specific shape they have a specific set specific shape they have a specific set of features of features of features uh they have a specific way that they uh they have a specific way that they uh they have a specific way that they look and you can see right now the way look and you can see right now the way look and you can see right now the way I'm talking about these features is very I'm talking about these features is very I'm talking about these features is very vague because it is very difficult for vague because it is very difficult for vague because it is very difficult for humans to provide a summary of all of humans to provide a summary of all of humans to provide a summary of all of these features or exactly pinpoint point these features or exactly pinpoint point these features or exactly pinpoint point the causes of variation in between the the causes of variation in between the the causes of variation in between the cats and that is exactly why we are cats and that is exactly why we are cats and that is exactly why we are learning this process [snorts] which can learning this process [snorts] which can learning this process [snorts] which can help us to predict this probability help us to predict this probability help us to predict this probability distribution. Okay. Uh I hope that was clear why we Okay. Uh I hope that was clear why we are trying to find a distribution which are trying to find a distribution which are trying to find a distribution which can sample images which look like cats can sample images which look like cats can sample images which look like cats and we will now take inspiration from and we will now take inspiration from and we will now take inspiration from the diffusion process. Now this is where the diffusion process. Now this is where

  15. L3 @ 864s

    the diffusion process. Now this is where it gets very interesting. In the next it gets very interesting. In the next it gets very interesting. In the next set of slides I'm going to replace the set of slides I'm going to replace the set of slides I'm going to replace the die with data. Okay. So we want to go from images to Okay. So we want to go from images to distribution which looks like in in in distribution which looks like in in in distribution which looks like in in in the figure I have shown a contour which the figure I have shown a contour which the figure I have shown a contour which is a just a schematic distribution which is a just a schematic distribution which is a just a schematic distribution which can be used to sample the images of can be used to sample the images of can be used to sample the images of cats. cats. cats. Now we don't have access to it. We only Now we don't have access to it. We only Now we don't have access to it. We only have images to images of the cats and have images to images of the cats and have images to images of the cats and based on these images we are expected to based on these images we are expected to based on these images we are expected to find this unknown magical distribution. find this unknown magical distribution. find this unknown magical distribution. [snorts] Okay. Now the question is Okay. Now the question is what if [snorts] we diffuse data instead what if [snorts] we diffuse data instead what if [snorts] we diffuse data instead of diffusing the die? of diffusing the die? of diffusing the die? What if we modify or transform the data such that modify or transform the data such that the structure of the data completely the structure of the data completely the structure of the data completely disappears which is the first disappears which is the first disappears which is the first observation that we looked at in the observation that we looked at in the observation that we looked at in the diffusion process [snorts] diffusion process [snorts] diffusion process [snorts] and then later we recover it. and then later we recover it.

  16. L3 @ 937s

    and then later we recover it. So we destroy something only to recover So we destroy something only to recover So we destroy something only to recover it after some point. it after some point. it after some point. Now if if someone would have come up Now if if someone would have come up Now if if someone would have come up with this idea maybe with this idea maybe with this idea maybe 10 years back 10 years back 10 years back it it sounds very cool because whenever it it sounds very cool because whenever it it sounds very cool because whenever there is a reversal of time involved there is a reversal of time involved there is a reversal of time involved there is a natural fascination towards there is a natural fascination towards there is a natural fascination towards it. it. it. I don't know maybe that's why people I don't know maybe that's why people I don't know maybe that's why people love to see movies with time machines in love to see movies with time machines in love to see movies with time machines in it and that was a part of the reason of it and that was a part of the reason of it and that was a part of the reason of my fascination towards this field as my fascination towards this field as my fascination towards this field as well. this this whole idea of diffusing well. this this whole idea of diffusing well. this this whole idea of diffusing data and later recovering it it sounds data and later recovering it it sounds data and later recovering it it sounds almost like uh almost like uh almost like uh too good to be true and and impossible too good to be true and and impossible too good to be true and and impossible right and how how will this even work right and how how will this even work right and how how will this even work [snorts] [snorts] [snorts] so we'll we'll do something like this uh so we'll we'll do something like this uh so we'll we'll do something like this uh when we are diffusing the dye we can when we are diffusing the dye we can when we are diffusing the dye we can visualize the dye being slowly filling

  17. L3 @ 996s

    visualize the dye being slowly filling visualize the dye being slowly filling the liquid and the structure the liquid and the structure the liquid and the structure disappearing disappearing disappearing When we are diffusing data, this is what When we are diffusing data, this is what When we are diffusing data, this is what we mean. We get to a point where it we mean. We get to a point where it we mean. We get to a point where it almost becomes pure noise. You might almost becomes pure noise. You might almost becomes pure noise. You might have seen this these displays on our TVs have seen this these displays on our TVs have seen this these displays on our TVs when we are not getting any signal. when we are not getting any signal. when we are not getting any signal. Right? There is a very uh Right? There is a very uh Right? There is a very uh harsh sound which comes on the screen harsh sound which comes on the screen harsh sound which comes on the screen and the picture looks like this. It's and the picture looks like this. It's and the picture looks like this. It's it's filled with these multicolored it's filled with these multicolored it's filled with these multicolored dots. dots. dots. And what you're seeing on the screen And what you're seeing on the screen And what you're seeing on the screen before you have no idea it has before you have no idea it has before you have no idea it has completely disappeared. completely disappeared. completely disappeared. This is exactly what we want to do in This is exactly what we want to do in This is exactly what we want to do in the forward process where we want to the forward process where we want to the forward process where we want to diffuse our data so that it becomes pure diffuse our data so that it becomes pure diffuse our data so that it becomes pure noise. The the the structure disappears. noise. The the the structure disappears. noise. The the the structure disappears. And then in the reverse process we will And then in the reverse process we will And then in the reverse process we will do the opposite. In the reverse process, do the opposite. In the reverse process,

  18. L3 @ 1053s

    do the opposite. In the reverse process, we will start with noise and then slowly we will start with noise and then slowly we will start with noise and then slowly we will move backwards and we will we will move backwards and we will we will move backwards and we will recover the original image. recover the original image. recover the original image. [snorts] And the first question that pops in the And the first question that pops in the mind is mind is mind is how do we even begin with this recovery how do we even begin with this recovery how do we even begin with this recovery process and where is the learning process and where is the learning process and where is the learning actually happening? actually happening? actually happening? Where am I learning the variations in Where am I learning the variations in Where am I learning the variations in the different features of the cats? the different features of the cats? the different features of the cats? Where am I learning that okay some cats Where am I learning that okay some cats Where am I learning that okay some cats have bigger eyes, some have smaller have bigger eyes, some have smaller have bigger eyes, some have smaller eyes, some have long whiskers, short eyes, some have long whiskers, short eyes, some have long whiskers, short whiskers, etc. whiskers, etc. whiskers, etc. So out of all of this, what is something So out of all of this, what is something So out of all of this, what is something that is in our control and what is that is in our control and what is that is in our control and what is something that we are planning to learn? something that we are planning to learn? something that we are planning to learn? At least that was the first question At least that was the first question At least that was the first question which came to my mind when I was looking which came to my mind when I was looking which came to my mind when I was looking at this process. at this process. at this process. [snorts] Okay, now comes the interesting part of Okay, now comes the interesting part of the lecture. the lecture. the lecture. We have already said that we are going

  19. L3 @ 1113s

    We have already said that we are going We have already said that we are going to apply the diffusion process to data. to apply the diffusion process to data. to apply the diffusion process to data. But to apply the diffusion, we need to But to apply the diffusion, we need to But to apply the diffusion, we need to first figure out how will I transform first figure out how will I transform first figure out how will I transform the data to noise. Visually, this looks the data to noise. Visually, this looks the data to noise. Visually, this looks very clear. But what is the mathematical very clear. But what is the mathematical very clear. But what is the mathematical process of doing that? process of doing that? process of doing that? And we are going to look at it from a And we are going to look at it from a And we are going to look at it from a microscopic lens because we have already microscopic lens because we have already microscopic lens because we have already built our intuition that microscopic built our intuition that microscopic built our intuition that microscopic processes are time reversible. So we are processes are time reversible. So we are processes are time reversible. So we are going to look at this forward diffusion going to look at this forward diffusion going to look at this forward diffusion process from a microscopic lens. Okay. So Okay. So we briefly looked at these motion of we briefly looked at these motion of we briefly looked at these motion of particles, right? particles, right? particles, right? >> [snorts] >> [snorts] >> [snorts] >> which are swiggling around >> which are swiggling around >> which are swiggling around uh and and they are moving in a very uh and and they are moving in a very uh and and they are moving in a very random fashion [snorts] random fashion [snorts] random fashion [snorts] and because I'm using the word random and because I'm using the word random and because I'm using the word random this is also called as a brownian motion this is also called as a brownian motion this is also called as a brownian motion and you see this motion in a lot of and you see this motion in a lot of

  20. L3 @ 1181s

    and you see this motion in a lot of physical processes you see these motions physical processes you see these motions physical processes you see these motions in polymers you see this even in the in polymers you see this even in the in polymers you see this even in the movement of stock prices [snorts] movement of stock prices [snorts] movement of stock prices [snorts] and here if you and here if you and here if you The particles are only moving around The particles are only moving around The particles are only moving around their center. They are not really their center. They are not really their center. They are not really drifting. drifting. drifting. So typically Brownian motion has two So typically Brownian motion has two So typically Brownian motion has two terms. The first term is the drift term terms. The first term is the drift term terms. The first term is the drift term and the second term is the noise term or and the second term is the noise term or and the second term is the noise term or the diffusion term. the diffusion term. the diffusion term. The drift term is what carries you with The drift term is what carries you with The drift term is what carries you with the liquid and the noise term is what the liquid and the noise term is what the liquid and the noise term is what causes these random perturbations. causes these random perturbations. causes these random perturbations. Now in this image which I have taken it Now in this image which I have taken it Now in this image which I have taken it almost looks like the fluid is static almost looks like the fluid is static almost looks like the fluid is static otherwise these particles will be otherwise these particles will be otherwise these particles will be flowing right. So there is only the flowing right. So there is only the flowing right. So there is only the noise term or the diffusion term which noise term or the diffusion term which noise term or the diffusion term which is being shown in this image. is being shown in this image. is being shown in this image. On the right hand side we are zooming in On the right hand side we are zooming in

  21. L3 @ 1236s

    On the right hand side we are zooming in on specific particles. So here I have on specific particles. So here I have on specific particles. So here I have taken taken taken seven particles as reference and I'm seven particles as reference and I'm seven particles as reference and I'm plotting their trajectory. plotting their trajectory. plotting their trajectory. Now here I have assumed there is some Now here I have assumed there is some Now here I have assumed there is some drift when plotting these trajectories drift when plotting these trajectories drift when plotting these trajectories but you can see these motions of these but you can see these motions of these but you can see these motions of these particles almost look like a movement of particles almost look like a movement of particles almost look like a movement of a stock price. a stock price. a stock price. The these are the exact trajectories of The these are the exact trajectories of The these are the exact trajectories of particles when they follow Brownian particles when they follow Brownian particles when they follow Brownian motion. motion. motion. And there is a mathematical description And there is a mathematical description And there is a mathematical description of this which is very specific. But for of this which is very specific. But for of this which is very specific. But for now we are interested in the intuition now we are interested in the intuition now we are interested in the intuition behind Brownian motion and we are going behind Brownian motion and we are going behind Brownian motion and we are going to subject our data to the Brownian to subject our data to the Brownian to subject our data to the Brownian motion. [snorts] motion. [snorts] motion. [snorts] Now the first thought is okay I have Now the first thought is okay I have Now the first thought is okay I have this image which has maybe it's a 28x 28 this image which has maybe it's a 28x 28 this image which has maybe it's a 28x 28 image. So there are 784 pixels. image. So there are 784 pixels. image. So there are 784 pixels. So what am I exactly doing here? Well So what am I exactly doing here? Well

  22. L3 @ 1295s

    So what am I exactly doing here? Well imagine that instead of these tiny dots imagine that instead of these tiny dots imagine that instead of these tiny dots you have 784 dots and all these dots are you have 784 dots and all these dots are you have 784 dots and all these dots are following a brownie in motion. [snorts] following a brownie in motion. [snorts] following a brownie in motion. [snorts] So imagine that we have a book here like So imagine that we have a book here like So imagine that we have a book here like this in front of the screen. This is our this in front of the screen. This is our this in front of the screen. This is our image. I divide this book into 784 image. I divide this book into 784 image. I divide this book into 784 pixels. I take out each of these pixels pixels. I take out each of these pixels pixels. I take out each of these pixels and arrange them as dots. And I allow and arrange them as dots. And I allow and arrange them as dots. And I allow these dots to transform themselves or these dots to transform themselves or these dots to transform themselves or follow a Brownian motion follow a Brownian motion follow a Brownian motion and completely change themselves using and completely change themselves using and completely change themselves using the Brownian trajectory. And at the end the Brownian trajectory. And at the end the Brownian trajectory. And at the end of it, I expect these molecules or these of it, I expect these molecules or these of it, I expect these molecules or these particles to move so far from their particles to move so far from their particles to move so far from their original value that the entire image original value that the entire image original value that the entire image will become noise. That is my will become noise. That is my will become noise. That is my expectation. Okay. So, uh Okay. So, uh this is exactly what I'm planning to do. this is exactly what I'm planning to do. this is exactly what I'm planning to do. I'm planning to take all these images, I'm planning to take all these images,

  23. L3 @ 1355s

    I'm planning to take all these images, divide them into pixels and subject each divide them into pixels and subject each divide them into pixels and subject each pixel to a brownian trajectory so that pixel to a brownian trajectory so that pixel to a brownian trajectory so that after enough time the data becomes after enough time the data becomes after enough time the data becomes noise. noise. noise. So you can see that I have taken an So you can see that I have taken an So you can see that I have taken an inspiration from the microscopic motion inspiration from the microscopic motion inspiration from the microscopic motion of fluid particles and I'm subjecting my of fluid particles and I'm subjecting my of fluid particles and I'm subjecting my data to this microscopic motion so that data to this microscopic motion so that data to this microscopic motion so that after enough time has passed it after enough time has passed it after enough time has passed it completely becomes noise. Now if you look at the two approaches Now if you look at the two approaches which I discussed earlier the DDPM and which I discussed earlier the DDPM and which I discussed earlier the DDPM and the score-based approach both of them the score-based approach both of them the score-based approach both of them have a forward diffusion process which have a forward diffusion process which have a forward diffusion process which transforms transforms transforms image to noise image to noise image to noise but but they have a specific form in which these they have a specific form in which these they have a specific form in which these forward diffusion processes are written forward diffusion processes are written forward diffusion processes are written which are special case of the Brownian which are special case of the Brownian which are special case of the Brownian motion itself. motion itself. motion itself. Now why is it a special case of the Now why is it a special case of the Now why is it a special case of the Brownian motion itself? Well, the Brownian motion itself? Well, the Brownian motion itself? Well, the Brownian motion as I mentioned it has Brownian motion as I mentioned it has Brownian motion as I mentioned it has two terms a drift term and a noise term.

  24. L3 @ 1423s

    two terms a drift term and a noise term. two terms a drift term and a noise term. So DDPM has a specific formulation for So DDPM has a specific formulation for So DDPM has a specific formulation for the drift term and the noise term and the drift term and the noise term and the drift term and the noise term and noise condition score network have a noise condition score network have a noise condition score network have a specific formulation of the drift term specific formulation of the drift term specific formulation of the drift term and the noise term. But essentially both and the noise term. But essentially both and the noise term. But essentially both the trajectories are Brownian motion are the trajectories are Brownian motion are the trajectories are Brownian motion are following a stochastic motion at the following a stochastic motion at the following a stochastic motion at the heart of it. heart of it. heart of it. Okay, I'm using the drift and uh the Okay, I'm using the drift and uh the Okay, I'm using the drift and uh the diffusion term a lot. Let us try to diffusion term a lot. Let us try to diffusion term a lot. Let us try to intuitively understand what these two intuitively understand what these two intuitively understand what these two terms mean. terms mean. terms mean. So let's take an example that you have a So let's take an example that you have a So let's take an example that you have a simple paper boat. You have a paper boat simple paper boat. You have a paper boat simple paper boat. You have a paper boat and you have put the paper boat on a and you have put the paper boat on a and you have put the paper boat on a river on a stream of river on a current. river on a stream of river on a current. river on a stream of river on a current. Now the question is what will affect the Now the question is what will affect the Now the question is what will affect the path of this boat? path of this boat? path of this boat? Well, the first thing I can see is that Well, the first thing I can see is that

  25. L3 @ 1472s

    Well, the first thing I can see is that I can obviously see some current lines I can obviously see some current lines I can obviously see some current lines in the river. So I'll say that well in the river. So I'll say that well in the river. So I'll say that well there are some current lines. So the there are some current lines. So the there are some current lines. So the boat will move according to the current. boat will move according to the current. boat will move according to the current. Right? That is true. That is the first Right? That is true. That is the first Right? That is true. That is the first part which is called as the steady part which is called as the steady part which is called as the steady drift. But there is a second part also drift. But there is a second part also drift. But there is a second part also which is the random ripples which are which is the random ripples which are which is the random ripples which are created at any point in in the lake. created at any point in in the lake. created at any point in in the lake. So these random ripples are going to So these random ripples are going to So these random ripples are going to swerve the boat left and right along swerve the boat left and right along swerve the boat left and right along with the drift. This component will be with the drift. This component will be with the drift. This component will be added to the motion of the boat. Now added to the motion of the boat. Now added to the motion of the boat. Now this is exactly the diffusion component this is exactly the diffusion component this is exactly the diffusion component and this is what is responsible for and this is what is responsible for and this is what is responsible for finally transforming the data to pure finally transforming the data to pure finally transforming the data to pure noise. The combined motion is a combination of The combined motion is a combination of a steady drift term which is the current a steady drift term which is the current

  26. L3 @ 1524s

    a steady drift term which is the current of the lake or the river and the second of the lake or the river and the second of the lake or the river and the second term is the diffusion term which are term is the diffusion term which are term is the diffusion term which are these random ripples which are causing these random ripples which are causing these random ripples which are causing the boat to swerve left and right. the boat to swerve left and right. the boat to swerve left and right. Now this analogy is confined to the Now this analogy is confined to the Now this analogy is confined to the example of boats but example of boats but example of boats but this is exactly what we are seeing over this is exactly what we are seeing over this is exactly what we are seeing over here when we are transforming the data here when we are transforming the data here when we are transforming the data to noise. We have a drift term and we to noise. We have a drift term and we to noise. We have a drift term and we have a diffusion term have a diffusion term have a diffusion term so far. Okay. Fine. We have looked at so far. Okay. Fine. We have looked at so far. Okay. Fine. We have looked at the forward diffusion process and uh the forward diffusion process and uh the forward diffusion process and uh we looked at there are two parts the we looked at there are two parts the we looked at there are two parts the drift part and the diffusion part and drift part and the diffusion part and drift part and the diffusion part and the combined motion is a combination of the combined motion is a combination of the combined motion is a combination of this. [snorts] this. [snorts] this. [snorts] Now what we will do is we will apply Now what we will do is we will apply Now what we will do is we will apply this analogy to our forward diffusion this analogy to our forward diffusion this analogy to our forward diffusion process and look at a mathematical form process and look at a mathematical form process and look at a mathematical form for this forward diffusion.

  27. L3 @ 1580s

    for this forward diffusion. for this forward diffusion. Okay. So we have converted data to noise Okay. So we have converted data to noise Okay. So we have converted data to noise and the change in the pixel values is a and the change in the pixel values is a and the change in the pixel values is a combination of deterministic drift and combination of deterministic drift and combination of deterministic drift and stochastic diffusion. stochastic diffusion. stochastic diffusion. Now the drift has a specific formula Now the drift has a specific formula Now the drift has a specific formula which comes as a function f of x t into which comes as a function f of x t into which comes as a function f of x t into change in time and the second is a change in time and the second is a change in time and the second is a diffusion term which depends on a diffusion term which depends on a diffusion term which depends on a function g of t. function g of t. function g of t. So the forward diffusion process depends So the forward diffusion process depends So the forward diffusion process depends on two terms. One is the drift term on two terms. One is the drift term on two terms. One is the drift term which is denoted as f and the second is which is denoted as f and the second is which is denoted as f and the second is the diffusion term which is denoted as the diffusion term which is denoted as the diffusion term which is denoted as g. g. g. Now the good part is that we know both Now the good part is that we know both Now the good part is that we know both of these terms which means that we of these terms which means that we of these terms which means that we exactly know how we are going to exactly know how we are going to exactly know how we are going to transform the data to noise. There is transform the data to noise. There is transform the data to noise. There is absolutely no learning happening here. absolutely no learning happening here. absolutely no learning happening here. You might relate this to a variational

  28. L3 @ 1638s

    You might relate this to a variational You might relate this to a variational autoenccoder where there is an encoder autoenccoder where there is an encoder autoenccoder where there is an encoder and there is a decoder and the learning and there is a decoder and the learning and there is a decoder and the learning happens in both the encoder and the happens in both the encoder and the happens in both the encoder and the decoder. This is very different. We are decoder. This is very different. We are decoder. This is very different. We are not learning anything in the forward not learning anything in the forward not learning anything in the forward process. It is completely known to us process. It is completely known to us process. It is completely known to us because we are the ones who are deciding because we are the ones who are deciding because we are the ones who are deciding the drift term and the diffusion term. the drift term and the diffusion term. the drift term and the diffusion term. In DDPM the drift term and diffusion In DDPM the drift term and diffusion In DDPM the drift term and diffusion term was different. In uh score- based term was different. In uh score- based term was different. In uh score- based approach the drift term and the approach the drift term and the approach the drift term and the diffusion term was different. diffusion term was different. diffusion term was different. But both of these approaches can be But both of these approaches can be But both of these approaches can be viewed from this microscopic lens where viewed from this microscopic lens where viewed from this microscopic lens where we are diffusing every single pixel in we are diffusing every single pixel in we are diffusing every single pixel in the data as a brownian trajectory. Okay, now we come to the second part Okay, now we come to the second part which is a very crucial part in uh which is a very crucial part in uh which is a very crucial part in uh today's lecture. Remember we started today's lecture. Remember we started today's lecture. Remember we started with our discussion with the time with our discussion with the time

  29. L3 @ 1689s

    with our discussion with the time machine. We were asking that can we machine. We were asking that can we machine. We were asking that can we reverse the reverse the reverse the diffusion of the dye? Can we bring back diffusion of the dye? Can we bring back diffusion of the dye? Can we bring back the original spherical nature of the the original spherical nature of the the original spherical nature of the drop? Can we understand it? And at a drop? Can we understand it? And at a drop? Can we understand it? And at a microscopic level we had no idea how to microscopic level we had no idea how to microscopic level we had no idea how to do it. But we looked at a we used a lens do it. But we looked at a we used a lens do it. But we looked at a we used a lens and looked at it at it microscopically and looked at it at it microscopically and looked at it at it microscopically and we realized that okay microscopic and we realized that okay microscopic and we realized that okay microscopic motions are diffused are are are time motions are diffused are are are time motions are diffused are are are time reversible reversible reversible that's why we subjected these pixels in that's why we subjected these pixels in that's why we subjected these pixels in these cat images to the brownian motion these cat images to the brownian motion these cat images to the brownian motion and now we come to the main part which and now we come to the main part which and now we come to the main part which is can we write the reverse process as is can we write the reverse process as is can we write the reverse process as well it turns out yes we can write the it turns out yes we can write the reverse process And uh we are helped and reverse process And uh we are helped and reverse process And uh we are helped and very grateful to this paper which was very grateful to this paper which was very grateful to this paper which was released released released more than 40 years back in the year 1982 more than 40 years back in the year 1982

  30. L3 @ 1745s

    more than 40 years back in the year 1982 by a single author Brian Anderson. by a single author Brian Anderson. by a single author Brian Anderson. And in this paper he wrote a formula And in this paper he wrote a formula And in this paper he wrote a formula which is so simple that it makes which is so simple that it makes which is so simple that it makes calculation of these reverse calculation of these reverse calculation of these reverse trajectories very easy. He actually gave trajectories very easy. He actually gave trajectories very easy. He actually gave a closed-ended formula using which we a closed-ended formula using which we a closed-ended formula using which we can reverse the can reverse the can reverse the forward diffusion process which is Brown forward diffusion process which is Brown forward diffusion process which is Brown in motion in motion in motion and uh it is incredible how works of and uh it is incredible how works of and uh it is incredible how works of these single authors are still being these single authors are still being these single authors are still being used in the AI literature and u they are used in the AI literature and u they are used in the AI literature and u they are at the heart of diffusion models which at the heart of diffusion models which at the heart of diffusion models which power image generation, video generation power image generation, video generation power image generation, video generation and and audio generation. and and audio generation. and and audio generation. Whenever I see examples like this, my Whenever I see examples like this, my Whenever I see examples like this, my belief in research always gets belief in research always gets belief in research always gets strengthened and u the incremental strengthened and u the incremental strengthened and u the incremental nature of research where each work nature of research where each work nature of research where each work builds upon the work which has come builds upon the work which has come builds upon the work which has come previously. previously. previously. It it really uh resonates with me and I It it really uh resonates with me and I It it really uh resonates with me and I appreciate all the researchers who spend

  31. L3 @ 1812s

    appreciate all the researchers who spend appreciate all the researchers who spend a lot of time in writing these papers. a lot of time in writing these papers. a lot of time in writing these papers. Okay. And what does the reverse formula Okay. And what does the reverse formula Okay. And what does the reverse formula look like? Well, the forward formula has look like? Well, the forward formula has look like? Well, the forward formula has two terms. The drift term and the two terms. The drift term and the two terms. The drift term and the diffusion term. As I have shown over diffusion term. As I have shown over diffusion term. As I have shown over here, here, here, the reverse formula the reverse formula the reverse formula also has two terms. The diffusion term also has two terms. The diffusion term also has two terms. The diffusion term remains the same. As you can see, we remains the same. As you can see, we remains the same. As you can see, we will focus on the drift term, which is will focus on the drift term, which is will focus on the drift term, which is the combination of the drift term for the combination of the drift term for the combination of the drift term for the forward diffusion process. And there the forward diffusion process. And there the forward diffusion process. And there is an additional factor being added is an additional factor being added is an additional factor being added here. We will only focus on that factor here. We will only focus on that factor here. We will only focus on that factor right now because that was the biggest right now because that was the biggest right now because that was the biggest contribution from the Anderson paper. contribution from the Anderson paper. contribution from the Anderson paper. And that factor is a multiplication of And that factor is a multiplication of And that factor is a multiplication of two terms. The first term is the square two terms. The first term is the square two terms. The first term is the square of G of T which is the diffusion term in of G of T which is the diffusion term in

  32. L3 @ 1859s

    of G of T which is the diffusion term in the forward process. And the second term the forward process. And the second term the forward process. And the second term is the score function. is the score function. is the score function. We have not discussed what is score We have not discussed what is score We have not discussed what is score function in this lecture. But uh function in this lecture. But uh function in this lecture. But uh you might have heard me talk about score you might have heard me talk about score you might have heard me talk about score based approach. based approach. based approach. And the reason why the score-based And the reason why the score-based And the reason why the score-based approach works so beautifully is because approach works so beautifully is because approach works so beautifully is because it naturally comes up in the formula for it naturally comes up in the formula for it naturally comes up in the formula for the reverse the reverse the reverse diffusion process. It actually comes up diffusion process. It actually comes up diffusion process. It actually comes up as a separate function. as a separate function. as a separate function. And you might look at this equation and And you might look at this equation and And you might look at this equation and say that I already know f, I already say that I already know f, I already say that I already know f, I already don't know g. So can I directly predict don't know g. So can I directly predict don't know g. So can I directly predict the reverse motion? Can I reverse from the reverse motion? Can I reverse from the reverse motion? Can I reverse from noise to the images of cats? The answer noise to the images of cats? The answer noise to the images of cats? The answer is no. because the score function is not is no. because the score function is not is no. because the score function is not very easy to calculate. If this was very very easy to calculate. If this was very very easy to calculate. If this was very easy then we would have directly easy then we would have directly easy then we would have directly reversed from noise to the image and

  33. L3 @ 1912s

    reversed from noise to the image and reversed from noise to the image and this is where the learning will happen. this is where the learning will happen. this is where the learning will happen. Why is the score function difficult to Why is the score function difficult to Why is the score function difficult to calculate? calculate? calculate? Well to understand that first we will Well to understand that first we will Well to understand that first we will have to look at what is a score function have to look at what is a score function have to look at what is a score function exactly exactly exactly and I'm going to give you a analogy to and I'm going to give you a analogy to and I'm going to give you a analogy to understand the score function. Imagine understand the score function. Imagine understand the score function. Imagine that you're out in the ocean and you're that you're out in the ocean and you're that you're out in the ocean and you're trying to find a shark. trying to find a shark. trying to find a shark. You go with a compass You go with a compass You go with a compass which gives you exactly where the sharks which gives you exactly where the sharks which gives you exactly where the sharks are present. are present. are present. So you start out with a random location. So you start out with a random location. So you start out with a random location. Your compass tells you go go left. You Your compass tells you go go left. You Your compass tells you go go left. You go towards the left direction. You move. go towards the left direction. You move. go towards the left direction. You move. Then your compos says oh no no no go go Then your compos says oh no no no go go Then your compos says oh no no no go go right. Go straight. Go left. and then it right. Go straight. Go left. and then it right. Go straight. Go left. and then it guides you until you are in the region guides you until you are in the region guides you until you are in the region where the density of the sharks is where the density of the sharks is

  34. L3 @ 1964s

    where the density of the sharks is maximum. maximum. maximum. Now score function is exactly like this Now score function is exactly like this Now score function is exactly like this compass and sharks are your data. You're compass and sharks are your data. You're compass and sharks are your data. You're using a compass to navigate yourself using a compass to navigate yourself using a compass to navigate yourself around the data field so that you find around the data field so that you find around the data field so that you find the region where the data is maximum the region where the data is maximum the region where the data is maximum where the data density is maximum. where the data density is maximum. where the data density is maximum. [snorts] So the score function points to [snorts] So the score function points to [snorts] So the score function points to the direction where the direction where the direction where the density of the data is maximum. the density of the data is maximum. the density of the data is maximum. If you look at this this example, the If you look at this this example, the If you look at this this example, the black dots is the region where the black dots is the region where the black dots is the region where the density of the data is the highest. That density of the data is the highest. That density of the data is the highest. That is where the sharks are present. And you is where the sharks are present. And you is where the sharks are present. And you can see all the arrows are kind of can see all the arrows are kind of can see all the arrows are kind of pointing towards those regions. Well, pointing towards those regions. Well, pointing towards those regions. Well, those arrows are the score function. And those arrows are the score function. And those arrows are the score function. And you can write them as gradient of log of you can write them as gradient of log of you can write them as gradient of log of P of T where P of T is the probability P of T where P of T is the probability P of T where P of T is the probability density. density.

  35. L3 @ 2024s

    density. Now where does the gradient and log come Now where does the gradient and log come Now where does the gradient and log come from? Well, the gradient comes from uh from? Well, the gradient comes from uh from? Well, the gradient comes from uh your objective of going from lower your objective of going from lower your objective of going from lower density to a higher density. So the density to a higher density. So the density to a higher density. So the gradient naturally puts you in that gradient naturally puts you in that gradient naturally puts you in that direction. direction. direction. And where does the log come from? If you And where does the log come from? If you And where does the log come from? If you are in a region where the probability is are in a region where the probability is are in a region where the probability is very low then you need a very large step very low then you need a very large step very low then you need a very large step right that's why you take a log because right that's why you take a log because right that's why you take a log because log of lower value is very high. So if log of lower value is very high. So if log of lower value is very high. So if you're very large if you're away from you're very large if you're away from you're very large if you're away from the sharks you take bigger steps and you the sharks you take bigger steps and you the sharks you take bigger steps and you move closer faster. move closer faster. move closer faster. That is what the score function does. That is what the score function does. That is what the score function does. Now if you look at this formula you can Now if you look at this formula you can Now if you look at this formula you can see that it has this probability see that it has this probability see that it has this probability function which is exactly what we want function which is exactly what we want function which is exactly what we want to predict. Remember we started with a to predict. Remember we started with a

  36. L3 @ 2074s

    to predict. Remember we started with a discussion that we want to predict the discussion that we want to predict the discussion that we want to predict the distribution of the cats and this distribution of the cats and this distribution of the cats and this probability function is something which probability function is something which probability function is something which we do not know. That is the reason we we do not know. That is the reason we we do not know. That is the reason we cannot substitute this value directly in cannot substitute this value directly in cannot substitute this value directly in the score function the score function the score function which becomes the bottleneck in this which becomes the bottleneck in this which becomes the bottleneck in this formula. formula. formula. So what is the way around? How do we So what is the way around? How do we So what is the way around? How do we learn the score function? In today's lecture, I will not be In today's lecture, I will not be covering how the score function is covering how the score function is covering how the score function is learned. But I am going to assume that learned. But I am going to assume that learned. But I am going to assume that we have learned the score function. we have learned the score function. we have learned the score function. To understand how the score function is To understand how the score function is To understand how the score function is learned, you can look at my previous two learned, you can look at my previous two learned, you can look at my previous two lectures which talk about score function lectures which talk about score function lectures which talk about score function in much more detail. But the process is again simple. Uh we But the process is again simple. Uh we add noise deliberately and then we learn add noise deliberately and then we learn add noise deliberately and then we learn how much noise is added and in that how much noise is added and in that how much noise is added and in that process we learn the score function. process we learn the score function. process we learn the score function. For now let's simplify things. Let us

  37. L3 @ 2134s

    For now let's simplify things. Let us For now let's simplify things. Let us assume that the score function is assume that the score function is assume that the score function is already learned. already learned. already learned. Okay. So uh let us say that we know the Okay. So uh let us say that we know the Okay. So uh let us say that we know the score function already and in the score function already and in the score function already and in the process of learning the score function process of learning the score function process of learning the score function we have learned a lot about our data as we have learned a lot about our data as we have learned a lot about our data as well. Okay. Now let's say we know the score Okay. Now let's say we know the score function. So we know f, we know g and we function. So we know f, we know g and we function. So we know f, we know g and we know s. The natural question is how do know s. The natural question is how do know s. The natural question is how do we sample from this? we sample from this? we sample from this? How do we go back from noise to the How do we go back from noise to the How do we go back from noise to the data? If we already have the reverse data? If we already have the reverse data? If we already have the reverse time process equation given by Anderson time process equation given by Anderson time process equation given by Anderson and we know all the terms, how do we go and we know all the terms, how do we go and we know all the terms, how do we go back back from noise to data? back back from noise to data? back back from noise to data? Let's look at uh the diffusion the the Let's look at uh the diffusion the the Let's look at uh the diffusion the the reverse process again and uh again I'm reverse process again and uh again I'm reverse process again and uh again I'm emphasizing that the drift term here emphasizing that the drift term here emphasizing that the drift term here comprises of the drift in the forward

  38. L3 @ 2190s

    comprises of the drift in the forward comprises of the drift in the forward diffusion and the factor which depends diffusion and the factor which depends diffusion and the factor which depends on the score function and the diffusion on the score function and the diffusion on the score function and the diffusion term remains the same. term remains the same. term remains the same. Now our only job is to discretise this Now our only job is to discretise this Now our only job is to discretise this which means that we are going to replace which means that we are going to replace which means that we are going to replace dx by delta x. We are going to replace dx by delta x. We are going to replace dx by delta x. We are going to replace dt by deltat t and we are going to dt by deltat t and we are going to dt by deltat t and we are going to replace dw by a random variable zed replace dw by a random variable zed replace dw by a random variable zed which takes a value between 0 and 1 and which takes a value between 0 and 1 and which takes a value between 0 and 1 and root of delta t. This is the diffusion root of delta t. This is the diffusion root of delta t. This is the diffusion term in the brownian motion. term in the brownian motion. term in the brownian motion. Now the key concept here is that the Now the key concept here is that the Now the key concept here is that the noise scales with the square root of noise scales with the square root of noise scales with the square root of time and not time itself. time and not time itself. time and not time itself. The reason is that if you take any The reason is that if you take any The reason is that if you take any brownian trajectory and you take any two brownian trajectory and you take any two brownian trajectory and you take any two points separated by a gap of delta t you points separated by a gap of delta t you points separated by a gap of delta t you ask the question what is the variation

  39. L3 @ 2246s

    ask the question what is the variation ask the question what is the variation in the values of these two points. It in the values of these two points. It in the values of these two points. It turns out that turns out that turns out that the variance in these values of the variance in these values of the variance in these values of difference between these two points difference between these two points difference between these two points scales as deltat t which is the scales as deltat t which is the scales as deltat t which is the difference in this time which means that difference in this time which means that difference in this time which means that as you move further and farther away if as you move further and farther away if as you move further and farther away if you fix one point if the second point you fix one point if the second point you fix one point if the second point moves farther and further away the moves farther and further away the moves farther and further away the variance will go on increasing as delta variance will go on increasing as delta variance will go on increasing as delta t. Now because the variance increases as t. Now because the variance increases as t. Now because the variance increases as delta t the standard deviation increases delta t the standard deviation increases delta t the standard deviation increases at as root of delta t and that is why at as root of delta t and that is why at as root of delta t and that is why you have a root of delta t in this you have a root of delta t in this you have a root of delta t in this formula. Okay. Now you might say okay what is the Okay. Now you might say okay what is the next step? Well now you have learned to next step? Well now you have learned to next step? Well now you have learned to find the increment of x which is the find the increment of x which is the find the increment of x which is the pixel intensity pixel intensity pixel intensity in terms of delta t. So if you have in terms of delta t. So if you have

  40. L3 @ 2295s

    in terms of delta t. So if you have delta t you calculate delta x then you delta t you calculate delta x then you delta t you calculate delta x then you again substitute delta t calculate delta again substitute delta t calculate delta again substitute delta t calculate delta x and you move from noise to the data x and you move from noise to the data x and you move from noise to the data recursively using this equation. recursively using this equation. recursively using this equation. Once you have discretized this you can Once you have discretized this you can Once you have discretized this you can enter into a loop where you can enter into a loop where you can enter into a loop where you can continuously run this discretization continuously run this discretization continuously run this discretization and uh you can reach to the original and uh you can reach to the original and uh you can reach to the original image. this equation or because of this image. this equation or because of this image. this equation or because of this nature of discretization and the way nature of discretization and the way nature of discretization and the way this equation appears, it is also called this equation appears, it is also called this equation appears, it is also called as a stochastic differential equation. as a stochastic differential equation. as a stochastic differential equation. An ordinary differential equation would An ordinary differential equation would An ordinary differential equation would not have this second part which comes up not have this second part which comes up not have this second part which comes up as root of delta t square root of time. as root of delta t square root of time. as root of delta t square root of time. It would only have this first part. But It would only have this first part. But It would only have this first part. But here we have the second part also. That here we have the second part also. That here we have the second part also. That is why it is called as a stochastic is why it is called as a stochastic is why it is called as a stochastic differential equation or an SDE. differential equation or an SDE.

  41. L3 @ 2344s

    differential equation or an SDE. This formulation was laid down in a This formulation was laid down in a This formulation was laid down in a paper by Song and Arman in the year paper by Song and Arman in the year paper by Song and Arman in the year 2020. I will link the description below 2020. I will link the description below 2020. I will link the description below in the description section. in the description section. in the description section. But essentially at the heart of the But essentially at the heart of the But essentially at the heart of the paper it's it it talks about paper it's it it talks about paper it's it it talks about discretizing this equation and forming a discretizing this equation and forming a discretizing this equation and forming a connection between the SDE framework connection between the SDE framework connection between the SDE framework which is the stochastic differential which is the stochastic differential which is the stochastic differential equation framework and the previous two equation framework and the previous two equation framework and the previous two approaches which were DDPM and score approaches which were DDPM and score approaches which were DDPM and score based function or NCSN noise conditional based function or NCSN noise conditional based function or NCSN noise conditional score networks. The sampling process looks something The sampling process looks something like this. We start with noise. like this. We start with noise. like this. We start with noise. Then um Then um Then um remember we have two terms. We have a remember we have two terms. We have a remember we have two terms. We have a diffusion term and we have a noise term. diffusion term and we have a noise term. diffusion term and we have a noise term. This is the sorry we have a drift term This is the sorry we have a drift term This is the sorry we have a drift term and a noise term. So the blue arrow and a noise term. So the blue arrow and a noise term. So the blue arrow represents the drift term. The red arrow represents the drift term. The red arrow represents the drift term. The red arrow represents the noise term. So we go from represents the noise term. So we go from

  42. L3 @ 2402s

    represents the noise term. So we go from the first iteration to the second the first iteration to the second the first iteration to the second iteration. Then we again do diffusion iteration. Then we again do diffusion iteration. Then we again do diffusion and drift. We go here. Again we do drift and drift. We go here. Again we do drift and drift. We go here. Again we do drift and diffusion. We go here. You can see and diffusion. We go here. You can see and diffusion. We go here. You can see the clarity of the image is slowly the clarity of the image is slowly the clarity of the image is slowly becoming better and better and better. becoming better and better and better. becoming better and better and better. And we do this until we get the original And we do this until we get the original And we do this until we get the original image. image. image. This is also called as the oiler maruama This is also called as the oiler maruama This is also called as the oiler maruama solver. This is the simplest way of solver. This is the simplest way of solver. This is the simplest way of solving this particular stoastic solving this particular stoastic solving this particular stoastic differential equation. And you can see differential equation. And you can see differential equation. And you can see it can be visualized in the drift and it can be visualized in the drift and it can be visualized in the drift and the noise components. the noise components. the noise components. So this is how we solve the forward So this is how we solve the forward So this is how we solve the forward process and the reverse process. Um and process and the reverse process. Um and process and the reverse process. Um and why we have used the microscopic nature why we have used the microscopic nature why we have used the microscopic nature of the diffusion which we discussed at of the diffusion which we discussed at of the diffusion which we discussed at the start. Now it might become clear to the start. Now it might become clear to the start. Now it might become clear to you because we know the exact equation you because we know the exact equation

  43. L3 @ 2454s

    you because we know the exact equation of the reverse process and we use that of the reverse process and we use that of the reverse process and we use that same equation for sampling. same equation for sampling. same equation for sampling. The only thing we have done here is we The only thing we have done here is we The only thing we have done here is we have different ways of or we have have different ways of or we have have different ways of or we have predicted the score function directly. predicted the score function directly. predicted the score function directly. We have assumed we know the score We have assumed we know the score We have assumed we know the score function but there is a way to predict function but there is a way to predict function but there is a way to predict the score function from the data itself. Okay. So uh now what we will do is we Okay. So uh now what we will do is we will take a example a practical example will take a example a practical example will take a example a practical example and we will understand the forward and and we will understand the forward and and we will understand the forward and diffusion process from a practical point diffusion process from a practical point diffusion process from a practical point of view and uh through this we will also of view and uh through this we will also of view and uh through this we will also get a firm understanding of the concepts get a firm understanding of the concepts get a firm understanding of the concepts we have discussed so far and uh we will we have discussed so far and uh we will we have discussed so far and uh we will get confidence in the SDE or the get confidence in the SDE or the get confidence in the SDE or the stoastic differential equation stoastic differential equation stoastic differential equation framework. framework. framework. The example we will take is imagine The example we will take is imagine The example we will take is imagine you're training a generative model on a you're training a generative model on a you're training a generative model on a data set consisting only of images of

  44. L3 @ 2509s

    data set consisting only of images of data set consisting only of images of handwritten digits which are ones and handwritten digits which are ones and handwritten digits which are ones and eights. eights. eights. Now instead of cats we are taking these Now instead of cats we are taking these Now instead of cats we are taking these two handwritten digits 1 and 8. two handwritten digits 1 and 8. two handwritten digits 1 and 8. And the objective is is to find out the And the objective is is to find out the And the objective is is to find out the probability density probability density probability density or the distribution of these images. or the distribution of these images. or the distribution of these images. Now let's say the distribution looks Now let's say the distribution looks Now let's say the distribution looks like a biodal distribution like a biodal distribution like a biodal distribution where the first mode corresponds to where the first mode corresponds to where the first mode corresponds to digit one and the second mode digit one and the second mode digit one and the second mode corresponds to digit two. I have taken a corresponds to digit two. I have taken a corresponds to digit two. I have taken a biodal data distribution because it's biodal data distribution because it's biodal data distribution because it's going to help me explain certain things. going to help me explain certain things. going to help me explain certain things. But this is the distribution that we But this is the distribution that we But this is the distribution that we want to sample from. But we have no idea want to sample from. But we have no idea want to sample from. But we have no idea about this distribution. We are only about this distribution. We are only about this distribution. We are only given a bunch of handwritten digits once given a bunch of handwritten digits once given a bunch of handwritten digits once and eights and we are asked to sample and eights and we are asked to sample and eights and we are asked to sample images from a distribution which look images from a distribution which look images from a distribution which look like these handwritten digits. like these handwritten digits.

  45. L3 @ 2566s

    like these handwritten digits. Okay. So uh we are expecting a biodal Okay. So uh we are expecting a biodal Okay. So uh we are expecting a biodal distribution. This is what where we want distribution. This is what where we want distribution. This is what where we want to go. to go. to go. Now imagine that this is the Now imagine that this is the Now imagine that this is the distribution which is the dye and you distribution which is the dye and you distribution which is the dye and you are diffusing the dye so that the are diffusing the dye so that the are diffusing the dye so that the structure slowly disappears. The red structure slowly disappears. The red structure slowly disappears. The red paint is slowly dissolving in water paint is slowly dissolving in water paint is slowly dissolving in water becoming uniform over time and uh we are becoming uniform over time and uh we are becoming uniform over time and uh we are using a forward diffusion process as the using a forward diffusion process as the using a forward diffusion process as the first step first step first step and we are going to use this microscopic and we are going to use this microscopic and we are going to use this microscopic approach. So we are going to subject approach. So we are going to subject approach. So we are going to subject every pixel in these handwritten digits every pixel in these handwritten digits every pixel in these handwritten digits to a brownian motion and then we will to a brownian motion and then we will to a brownian motion and then we will learn to reverse the brownian motion. learn to reverse the brownian motion. learn to reverse the brownian motion. Okay. So we will diffuse each Okay. So we will diffuse each Okay. So we will diffuse each handwritten digit such such that like a handwritten digit such such that like a handwritten digit such such that like a particle each image follows a brownie in particle each image follows a brownie in particle each image follows a brownie in motion or more specifically each pixel motion or more specifically each pixel motion or more specifically each pixel in the image follows a brownie in in the image follows a brownie in

  46. L3 @ 2621s

    in the image follows a brownie in motion. motion. Okay. So just like a paint is diffusing Okay. So just like a paint is diffusing Okay. So just like a paint is diffusing in water our in water our in water our original distribution is diffusing and original distribution is diffusing and original distribution is diffusing and we are transforming it to noise using we are transforming it to noise using we are transforming it to noise using brownian motion. How does that look brownian motion. How does that look brownian motion. How does that look like? Well, on the left you can see this like? Well, on the left you can see this like? Well, on the left you can see this is the distribution which we don't have is the distribution which we don't have is the distribution which we don't have any idea about. But we are transforming any idea about. But we are transforming any idea about. But we are transforming every image through a trajectory which every image through a trajectory which every image through a trajectory which looks like this. You can see as we move looks like this. You can see as we move looks like this. You can see as we move from left to right the myodel from left to right the myodel from left to right the myodel distribution gets converted to a distribution gets converted to a distribution gets converted to a unimodel distribution because it gets unimodel distribution because it gets unimodel distribution because it gets converted to noise. converted to noise. converted to noise. And all of these trajectories And all of these trajectories And all of these trajectories these are the brownian motion followed these are the brownian motion followed these are the brownian motion followed by every single pixel or we can rather by every single pixel or we can rather by every single pixel or we can rather say image in this trajectory. say image in this trajectory. say image in this trajectory. Now this is a simplified picture. In Now this is a simplified picture. In Now this is a simplified picture. In reality reality reality uh this is just a one-dimensional uh this is just a one-dimensional uh this is just a one-dimensional trajectory I have shown but in reality trajectory I have shown but in reality

  47. L3 @ 2680s

    trajectory I have shown but in reality we have these 784 pixels. So it will be we have these 784 pixels. So it will be we have these 784 pixels. So it will be in a 784dimensional space which is in a 784dimensional space which is in a 784dimensional space which is impossible for us to visualize impossible for us to visualize impossible for us to visualize but for clarity we have assumed that but for clarity we have assumed that but for clarity we have assumed that it's a 1D for for reference and the it's a 1D for for reference and the it's a 1D for for reference and the particles look like you can see there is particles look like you can see there is particles look like you can see there is a drift term and a diffusion term both a drift term and a diffusion term both a drift term and a diffusion term both because if there was just a diffusion because if there was just a diffusion because if there was just a diffusion term you would see the particles moving term you would see the particles moving term you would see the particles moving around the mean but now we are seeing around the mean but now we are seeing around the mean but now we are seeing the particles also drifting with time. the particles also drifting with time. the particles also drifting with time. So this is the forward diffusion So this is the forward diffusion So this is the forward diffusion process. We have transformed the process. We have transformed the process. We have transformed the data to noise. data to noise. data to noise. Now we are going to and and and here we Now we are going to and and and here we Now we are going to and and and here we know the drift term, the diffusion term know the drift term, the diffusion term know the drift term, the diffusion term F and G are decided by us. We know them F and G are decided by us. We know them F and G are decided by us. We know them completely. completely. completely. Now the next step is we know that the Now the next step is we know that the

  48. L3 @ 2731s

    Now the next step is we know that the microscopic brownian motion is microscopic brownian motion is microscopic brownian motion is reversible. Uh because as you can see we reversible. Uh because as you can see we reversible. Uh because as you can see we cannot tell if this video is being cannot tell if this video is being cannot tell if this video is being played forward or backward. played forward or backward. played forward or backward. and Anderson has given us the formula and Anderson has given us the formula and Anderson has given us the formula for the reverse motion. for the reverse motion. for the reverse motion. And we are going to assume that we have And we are going to assume that we have And we are going to assume that we have trained a neural network to learn the trained a neural network to learn the trained a neural network to learn the score function. That is one block which score function. That is one block which score function. That is one block which is not covered in this lecture. Uh but is not covered in this lecture. Uh but is not covered in this lecture. Uh but we are going to assume for now that the we are going to assume for now that the we are going to assume for now that the score function is learned. score function is learned. score function is learned. uh typically the score function assumes uh typically the score function assumes uh typically the score function assumes a a a unit as as an architecture but uh we unit as as an architecture but uh we unit as as an architecture but uh we will cover that briefly towards the end will cover that briefly towards the end will cover that briefly towards the end of this lecture. Okay. So the score function is predicted Okay. So the score function is predicted using a model which we train and once we using a model which we train and once we using a model which we train and once we know that we have access to F, G and S know that we have access to F, G and S know that we have access to F, G and S as well. as well.

  49. L3 @ 2788s

    as well. And now you can see on the left hand And now you can see on the left hand And now you can see on the left hand side we have noise and we are using the side we have noise and we are using the side we have noise and we are using the discretized formulation of the SD which discretized formulation of the SD which discretized formulation of the SD which is the oiler maruama equation and we are is the oiler maruama equation and we are is the oiler maruama equation and we are going from noise to data and we know going from noise to data and we know going from noise to data and we know exactly what the trajectory to follow exactly what the trajectory to follow exactly what the trajectory to follow because the microscopic motion is time because the microscopic motion is time because the microscopic motion is time reversible right so we know F we know G reversible right so we know F we know G reversible right so we know F we know G and we know S. So as long as the score and we know S. So as long as the score and we know S. So as long as the score function is trained properly, we should function is trained properly, we should function is trained properly, we should be able to go from noise to images which be able to go from noise to images which be able to go from noise to images which you can see from uh you can see from uh you can see from uh this image which articulates the this image which articulates the this image which articulates the brownian motion trajectories but rather brownian motion trajectories but rather brownian motion trajectories but rather in a reverse way. So you can see that in a reverse way. So you can see that in a reverse way. So you can see that the reverse motion also looks like a the reverse motion also looks like a the reverse motion also looks like a brownian motion brownian motion brownian motion uh where there is a drift term and a uh where there is a drift term and a uh where there is a drift term and a diffusion term but this time the diffusion term but this time the

  50. L3 @ 2836s

    diffusion term but this time the trajectory is spread into two very trajectory is spread into two very trajectory is spread into two very nicely and uh we do get a biodal nicely and uh we do get a biodal nicely and uh we do get a biodal distribution at the end which is very distribution at the end which is very distribution at the end which is very interesting. This is how we learn to generate samples This is how we learn to generate samples from a distribution which is not known from a distribution which is not known from a distribution which is not known to us. But we are only given some to us. But we are only given some to us. But we are only given some handful of images and uh this is exactly handful of images and uh this is exactly handful of images and uh this is exactly the framework which current diffusion the framework which current diffusion the framework which current diffusion models follow. models follow. models follow. Uh there are two tracks which is first Uh there are two tracks which is first Uh there are two tracks which is first is DDPM and the second is the energy is DDPM and the second is the energy is DDPM and the second is the energy based track which is the score function. based track which is the score function. based track which is the score function. But the stochastic differential equation But the stochastic differential equation But the stochastic differential equation approach SD approach unifies both of approach SD approach unifies both of approach SD approach unifies both of these where both of these are just these where both of these are just these where both of these are just special cases of selecting the drift special cases of selecting the drift special cases of selecting the drift term and the diffusion term. term and the diffusion term. term and the diffusion term. And we can see now why the score And we can see now why the score And we can see now why the score function becomes so important. You might function becomes so important. You might function becomes so important. You might have seen this word score function in have seen this word score function in

  51. L3 @ 2888s

    have seen this word score function in many papers but only after going through many papers but only after going through many papers but only after going through the SD framework we really understand the SD framework we really understand the SD framework we really understand why it becomes plays such an important why it becomes plays such an important why it becomes plays such an important role in the formulation. role in the formulation. role in the formulation. Now we actually solve this problem uh Now we actually solve this problem uh Now we actually solve this problem uh using Google Collab. I have shared the using Google Collab. I have shared the using Google Collab. I have shared the link in the description section. I want link in the description section. I want link in the description section. I want all of you to go through the details of all of you to go through the details of all of you to go through the details of this notebook properly, understand all this notebook properly, understand all this notebook properly, understand all the different components the different components the different components and uh finally the results we get after and uh finally the results we get after and uh finally the results we get after 500 iterations. It's it's it's not very 500 iterations. It's it's it's not very 500 iterations. It's it's it's not very good. After 5,000 iterations, you can good. After 5,000 iterations, you can good. After 5,000 iterations, you can see we are slowly starting to predict see we are slowly starting to predict see we are slowly starting to predict the digits as 8 and 1. the digits as 8 and 1. the digits as 8 and 1. um I I could have continued to 50,000 um I I could have continued to 50,000 um I I could have continued to 50,000 where it can give more clarity to these where it can give more clarity to these where it can give more clarity to these images but you can see we are slowly images but you can see we are slowly images but you can see we are slowly starting to learn how to generate images starting to learn how to generate images starting to learn how to generate images using a very simple approach which can

  52. L3 @ 2939s

    using a very simple approach which can using a very simple approach which can be implemented in Google Collab and uh be implemented in Google Collab and uh be implemented in Google Collab and uh this is a playground for all of you just this is a playground for all of you just this is a playground for all of you just feel free to experiment dive deeper into feel free to experiment dive deeper into feel free to experiment dive deeper into the individual cells get your hands the individual cells get your hands the individual cells get your hands dirty with the diffusion process because dirty with the diffusion process because dirty with the diffusion process because it is an incredibly useful skill to it is an incredibly useful skill to it is an incredibly useful skill to master diffusion master diffusion master diffusion And again as I said in the middle of the And again as I said in the middle of the And again as I said in the middle of the lecture there is a certain fascination lecture there is a certain fascination lecture there is a certain fascination towards the diffusion process. I think towards the diffusion process. I think towards the diffusion process. I think it stems from the fascination towards it stems from the fascination towards it stems from the fascination towards reversal of time itself which is what reversal of time itself which is what reversal of time itself which is what diffusion processes achieve. There are diffusion processes achieve. There are diffusion processes achieve. There are applications in uh video generation applications in uh video generation applications in uh video generation image generation even robotics there are image generation even robotics there are image generation even robotics there are some amazing policies which generate some amazing policies which generate some amazing policies which generate actions using diffusion. actions using diffusion. actions using diffusion. I will link the main paper which I will link the main paper which I will link the main paper which describes the SD frame uh framework in describes the SD frame uh framework in describes the SD frame uh framework in the description. Please go through it. the description. Please go through it. the description. Please go through it. It's a very nicely written paper and It's a very nicely written paper and

  53. L3 @ 2990s

    It's a very nicely written paper and after this introduction you should be after this introduction you should be after this introduction you should be able to understand it. Thank you very able to understand it. Thank you very able to understand it. Thank you very much everyone and I I hope you enjoyed much everyone and I I hope you enjoyed much everyone and I I hope you enjoyed this new approach which I taught this this new approach which I taught this this new approach which I taught this lecture with and uh I will see you in lecture with and uh I will see you in lecture with and uh I will see you in the next lecture.

  54. L6 @ 4s

    Hello everyone and welcome to this next Hello everyone and welcome to this next lecture in of the course principles of lecture in of the course principles of lecture in of the course principles of diffusion models. diffusion models. diffusion models. Firstly before we get started let us Firstly before we get started let us Firstly before we get started let us quickly recap what all things we have quickly recap what all things we have quickly recap what all things we have done in this course so far. done in this course so far. done in this course so far. We started off with We started off with We started off with deep generative models. We looked at these kind of models try to We looked at these kind of models try to predict the true distribution predict the true distribution predict the true distribution uh which governs the underlying data uh which governs the underlying data uh which governs the underlying data and uh we also looked at and uh we also looked at and uh we also looked at we do not have access to the true we do not have access to the true we do not have access to the true distribution. The only thing we have distribution. The only thing we have distribution. The only thing we have access to is samples from the data. And access to is samples from the data. And access to is samples from the data. And given the data samples, our objective is given the data samples, our objective is given the data samples, our objective is to predict the true distribution which to predict the true distribution which to predict the true distribution which governs this governs this governs this data distribution. data distribution. data distribution. And uh to do that, let's say the true And uh to do that, let's say the true And uh to do that, let's say the true distribution is given by P data of X. distribution is given by P data of X. distribution is given by P data of X. People generally use a neural network to People generally use a neural network to People generally use a neural network to approximate this two distribution as P5 approximate this two distribution as P5 approximate this two distribution as P5 of X. of X.

  55. L6 @ 75s

    of X. And our objective then becomes to And our objective then becomes to And our objective then becomes to determine determine determine the probability distribution P5 of X the probability distribution P5 of X the probability distribution P5 of X over here. over here. over here. Okay. Now what are the different methods Okay. Now what are the different methods Okay. Now what are the different methods which we will use to determine this which we will use to determine this which we will use to determine this probability distribution. probability distribution. probability distribution. And uh the first several lectures are And uh the first several lectures are And uh the first several lectures are going to be focused on going to be focused on going to be focused on these techniques. And the first these techniques. And the first these techniques. And the first technique that we looked at which is technique that we looked at which is technique that we looked at which is used to predict the probability used to predict the probability used to predict the probability distribution of the underlying data distribution of the underlying data distribution of the underlying data uh is called as variational uh is called as variational uh is called as variational autoenccoders or VAEs. In variation autoenccoders, what we do In variation autoenccoders, what we do is we first convert the data is we first convert the data is we first convert the data from real space to a latin space from real space to a latin space from real space to a latin space and then we convert it back to the real and then we convert it back to the real and then we convert it back to the real space. So the typical architecture of the VA So the typical architecture of the VA looks like this looks like this looks like this and uh the the first part is called as and uh the the first part is called as and uh the the first part is called as the encoder over here and the second the encoder over here and the second the encoder over here and the second part is called as the decoder. part is called as the decoder. part is called as the decoder. The objective is to match the output The objective is to match the output

  56. L6 @ 150s

    The objective is to match the output image with the input image as close as image with the input image as close as image with the input image as close as possible. possible. possible. And here we looked at there are two And here we looked at there are two And here we looked at there are two types of losses. The first loss is the types of losses. The first loss is the types of losses. The first loss is the reconstruction loss and the second loss is called as the and the second loss is called as the regularization loss. The practical example that we looked at The practical example that we looked at for understanding VAS is that of for understanding VAS is that of for understanding VAS is that of handwritten digits. handwritten digits. handwritten digits. So for handwritten digits, our objective So for handwritten digits, our objective So for handwritten digits, our objective was to feed in several data samples of was to feed in several data samples of was to feed in several data samples of handwritten uh digits and we wanted our handwritten uh digits and we wanted our handwritten uh digits and we wanted our VA to predict VA to predict VA to predict first of all first of all first of all predict this underlying data predict this underlying data predict this underlying data distribution and after you predict you distribution and after you predict you distribution and after you predict you can also sample from uh the probability can also sample from uh the probability can also sample from uh the probability distribution. distribution. distribution. One of the main drawbacks of VAS we One of the main drawbacks of VAS we One of the main drawbacks of VAS we understood is that it produces outputs understood is that it produces outputs understood is that it produces outputs which are blurry which are blurry which are blurry and it requires training of both these and it requires training of both these and it requires training of both these neural networks parallelly neural networks parallelly neural networks parallelly which is computationally intensive. So VAS were the first class of methods So VAS were the first class of methods which could show that artificial which could show that artificial which could show that artificial intelligence can be used to produce

  57. L6 @ 226s

    intelligence can be used to produce intelligence can be used to produce images which look like they belong to images which look like they belong to images which look like they belong to the underlying data distribution. the underlying data distribution. the underlying data distribution. Okay. Now we proceeded ahead from VAS Okay. Now we proceeded ahead from VAS Okay. Now we proceeded ahead from VAS and we moved to this topic called as and we moved to this topic called as and we moved to this topic called as diffusion. Now diffusion was a very interesting Now diffusion was a very interesting topic because we started with a very topic because we started with a very topic because we started with a very nice intuition about diffusion that if nice intuition about diffusion that if nice intuition about diffusion that if you spray a perfume in one corner of the you spray a perfume in one corner of the you spray a perfume in one corner of the room the molecules are going to diffuse room the molecules are going to diffuse room the molecules are going to diffuse out throughout the room. So the smell is out throughout the room. So the smell is out throughout the room. So the smell is going to percolate from one corner of going to percolate from one corner of going to percolate from one corner of the room to another corner of the room. the room to another corner of the room. the room to another corner of the room. This is the concept of diffusion and This is the concept of diffusion and This is the concept of diffusion and there are two main pillars which kind of there are two main pillars which kind of there are two main pillars which kind of govern this concept. The first is that govern this concept. The first is that govern this concept. The first is that the distribution becomes uniform and uh the distribution becomes uniform and uh the distribution becomes uniform and uh all you have is noise at the end. The all you have is noise at the end. The all you have is noise at the end. The original structure slowly disappears and original structure slowly disappears and original structure slowly disappears and it becomes uniform. it becomes uniform.

  58. L6 @ 282s

    it becomes uniform. We took these key ideas and we We took these key ideas and we We took these key ideas and we constructed a diffusion model which is constructed a diffusion model which is constructed a diffusion model which is inspired from the variational inspired from the variational inspired from the variational autoenccoder. autoenccoder. autoenccoder. Now how do we construct something that Now how do we construct something that Now how do we construct something that is inspired from the variation is inspired from the variation is inspired from the variation autoenccoder? First of all you replace autoenccoder? First of all you replace autoenccoder? First of all you replace encoder by something which is let's say encoder by something which is let's say encoder by something which is let's say called a diffuser. called a diffuser. called a diffuser. That is it takes an input data. Let's That is it takes an input data. Let's That is it takes an input data. Let's say we take an image of a house say we take an image of a house say we take an image of a house and in a series of steps it gradually and in a series of steps it gradually and in a series of steps it gradually corrupts the input data. It adds noise corrupts the input data. It adds noise corrupts the input data. It adds noise in each of these step and finally what in each of these step and finally what in each of these step and finally what we get is complete noise. So this is called as the forward So this is called as the forward process. We looked at exactly how this noise is We looked at exactly how this noise is is injected. What is the mathematical is injected. What is the mathematical is injected. What is the mathematical formulation of this noise and how you formulation of this noise and how you formulation of this noise and how you can convert any image to something where can convert any image to something where can convert any image to something where the structure slowly disappears and it the structure slowly disappears and it the structure slowly disappears and it becomes uniform. becomes uniform. becomes uniform. So you inject noise at every transition. So you inject noise at every transition.

  59. L6 @ 349s

    So you inject noise at every transition. So this is the first transition, second, So this is the first transition, second, So this is the first transition, second, third etc. third etc. third etc. Now this is the encoder which is fixed. Now this is the encoder which is fixed. Now this is the encoder which is fixed. This is very different from the This is very different from the This is very different from the variational autoenccoder where we learn variational autoenccoder where we learn variational autoenccoder where we learn the parameters of the encoder and the the parameters of the encoder and the the parameters of the encoder and the decoder. But in this case the encoder decoder. But in this case the encoder decoder. But in this case the encoder parameters are completely fixed. parameters are completely fixed. parameters are completely fixed. Now the next process is that of the Now the next process is that of the Now the next process is that of the decoder. So you need to take the noise decoder. So you need to take the noise decoder. So you need to take the noise and produce the given image from the and produce the given image from the and produce the given image from the noise. noise. noise. Now this itself looks like it is a bit Now this itself looks like it is a bit Now this itself looks like it is a bit challenging but we realize that the challenging but we realize that the challenging but we realize that the final formula final formula final formula in this diffusion model comes to a very in this diffusion model comes to a very in this diffusion model comes to a very simple concept. Let's say the noise that simple concept. Let's say the noise that simple concept. Let's say the noise that we have injected is epsilon and in the we have injected is epsilon and in the we have injected is epsilon and in the reverse process we are trying to predict reverse process we are trying to predict reverse process we are trying to predict the noise through our neural network the noise through our neural network the noise through our neural network which is epsilon hat. which is epsilon hat. which is epsilon hat. So our objective is to minimize the mean

  60. L6 @ 405s

    So our objective is to minimize the mean So our objective is to minimize the mean square error between epsilon and epsilon square error between epsilon and epsilon square error between epsilon and epsilon hat for all these different points in the for all these different points in the transition process. For example, if we transition process. For example, if we transition process. For example, if we take this transition and we ask the take this transition and we ask the take this transition and we ask the question that for this image, if I want question that for this image, if I want question that for this image, if I want to reconstruct the original image, how to reconstruct the original image, how to reconstruct the original image, how much noise should I remove from this much noise should I remove from this much noise should I remove from this image? And the noise that you remove image? And the noise that you remove image? And the noise that you remove should match very close to the noise should match very close to the noise should match very close to the noise which is injected. So essentially this which is injected. So essentially this which is injected. So essentially this is also can be called as a noise is also can be called as a noise is also can be called as a noise predictor. And uh the reason this works is because And uh the reason this works is because let's say you you consider any image in let's say you you consider any image in let's say you you consider any image in the world you apply the forward process. the world you apply the forward process. the world you apply the forward process. So every single image in the world can So every single image in the world can So every single image in the world can be represented as noise. And this is be represented as noise. And this is be represented as noise. And this is something which was quite interesting. something which was quite interesting. something which was quite interesting. We we asked the question that what is We we asked the question that what is We we asked the question that what is something which is common in every something which is common in every

  61. L6 @ 461s

    something which is common in every single image on earth. And at a first single image on earth. And at a first single image on earth. And at a first glance you're not able to answer it glance you're not able to answer it glance you're not able to answer it because the images look so different. because the images look so different. because the images look so different. Right? A laptop looks different, a chair Right? A laptop looks different, a chair Right? A laptop looks different, a chair looks different. But every single image looks different. But every single image looks different. But every single image can be reconstructed back from noise if can be reconstructed back from noise if can be reconstructed back from noise if you know how much noise to remove you know how much noise to remove you know how much noise to remove at every step. So that finally you get at every step. So that finally you get at every step. So that finally you get the actual image. Now instead of the actual image. Now instead of the actual image. Now instead of removing noise in one single step, if we removing noise in one single step, if we removing noise in one single step, if we remove the noise step by step, we can remove the noise step by step, we can remove the noise step by step, we can have a granular access to the reverse have a granular access to the reverse have a granular access to the reverse transition process. transition process. transition process. And this granular access to the reverse And this granular access to the reverse And this granular access to the reverse transition process helps us to train the transition process helps us to train the transition process helps us to train the neural network properly. neural network properly. neural network properly. We looked at an example where we could We looked at an example where we could We looked at an example where we could generate these images very nicely and generate these images very nicely and generate these images very nicely and diffusion models were a big revolution diffusion models were a big revolution diffusion models were a big revolution in the field of image generation through in the field of image generation through

  62. L6 @ 516s

    in the field of image generation through generative AI. Models were extremely generative AI. Models were extremely generative AI. Models were extremely important in creating a new wave in the important in creating a new wave in the important in creating a new wave in the image generation process and uh it was image generation process and uh it was image generation process and uh it was something which offered a very credible something which offered a very credible something which offered a very credible alternative to variational alternative to variational alternative to variational autoenccoders. Uh okay. So having understood both Uh okay. So having understood both variational autoenccoders and diffusion variational autoenccoders and diffusion variational autoenccoders and diffusion models, we move to another important models, we move to another important models, we move to another important concept which as we will look at it will concept which as we will look at it will concept which as we will look at it will closely intersect with the diffusion closely intersect with the diffusion closely intersect with the diffusion framework. However, before looking at framework. However, before looking at framework. However, before looking at that inter uh that that intersection, we that inter uh that that intersection, we that inter uh that that intersection, we should first understand about the should first understand about the should first understand about the concept independently. concept independently. concept independently. So the concept uh or the framework that So the concept uh or the framework that So the concept uh or the framework that we are going to learn today is called as we are going to learn today is called as we are going to learn today is called as energy based models energy based models energy based models and initially we will look at an and initially we will look at an and initially we will look at an independent analysis of energy based independent analysis of energy based independent analysis of energy based models and finally we will look at in in models and finally we will look at in in models and finally we will look at in in in detail how energy based models in detail how energy based models in detail how energy based models combine very nicely with diffusion combine very nicely with diffusion combine very nicely with diffusion models in a single cohesive framework.

  63. L6 @ 585s

    models in a single cohesive framework. models in a single cohesive framework. So this is something which is very So this is something which is very So this is something which is very unique and and very different to what unique and and very different to what unique and and very different to what people in the field of a IML are people in the field of a IML are people in the field of a IML are generally used to. We don't really work generally used to. We don't really work generally used to. We don't really work with energy based models that much. with energy based models that much. with energy based models that much. However, it is very important to However, it is very important to However, it is very important to understand the evolution of this thread. understand the evolution of this thread. understand the evolution of this thread. uh and when we understand the evolution uh and when we understand the evolution uh and when we understand the evolution of this thread properly, we will truly of this thread properly, we will truly of this thread properly, we will truly be be able to understand be be able to understand be be able to understand uh uh uh this the current state-of-the-art this the current state-of-the-art this the current state-of-the-art diffusion models and appreciate them as diffusion models and appreciate them as diffusion models and appreciate them as well. well. well. Okay. So, at the heart of energy based Okay. So, at the heart of energy based Okay. So, at the heart of energy based models models models is modeling probability distributions is modeling probability distributions is modeling probability distributions using energy functions. using energy functions. using energy functions. Remember our objective is to predict the Remember our objective is to predict the Remember our objective is to predict the probability distribution which matches probability distribution which matches probability distribution which matches as close as possible to the true data as close as possible to the true data as close as possible to the true data distribution. So let us denote the distribution. So let us denote the distribution. So let us denote the probability distribution that we are probability distribution that we are probability distribution that we are predicting as P5. predicting as P5. predicting as P5. So in the energy based models our

  64. L6 @ 648s

    So in the energy based models our So in the energy based models our objective is to predict P5 of X using objective is to predict P5 of X using objective is to predict P5 of X using energy functions. energy functions. energy functions. Now what exactly is this energy Now what exactly is this energy Now what exactly is this energy function? Let's let's try to understand function? Let's let's try to understand function? Let's let's try to understand it. it. it. Let this X Let this X Let this X be a data point be a data point be a data point and the energy function for X is defined and the energy function for X is defined and the energy function for X is defined as E5 of X. So let's let's take a So let's let's take a simple example where x which is your simple example where x which is your simple example where x which is your data points they take values from minus4 data points they take values from minus4 data points they take values from minus4 to +4 to +4 to +4 and the energy functions which is and the energy functions which is and the energy functions which is denoted as ei of x looks like this. denoted as ei of x looks like this. denoted as ei of x looks like this. It appears like a curve with a cusp at It appears like a curve with a cusp at It appears like a curve with a cusp at the bottom. the bottom. the bottom. Now at the first glance when I look at Now at the first glance when I look at Now at the first glance when I look at this curve what strikes me is that there this curve what strikes me is that there this curve what strikes me is that there are two points which are at the minima are two points which are at the minima are two points which are at the minima of of this curve. of of this curve. of of this curve. So So So imagine that you take a ball and you imagine that you take a ball and you imagine that you take a ball and you drop this ball in this curve. drop this ball in this curve.

  65. L6 @ 717s

    drop this ball in this curve. What will happen is that the ball will What will happen is that the ball will What will happen is that the ball will slowly settle down, slowly fall down and slowly settle down, slowly fall down and slowly settle down, slowly fall down and settle into let's say this configuration settle into let's say this configuration settle into let's say this configuration because it appears to be a valley. because it appears to be a valley. because it appears to be a valley. But if you push it even further, it will But if you push it even further, it will But if you push it even further, it will go ahead and it will fall into this go ahead and it will fall into this go ahead and it will fall into this valley. valley. valley. So the energy landscape has multiple So the energy landscape has multiple So the energy landscape has multiple valleys in this sample example that we valleys in this sample example that we valleys in this sample example that we have taken. have taken. have taken. But these valleys mean something very But these valleys mean something very But these valleys mean something very specific. specific. specific. So when we look at an energy landscape, So when we look at an energy landscape, So when we look at an energy landscape, the energies which are the minimum, the the energies which are the minimum, the the energies which are the minimum, the lowest energy configurations, lowest energy configurations, lowest energy configurations, these are the configurations which are these are the configurations which are these are the configurations which are the most preferable or preferred the most preferable or preferred the most preferable or preferred configurations. configurations. configurations. This is inspired from physics. Let's say This is inspired from physics. Let's say This is inspired from physics. Let's say we take an apple and I take a apple and we take an apple and I take a apple and we take an apple and I take a apple and I drop it. So why does the apple come I drop it. So why does the apple come I drop it. So why does the apple come down and settle on the ground? The down and settle on the ground? The

  66. L6 @ 777s

    down and settle on the ground? The reason is that at on on on the ground reason is that at on on on the ground reason is that at on on on the ground the potential energy of the apple is is the potential energy of the apple is is the potential energy of the apple is is minimum. And we apply a very similar minimum. And we apply a very similar minimum. And we apply a very similar sort of a logic here for our data sort of a logic here for our data sort of a logic here for our data points. What we say is that if we are points. What we say is that if we are points. What we say is that if we are able to pinpoint the energy function for able to pinpoint the energy function for able to pinpoint the energy function for all the data points which are given to all the data points which are given to all the data points which are given to me, I can get a sense of how probable me, I can get a sense of how probable me, I can get a sense of how probable that each sample is. For example, that each sample is. For example, that each sample is. For example, here I can say that the probability of x here I can say that the probability of x here I can say that the probability of x = -4 = -4 = -4 is much less than the probability of x = is much less than the probability of x = is much less than the probability of x = -1.5 -1.5 -1.5 because the energy of this is because the energy of this is because the energy of this is considerably less compared to the energy considerably less compared to the energy considerably less compared to the energy of this. of this. of this. So the energy function becomes a proxy So the energy function becomes a proxy So the energy function becomes a proxy for us to comment about the preference for us to comment about the preference for us to comment about the preference or the probability of the actions. Okay, now that we have seen how the Okay, now that we have seen how the energy function can be a proxy for how

  67. L6 @ 835s

    energy function can be a proxy for how energy function can be a proxy for how probable that particular data point or probable that particular data point or probable that particular data point or sample is, let us dig even deeper. How sample is, let us dig even deeper. How sample is, let us dig even deeper. How do we convert this energy landscape into do we convert this energy landscape into do we convert this energy landscape into a set of probabilities? a set of probabilities? a set of probabilities? Okay, so first of all, the points which Okay, so first of all, the points which Okay, so first of all, the points which lower energy should have higher lower energy should have higher lower energy should have higher probability probability probability and the points with higher energy should and the points with higher energy should and the points with higher energy should have lower probability. This much we have lower probability. This much we have lower probability. This much we understand from the physical intuition understand from the physical intuition understand from the physical intuition that we just developed. So the that we just developed. So the that we just developed. So the probability curve should look something probability curve should look something probability curve should look something like this. Here it should have high like this. Here it should have high like this. Here it should have high values values values and here it should have high values. So and here it should have high values. So and here it should have high values. So something like this. something like this. something like this. So you can see that these are the exact So you can see that these are the exact So you can see that these are the exact same points which correspond to the same points which correspond to the same points which correspond to the energy minima. Now if you superimpose both these curves Now if you superimpose both these curves you you get something like this. you you get something like this. you you get something like this. Now the question is that uh we can of Now the question is that uh we can of Now the question is that uh we can of course draw this schematically and we course draw this schematically and we

  68. L6 @ 892s

    course draw this schematically and we can say that okay we want this energy or can say that okay we want this energy or can say that okay we want this energy or or this probability curve to follow or this probability curve to follow or this probability curve to follow these rules that it should be inversely these rules that it should be inversely these rules that it should be inversely proportional to the energy at at that proportional to the energy at at that proportional to the energy at at that point. But when we are trying to develop point. But when we are trying to develop point. But when we are trying to develop any framework we need to have the exact any framework we need to have the exact any framework we need to have the exact mathematical formulation for any mathematical formulation for any mathematical formulation for any transformation. So here the transformation. So here the transformation. So here the transformation is converting from energy transformation is converting from energy transformation is converting from energy to probability and you need to pinpoint to probability and you need to pinpoint to probability and you need to pinpoint exactly how you are going to exactly how you are going to exactly how you are going to complete this transformation. complete this transformation. complete this transformation. So let's let's try to understand can we So let's let's try to understand can we So let's let's try to understand can we think of a mathematical function which think of a mathematical function which think of a mathematical function which can take us from this energy curve to can take us from this energy curve to can take us from this energy curve to the probability curve. the probability curve. the probability curve. The function should have higher higher The function should have higher higher The function should have higher higher energy with for lower probabilities energy with for lower probabilities energy with for lower probabilities lower energy for higher probabilities lower energy for higher probabilities lower energy for higher probabilities and it should have only positive values. and it should have only positive values. and it should have only positive values. So these are the three criterias which So these are the three criterias which So these are the three criterias which my function should satisfy

  69. L6 @ 951s

    my function should satisfy my function should satisfy and people use an exponential function and people use an exponential function and people use an exponential function to relate the energy to the probability to relate the energy to the probability to relate the energy to the probability because it satisfies all these pro all because it satisfies all these pro all because it satisfies all these pro all these properties. these properties. these properties. So here you can see I have taken a So here you can see I have taken a So here you can see I have taken a example of the exponential function e to example of the exponential function e to example of the exponential function e to minus x. minus x. minus x. So the value of this function at x=0 is So the value of this function at x=0 is So the value of this function at x=0 is 1 and it gradually decreases as you 1 and it gradually decreases as you 1 and it gradually decreases as you increase the value of x. increase the value of x. increase the value of x. So for higher So for higher So for higher energy levels you can see the energy levels you can see the energy levels you can see the probability is minimum whereas for lower probability is minimum whereas for lower probability is minimum whereas for lower energy levels the probability is maximum energy levels the probability is maximum energy levels the probability is maximum and the probability value should lie and the probability value should lie and the probability value should lie between 0 and one. We should add this between 0 and one. We should add this between 0 and one. We should add this this point also should lie between 0 and this point also should lie between 0 and this point also should lie between 0 and one which also the exponential function one which also the exponential function one which also the exponential function very nicely satisfies. Okay. So this is the exponential curve Okay. So this is the exponential curve which we will use as the transformation which we will use as the transformation which we will use as the transformation from the energies to the probabilities. Now let's say we denote the probability Now let's say we denote the probability density as p5 of x.

  70. L6 @ 1023s

    density as p5 of x. density as p5 of x. Then from the above graph we can find Then from the above graph we can find Then from the above graph we can find out a function which represents the out a function which represents the out a function which represents the uh uh energy and the probability relation. So energy and the probability relation. So energy and the probability relation. So let's let's try to understand this. So let's let's try to understand this. So let's let's try to understand this. So the probability is given as P5 of X and the probability is given as P5 of X and the probability is given as P5 of X and the energy is given as E5 of X. So this the energy is given as E5 of X. So this the energy is given as E5 of X. So this is how we relate the energy function to is how we relate the energy function to is how we relate the energy function to the probability. So now if you give me the probability. So now if you give me the probability. So now if you give me any value for the energy function, I can any value for the energy function, I can any value for the energy function, I can give you exactly what is the probability give you exactly what is the probability give you exactly what is the probability for that data sample for that data sample for that data sample u to be sampled from that from from the u to be sampled from that from from the u to be sampled from that from from the given distribution. given distribution. given distribution. Okay. So this is a great starting point. Okay. So this is a great starting point. Okay. So this is a great starting point. uh we have started with energy functions uh we have started with energy functions uh we have started with energy functions which derives its its intuition from which derives its its intuition from which derives its its intuition from physics. In physics if you look at any physics. In physics if you look at any physics. In physics if you look at any system u every system finally tries to system u every system finally tries to

  71. L6 @ 1076s

    system u every system finally tries to go in a position with the minimum go in a position with the minimum go in a position with the minimum possible energy and this is something possible energy and this is something possible energy and this is something which is quite well known. People use it which is quite well known. People use it which is quite well known. People use it to derive formulas for the variation for to derive formulas for the variation for to derive formulas for the variation for different quantities. different quantities. different quantities. Now here we are going to use a very Now here we are going to use a very Now here we are going to use a very similar principle but we are going to similar principle but we are going to similar principle but we are going to use that principle to connect the use that principle to connect the use that principle to connect the energies to the probabilities of that energies to the probabilities of that energies to the probabilities of that individual data points. For example, if individual data points. For example, if individual data points. For example, if you give me 100 images of cats you give me 100 images of cats you give me 100 images of cats and if you tell me pick one image and and if you tell me pick one image and and if you tell me pick one image and tell me what is the probability of tell me what is the probability of tell me what is the probability of finding this image finding this image finding this image in your given data set. Now to do that in your given data set. Now to do that in your given data set. Now to do that I'll first look at okay what's the I'll first look at okay what's the I'll first look at okay what's the energy function value for this image and energy function value for this image and energy function value for this image and then I'll calculate the probability for then I'll calculate the probability for then I'll calculate the probability for that using the exponential curve that is that using the exponential curve that is that using the exponential curve that is the whole idea behind energy based the whole idea behind energy based

  72. L6 @ 1126s

    the whole idea behind energy based models models okay now let us discuss after this point okay now let us discuss after this point okay now let us discuss after this point let's take an example suppose we have a let's take an example suppose we have a let's take an example suppose we have a set of discrete states which are -3 -2 set of discrete states which are -3 -2 set of discrete states which are -3 -2 -1 0 1 2 and 3 let's Say we have seven -1 0 1 2 and 3 let's Say we have seven -1 0 1 2 and 3 let's Say we have seven values. values. values. Okay. And now let us say that we use the Okay. And now let us say that we use the Okay. And now let us say that we use the above formula and we calculate the above formula and we calculate the above formula and we calculate the probability densities for all these probability densities for all these probability densities for all these states. Let's say we know the energy states. Let's say we know the energy states. Let's say we know the energy values for all these states and then we values for all these states and then we values for all these states and then we calculate the probability densities. So calculate the probability densities. So calculate the probability densities. So the probability density for the probability density for the probability density for minus3 is here -2 -1 0 minus3 is here -2 -1 0 minus3 is here -2 -1 0 1 2 and 3. 1 2 and 3. 1 2 and 3. Here I have used the formula probability Here I have used the formula probability Here I have used the formula probability of x = e minus e of x. The problem with this method is that the The problem with this method is that the probabilities do not add up to one. probabilities do not add up to one. probabilities do not add up to one. And why do we need the probability to And why do we need the probability to And why do we need the probability to add up to one? We need the probability add up to one? We need the probability

  73. L6 @ 1200s

    add up to one? We need the probability to add up to one because if we are to add up to one because if we are to add up to one because if we are creating a lot of samples and we are creating a lot of samples and we are creating a lot of samples and we are creating a probability distribution for creating a probability distribution for creating a probability distribution for all those samples, all those samples, all those samples, the probability distribution should the probability distribution should the probability distribution should satisfy the addition of all satisfy the addition of all satisfy the addition of all probabilities to be one. Imagine you uh probabilities to be one. Imagine you uh probabilities to be one. Imagine you uh you have a coin and the coin has two you have a coin and the coin has two you have a coin and the coin has two faces. It can either faces. It can either faces. It can either come up as heads or it can come up as come up as heads or it can come up as come up as heads or it can come up as tails. So the probability for each side tails. So the probability for each side tails. So the probability for each side is 0.5 and.5. It adds up to one. Let's is 0.5 and.5. It adds up to one. Let's is 0.5 and.5. It adds up to one. Let's take an example of a dice. The take an example of a dice. The take an example of a dice. The probability of rolling the dice and probability of rolling the dice and probability of rolling the dice and getting the number 1 is 1x 6. Similarly, getting the number 1 is 1x 6. Similarly, getting the number 1 is 1x 6. Similarly, getting the number 2 is 1x 6. If you add getting the number 2 is 1x 6. If you add getting the number 2 is 1x 6. If you add all these probabilities 1x 6 + 2x 6 + 3x all these probabilities 1x 6 + 2x 6 + 3x all these probabilities 1x 6 + 2x 6 + 3x 6 + 4x 6. 6 + 4x 6. 6 + 4x 6. Uh Uh Uh so let's let's take that example again.

  74. L6 @ 1252s

    so let's let's take that example again. so let's let's take that example again. Uh let's say you have a dice. Okay. Uh Uh let's say you have a dice. Okay. Uh Uh let's say you have a dice. Okay. Uh and and the probability of finding one and and the probability of finding one and and the probability of finding one is 1x 6. The probability of getting two is 1x 6. The probability of getting two is 1x 6. The probability of getting two is 1x 6. I think I made an error by is 1x 6. I think I made an error by is 1x 6. I think I made an error by saying the probability is 2x 6. saying the probability is 2x 6. saying the probability is 2x 6. Similarly, the probability of getting Similarly, the probability of getting Similarly, the probability of getting the number six is 1x 6 and you add this the number six is 1x 6 and you add this the number six is 1x 6 and you add this up together you get the number one. So up together you get the number one. So up together you get the number one. So this is a very fundamental property this is a very fundamental property this is a very fundamental property which any probability distribution which any probability distribution which any probability distribution should satisfy and it doesn't satisfy should satisfy and it doesn't satisfy should satisfy and it doesn't satisfy here in this case. The sum of all these here in this case. The sum of all these here in this case. The sum of all these probabilities is 25066. probabilities is 25066. probabilities is 25066. Now the question is that how do we Now the question is that how do we Now the question is that how do we convert this to a distribution which convert this to a distribution which convert this to a distribution which satisfies the addition of all these satisfies the addition of all these satisfies the addition of all these probabilities to be one. probabilities to be one. probabilities to be one. There is a very simple solution to this There is a very simple solution to this There is a very simple solution to this and the solution is that we can simply

  75. L6 @ 1302s

    and the solution is that we can simply and the solution is that we can simply normalize the probabilities by dividing normalize the probabilities by dividing normalize the probabilities by dividing it by the summation of the weights. it by the summation of the weights. it by the summation of the weights. For example, For example, For example, here what I have done is I have taken here what I have done is I have taken here what I have done is I have taken these same values for all these discrete these same values for all these discrete these same values for all these discrete states but I have divided it by 25066. states but I have divided it by 25066. states but I have divided it by 25066. Here you can see in this formula the Here you can see in this formula the Here you can see in this formula the formula which I'm using is = e^ minus e5 formula which I'm using is = e^ minus e5 formula which I'm using is = e^ minus e5 of x divided by 2.5066. Now if you sum up all these Now if you sum up all these probabilities they will definitely sum probabilities they will definitely sum probabilities they will definitely sum up to one. So our our requirement is is up to one. So our our requirement is is up to one. So our our requirement is is satisfied. satisfied. satisfied. So the number 2.566 is also called as So the number 2.566 is also called as So the number 2.566 is also called as the partition function. the partition function. the partition function. And the final relation between the And the final relation between the And the final relation between the energy and the probability density energy and the probability density energy and the probability density function then looks as follows. We are function then looks as follows. We are function then looks as follows. We are keeping the numerator intact exponential keeping the numerator intact exponential keeping the numerator intact exponential of minus e5 of x but we are dividing it of minus e5 of x but we are dividing it of minus e5 of x but we are dividing it by the partition function. by the partition function. by the partition function. So for the discrete states it looked

  76. L6 @ 1368s

    So for the discrete states it looked So for the discrete states it looked like this summation of e ra to e5 of x like this summation of e ra to e5 of x like this summation of e ra to e5 of x x goes from let's say -4 to 4. x goes from let's say -4 to 4. x goes from let's say -4 to 4. But if you have a continuous state then But if you have a continuous state then But if you have a continuous state then usually people represent this in the usually people represent this in the usually people represent this in the form of this integral symbol. form of this integral symbol. form of this integral symbol. And uh this is the partition function And uh this is the partition function And uh this is the partition function which represents the summation of which represents the summation of which represents the summation of all the exponentials in your data set. all the exponentials in your data set. all the exponentials in your data set. So this is what we want to predict. So this is what we want to predict. So this is what we want to predict. We want something which can we want the We want something which can we want the We want something which can we want the framework which can give us this p5 of x framework which can give us this p5 of x framework which can give us this p5 of x and uh this this looks great right we and uh this this looks great right we and uh this this looks great right we have expressed p5 of x in terms of the have expressed p5 of x in terms of the have expressed p5 of x in terms of the energy functions for all these discrete energy functions for all these discrete energy functions for all these discrete states or continuous states. Now the states or continuous states. Now the states or continuous states. Now the task boils down to how do we train these task boils down to how do we train these task boils down to how do we train these energy based models. In other words, how energy based models. In other words, how

  77. L6 @ 1426s

    energy based models. In other words, how do we find these energy functions which do we find these energy functions which do we find these energy functions which correspond to each of these states? correspond to each of these states? correspond to each of these states? So that is the problem which we want to So that is the problem which we want to So that is the problem which we want to address in in this section. Okay. So conceptually we want to do Okay. So conceptually we want to do something like this. something like this. Let's say Let's say Let's say you have on the x-axis you have your data samples which have been given your data samples which have been given to you and on the y-axis you have these to you and on the y-axis you have these to you and on the y-axis you have these probabilities. probabilities. probabilities. So this is the data which is good data So this is the data which is good data So this is the data which is good data which is drawn from your true which is drawn from your true which is drawn from your true probability distribution and this is the probability distribution and this is the probability distribution and this is the bad data. bad data. bad data. Now Now Now if this is my probability curve, what if this is my probability curve, what if this is my probability curve, what can I infer from this curve? can I infer from this curve? can I infer from this curve? The first thing I can infer is that the The first thing I can infer is that the The first thing I can infer is that the probability for good data probability for good data probability for good data is less than the probability for bad is less than the probability for bad is less than the probability for bad data, data, data, which is the opposite of what I want. which is the opposite of what I want. which is the opposite of what I want. Ideally, I want to push the bad data Ideally, I want to push the bad data Ideally, I want to push the bad data down, push the probability down, and down, push the probability down, and down, push the probability down, and push the good data up.

  78. L6 @ 1498s

    push the good data up. push the good data up. So I want to move this curve like this. So I want to move this curve like this. So I want to move this curve like this. The left side I want to push down and The left side I want to push down and The left side I want to push down and the right side I want to push up. This the right side I want to push up. This the right side I want to push up. This is exactly what the training of energy is exactly what the training of energy is exactly what the training of energy based model achieves. Here you can see based model achieves. Here you can see based model achieves. Here you can see uh after the training is completed. uh after the training is completed. uh after the training is completed. This is the original curve and this is This is the original curve and this is This is the original curve and this is the curve which we have after the the curve which we have after the the curve which we have after the training is completed. training is completed. training is completed. So we can clearly see that the So we can clearly see that the So we can clearly see that the probability for the bad data has been probability for the bad data has been probability for the bad data has been reduced and the probability for the good reduced and the probability for the good reduced and the probability for the good data has been increased. data has been increased. data has been increased. So schematically this looks fine and So schematically this looks fine and So schematically this looks fine and theoretically this looks great that we theoretically this looks great that we theoretically this looks great that we start with the random energy start with the random energy start with the random energy configuration and slowly modify the configuration and slowly modify the configuration and slowly modify the energy landscape so that bad data have energy landscape so that bad data have energy landscape so that bad data have lower probability and the good data have lower probability and the good data have lower probability and the good data have higher probability. All of us understand

  79. L6 @ 1549s

    higher probability. All of us understand higher probability. All of us understand this. this. this. Now we will again use the same Now we will again use the same Now we will again use the same formulation that we used for variational formulation that we used for variational formulation that we used for variational autoenccoders autoenccoders autoenccoders to to to understand the training of energy based understand the training of energy based understand the training of energy based models. Remember what we did with models. Remember what we did with models. Remember what we did with variational autoenccoders. We started variational autoenccoders. We started variational autoenccoders. We started out with maximizing this likelihood log out with maximizing this likelihood log out with maximizing this likelihood log of p5 of x. of p5 of x. of p5 of x. What this means is that What this means is that What this means is that if you take samples from the true if you take samples from the true if you take samples from the true distribution, let's say the samples distribution, let's say the samples distribution, let's say the samples which are given to you and you put it in which are given to you and you put it in which are given to you and you put it in your predicted distribution, you should your predicted distribution, you should your predicted distribution, you should get higher probability. This is exactly get higher probability. This is exactly get higher probability. This is exactly what we are doing here. This is the good what we are doing here. This is the good what we are doing here. This is the good data which we are talking about. So we data which we are talking about. So we data which we are talking about. So we want to maximize this want to maximize this want to maximize this which means we want to maximize this. We which means we want to maximize this. We which means we want to maximize this. We are simply substituting the formula for are simply substituting the formula for are simply substituting the formula for the probability based on the energy the probability based on the energy the probability based on the energy functions. functions. functions. Uh here E5 of X denotes the energy Uh here E5 of X denotes the energy

  80. L6 @ 1607s

    Uh here E5 of X denotes the energy function. Okay. So now if you break this down, Okay. So now if you break this down, this is the first term that you get. uh this is the first term that you get. uh this is the first term that you get. uh remember log of remember log of remember log of a by b is equal to log a a by b is equal to log a a by b is equal to log a minus log b. minus log b. minus log b. So this is the formula that I have used So this is the formula that I have used So this is the formula that I have used to uh make this simplification. Now once we reach this step we get to a Now once we reach this step we get to a stage where we have to calculate the log stage where we have to calculate the log stage where we have to calculate the log of this partition function which let's of this partition function which let's of this partition function which let's call it zed. call it zed. call it zed. The biggest problem with this The biggest problem with this The biggest problem with this formulation is that the partition formulation is that the partition formulation is that the partition function is intractable. It is function is intractable. It is function is intractable. It is impossible to calculate the partition impossible to calculate the partition impossible to calculate the partition function. function. function. And in fact we have faced a very similar And in fact we have faced a very similar And in fact we have faced a very similar issue. Uh do you remember how we face issue. Uh do you remember how we face issue. Uh do you remember how we face this issue while training the this issue while training the this issue while training the variational autoenccoder? variational autoenccoder? variational autoenccoder? What we said was if you want to What we said was if you want to What we said was if you want to calculate this calculate this calculate this we want to maximize we want to maximize we want to maximize so p5 of x can be written as integral of so p5 of x can be written as integral of

  81. L6 @ 1680s

    so p5 of x can be written as integral of p5 of x given z into p of z dz. p5 of x given z into p of z dz. p5 of x given z into p of z dz. And there we said that this integral is And there we said that this integral is And there we said that this integral is not possible to calculate. not possible to calculate. not possible to calculate. And uh what was the solution? How did we And uh what was the solution? How did we And uh what was the solution? How did we get over this? There was a very simple get over this? There was a very simple get over this? There was a very simple trick that we did. And uh the trick that trick that we did. And uh the trick that trick that we did. And uh the trick that we did was we defined an objective which we did was we defined an objective which we did was we defined an objective which is always lower than this maximum is always lower than this maximum is always lower than this maximum likelihood. likelihood. likelihood. And we call that objective as elbow. And we call that objective as elbow. And we call that objective as elbow. So instead of maximizing this which is So instead of maximizing this which is So instead of maximizing this which is non-tractable, we maximize the elbow non-tractable, we maximize the elbow non-tractable, we maximize the elbow which is tractable. And in energy based methods And in energy based methods we actually do not use the elbow term. we actually do not use the elbow term. we actually do not use the elbow term. We do not use the elbow approach We do not use the elbow approach We do not use the elbow approach but instead we introduce the notion of but instead we introduce the notion of but instead we introduce the notion of something called uh something called uh something called uh score function and we use a theory score function and we use a theory score function and we use a theory called score matching which completely called score matching which completely called score matching which completely bypasses the partition function. bypasses the partition function.

  82. L6 @ 1740s

    bypasses the partition function. So you might be thinking that what is So you might be thinking that what is So you might be thinking that what is this new terminology which I've this new terminology which I've this new terminology which I've introduced. What is this core matching? introduced. What is this core matching? introduced. What is this core matching? It looks like we are It looks like we are It looks like we are defining a score function, right? But defining a score function, right? But defining a score function, right? But haven't I already introduced the energy haven't I already introduced the energy haven't I already introduced the energy function? What does the score function function? What does the score function function? What does the score function even mean? even mean? even mean? So this is something which we will So this is something which we will So this is something which we will understand in the next section in this understand in the next section in this understand in the next section in this lecture. lecture. lecture. But as of now what we need to understand But as of now what we need to understand But as of now what we need to understand is to design the energy based models is to design the energy based models is to design the energy based models framework. We first find the relation framework. We first find the relation framework. We first find the relation between the energy and the probability between the energy and the probability between the energy and the probability that we have successfully done. Next we that we have successfully done. Next we that we have successfully done. Next we try to maximize the log likelihood of try to maximize the log likelihood of try to maximize the log likelihood of the data. the data. the data. But when we expand the term we encounter But when we expand the term we encounter But when we expand the term we encounter a term which a term which a term which includes the integral of the partition includes the integral of the partition includes the integral of the partition function which is impossible to function which is impossible to function which is impossible to calculate. calculate. calculate. And because of that we come up with a And because of that we come up with a

  83. L6 @ 1796s

    And because of that we come up with a alternative objective function which alternative objective function which alternative objective function which uses something called as the score uses something called as the score uses something called as the score function. So let us understand what is function. So let us understand what is function. So let us understand what is this score function and uh this score function and uh this score function and uh how does it get rid of the partition how does it get rid of the partition how does it get rid of the partition function. function. Okay. So let's look at it. Okay. So let's look at it. Okay. So let's look at it. Firstly, this this whole challenge of Firstly, this this whole challenge of Firstly, this this whole challenge of calculating an integral which is calculating an integral which is calculating an integral which is intractable and coming up with a intractable and coming up with a intractable and coming up with a solution. solution. solution. This has been a a common challenge in This has been a a common challenge in This has been a a common challenge in almost all the deep generative models almost all the deep generative models almost all the deep generative models and it and it and it uh it is mathematicians who help us to uh it is mathematicians who help us to uh it is mathematicians who help us to come out of this challenge and design an come out of this challenge and design an come out of this challenge and design an objective function which is tractable objective function which is tractable objective function which is tractable which which which clearly clearly clearly shows that mathematics is extremely shows that mathematics is extremely shows that mathematics is extremely important and it is the foundations important and it is the foundations important and it is the foundations behind all of these models for image generation that that we models for image generation that that we see around us. So whenever I look at see around us. So whenever I look at see around us. So whenever I look at this I I I have a deep sense of this I I I have a deep sense of this I I I have a deep sense of appreciation for people who actually sit

  84. L6 @ 1865s

    appreciation for people who actually sit appreciation for people who actually sit down, they do the mathematics and they down, they do the mathematics and they down, they do the mathematics and they come up with these objective functions. come up with these objective functions. come up with these objective functions. Okay. So what is the score and how do we Okay. So what is the score and how do we Okay. So what is the score and how do we calculate the score? Let's let's try to calculate the score? Let's let's try to calculate the score? Let's let's try to understand that in detail. Okay. So for a density P of X, the score Okay. So for a density P of X, the score function has a formula which is given by function has a formula which is given by function has a formula which is given by gradient of the log of P of X. And this might be a bit difficult for us And this might be a bit difficult for us to understand because there are two to understand because there are two to understand because there are two terms involved here. There is a gradient terms involved here. There is a gradient terms involved here. There is a gradient term which is involved here and there is term which is involved here and there is term which is involved here and there is a log term which is involved here. a log term which is involved here. a log term which is involved here. But this score function actually means But this score function actually means But this score function actually means something very intuitive and that is why something very intuitive and that is why something very intuitive and that is why it is fun to work with. it is fun to work with. it is fun to work with. Intuitively the score forms a vector Intuitively the score forms a vector Intuitively the score forms a vector field that points towards regions of field that points towards regions of field that points towards regions of higher probability. So it gives a local higher probability. So it gives a local higher probability. So it gives a local guide to where the data is most likely guide to where the data is most likely guide to where the data is most likely to occur.

  85. L6 @ 1937s

    to occur. to occur. Let's let let's take a sample example. Let's let let's take a sample example. Let's let let's take a sample example. Let us say these black dots these are Let us say these black dots these are Let us say these black dots these are your data samples. So the color uh the color orange So the color uh the color orange indicates that the probability of indicates that the probability of indicates that the probability of finding the samples is very high in this finding the samples is very high in this finding the samples is very high in this region and in this region. Now apart from this you see arrows all Now apart from this you see arrows all around the place. These arrows are the around the place. These arrows are the around the place. These arrows are the score function for every single point. score function for every single point. score function for every single point. For this point this is the score For this point this is the score For this point this is the score function. For this point this is the function. For this point this is the function. For this point this is the score function. And you can see that the score function. And you can see that the score function. And you can see that the score function is actually pointing us score function is actually pointing us score function is actually pointing us towards where the data is. Imagine you towards where the data is. Imagine you towards where the data is. Imagine you are moving out in the open sea and you are moving out in the open sea and you are moving out in the open sea and you want to find some sharks. And imagine want to find some sharks. And imagine want to find some sharks. And imagine you have a compass which points to the you have a compass which points to the you have a compass which points to the direction. Oh, there is the shark. I direction. Oh, there is the shark. I direction. Oh, there is the shark. I need to go there. This is exactly what need to go there. This is exactly what need to go there. This is exactly what score function does. It tells you the

  86. L6 @ 1998s

    score function does. It tells you the score function does. It tells you the direction to where the data is located. direction to where the data is located. direction to where the data is located. So in a way it is incredibly useful, So in a way it is incredibly useful, So in a way it is incredibly useful, right? right? right? uh and we'll we'll try to use this score uh and we'll we'll try to use this score uh and we'll we'll try to use this score function to actually train our model. function to actually train our model. function to actually train our model. But this is the idea of score function. But this is the idea of score function. But this is the idea of score function. So in the above figure the arrows So in the above figure the arrows So in the above figure the arrows represent the score field which are represent the score field which are represent the score field which are pointed towards the direction where the pointed towards the direction where the pointed towards the direction where the density of the data is the maximum. density of the data is the maximum. density of the data is the maximum. And uh to imagine this happening for a And uh to imagine this happening for a And uh to imagine this happening for a bunch of let's say we have images of bunch of let's say we have images of bunch of let's say we have images of cats and we find the score function for cats and we find the score function for cats and we find the score function for that. So the cat data itself is let's that. So the cat data itself is let's that. So the cat data itself is let's say in 784 dimensions. say in 784 dimensions. say in 784 dimensions. So humans cannot even comprehend this So humans cannot even comprehend this So humans cannot even comprehend this right? We cannot imagine but intuitively right? We cannot imagine but intuitively right? We cannot imagine but intuitively we understand that at every point if we we understand that at every point if we we understand that at every point if we know where the data is where the score know where the data is where the score

  87. L6 @ 2049s

    know where the data is where the score is the maximum is the maximum is the maximum we can start from any point and finally we can start from any point and finally we can start from any point and finally we will land up at the sample we will we will land up at the sample we will we will land up at the sample we will find the sample finally find the sample finally find the sample finally or we will find data points which are or we will find data points which are or we will find data points which are most likely to be sampled from the true most likely to be sampled from the true most likely to be sampled from the true probability distribution. probability distribution. So imagine that you are moving out in an So imagine that you are moving out in an So imagine that you are moving out in an open sea and uh you want to find these open sea and uh you want to find these open sea and uh you want to find these sharks right which are our data. So you sharks right which are our data. So you sharks right which are our data. So you move you find the score move in the move you find the score move in the move you find the score move in the direction again find the score and direction again find the score and direction again find the score and slowly you move and then you finally slowly you move and then you finally slowly you move and then you finally land up in an area where you find the land up in an area where you find the land up in an area where you find the actual sample. So for example, if I actual sample. So for example, if I actual sample. So for example, if I start here, I can slowly move towards start here, I can slowly move towards start here, I can slowly move towards the area where I find the samples. the area where I find the samples. the area where I find the samples. Now, but but how how do you move in this Now, but but how how do you move in this Now, but but how how do you move in this trajectory? trajectory?

  88. L6 @ 2098s

    trajectory? Is is is it can I define it? Can I write Is is is it can I define it? Can I write Is is is it can I define it? Can I write an equation which can tell me how do I an equation which can tell me how do I an equation which can tell me how do I move in this trajectory? Let's let's try move in this trajectory? Let's let's try move in this trajectory? Let's let's try to unpack that as well in in this to unpack that as well in in this to unpack that as well in in this lecture. lecture. But before that, we will take uh the But before that, we will take uh the But before that, we will take uh the simplest example possible. simplest example possible. simplest example possible. Let's assume that our probability Let's assume that our probability Let's assume that our probability density curve is the goshian. density curve is the goshian. density curve is the goshian. What does this mean? What does this mean? What does this mean? This means that the probability of x= This means that the probability of x= This means that the probability of x= to0 is the maximum and the probability to0 is the maximum and the probability to0 is the maximum and the probability dies off as you go towards the right or dies off as you go towards the right or dies off as you go towards the right or the left. This is the standard normal the left. This is the standard normal the left. This is the standard normal gshian distribution. We have seen this gshian distribution. We have seen this gshian distribution. We have seen this distribution in the distribution in the distribution in the forward transition process in the DDPM forward transition process in the DDPM forward transition process in the DDPM framework and also in the reverse framework and also in the reverse framework and also in the reverse transition process. transition process. Okay. So this is the standard normal Okay. So this is the standard normal Okay. So this is the standard normal distribution which is great. Now distribution which is great. Now distribution which is great. Now intuitively what do you expect? intuitively what do you expect? intuitively what do you expect? Let's say I calculate the score for this

  89. L6 @ 2158s

    Let's say I calculate the score for this Let's say I calculate the score for this point. Where will this score point point. Where will this score point point. Where will this score point towards to this score will the score towards to this score will the score towards to this score will the score function for this point will be might be function for this point will be might be function for this point will be might be something like this. Right? because it something like this. Right? because it something like this. Right? because it points towards the center points towards the center points towards the center which has the maximum probability. which has the maximum probability. which has the maximum probability. Similarly, the score function for this Similarly, the score function for this Similarly, the score function for this data point data point data point would probably look something like this. would probably look something like this. would probably look something like this. It it points towards the data point It it points towards the data point It it points towards the data point which has the maximum probability which has the maximum probability which has the maximum probability and uh I I'll probably get arrows which and uh I I'll probably get arrows which and uh I I'll probably get arrows which look like this. Remember this is just a look like this. Remember this is just a look like this. Remember this is just a one-dimensional data. So we have to plot one-dimensional data. So we have to plot one-dimensional data. So we have to plot our arrows on the x-axis. our arrows on the x-axis. our arrows on the x-axis. So let's see. So the mathematical form So let's see. So the mathematical form So let's see. So the mathematical form for the probability looks like this. You can see that if you substitute x= You can see that if you substitute x= to0 here you will get maximum value 1x to0 here you will get maximum value 1x to0 here you will get maximum value 1x <unk>2 pi and as you move towards the <unk>2 pi and as you move towards the <unk>2 pi and as you move towards the right or the left uh the value will right or the left uh the value will

  90. L6 @ 2213s

    right or the left uh the value will decline at x= to infinity or minus decline at x= to infinity or minus decline at x= to infinity or minus infinity the value becomes zero. infinity the value becomes zero. infinity the value becomes zero. Now let us calculate the score function. Now let us calculate the score function. Now let us calculate the score function. uh remember the formula for score uh remember the formula for score uh remember the formula for score function was gradient of log of p of x. function was gradient of log of p of x. function was gradient of log of p of x. So let's take the log of this. What is So let's take the log of this. What is So let's take the log of this. What is the log of this? Remember log of the log of this? Remember log of the log of this? Remember log of exponential e ra to any quantity let's exponential e ra to any quantity let's exponential e ra to any quantity let's say p is equal to p itself. say p is equal to p itself. say p is equal to p itself. So if I take the log of this I will get So if I take the log of this I will get So if I take the log of this I will get and and log of and and log of and and log of a a a is equal to log a + log b. So first of all I do that I split this So first of all I do that I split this logarithm into two terms. This is term logarithm into two terms. This is term logarithm into two terms. This is term one and this is term two. one and this is term two. one and this is term two. Now this is just some constant log of Now this is just some constant log of Now this is just some constant log of some constant value some constant value some constant value and this is - x square by 2. and this is - x square by 2. and this is - x square by 2. Now this is just the first term. We have Now this is just the first term. We have

  91. L6 @ 2275s

    Now this is just the first term. We have to take a gradient of this also. to take a gradient of this also. to take a gradient of this also. Gradient of a constant is zero and the Gradient of a constant is zero and the Gradient of a constant is zero and the gradient of this is minus x. gradient of this is minus x. gradient of this is minus x. So the score function that we get is So the score function that we get is So the score function that we get is simply minus x. Now what does this mean? simply minus x. Now what does this mean? simply minus x. Now what does this mean? This means that for x = +2 the score is This means that for x = +2 the score is This means that for x = +2 the score is minus2 which is exactly what we minus2 which is exactly what we minus2 which is exactly what we expected. At x= minus2 the score is +2 expected. At x= minus2 the score is +2 expected. At x= minus2 the score is +2 and the magnitude of the score increases and the magnitude of the score increases and the magnitude of the score increases as we move away from the mean. So if you as we move away from the mean. So if you as we move away from the mean. So if you are very far the score tells you that are very far the score tells you that are very far the score tells you that okay this is the direction of the data okay this is the direction of the data okay this is the direction of the data but the magnitude is also high which but the magnitude is also high which but the magnitude is also high which means that you are very far from the means that you are very far from the means that you are very far from the data. you need to go there with a higher data. you need to go there with a higher data. you need to go there with a higher magnitude. magnitude. magnitude. So uh this is what we get if we actually So uh this is what we get if we actually

  92. L6 @ 2324s

    So uh this is what we get if we actually plot the score function. You can see plot the score function. You can see plot the score function. You can see that the arrows are pointed towards the that the arrows are pointed towards the that the arrows are pointed towards the direction where the probability for the direction where the probability for the direction where the probability for the data sample is the maximum and the data sample is the maximum and the data sample is the maximum and the magnitude of the arrows also increase as magnitude of the arrows also increase as magnitude of the arrows also increase as we go towards the left and right because we go towards the left and right because we go towards the left and right because the distance from the mean increases. So this is the idea of score function So this is the idea of score function and uh what it represents and uh what it represents and uh what it represents quite quite interesting uh in in my quite quite interesting uh in in my quite quite interesting uh in in my opinion I I think scores when I first opinion I I think scores when I first opinion I I think scores when I first looked at this in the literature for looked at this in the literature for looked at this in the literature for example I was reading the DDPM paper example I was reading the DDPM paper example I was reading the DDPM paper first and there they have very casually first and there they have very casually first and there they have very casually mentioned that mentioned that mentioned that uh you know I developed a framework uh you know I developed a framework uh you know I developed a framework where where where uh both the diffusion and this course uh both the diffusion and this course uh both the diffusion and this course can be unified from the lens of this can be unified from the lens of this can be unified from the lens of this same framework same framework same framework and at that time I was not aware of the and at that time I was not aware of the and at that time I was not aware of the score-based framework but

  93. L6 @ 2381s

    score-based framework but score-based framework but score-based methods score-based methods score-based methods is something which has come before is something which has come before is something which has come before diffusion models and in in the year the diffusion models and in in the year the diffusion models and in in the year the decade of 2010s itself. decade of 2010s itself. decade of 2010s itself. So there have been these two frameworks So there have been these two frameworks So there have been these two frameworks which have made parallel progress in the which have made parallel progress in the which have made parallel progress in the deep generative modeling and uh that is deep generative modeling and uh that is deep generative modeling and uh that is the score based approach and diffusion the score based approach and diffusion the score based approach and diffusion based approach. based approach. based approach. Okay. So uh in this example we can Okay. So uh in this example we can Okay. So uh in this example we can clearly see that all the points are clearly see that all the points are clearly see that all the points are pointed towards the center because the pointed towards the center because the pointed towards the center because the origin has the maximum probability origin has the maximum probability origin has the maximum probability density and the further you are away density and the further you are away density and the further you are away from the center the magnitude of the from the center the magnitude of the from the center the magnitude of the arrows increases because it will require arrows increases because it will require arrows increases because it will require more force to pull it to the center. Now the question is okay that is great Now the question is okay that is great but how does the score helps us in but how does the score helps us in but how does the score helps us in getting rid of the partition function. getting rid of the partition function. getting rid of the partition function. Remember that was one of the main Remember that was one of the main Remember that was one of the main challenges which we started out with. We challenges which we started out with. We

  94. L6 @ 2440s

    challenges which we started out with. We wanted something which can uh help us wanted something which can uh help us wanted something which can uh help us offer a tractable alternative to offer a tractable alternative to offer a tractable alternative to calculate the maximum likelihood of the calculate the maximum likelihood of the calculate the maximum likelihood of the data. data. data. So let's let's understand that. So this uh subsection is titled freedom So this uh subsection is titled freedom from partition function. from partition function. from partition function. So we had seen before that calculating So we had seen before that calculating So we had seen before that calculating the partition function was intractable. the partition function was intractable. the partition function was intractable. It was difficult to calculate the It was difficult to calculate the It was difficult to calculate the partition function. partition function. partition function. Because of this we could not find an Because of this we could not find an Because of this we could not find an expression for maximizing the expression for maximizing the expression for maximizing the probability density likelihood. probability density likelihood. probability density likelihood. Now let's see how this formulation Now let's see how this formulation Now let's see how this formulation changes. changes. changes. So I would like to actually prove this So I would like to actually prove this So I would like to actually prove this to see uh whether we are able to do to see uh whether we are able to do to see uh whether we are able to do this. So okay now the score is given by this. So okay now the score is given by this. So okay now the score is given by grad of log of p of x right log of a / b is log a minus log b. So it log of a / b is log a minus log b. So it is log of e^ - e5 of x minus log of integral of e to minus e5 minus log of integral of e to minus e5 of x dx. Now this is simply minus ei of x and Now this is simply minus ei of x and this this remains same. Now what what happens is that when you

  95. L6 @ 2529s

    Now what what happens is that when you take this gradient uh the the first term take this gradient uh the the first term take this gradient uh the the first term is minus grad of e5 of x which is simply is minus grad of e5 of x which is simply is minus grad of e5 of x which is simply the gradient of the energy function with the gradient of the energy function with the gradient of the energy function with a negative sign and this term actually a negative sign and this term actually a negative sign and this term actually cancels out to zero because because we cancels out to zero because because we cancels out to zero because because we are summing up over all the data are summing up over all the data are summing up over all the data samples. it ends up being a uh being a samples. it ends up being a uh being a samples. it ends up being a uh being a term or an expression which is term or an expression which is term or an expression which is independent of x and taking a derivative independent of x and taking a derivative independent of x and taking a derivative of this becomes zero because it is of this becomes zero because it is of this becomes zero because it is independent of x. So this is how we get independent of x. So this is how we get independent of x. So this is how we get rid of the partition function rid of the partition function rid of the partition function uh to calculate the score function. uh to calculate the score function. uh to calculate the score function. So So if you just plainly look at s of x right if you just plainly look at s of x right if you just plainly look at s of x right to calculate s of x we don't need to to calculate s of x we don't need to to calculate s of x we don't need to worry about partition function. It worry about partition function. It worry about partition function. It simply becomes minus grad of e5 of x simply becomes minus grad of e5 of x simply becomes minus grad of e5 of x that's all. that's all.

  96. L6 @ 2586s

    that's all. So the partition function becomes zero So the partition function becomes zero So the partition function becomes zero and is not included in the calculation and is not included in the calculation and is not included in the calculation of the score function. of the score function. of the score function. Which is why we get the freedom from the Which is why we get the freedom from the Which is why we get the freedom from the partition function calculation. Now let us uh move ahead and uh see how Now let us uh move ahead and uh see how this formulation of the score function this formulation of the score function this formulation of the score function can be used for training energy based can be used for training energy based can be used for training energy based models which is our objective in the models which is our objective in the models which is our objective in the first place. first place. first place. So now that we have understood the So now that we have understood the So now that we have understood the intuition behind s of x which is the intuition behind s of x which is the intuition behind s of x which is the score function and score function and score function and also we have understood that to also we have understood that to also we have understood that to calculate the score you don't need to calculate the score you don't need to calculate the score you don't need to rely on the partition function. rely on the partition function. rely on the partition function. Let us go ahead and understand how how Let us go ahead and understand how how Let us go ahead and understand how how do we train these energy based models. do we train these energy based models. do we train these energy based models. Okay. So, uh energy based models are Okay. So, uh energy based models are Okay. So, uh energy based models are trained using a concept which is called trained using a concept which is called trained using a concept which is called as score matching. What is score matching? Score matching What is score matching? Score matching makes sure that the score of the makes sure that the score of the

  97. L6 @ 2642s

    makes sure that the score of the predicted model matches the score of the predicted model matches the score of the predicted model matches the score of the true probability distribution. true probability distribution. true probability distribution. So, uh we have two scores over here. The So, uh we have two scores over here. The So, uh we have two scores over here. The first is the score of the true first is the score of the true first is the score of the true distribution which makes uses which distribution which makes uses which distribution which makes uses which which uses the which uses the which uses the probability distribution of the probability distribution of the probability distribution of the underlying data. Remember we do not have underlying data. Remember we do not have underlying data. Remember we do not have access to this and the second is the score for the and the second is the score for the predicted distribution. This is what we predicted distribution. This is what we predicted distribution. This is what we want to train. So the score matching loss looks like So the score matching loss looks like this. this. We want to calculate uh We want to calculate uh We want to calculate uh we want to train our model such that the we want to train our model such that the we want to train our model such that the score function of our predicted model score function of our predicted model score function of our predicted model matches the score of the true matches the score of the true matches the score of the true probability distribution. probability distribution. And uh this looks fine uh in in And uh this looks fine uh in in And uh this looks fine uh in in principle. If you expand this uh you principle. If you expand this uh you principle. If you expand this uh you would know that it it it boils down to would know that it it it boils down to would know that it it it boils down to uh if if we just use this formulation uh if if we just use this formulation uh if if we just use this formulation this boils down to this boils down to this boils down to ei So ideally we want

  98. L6 @ 2733s

    So ideally we want a loss which looks like this. Now the a loss which looks like this. Now the a loss which looks like this. Now the question again becomes how do we question again becomes how do we question again becomes how do we calculate this uh probability data or calculate this uh probability data or calculate this uh probability data or the the energy function for your the the energy function for your the the energy function for your underlying samples. You have absolutely underlying samples. You have absolutely underlying samples. You have absolutely no idea how to calculate this right. no idea how to calculate this right. no idea how to calculate this right. It's it's it's difficult because we do It's it's it's difficult because we do It's it's it's difficult because we do not know this. We only have these not know this. We only have these not know this. We only have these samples. We do not know what is the samples. We do not know what is the samples. We do not know what is the probability associated with them. what probability associated with them. what probability associated with them. what is the energy associated with them? If is the energy associated with them? If is the energy associated with them? If we knew that we could plot the original we knew that we could plot the original we knew that we could plot the original distribution as it is, right? So let us distribution as it is, right? So let us distribution as it is, right? So let us park this for now. We will come back to park this for now. We will come back to park this for now. We will come back to this after some time. this after some time. this after some time. Let us first assume that we have trained Let us first assume that we have trained Let us first assume that we have trained our score function. Okay. So uh let us our score function. Okay. So uh let us our score function. Okay. So uh let us assume that assume that assume that this is known to us. this is known to us. this is known to us. Let us assume that we have done the Let us assume that we have done the

  99. L6 @ 2782s

    Let us assume that we have done the training. We have matched this score training. We have matched this score training. We have matched this score with the uh score for the true model for with the uh score for the true model for with the uh score for the true model for the true distribution and we have got the true distribution and we have got the true distribution and we have got the score. The main question is that the score. The main question is that the score. The main question is that once you get the score, how do you once you get the score, how do you once you get the score, how do you sample from the score? How do you sample sample from the score? How do you sample sample from the score? How do you sample the images the images the images in in the diffusion model? We were in in the diffusion model? We were in in the diffusion model? We were directly predicting the probability directly predicting the probability directly predicting the probability distribution or the noise level. So we distribution or the noise level. So we distribution or the noise level. So we could subtract the noise iteratively could subtract the noise iteratively could subtract the noise iteratively from the from the from the uh from pure noise and we could get the uh from pure noise and we could get the uh from pure noise and we could get the image right we could sample but how do image right we could sample but how do image right we could sample but how do we sample from this score function we sample from this score function we sample from this score function all we know is the direction of where to all we know is the direction of where to all we know is the direction of where to go and intuitively it it looks like it go and intuitively it it looks like it go and intuitively it it looks like it should be possible because if I am in a should be possible because if I am in a should be possible because if I am in a data point and I know where to go I can data point and I know where to go I can

  100. L6 @ 2829s

    data point and I know where to go I can navigate my way and I can find samples navigate my way and I can find samples navigate my way and I can find samples from the distribution Okay. And uh this from the distribution Okay. And uh this from the distribution Okay. And uh this this this this is very interesting this this this is very interesting this this this is very interesting because it really uh it will help us to because it really uh it will help us to because it really uh it will help us to understand a bit deeper about understand a bit deeper about understand a bit deeper about understand the score function in a more understand the score function in a more understand the score function in a more deeper way. Okay. So the question that deeper way. Okay. So the question that deeper way. Okay. So the question that we will address is how do you sample the we will address is how do you sample the we will address is how do you sample the data if you have the score function. Okay. So let's let's try to understand Okay. So let's let's try to understand this in detail. So to understand this we will start from So to understand this we will start from something which is uh even more at a simpler level. Okay. We even more at a simpler level. Okay. We will start from uh this question that if will start from uh this question that if will start from uh this question that if you are dropped into a thick fog in a you are dropped into a thick fog in a you are dropped into a thick fog in a vast landscape and vast landscape and vast landscape and your goal is to find the deepest valley your goal is to find the deepest valley your goal is to find the deepest valley because that is where the treasure is because that is where the treasure is because that is where the treasure is hidden. hidden. hidden. How will you go to the deepest valley? How will you go to the deepest valley? How will you go to the deepest valley? Here I can visually see that this is the Here I can visually see that this is the

  101. L6 @ 2898s

    Here I can visually see that this is the deepest valley and I can see the deepest valley and I can see the deepest valley and I can see the treasure is hidden there. treasure is hidden there. treasure is hidden there. But if your goal is to reach the deepest But if your goal is to reach the deepest But if your goal is to reach the deepest valley, how would you go to the deepest valley, how would you go to the deepest valley, how would you go to the deepest valley if you cannot directly visually valley if you cannot directly visually valley if you cannot directly visually see this? see this? see this? Okay, so uh you would like to trace a Okay, so uh you would like to trace a Okay, so uh you would like to trace a route which looks something like this, route which looks something like this, route which looks something like this, right? How can you frame it in a different way? How can you frame it in a different way? The strategy that you will use is since The strategy that you will use is since The strategy that you will use is since you know that the pressure is in the you know that the pressure is in the you know that the pressure is in the valley somewhere, you know that going valley somewhere, you know that going valley somewhere, you know that going down is good and going up is bad. down is good and going up is bad. down is good and going up is bad. And to reach the valley in the quickest And to reach the valley in the quickest And to reach the valley in the quickest possible time, you will go in the possible time, you will go in the possible time, you will go in the direction where the downward slope is direction where the downward slope is direction where the downward slope is maximum. So this is this is quite intuitive, So this is this is quite intuitive, right? Let's say this is the this is my right? Let's say this is the this is my right? Let's say this is the this is my valley and I start here. and I want to valley and I start here. and I want to

  102. L6 @ 2950s

    valley and I start here. and I want to reach the bottom of the valley. So I'll reach the bottom of the valley. So I'll reach the bottom of the valley. So I'll make a step like this. I'll again make a make a step like this. I'll again make a make a step like this. I'll again make a step. I'll again make a step and I will step. I'll again make a step and I will step. I'll again make a step and I will reach the bottom of the valley. So let reach the bottom of the valley. So let reach the bottom of the valley. So let us say the slope is given by the symbol us say the slope is given by the symbol us say the slope is given by the symbol uh q and if your current position is xt uh q and if your current position is xt uh q and if your current position is xt and your next position is xt + 1 then and your next position is xt + 1 then and your next position is xt + 1 then you can write your next position like you can write your next position like you can write your next position like this. So this is actually So this is actually the slope. So what we are doing here is let's say So what we are doing here is let's say this is your valley this is your valley this is your valley and you are here at XT. Okay. So x t + 1 Okay. So x t + 1 can be written in terms of if you know can be written in terms of if you know can be written in terms of if you know this slope you can multiply this slope this slope you can multiply this slope this slope you can multiply this slope by so this this would be delta t in this by so this this would be delta t in this by so this this would be delta t in this case you multiply it by delta t and you case you multiply it by delta t and you case you multiply it by delta t and you would get your next location. would get your next location.

  103. L6 @ 3012s

    would get your next location. But what is the slope here? How do we But what is the slope here? How do we But what is the slope here? How do we find the slope? This is my energy find the slope? This is my energy find the slope? This is my energy function ei of x which we started out in function ei of x which we started out in function ei of x which we started out in the first place. the first place. the first place. and uh the the slope is given by grad of and uh the the slope is given by grad of and uh the the slope is given by grad of E5 of X. E5 of X. E5 of X. So then my equation becomes So then my equation becomes So then my equation becomes Q q is negative of grad because we are Q q is negative of grad because we are Q q is negative of grad because we are going in the reverse direction. Here we going in the reverse direction. Here we going in the reverse direction. Here we can write XT + 1 = XT minus N into can write XT + 1 = XT minus N into can write XT + 1 = XT minus N into gradient of EI of X. gradient of EI of X. gradient of EI of X. This is very similar to the gradient This is very similar to the gradient This is very similar to the gradient descent equation. uh remember how we descent equation. uh remember how we descent equation. uh remember how we let's say this is your parameter let's say this is your parameter let's say this is your parameter w and uh this is your loss. w and uh this is your loss. w and uh this is your loss. So how do we update the parameter wt + 1 So how do we update the parameter wt + 1 So how do we update the parameter wt + 1 equal to wt minus alpha step size into equal to wt minus alpha step size into equal to wt minus alpha step size into gradient of the loss function. This is gradient of the loss function. This is gradient of the loss function. This is exactly what we are doing here. We are

  104. L6 @ 3067s

    exactly what we are doing here. We are exactly what we are doing here. We are calculating the gradient of the energy calculating the gradient of the energy calculating the gradient of the energy and u and u and u we are taking a step in the this is what we are taking a step in the this is what we are taking a step in the this is what this minus sign encodes this gradient this minus sign encodes this gradient this minus sign encodes this gradient will always have a positive value that's will always have a positive value that's will always have a positive value that's what that's how we define it what that's how we define it what that's how we define it and uh the the negative sign ensures and uh the the negative sign ensures and uh the the negative sign ensures that we are going in the direction of that we are going in the direction of that we are going in the direction of the slope like the bottom of of the the slope like the bottom of of the the slope like the bottom of of the valley. valley. So this is how we define our movement. So this is how we define our movement. So this is how we define our movement. If you go up according to the above If you go up according to the above If you go up according to the above rule, you are guaranteed to move towards rule, you are guaranteed to move towards rule, you are guaranteed to move towards regions where the energy function is regions where the energy function is regions where the energy function is minimum. minimum. minimum. Now you might see that oh this is this Now you might see that oh this is this Now you might see that oh this is this looks somewhat familiar, right? How does looks somewhat familiar, right? How does looks somewhat familiar, right? How does this look familiar? this look familiar? this look familiar? Remember uh we looked at the score Remember uh we looked at the score Remember uh we looked at the score function and it it looked something like function and it it looked something like function and it it looked something like this which is gradient of the energy this which is gradient of the energy

  105. L6 @ 3119s

    this which is gradient of the energy function. So the negative of this function. So the negative of this function. So the negative of this actually is the score function itself. actually is the score function itself. actually is the score function itself. So this can be written as XT + N into S5 So this can be written as XT + N into S5 So this can be written as XT + N into S5 of XT. of XT. of XT. So does that mean we are done here? So does that mean we are done here? So does that mean we are done here? Because if we calculate the Because if we calculate the Because if we calculate the we are at any point we calculate the we are at any point we calculate the we are at any point we calculate the score score score we take a step size of n and we take we take a step size of n and we take we take a step size of n and we take that step we move towards xt + 1 we take that step we move towards xt + 1 we take that step we move towards xt + 1 we take another step we move towards xt +2 so another step we move towards xt +2 so another step we move towards xt +2 so does that mean our our job is done? Well does that mean our our job is done? Well does that mean our our job is done? Well we are not done yet because this is only we are not done yet because this is only we are not done yet because this is only half of the entire puzzle which we have half of the entire puzzle which we have half of the entire puzzle which we have solved. solved. solved. The next half becomes very crucial. The next half becomes very crucial. The next half becomes very crucial. Consider this scenario where you have Consider this scenario where you have Consider this scenario where you have reached the bottom of the pit and the reached the bottom of the pit and the reached the bottom of the pit and the gradient is zero. So you are just gradient is zero. So you are just

  106. L6 @ 3172s

    gradient is zero. So you are just sitting there but there is another pit sitting there but there is another pit sitting there but there is another pit which is even below which is even below which is even below even below this. So this looks something even below this. So this looks something even below this. So this looks something like this. like this. like this. You are stuck somewhere here but there You are stuck somewhere here but there You are stuck somewhere here but there is another pit which is somewhere over is another pit which is somewhere over is another pit which is somewhere over here. How will you find this? And in here. How will you find this? And in here. How will you find this? And in fact the the first curve that we saw was fact the the first curve that we saw was fact the the first curve that we saw was like this when we started our discussion like this when we started our discussion like this when we started our discussion on energy based models. Using this on energy based models. Using this on energy based models. Using this approach you will reach this and the approach you will reach this and the approach you will reach this and the gradient will be zero and then XT + 1 gradient will be zero and then XT + 1 gradient will be zero and then XT + 1 will be same as XT. You will just stay will be same as XT. You will just stay will be same as XT. You will just stay here. here. here. I need something which can push me out I need something which can push me out I need something which can push me out of this and take me towards the next of this and take me towards the next of this and take me towards the next minima which might be lower. So we are here and the global minima we So we are here and the global minima we will never find. To solve this issue, we will never find. To solve this issue, we will never find. To solve this issue, we need to provide a shake which gives you need to provide a shake which gives you

  107. L6 @ 3220s

    need to provide a shake which gives you enough random energy to kick you out of enough random energy to kick you out of enough random energy to kick you out of these small potholes so that you can these small potholes so that you can these small potholes so that you can keep moving towards the bottom of the keep moving towards the bottom of the keep moving towards the bottom of the valley which is where the global minima valley which is where the global minima valley which is where the global minima lies. lies. lies. So you need something which can give you So you need something which can give you So you need something which can give you that shake right which can push you. that shake right which can push you. that shake right which can push you. And remember in the lecture on diffusion And remember in the lecture on diffusion And remember in the lecture on diffusion we had discussed about adding noise to we had discussed about adding noise to we had discussed about adding noise to the data and we had written a simple the data and we had written a simple the data and we had written a simple expression to add noise as follows. expression to add noise as follows. expression to add noise as follows. where x i + 1 is the image in the where x i + 1 is the image in the where x i + 1 is the image in the forward step at the next time step. This forward step at the next time step. This forward step at the next time step. This is the current time step and this is is the current time step and this is is the current time step and this is what introduces the noise. what introduces the noise. what introduces the noise. Uh this epsilon is a random variable Uh this epsilon is a random variable Uh this epsilon is a random variable which can take any value between 0 to 1. which can take any value between 0 to 1. which can take any value between 0 to 1. So this above expression also means that So this above expression also means that So this above expression also means that we are sampling from a gshian

  108. L6 @ 3269s

    we are sampling from a gshian we are sampling from a gshian distribution with a mean of xi and a distribution with a mean of xi and a distribution with a mean of xi and a standard deviation of beta. standard deviation of beta. standard deviation of beta. uh this we also saw in variational uh this we also saw in variational uh this we also saw in variational autoenccoders where given sigma and epi autoenccoders where given sigma and epi autoenccoders where given sigma and epi sigma and mu for the latent variable the sigma and mu for the latent variable the sigma and mu for the latent variable the way we wrote zed was mu + sigma * way we wrote zed was mu + sigma * way we wrote zed was mu + sigma * epsilon epsilon epsilon so okay so now we can think okay fine so okay so now we can think okay fine so okay so now we can think okay fine can we add noise using something like can we add noise using something like can we add noise using something like this and it turns out we can definitely this and it turns out we can definitely this and it turns out we can definitely we can add noise which looks something we can add noise which looks something we can add noise which looks something like So th this part of the expression is So th this part of the expression is exactly the same that we looked at exactly the same that we looked at exactly the same that we looked at before. There is only this new term before. There is only this new term before. There is only this new term which is added which uh which is added which uh which is added which uh has this variable which is the random has this variable which is the random has this variable which is the random variable epsilon and <unk>2 ea is just a constant which and <unk>2 ea is just a constant which is probably exists there to make sure is probably exists there to make sure is probably exists there to make sure the total variance of this xt + 1 is the total variance of this xt + 1 is

  109. L6 @ 3330s

    the total variance of this xt + 1 is maybe a constant or it's just one that maybe a constant or it's just one that maybe a constant or it's just one that is my hypothesis is my hypothesis is my hypothesis this is also called as discrete time this is also called as discrete time this is also called as discrete time lang update lang update lang update Why it is called languin update and not Why it is called languin update and not Why it is called languin update and not just a gradient descent update? It's just a gradient descent update? It's just a gradient descent update? It's because we are adding this stoastic term because we are adding this stoastic term because we are adding this stoastic term which is like a shake. So what this term which is like a shake. So what this term which is like a shake. So what this term will do is I reach at the bottom of this will do is I reach at the bottom of this will do is I reach at the bottom of this pit. pit. pit. Now xt + 1 is equal to xt Now xt + 1 is equal to xt Now xt + 1 is equal to xt plus plus plus uh what was the term over here? Yeah. uh what was the term over here? Yeah. uh what was the term over here? Yeah. Minus n of Minus n of Minus n of gradient of the energy plus this stochastic term which is plus this stochastic term which is <unk>2n into epsilon. So this is zero. <unk>2n into epsilon. So this is zero. <unk>2n into epsilon. So this is zero. So because the gradient is zero. But So because the gradient is zero. But So because the gradient is zero. But because of this term I'll be forced to because of this term I'll be forced to because of this term I'll be forced to move out of this bit and explore new move out of this bit and explore new move out of this bit and explore new regions which is exactly what we want. regions which is exactly what we want. regions which is exactly what we want. We want to deliberately add this We want to deliberately add this We want to deliberately add this stoasticity.

  110. L6 @ 3392s

    Now we what we do is we replace this Now we what we do is we replace this grad of energy by s of x which we have grad of energy by s of x which we have grad of energy by s of x which we have already seen before and this is the already seen before and this is the already seen before and this is the final expression that we get. final expression that we get. final expression that we get. Substituting this in the equation gives Substituting this in the equation gives Substituting this in the equation gives us the us the us the update rule which we can use to sample update rule which we can use to sample update rule which we can use to sample when we know the predicted score when we know the predicted score when we know the predicted score function. So this should ideally be si function. So this should ideally be si function. So this should ideally be si because it is the predicted score because it is the predicted score because it is the predicted score function. We have not yet looked at how function. We have not yet looked at how function. We have not yet looked at how to train the score function to match it to train the score function to match it to train the score function to match it with the real score function. There are with the real score function. There are with the real score function. There are challenges there. But now we know that challenges there. But now we know that challenges there. But now we know that once we have the score function, we can once we have the score function, we can once we have the score function, we can use the langin use the langin use the langin sampling method to sample from the score sampling method to sample from the score sampling method to sample from the score function and we can find a trajectory. function and we can find a trajectory. function and we can find a trajectory. So ideally if we start from some So ideally if we start from some So ideally if we start from some trajectory let's say I start from here trajectory let's say I start from here

  111. L6 @ 3436s

    trajectory let's say I start from here with this update I'll go like this with this update I'll go like this with this update I'll go like this and then I'll finally I I I need to come and then I'll finally I I I need to come and then I'll finally I I I need to come up to a space where the probability up to a space where the probability up to a space where the probability density density density of uh my my data is maximum and and I of uh my my data is maximum and and I of uh my my data is maximum and and I can safely sample from it. can safely sample from it. can safely sample from it. So this is how you would move. You start So this is how you would move. You start So this is how you would move. You start with a point, you end up with a point with a point, you end up with a point with a point, you end up with a point and then that point is your sample. And and then that point is your sample. And and then that point is your sample. And uh why is that point your sample? uh why is that point your sample? uh why is that point your sample? Because you're ending up at a point Because you're ending up at a point Because you're ending up at a point which the where the score function has which the where the score function has which the where the score function has guided you. So this is like a compass guided you. So this is like a compass guided you. So this is like a compass which the score function guides you which the score function guides you which the score function guides you towards. You start from anywhere. You towards. You start from anywhere. You towards. You start from anywhere. You have no idea where the data is but you have no idea where the data is but you have no idea where the data is but you have a small compass with you which is have a small compass with you which is have a small compass with you which is the score function which takes you from the score function which takes you from

  112. L6 @ 3479s

    the score function which takes you from one location to the another and finally one location to the another and finally one location to the another and finally you end up at a space or a region where you end up at a space or a region where you end up at a space or a region where the probability density is the maximum. the probability density is the maximum. the probability density is the maximum. So we are going to now look at exactly So we are going to now look at exactly So we are going to now look at exactly how to use lang dynamics to sample from how to use lang dynamics to sample from how to use lang dynamics to sample from a known probability distribution. a known probability distribution. a known probability distribution. uh remember we'll consider that the uh remember we'll consider that the uh remember we'll consider that the probability distribution is already probability distribution is already probability distribution is already known because we want to uh consider known because we want to uh consider known because we want to uh consider that the score function is already that the score function is already that the score function is already known. Okay. So now let's take a practical Okay. So now let's take a practical example and dive into the Google collab example and dive into the Google collab example and dive into the Google collab notebook. Okay. So the example that we notebook. Okay. So the example that we notebook. Okay. So the example that we are going to look at is uh are going to look at is uh are going to look at is uh we are going to consider that we know we are going to consider that we know we are going to consider that we know this true probability distribution which this true probability distribution which this true probability distribution which is p data of x. is p data of x. is p data of x. So we are going to assume that p data of So we are going to assume that p data of So we are going to assume that p data of x is known to us. It looks something x is known to us. It looks something

  113. L6 @ 3535s

    x is known to us. It looks something like this. The probability of finding like this. The probability of finding like this. The probability of finding the samples is maximum at these two the samples is maximum at these two the samples is maximum at these two locations where you see the spots are locations where you see the spots are locations where you see the spots are very hot. Right? And as you move away very hot. Right? And as you move away very hot. Right? And as you move away from uh these points we can see that the from uh these points we can see that the from uh these points we can see that the probability dies down slowly. So ideally what should happen is So ideally what should happen is whenever I start from any point let's whenever I start from any point let's whenever I start from any point let's say here say here say here uh my languin dynamic sampling should uh my languin dynamic sampling should uh my languin dynamic sampling should take me from this point to a point which take me from this point to a point which take me from this point to a point which is either here or here. is either here or here. is either here or here. then I can be sure that my method of then I can be sure that my method of then I can be sure that my method of sampling using the languin dynamics sampling using the languin dynamics sampling using the languin dynamics technique has actually worked. technique has actually worked. technique has actually worked. So let's look at an example to see So let's look at an example to see So let's look at an example to see whether that actually happens or not. whether that actually happens or not. whether that actually happens or not. We will look at a Google Collab We will look at a Google Collab We will look at a Google Collab notebook. I have shared all these links notebook. I have shared all these links notebook. I have shared all these links down below in the chat uh in in in the down below in the chat uh in in in the down below in the chat uh in in in the comments. comments.

  114. L6 @ 3595s

    comments. So first we define the setup which are So first we define the setup which are So first we define the setup which are the two mountain setup as as we saw the two mountain setup as as we saw the two mountain setup as as we saw there are two peaks in our data there are two peaks in our data there are two peaks in our data and in this block we calculate the and in this block we calculate the and in this block we calculate the probability and the scores also. probability and the scores also. probability and the scores also. Remember we are assuming that the scores Remember we are assuming that the scores Remember we are assuming that the scores are known to us. We are not discussing are known to us. We are not discussing are known to us. We are not discussing the training process right now. We are the training process right now. We are the training process right now. We are only discussing the sampling process. only discussing the sampling process. only discussing the sampling process. So now what we do is we run the lang So now what we do is we run the lang So now what we do is we run the lang dynamics here. I want you to take a very dynamics here. I want you to take a very dynamics here. I want you to take a very close attention to this step which is close attention to this step which is close attention to this step which is exactly what we wrote before. exactly what we wrote before. exactly what we wrote before. The position at the next time step is The position at the next time step is The position at the next time step is the position at the current time step the position at the current time step the position at the current time step plus EA time the score plus the noise plus EA time the score plus the noise plus EA time the score plus the noise scale which is square root of 2 EA into scale which is square root of 2 EA into scale which is square root of 2 EA into noise. We use the exact same equation noise. We use the exact same equation

  115. L6 @ 3640s

    noise. We use the exact same equation which we discuss in theory which we discuss in theory which we discuss in theory and then we visualize the path. and then we visualize the path. and then we visualize the path. So we start from here and we end up over So we start from here and we end up over So we start from here and we end up over here. This is the end path. here. This is the end path. here. This is the end path. Now I'll again run this. We start and we Now I'll again run this. We start and we Now I'll again run this. We start and we again end somewhere here. We will not again end somewhere here. We will not again end somewhere here. We will not end up at the exact same point every end up at the exact same point every end up at the exact same point every time because the process is stoastic in time because the process is stoastic in time because the process is stoastic in nature. But we will end up at a place nature. But we will end up at a place nature. But we will end up at a place where we are very close to the where we are very close to the where we are very close to the area where the probability of finding area where the probability of finding area where the probability of finding data is the maximum. data is the maximum. data is the maximum. Now one question you might have is why Now one question you might have is why Now one question you might have is why do we see these zigzag paths? It almost do we see these zigzag paths? It almost do we see these zigzag paths? It almost appears like a drunk person is walking, appears like a drunk person is walking, appears like a drunk person is walking, right? In fact, this is an analogy which right? In fact, this is an analogy which right? In fact, this is an analogy which is used in a lot of classical textbooks. is used in a lot of classical textbooks. is used in a lot of classical textbooks. It's it's called a drunk hiker. It's it's called a drunk hiker.

  116. L6 @ 3688s

    It's it's called a drunk hiker. It's like someone is drunk and walking It's like someone is drunk and walking It's like someone is drunk and walking here and there and finally reaching the here and there and finally reaching the here and there and finally reaching the final solution. And this this this final solution. And this this this final solution. And this this this apparent drunkenness is because of the apparent drunkenness is because of the apparent drunkenness is because of the the third term which we have added which the third term which we have added which the third term which we have added which which introduces the noise in the update which introduces the noise in the update which introduces the noise in the update rule because we have that epsilon rule because we have that epsilon rule because we have that epsilon sitting over there. we have these paths sitting over there. we have these paths sitting over there. we have these paths which appear zigzag otherwise we will which appear zigzag otherwise we will which appear zigzag otherwise we will get an update which appears probably get an update which appears probably get an update which appears probably smooth with continuous derivatives at at smooth with continuous derivatives at at smooth with continuous derivatives at at all time steps. all time steps. all time steps. So this is amazing right we are actually So this is amazing right we are actually So this is amazing right we are actually able to simulate trajectories once we able to simulate trajectories once we able to simulate trajectories once we have the score function we can reach a have the score function we can reach a have the score function we can reach a point where point where point where it looks like we are reaching the point it looks like we are reaching the point it looks like we are reaching the point where the probability of that data being where the probability of that data being where the probability of that data being sampled from the two distribution is sampled from the two distribution is sampled from the two distribution is maximum. maximum. maximum. So I want all of you to run this tweak a So I want all of you to run this tweak a

  117. L6 @ 3738s

    So I want all of you to run this tweak a little things here and there have fun little things here and there have fun little things here and there have fun with the notebook and really develop with the notebook and really develop with the notebook and really develop confidence in the lang dynamics confidence in the lang dynamics confidence in the lang dynamics technique. technique. technique. Later we will see a very close parallel Later we will see a very close parallel Later we will see a very close parallel between the lang dynamics and the between the lang dynamics and the between the lang dynamics and the diffusion method as well. diffusion method as well. diffusion method as well. And it is amazing that this lang dynamic And it is amazing that this lang dynamic And it is amazing that this lang dynamic sampling is in fact related to sampling is in fact related to sampling is in fact related to diffusion. But that is the beauty of diffusion. But that is the beauty of diffusion. But that is the beauty of this course. we will slowly unpack it. this course. we will slowly unpack it. this course. we will slowly unpack it. Okay. So now we will look at how exactly Okay. So now we will look at how exactly Okay. So now we will look at how exactly does the score function training happen. does the score function training happen. does the score function training happen. How is the score function trained? How is the score function trained? How is the score function trained? We we we briefly mentioned something We we we briefly mentioned something We we we briefly mentioned something called score matching but now we are called score matching but now we are called score matching but now we are going to look at score matching in more going to look at score matching in more going to look at score matching in more detail. detail. detail. This is also called as scorebased This is also called as scorebased This is also called as scorebased generative models. generative models. generative models. The key idea is that since sampling with The key idea is that since sampling with The key idea is that since sampling with since sampling with langu dynamics only since sampling with langu dynamics only

  118. L6 @ 3795s

    since sampling with langu dynamics only requires the score like we saw in this requires the score like we saw in this requires the score like we saw in this formula it only requires the score we formula it only requires the score we formula it only requires the score we can learn it directly from a neural can learn it directly from a neural can learn it directly from a neural network. So let's say this is your data network. So let's say this is your data network. So let's say this is your data and uh the ground truth score is given and uh the ground truth score is given and uh the ground truth score is given in black. These are the black arrows in black. These are the black arrows in black. These are the black arrows and the blue arrows is our predictions. So we can see that it's not exactly So we can see that it's not exactly accurate. Um in many places our accurate. Um in many places our accurate. Um in many places our predictions are slightly predictions are slightly predictions are slightly different different different than the ground truth. But the whole than the ground truth. But the whole than the ground truth. But the whole objective of the scorebased generative objective of the scorebased generative objective of the scorebased generative models is to predict the score function models is to predict the score function models is to predict the score function such that it matches the ground truth such that it matches the ground truth such that it matches the ground truth score as closely as possible. So this is score as closely as possible. So this is score as closely as possible. So this is just a visual representation of that. Now this true score function is given by Now this true score function is given by this and the predicted score function is this and the predicted score function is this and the predicted score function is given by given by given by this where the only thing we have done this where the only thing we have done this where the only thing we have done is replace data by five over here is replace data by five over here is replace data by five over here and score matching fits this vector

  119. L6 @ 3867s

    and score matching fits this vector and score matching fits this vector field by minimizing the mean square field by minimizing the mean square field by minimizing the mean square error between the true and the estimated error between the true and the estimated error between the true and the estimated scores which looks something like this. scores which looks something like this. scores which looks something like this. Now the main issue is that it is very Now the main issue is that it is very Now the main issue is that it is very difficult to find this. This is difficult to find this. This is difficult to find this. This is intractable. Why is it interractable? We don't know P Why is it interractable? We don't know P data. data. How are we going to calculate grad of How are we going to calculate grad of How are we going to calculate grad of log of P of data? It's it's it it log of P of data? It's it's it it log of P of data? It's it's it it appears impossible. appears impossible. appears impossible. However, there was a paper there was a paper which was released by there was a paper which was released by these two people in the year 2005 which showed that this score matching showed that this score matching showed that this score matching objective can be converted into a objective can be converted into a objective can be converted into a tractable score matching objective. Let's try to look at this paper so that Let's try to look at this paper so that we develop some kind of an appreciation we develop some kind of an appreciation we develop some kind of an appreciation for the work which has happened over two for the work which has happened over two for the work which has happened over two decades of of literature to reach where decades of of literature to reach where decades of of literature to reach where we are right now. we are right now. we are right now. So these are the authors. I I I do not So these are the authors. I I I do not So these are the authors. I I I do not know how to pronounce this name. know how to pronounce this name.

  120. L6 @ 3940s

    know how to pronounce this name. uh however they they came up with a uh however they they came up with a uh however they they came up with a technique to technique to technique to distill that objective function into distill that objective function into distill that objective function into something which is tractable. This is something which is tractable. This is something which is tractable. This is very similar to what we did with the very similar to what we did with the very similar to what we did with the elbow objective in variation elbow objective in variation elbow objective in variation autoenccoders. autoenccoders. autoenccoders. However, this is a bit different because However, this is a bit different because However, this is a bit different because we are not directly we are not directly we are not directly calculating the lower bound on it, but calculating the lower bound on it, but calculating the lower bound on it, but rather we are finding a rather we are finding a rather we are finding a uh a term or an expression whose uh a term or an expression whose uh a term or an expression whose expected value is exactly the same as expected value is exactly the same as expected value is exactly the same as the expected value of the term that we the expected value of the term that we the expected value of the term that we want to calculate. want to calculate. want to calculate. So in a way it's not an approximation So in a way it's not an approximation So in a way it's not an approximation like elbow but it's it's it's a like elbow but it's it's it's a like elbow but it's it's it's a different representation of the same different representation of the same different representation of the same expression. Okay. So this was this is the uh this is Okay. So this was this is the uh this is what we want what we want what we want what we want to calculate and uh this this paper kind of and uh this this paper kind of reformulated this objective as reformulated this objective as reformulated this objective as a summation of this and a constant. a summation of this and a constant.

  121. L6 @ 4007s

    a summation of this and a constant. So basically minimizing this meant that So basically minimizing this meant that So basically minimizing this meant that we want to minimize this and you see here we do not have the SS and you see here we do not have the SS data anymore over here or S of X which data anymore over here or S of X which data anymore over here or S of X which is the true is the true is the true we do not have we do not have we do not have somehow magically somehow magically somehow magically the score function has disappeared from the score function has disappeared from the score function has disappeared from the loss the loss the loss and it has been replaced by two terms. and it has been replaced by two terms. and it has been replaced by two terms. The first term is the trace The first term is the trace The first term is the trace of grad of S S5 of X. So we are taking of grad of S S5 of X. So we are taking of grad of S S5 of X. So we are taking the gradient of the score function and the gradient of the score function and the gradient of the score function and the second term is the the second term is the the second term is the square error of the square error of the square error of the score function matrix. score function matrix. score function matrix. So let us try to understand both of So let us try to understand both of So let us try to understand both of these terms and exactly what they mean. these terms and exactly what they mean. these terms and exactly what they mean. First let us look at the trace. First let us look at the trace. First let us look at the trace. What is the meaning of a trace of a What is the meaning of a trace of a What is the meaning of a trace of a matrix? matrix? matrix? Let us say we have a matrix which looks Let us say we have a matrix which looks Let us say we have a matrix which looks like this. A1 A2 A3 like this. A1 A2 A3

  122. L6 @ 4081s

    like this. A1 A2 A3 A4 A5 A6 A4 A5 A6 A4 A5 A6 A7 A8 and A9. So the trace of this A7 A8 and A9. So the trace of this A7 A8 and A9. So the trace of this matrix is simply the addition of a1 + a5 matrix is simply the addition of a1 + a5 matrix is simply the addition of a1 + a5 plus a9. This is the trace and This is the trace and what we are doing here is we are simply what we are doing here is we are simply what we are doing here is we are simply taking a1 squared plus taking a1 squared plus taking a1 squared plus a2 squar + a3 squar + dot dot dot a9 a2 squar + a3 squar + dot dot dot a9 a2 squar + a3 squar + dot dot dot a9 squared and dividing it by 2. squared and dividing it by 2. squared and dividing it by 2. And this is something which we can And this is something which we can And this is something which we can easily calculate. Right? Once we have easily calculate. Right? Once we have easily calculate. Right? Once we have this this this uh this is only dependent on the network uh this is only dependent on the network uh this is only dependent on the network that we want to train. that we want to train. that we want to train. So now suddenly we have an objective So now suddenly we have an objective So now suddenly we have an objective function where function where function where every single entity depends on the every single entity depends on the every single entity depends on the neural network that we want to train. So neural network that we want to train. So neural network that we want to train. So we are no longer dependent on something we are no longer dependent on something we are no longer dependent on something which is interactable. Now both of these terms they mean Now both of these terms they mean something very unique. Even if you don't something very unique. Even if you don't something very unique. Even if you don't understand the derivation of this understand the derivation of this understand the derivation of this uh it is it is completely fine for our

  123. L6 @ 4151s

    uh it is it is completely fine for our uh it is it is completely fine for our purposes what we need to understand is purposes what we need to understand is purposes what we need to understand is the intuitive meaning behind both of the intuitive meaning behind both of the intuitive meaning behind both of these terms. The first term which is the these terms. The first term which is the these terms. The first term which is the trace let's see trace let's see trace let's see the trace term measures the divergence the trace term measures the divergence the trace term measures the divergence of your arrows. It measures if the of your arrows. It measures if the of your arrows. It measures if the arrows are spreading out or if the arrows are spreading out or if the arrows are spreading out or if the arrows are arrows are arrows are are converging at a specific point. are converging at a specific point. are converging at a specific point. So if we have a trace which is positive, So if we have a trace which is positive, So if we have a trace which is positive, the arrows are exploding outwards like a the arrows are exploding outwards like a the arrows are exploding outwards like a bomb which is going off. And if we have bomb which is going off. And if we have bomb which is going off. And if we have a negative trace, the arrows are sucking a negative trace, the arrows are sucking a negative trace, the arrows are sucking inwards. inwards. inwards. And since we are minimizing the loss, we And since we are minimizing the loss, we And since we are minimizing the loss, we are trying to minimize this trace. Which are trying to minimize this trace. Which are trying to minimize this trace. Which means we want to make it as negative as means we want to make it as negative as means we want to make it as negative as possible. Which means that we want to possible. Which means that we want to possible. Which means that we want to make our data samples look as syncs. make our data samples look as syncs. make our data samples look as syncs. Which makes a lot of sense, right? In

  124. L6 @ 4205s

    Which makes a lot of sense, right? In Which makes a lot of sense, right? In the previous example that we looked at the previous example that we looked at the previous example that we looked at these these were the two let's say these these were the two let's say these these were the two let's say probability contours. So what this first probability contours. So what this first probability contours. So what this first term does is it makes the score function term does is it makes the score function term does is it makes the score function looks like this. So that the true data the region around So that the true data the region around the true data appears like a drain the true data appears like a drain the true data appears like a drain which is what we want because if if which is what we want because if if which is what we want because if if someone starts from here because of this sink or drain like because of this sink or drain like structure you'll move towards the actual structure you'll move towards the actual structure you'll move towards the actual data point. data point. data point. So So positive trace looks like this. It's positive trace looks like this. It's positive trace looks like this. It's almost like it's diverging outwards and almost like it's diverging outwards and almost like it's diverging outwards and negative trace looks like this. It's negative trace looks like this. It's negative trace looks like this. It's diverging inwards. diverging inwards. diverging inwards. In fluid flows, people often equate In fluid flows, people often equate In fluid flows, people often equate these trace terms to zero these trace terms to zero these trace terms to zero in in in many of the cases because you in in in many of the cases because you in in in many of the cases because you cannot have fluid particles which are cannot have fluid particles which are cannot have fluid particles which are expanding or contracting. expanding or contracting. expanding or contracting. Okay. Anyway, so this is the first term Okay. Anyway, so this is the first term Okay. Anyway, so this is the first term where we are trying to make the data where we are trying to make the data

  125. L6 @ 4274s

    where we are trying to make the data samples appear as syncs. samples appear as syncs. samples appear as syncs. which is what you see over here also. If which is what you see over here also. If which is what you see over here also. If you see this this this uh you see this this this uh you see this this this uh probably that image we do not have at probably that image we do not have at probably that image we do not have at the moment. But if you just look at this the moment. But if you just look at this the moment. But if you just look at this image, this area is like a sink, right? image, this area is like a sink, right? image, this area is like a sink, right? The arrows appear like this oriented The arrows appear like this oriented The arrows appear like this oriented towards that region from all directions. towards that region from all directions. towards that region from all directions. So this term is intuitive. The second So this term is intuitive. The second So this term is intuitive. The second term, it measures the length or the term, it measures the length or the term, it measures the length or the strength of your arrows. It wants the strength of your arrows. It wants the strength of your arrows. It wants the arrows to be as as small as possible. So arrows to be as as small as possible. So arrows to be as as small as possible. So you want this squared term to be zero. you want this squared term to be zero. you want this squared term to be zero. So you want the arrows to be as small as So you want the arrows to be as small as So you want the arrows to be as small as possible. Why do you want small arrows? possible. Why do you want small arrows? possible. Why do you want small arrows? It's because regions where P of data is It's because regions where P of data is It's because regions where P of data is high will have more score and contribute high will have more score and contribute high will have more score and contribute more to this term. more to this term.

  126. L6 @ 4327s

    more to this term. So the this term drives So the this term drives So the this term drives scores to zero in the higher probability scores to zero in the higher probability scores to zero in the higher probability areas and those locations become areas and those locations become areas and those locations become stationary. So now let's say in in the stationary. So now let's say in in the stationary. So now let's say in in the example of the two mountain case where example of the two mountain case where example of the two mountain case where this was the probability this was the probability this was the probability this was the area where the probability this was the area where the probability this was the area where the probability was maximum. What our formulation does was maximum. What our formulation does was maximum. What our formulation does is areas where the probability is is areas where the probability is is areas where the probability is maximum high probability areas it will maximum high probability areas it will maximum high probability areas it will force the score to be zero. Which means force the score to be zero. Which means force the score to be zero. Which means that once someone comes here they will that once someone comes here they will that once someone comes here they will not be guided anywhere. It is a not be guided anywhere. It is a not be guided anywhere. It is a stationary point. They will just sit stationary point. They will just sit stationary point. They will just sit there which is what we want. Right? We there which is what we want. Right? We there which is what we want. Right? We want the sink to divert the want the sink to divert the want the sink to divert the people towards the actual samples and people towards the actual samples and people towards the actual samples and then once they are diverted you want then once they are diverted you want then once they are diverted you want them to sit there which is what uh them to sit there which is what uh them to sit there which is what uh happens with this term. So the locations happens with this term. So the locations happens with this term. So the locations become stationary.

  127. L6 @ 4381s

    Now uh once we have understood this, Now uh once we have understood this, this is where the actual training of the this is where the actual training of the this is where the actual training of the uh scores happen so that they match with uh scores happen so that they match with uh scores happen so that they match with the the the real scores as much as possible. And we real scores as much as possible. And we real scores as much as possible. And we will take a very interesting practical will take a very interesting practical will take a very interesting practical example to understand how this example to understand how this example to understand how this formulation is used formulation is used formulation is used to learn the score function. to learn the score function. to learn the score function. The example that we will take is we will The example that we will take is we will The example that we will take is we will assume that these are the data samples. assume that these are the data samples. assume that these are the data samples. Remember we are not assuming that the Remember we are not assuming that the Remember we are not assuming that the probability is known here. We are only probability is known here. We are only probability is known here. We are only assuming that the data samples is known. assuming that the data samples is known. assuming that the data samples is known. And from this we want to predict the And from this we want to predict the And from this we want to predict the score function from this data. We will not stop at this though. We'll We will not stop at this though. We'll predict the score function and then predict the score function and then predict the score function and then we'll sample through land dynamics to we'll sample through land dynamics to we'll sample through land dynamics to see if we can find any new points. see if we can find any new points. see if we can find any new points. uh which belong to this underlying uh which belong to this underlying uh which belong to this underlying distribution. So in a way we will take a distribution. So in a way we will take a

  128. L6 @ 4437s

    distribution. So in a way we will take a look at the whole uh whole pipeline. look at the whole uh whole pipeline. look at the whole uh whole pipeline. Okay, let's try to understand this. Okay, let's try to understand this. Okay, let's try to understand this. We'll look at uh this very interesting We'll look at uh this very interesting We'll look at uh this very interesting Google collab notebook. Google collab notebook. Google collab notebook. So when I come across core function, it So when I come across core function, it So when I come across core function, it reminds me of some of the concepts in reminds me of some of the concepts in reminds me of some of the concepts in fluid mechanics which I had learned. So fluid mechanics which I had learned. So fluid mechanics which I had learned. So it gives me a lot of pleasure to it gives me a lot of pleasure to it gives me a lot of pleasure to understand them. understand them. understand them. Okay. So in the first uh step we Okay. So in the first uh step we Okay. So in the first uh step we set up the data. We generate two set up the data. We generate two set up the data. We generate two interlocking moon shapes interlocking moon shapes interlocking moon shapes which look something like this. Okay. So the training data looks like Okay. So the training data looks like this. And in the in the next step what this. And in the in the next step what this. And in the in the next step what we do is we define the score neural we do is we define the score neural we do is we define the score neural network which network which network which looks like this. It is a simple multi-layer perceptron It is a simple multi-layer perceptron where as an output we predict the score where as an output we predict the score where as an output we predict the score values and this is where you and this is where you uh uh define your loss. We can see that the define your loss. We can see that the define your loss. We can see that the loss is made up of two terms. The first

  129. L6 @ 4518s

    loss is made up of two terms. The first loss is made up of two terms. The first is the trace of the Jacobian is the trace of the Jacobian is the trace of the Jacobian which is what we discussed before which is what we discussed before which is what we discussed before and the second is the nom square which and the second is the nom square which and the second is the nom square which is over here. is over here. is over here. Now you can see that both of these terms Now you can see that both of these terms Now you can see that both of these terms depend on this score prediction matrix depend on this score prediction matrix depend on this score prediction matrix which is what our model has predicted. which is what our model has predicted. which is what our model has predicted. It does not depend on the true scores at It does not depend on the true scores at It does not depend on the true scores at all. So we can easily define the loss all. So we can easily define the loss all. So we can easily define the loss function without knowing the true function without knowing the true function without knowing the true scores. So this is one of the most scores. So this is one of the most scores. So this is one of the most important step which is the score important step which is the score important step which is the score matching loss which all of us need to matching loss which all of us need to matching loss which all of us need to understand very clearly. This is exactly understand very clearly. This is exactly understand very clearly. This is exactly the same that we have discussed in the same that we have discussed in the same that we have discussed in theory. Okay. Uh let's let's go ahead. Okay. Uh let's let's go ahead. The output of the score network is two The output of the score network is two The output of the score network is two because it is a vector and we have because it is a vector and we have because it is a vector and we have predictions for the x score and the y predictions for the x score and the y

  130. L6 @ 4570s

    predictions for the x score and the y score to create a vector. score to create a vector. score to create a vector. Next we have a training loop which is a Next we have a training loop which is a Next we have a training loop which is a standard training loop in machine standard training loop in machine standard training loop in machine learning. And here I have trained for learning. And here I have trained for learning. And here I have trained for 800 epochs and you can see the loss 800 epochs and you can see the loss 800 epochs and you can see the loss going down with the time. going down with the time. going down with the time. And then finally as you uh train uh your And then finally as you uh train uh your And then finally as you uh train uh your scores to scores to scores to predict the true scores even without predict the true scores even without predict the true scores even without knowing the value of them just from the knowing the value of them just from the knowing the value of them just from the data samples. data samples. data samples. We ask this question that which way We ask this question that which way We ask this question that which way towards the data and uh this is what we towards the data and uh this is what we towards the data and uh this is what we get towards the end which is amazing get towards the end which is amazing get towards the end which is amazing right this this this looks like a valid right this this this looks like a valid right this this this looks like a valid score speed because we are pointing score speed because we are pointing score speed because we are pointing towards the data and if you look at any towards the data and if you look at any towards the data and if you look at any single point which is near your data single point which is near your data single point which is near your data distribution you can see that it doesn't distribution you can see that it doesn't distribution you can see that it doesn't completely appear like a sync but there

  131. L6 @ 4620s

    completely appear like a sync but there completely appear like a sync but there are arrows coming in from the left and are arrows coming in from the left and are arrows coming in from the left and the right arrows are not coming in from the right arrows are not coming in from the right arrows are not coming in from the top or the bottom here which is the top or the bottom here which is the top or the bottom here which is interesting interesting interesting But it it it does appear like it's But it it it does appear like it's But it it it does appear like it's creating syncs over there and these data creating syncs over there and these data creating syncs over there and these data points are stationary. So you will find points are stationary. So you will find points are stationary. So you will find you not find arrows which are very close you not find arrows which are very close you not find arrows which are very close to your data samples. to your data samples. to your data samples. Now you might say that Rajat we have now Now you might say that Rajat we have now Now you might say that Rajat we have now predicted the scores. Let's use langin predicted the scores. Let's use langin predicted the scores. Let's use langin dynamics and sample from it. Right? And dynamics and sample from it. Right? And dynamics and sample from it. Right? And what what we are going to do with what what we are going to do with what what we are going to do with languin dynamics is that languin dynamics is that languin dynamics is that let's say this is the uh score field. We let's say this is the uh score field. We let's say this is the uh score field. We start with any point and langaming start with any point and langaming start with any point and langaming dynamics is going to take us to a final dynamics is going to take us to a final dynamics is going to take us to a final point. point. point. Similarly, we start from any other point Similarly, we start from any other point Similarly, we start from any other point we go to any other point. So we are

  132. L6 @ 4669s

    we go to any other point. So we are we go to any other point. So we are going to see trajectories like these. going to see trajectories like these. going to see trajectories like these. And in this example, I have shown one And in this example, I have shown one And in this example, I have shown one single trajectory which is appearing in single trajectory which is appearing in single trajectory which is appearing in uh blue. So the trajectory starts over uh blue. So the trajectory starts over uh blue. So the trajectory starts over here which is the green dot and it ends here which is the green dot and it ends here which is the green dot and it ends over here. So you can see that it is over here. So you can see that it is over here. So you can see that it is ending at a place which is almost ending at a place which is almost ending at a place which is almost similar to where the data samples lie similar to where the data samples lie similar to where the data samples lie which is which is amazing right and it which is which is amazing right and it which is which is amazing right and it appears like again a drunken hiker which appears like again a drunken hiker which appears like again a drunken hiker which is going zigzag and the hiker finally is going zigzag and the hiker finally is going zigzag and the hiker finally lands to a place where you find the lands to a place where you find the lands to a place where you find the samples. So this lecture felt like a samples. So this lecture felt like a samples. So this lecture felt like a treasure hunt in a way that the treasure treasure hunt in a way that the treasure treasure hunt in a way that the treasure is located in some valleys with a is located in some valleys with a is located in some valleys with a minimum energy and we are trying to minimum energy and we are trying to minimum energy and we are trying to locate those valleys but we also locate those valleys but we also locate those valleys but we also perturve ourselves from the valleys so

  133. L6 @ 4713s

    perturve ourselves from the valleys so perturve ourselves from the valleys so that we don't miss any other treasures that we don't miss any other treasures that we don't miss any other treasures which are even bigger than what we have which are even bigger than what we have which are even bigger than what we have found. So we appear like a drunken hiker found. So we appear like a drunken hiker found. So we appear like a drunken hiker and we find these treasures finally and we find these treasures finally and we find these treasures finally and uh this paper is immensely helpful and uh this paper is immensely helpful and uh this paper is immensely helpful for us to uh derive these score matching for us to uh derive these score matching for us to uh derive these score matching loss. We have not gone into the uh loss. We have not gone into the uh loss. We have not gone into the uh details of it. It is slightly involved details of it. It is slightly involved details of it. It is slightly involved but I'm very grateful to their but I'm very grateful to their but I'm very grateful to their contribution because without that we contribution because without that we contribution because without that we will not have diffusion models that we will not have diffusion models that we will not have diffusion models that we see right now. see right now. see right now. Okay. So uh in the next chapter we will Okay. So uh in the next chapter we will Okay. So uh in the next chapter we will look at the foundational role of score look at the foundational role of score look at the foundational role of score function in modern diffusion models. function in modern diffusion models. function in modern diffusion models. Remember there are these two parallel Remember there are these two parallel Remember there are these two parallel tracks which I keep talking about but I tracks which I keep talking about but I tracks which I keep talking about but I have never made the connection between have never made the connection between have never made the connection between them yet. We will make the connection in them yet. We will make the connection in

  134. L6 @ 4759s

    them yet. We will make the connection in the next lecture. the next lecture. the next lecture. So initially used to enable efficient So initially used to enable efficient So initially used to enable efficient training of EBMS the score function has training of EBMS the score function has training of EBMS the score function has now evolved into a central component of now evolved into a central component of now evolved into a central component of a new generation of generative models a new generation of generative models a new generation of generative models which is which is amazing. which is which is amazing. which is which is amazing. And uh in this lecture we looked at a And uh in this lecture we looked at a And uh in this lecture we looked at a whole new framework which is based on whole new framework which is based on whole new framework which is based on energies. energies. energies. And uh similar to diffusion this is also And uh similar to diffusion this is also And uh similar to diffusion this is also inspired from physics. inspired from physics. inspired from physics. We want to we want our data to lie in We want to we want our data to lie in We want to we want our data to lie in configurations where the energy is configurations where the energy is configurations where the energy is minimum. minimum. And since we cannot predict the energy And since we cannot predict the energy And since we cannot predict the energy function directly, function directly, function directly, since the uh partition function which is since the uh partition function which is since the uh partition function which is the integral of the energy function over the integral of the energy function over the integral of the energy function over all possible data points is intractable, all possible data points is intractable, all possible data points is intractable, we define another objective function we define another objective function we define another objective function which is tractable and that is based on which is tractable and that is based on which is tractable and that is based on a term which is called as scores. a term which is called as scores. a term which is called as scores. Scores are given by gradient of Scores are given by gradient of

  135. L6 @ 4815s

    Scores are given by gradient of logarithm of uh the probability. It logarithm of uh the probability. It logarithm of uh the probability. It turns out that the scores appear as turns out that the scores appear as turns out that the scores appear as compass for us towards the data. If you compass for us towards the data. If you compass for us towards the data. If you if you are at any point in the grid, you if you are at any point in the grid, you if you are at any point in the grid, you take your compass out, it points to the take your compass out, it points to the take your compass out, it points to the direction of where exactly is the data. direction of where exactly is the data. direction of where exactly is the data. Now if we have this core function, we Now if we have this core function, we Now if we have this core function, we looked at a technique called long lang looked at a technique called long lang looked at a technique called long lang dynamics which allows us to sample from dynamics which allows us to sample from dynamics which allows us to sample from sample data sample data sample data which appears to be very close to the which appears to be very close to the which appears to be very close to the points where the distribution of the points where the distribution of the points where the distribution of the data lies in. We looked at an example data lies in. We looked at an example data lies in. We looked at an example where it's like two mountains and we where it's like two mountains and we where it's like two mountains and we started from any point and we reached started from any point and we reached started from any point and we reached towards one of the peak of the mountains towards one of the peak of the mountains towards one of the peak of the mountains using lang dynamics. using lang dynamics. using lang dynamics. Now the question is how do we train our Now the question is how do we train our Now the question is how do we train our neural network to match the scores and

  136. L6 @ 4862s

    neural network to match the scores and neural network to match the scores and to do that we use a score matching loss to do that we use a score matching loss to do that we use a score matching loss where we use an alternative expression where we use an alternative expression where we use an alternative expression which is tractable which which is tractable which which is tractable which very nicely does not involve anything to very nicely does not involve anything to very nicely does not involve anything to do with the probability of the data or do with the probability of the data or do with the probability of the data or the energy of the true data. I'm again the energy of the true data. I'm again the energy of the true data. I'm again very fascinated with the mathematics of very fascinated with the mathematics of very fascinated with the mathematics of it all. simple mathematical tricks can it all. simple mathematical tricks can it all. simple mathematical tricks can uh uh become the markers towards the starter become the markers towards the starter become the markers towards the starter of a whole new framework where now we of a whole new framework where now we of a whole new framework where now we are seeing energy based models used in are seeing energy based models used in are seeing energy based models used in diffusion as well. Thank you everyone. diffusion as well. Thank you everyone. diffusion as well. Thank you everyone. We are proceeding very nicely in this We are proceeding very nicely in this We are proceeding very nicely in this course and I'm excited to teach the next course and I'm excited to teach the next course and I'm excited to teach the next concept which is the role of score concept which is the role of score concept which is the role of score function in modern diffusion models.