cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 00:
04 | So let's begin. Previously last class uh So let's begin. Previously last class uh
we have talked about what is diffusion we have talked about what is diffusion we have talked about what is diffusion
m… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 00:
57 | reserve sorry um all right so just a reserve sorry um all right so just a
reminder what is the forward process reminder what is the forward process reminder what is the forward process
basically ally … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 01:
44 | sums up together is another gausian and
you can just sum up a bunch of gausians. you can just sum up a bunch of gausians. you can just sum up a bunch of gausians.
So that's why in the forward process … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 02:
35 | is the diffusion for process. All right.
Now, uh how now that we uh we have the Now, uh how now that we uh we have the Now, uh how now that we uh we have the
forward process to train to construct forw… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 03:
24 | mean of the gausian uh and in addition
we can actually rewrite this mean this we can actually rewrite this mean this we can actually rewrite this mean this
prediction of mean into something that predi… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 04:
08 | last time there's some questions about
why should we do this reparameterization why should we do this reparameterization why should we do this reparameterization
well because doing this regression is … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 04:
54 | this uh before I do that does anyone
know in the audience what is normalizing know in the audience what is normalizing know in the audience what is normalizing
flow by any No way. Nobody know. No way.… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 05:
45 | to uh latent is the inverse of the
mapping from latent to data. So this is mapping from latent to data. So this is mapping from latent to data. So this is
sort of like normalizing flows. Um and sort o… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 06:
40 | if you have a auto reggressive models if you have a auto reggressive models
then you need to break things up by the then you need to break things up by the then you need to break things up by the
chai… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 07:
25 | And uh for normalizing flow just by
looking at this can anyone tell me what looking at this can anyone tell me what looking at this can anyone tell me what
potential problem it can it can have. Yes. Y… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 08:
23 | least. Um but then it as we have learned
last time it also has a problem of like last time it also has a problem of like last time it also has a problem of like
very unstable training and also it very… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 09:
24 | of images rather than actually having to of images rather than actually having to
comput the comput the comput the
>> yes that's right so you just need to >> yes that's right so you just need to >> ye… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 10:
13 | model so what you do is you sort of just
like follow the gra so say suppose you like follow the gra so say suppose you like follow the gra so say suppose you
have a gradient that gives you like u have… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 11:
00 | space right so basically what you can
imagine is that you can start from imagine is that you can start from imagine is that you can start from
anywhere on the data space so in this anywhere on the dat… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 11:
45 | respect to the data point of the log
likelihood so what is happening is that likelihood so what is happening is that likelihood so what is happening is that
like so and then again like what you can li… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 12:
31 | just like in diffusion right your source
distribution and your target distribution and your target distribution and your target
distribution should always be in the distribution should always be in th… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 13:
28 | gradient of uh of of P
X uh with respect to X. That's fine. But X uh with respect to X. That's fine. But X uh with respect to X. That's fine. But
later on we're going to see why because later on we're… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 14:
15 | is uh machine learning 101 right. So
very easy very easy uh uh uh very easy very easy uh uh uh very easy very easy uh uh uh
formulation. And this L2 here uh it has formulation. And this L2 here uh it … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 15:
08 | like auto reggressive models. So it can like auto reggressive models. So it can
sample or generate everything all at sample or generate everything all at sample or generate everything all at
once once… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 15:
47 | this L2 without this ground truth score.
Turns out you don't even need to use the Turns out you don't even need to use the Turns out you don't even need to use the
ground truth score in your loss grou… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 16:
35 | simplify uh the L2 the the original L2 simplify uh the L2 the the original L2
into two parts here. And then the first into two parts here. And then the first into two parts here. And then the first
pa… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 17:
27 | can have the gradient of uh log of p
with respect to x into you can break it with respect to x into you can break it with respect to x into you can break it
into this thing right uh and then these int… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 18:
13 | and then you can basically just get
something like this. So um yeah this is something like this. So um yeah this is something like this. So um yeah this is
a this is like we we probably wouldn't a thi… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 18:
58 | need to remember is that the second part
of the two things left. It's going to of the two things left. It's going to of the two things left. It's going to
come boils down to the negative of the come b… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 19:
41 | loss that we get, we don't really need a
ground true score. It's it's like ground true score. It's it's like ground true score. It's it's like
actually pretty pretty sick. So all you actually pretty p… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 20:
38 | function has the same dimensionality of function has the same dimensionality of
the of the data, right? Yeah. So, you the of the data, right? Yeah. So, you the of the data, right? Yeah. So, you
can ju… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 21:
37 | no one else. Do you have a Do you have a
Do you have a Do you have a Do you have a
>> Well, yes, but it's kind of related to >> Well, yes, but it's kind of related to >> Well, yes, but it's kind of re… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 22:
42 | Literally Gausian. Okay. So basically
what you do is uh and this is what we what you do is uh and this is what we what you do is uh and this is what we
call dnoising score matching. Why so get call dn… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 23:
25 | is you just need a sample data and then
perturb it a little bit and now it perturb it a little bit and now it perturb it a little bit and now it
becomes this like it's easier to becomes this like it's… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 24:
15 | literally this. Uh so yeah, for for literally this. Uh so yeah, for for
those of you who wonder why log density those of you who wonder why log density those of you who wonder why log density
easier b… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 25:
25 | truth it's a perturbed ground truth then truth it's a perturbed ground truth then
you may have a question but like let's you may have a question but like let's you may have a question but like let's
m… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 26:
04 | anyway it's fine. Um uh so what you do
is you first draw draw a sample from is you first draw draw a sample from is you first draw draw a sample from
your data space using some easy to your data space… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 26:
50 | So this is what could have happened if
you do laundry dynamic essentially. So you do laundry dynamic essentially. So you do laundry dynamic essentially. So
at the beginning, right, we just like at the… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 27:
44 | anyway you just train your model to
predict the score and then you use predict the score and then you use predict the score and then you use
dynamic to sample. Uh however does it dynamic to sample. Uh… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 29:
00 | survey. Let's survey. Uh we are starting
from this column again. from this column again. from this column again.
Hello friends. Uh Hello friends. Uh Hello friends. Uh
does anyone want to share their t… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 30:
06 | haven't been that I haven't been talked haven't been that I haven't been talked
to today. There's so many people in the to today. There's so many people in the to today. There's so many people in the
… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 31:
12 | is what we call the manifold
hypothesis. What does it mean is that hypothesis. What does it mean is that hypothesis. What does it mean is that
basically usually the data that we want basically usually… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 31:
59 | what here says like some area of the what here says like some area of the
data space will not have support. What data space will not have support. What data space will not have support. What
it means … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 32:
43 | manifold from some point from on the
manifold to some point that's not on the manifold to some point that's not on the manifold to some point that's not on the
manifold the score is going to blow up m… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 33:
28 | then the second thing. Yeah. Yeah. Yeah.
>> What is this the left plot even showing >> What is this the left plot even showing >> What is this the left plot even showing
either? Oh sorry the the the s… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 34:
10 | space or entire data space now has
support uh and then the score matching support uh and then the score matching support uh and then the score matching
actually behave really really nicely actually be… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 34:
57 | image is eventually boiled down to say
10 features or something in that case 10 features or something in that case 10 features or something in that case
the manifold is load eventually how is the mani… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 35:
45 | there's no probability and that's why there's no probability and that's why
the score is going to blow up and adding the score is going to blow up and adding the score is going to blow up and adding
s… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 36:
42 | is like they they they train on the is like they they they train on the
clean data. So it's like not I mean yes clean data. So it's like not I mean yes clean data. So it's like not I mean yes
obviousl… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 37:
28 | without any data right? So what's going
to happen here is that um so say this is to happen here is that um so say this is to happen here is that um so say this is
like your actual data score that your… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 38:
14 | really know where to move in the low
density region. So you're just going to density region. So you're just going to density region. So you're just going to
move sort of randomly and then you you move… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 39:
02 | high density from higher density right high density from higher density right
so like say this is your like intended so like say this is your like intended so like say this is your like intended
data … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 39:
48 | >> it it depends on like what exactly is >> it it depends on like what exactly is
this distribution say if this uh this distribution say if this uh this distribution say if this uh
distribution is a m… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 40:
34 | Okay.
All right. Any other question? Cool. Cool.
All right. So, we have all these All right. So, we have all these All right. So, we have all these
problems. problems. problems.
Is it problem? Is it p… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 41:
48 | thing doesn't exist anymore, right? And thing doesn't exist anymore, right? And
then now the perturbed like say you have then now the perturbed like say you have then now the perturbed like say you ha… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 42:
37 | large of a of a noise, then it you're
just learning a very noisy distribution. just learning a very noisy distribution. just learning a very noisy distribution.
That's not what we want. So how much Th… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 43:
33 | about which region that we are uh that about which region that we are uh that
we're s we should be sampling at, then we're s we should be sampling at, then we're s we should be sampling at, then
we're… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 44:
19 | sample from, you are going to just
gradually decrease the noise or the gradually decrease the noise or the gradually decrease the noise or the
exploration term that you're adding at exploration term t… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 45:
05 | have that next? uh we have the pseudo
code and you know the long dynam dynamic code and you know the long dynam dynamic code and you know the long dynam dynamic
left over. Yeah. But basically literall… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 45:
50 | And then at training time you do this
multi-level den noising score matching multi-level den noising score matching multi-level den noising score matching
because at each time you basically because at… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 47:
23 | Yeah, just just think about it. Uh,
basically the difference between DDPM, basically the difference between DDPM, basically the difference between DDPM,
which is what we learned last last time, which … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 48:
48 | that the the the score is sort of it has that the the the score is sort of it has
like another factor or it has another like another factor or it has another like another factor or it has another
coef… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 49:
49 | we have infinite numbers of noise level? we have infinite numbers of noise level?
What do you what do we think is going to What do you what do we think is going to What do you what do we think is goin… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 50:
47 | tailor estimation of of of the real
thing and you can actually write both of thing and you can actually write both of thing and you can actually write both of
them into this this format and uh if you … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 51:
42 | because you see how like basically if
you do xt plus delta t minus xt is going you do xt plus delta t minus xt is going you do xt plus delta t minus xt is going
to be this thing right. So it's going t… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 52:
28 | you just add some stoasticity. So this you just add some stoasticity. So this
is why it's called stochastic is why it's called stochastic is why it's called stochastic
differential equation. different… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 53:
28 | and we choose the for process right so
we know f and g uh and then it's we know f and g uh and then it's we know f and g uh and then it's
literally just gausian right so who literally just gausian rig… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 54:
21 | to use a oiler solver which is like to use a oiler solver which is like
probably simple simplest solver uh that probably simple simplest solver uh that probably simple simplest solver uh that
then you… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 55:
09 | apply the change in x to x and then we apply the change in x to x and then we
apply delta t to t apply delta t to t apply delta t to t
and that's it. and that's it. and that's it.
And then we iterate … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 56:
13 | written as this like this function this written as this like this function this
this uh so there are four process can be this uh so there are four process can be this uh so there are four process can … |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 57:
20 | the G as variance and in the VE
formulation we do not have any formulation we do not have any formulation we do not have any
adjustment of the X like whatever X you adjustment of the X like whatever X… |
cmu-10799-diffusion-flow |
L1: CMU 10799 S26: Lecture 4 - Score-based Models - Diffusion & Flow Matching | 58:
09 | did diffusion. This class we did did diffusion. This class we did
scorebased model and we also learned scorebased model and we also learned scorebased model and we also learned
that they're basically … |
cmu-10799-diffusion-flow |
L9: CMU 10799 S26: Lecture 10 - Distillation, Consistency Models & Flow Maps - Diffusion & Flow Matching | 47:
17 | otherwise it doesn't make sense I guess.
Yeah. >> Yeah. Oh sorry. Yeah. >> Yeah. Oh sorry. Yeah.
>> How do you weigh the different loss >> How do you weigh the different loss >> How do you weigh the d… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 00:
04 | Hello everyone, welcome to the next Hello everyone, welcome to the next
lecture of the course principles of lecture of the course principles of lecture of the course principles of
diffusion models. di… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 01:
07 | calculate the magnetic field for all the
magnets and superimpose them together so magnets and superimpose them together so magnets and superimpose them together so
that we have a cumulative universal … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 02:
13 | idea where this ball came from. idea where this ball came from.
But we have an idea where it came from. But we have an idea where it came from. But we have an idea where it came from.
And essentially … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 03:
31 | Okay. And
the force required to pull the ball back the force required to pull the ball back the force required to pull the ball back
to the start is the score field for the to the start is the score f… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 04:
34 | Now this can also be represented as a
noise vector. Right? noise vector. Right? noise vector. Right?
[snorts] This is our flick. So [snorts] This is our flick. So [snorts] This is our flick. So
essent… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 05:
37 | And if this perturbed noise is very And if this perturbed noise is very
small, you could assume that whatever small, you could assume that whatever small, you could assume that whatever
score my I am … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 06:
33 | exactly what we saw here. Minus noise
divided by sigma. divided by sigma. divided by sigma.
And the predicted score is what our And the predicted score is what our And the predicted score is what our
… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 07:
41 | as I said before uh the field of score
matching has evolved matching has evolved matching has evolved
through years. So in 2005 was the proof through years. So in 2005 was the proof through years. So … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 08:
49 | that.
It's a very beautifully written paper It's a very beautifully written paper It's a very beautifully written paper
and in fact the textbook which I'm and in fact the textbook which I'm and in fac… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 09:
43 | image or data sample into noise and then
we predicted the reverse distribution we predicted the reverse distribution we predicted the reverse distribution
from the noise back to the data. from the noi… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 10:
59 | Now look at what is happening here. Your Now look at what is happening here. Your
space is huge. This is the your entire space is huge. This is the your entire space is huge. This is the your entire
s… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 12:
08 | 2005 which we explained and this is the
loss that we try to minimize. Okay. Now
this score is given by gradient of this score is given by gradient of this score is given by gradient of
log of p of x. … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 13:
29 | completely fine. Everything is defined.
This the slope is defined right? You This the slope is defined right? You This the slope is defined right? You
know exactly how your data changes as know exactl… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 14:
27 | score vector will not be defined. score vector will not be defined.
And imagine that you're going with a And imagine that you're going with a And imagine that you're going with a
compass in your hand … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 15:
28 | transformed from a low dimensional
manifold to a higher dimensional space. manifold to a higher dimensional space. manifold to a higher dimensional space.
It's taking up the space It's taking up the s… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 17:
00 | Remember in the previous formulation by Remember in the previous formulation by
Vincent, we just had one noise. Vincent, we just had one noise. Vincent, we just had one noise.
We we added one noise wi… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 18:
24 | you are flicking the magnet
not just once with one level, but you're not just once with one level, but you're not just once with one level, but you're
flicking it with multiple levels. flicking it wit… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 19:
42 | multiple levels of noise? multiple levels of noise?
We don't do this in diffusion also. In We don't do this in diffusion also. In We don't do this in diffusion also. In
diffusion there is one constant… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 20:
48 | is it a single neural network which is
trained on trained on trained on
uh for all these losses and if it's a uh for all these losses and if it's a uh for all these losses and if it's a
single neural … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 22:
13 | Remember now the score does not just Remember now the score does not just
depend on X depend on X depend on X
but it depends on X comma sigma where but it depends on X comma sigma where but it depends… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 23:
27 | and land as close as possible to the and land as close as possible to the
data point. Now the interesting thing is that I have Now the interesting thing is that I have
a score function for a score fun… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 24:
58 | The only difference is that it is
performed sequentially performed sequentially performed sequentially
multiple times. Let us understand how we multiple times. Let us understand how we multiple times.… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 26:
21 | Okay. So you are purposely deliberately
inputting a very high noise level so inputting a very high noise level so inputting a very high noise level so
that that
uh you move to areas where you probably… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 27:
17 | here [snorts] the destination is not our
data but it's far away from our data data but it's far away from our data data but it's far away from our data
because we are adding so much noise. So because … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 28:
10 | we are moving according to the lang
dynamics update rule. This time the dynamics update rule. This time the dynamics update rule. This time the
variance is a bit lower and um variance is a bit lower a… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 29:
07 | sample the entire data, right? Even if, sample the entire data, right? Even if,
let's say, you have handwritten digits let's say, you have handwritten digits let's say, you have handwritten digits
and… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 30:
00 | matching formula that it it it finally
tries to predict the noise itself and tries to predict the noise itself and tries to predict the noise itself and
that that is exactly what the diffusion that th… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 31:
01 | please go through this for people who please go through this for people who
like to see how both are connected and like to see how both are connected and like to see how both are connected and
it's it… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 32:
14 | uh and then
which is exactly what which is exactly what which is exactly what
we want to predict right we want to we want to predict right we want to we want to predict right we want to
understand tha… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 33:
12 | the reverse transition kernel is we are the reverse transition kernel is we are
trying to predict how much noise was trying to predict how much noise was trying to predict how much noise was
added in … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 34:
09 | when we are inferring we do it n uh we when we are inferring we do it n uh we
do it l times as as we saw here there do it l times as as we saw here there do it l times as as we saw here there
are l di… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 34:
56 | this and you do it for some specific
time steps t so if the time step is time steps t so if the time step is time steps t so if the time step is
greater than t you just stop this greater than t you ju… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 35:
59 | So this is exactly how the anal So this is exactly how the anal
land dynamics loop looks like. land dynamics loop looks like. land dynamics loop looks like.
Let's take a look at a practical example Le… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 37:
01 | you
uh define your score network which is uh define your score network which is uh define your score network which is
typically a conditional unit. typically a conditional unit. typically a conditiona… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 38:
08 | And that is exactly what we see in the
practical U notebook as well. practical U notebook as well. practical U notebook as well.
[snorts] But this is done for different [snorts] But this is done for d… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 39:
15 | if I run this inference for this epox
where the loss is very low, you can see where the loss is very low, you can see where the loss is very low, you can see
it's almost 554, it's almost 554, it's alm… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 40:
06 | manifold
and by adding noise we are spreading it and by adding noise we are spreading it and by adding noise we are spreading it
so that it occupies the entire space. so that it occupies the entire sp… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 40:
48 | which have very low density of of the
data. And adding noise helps us to do data. And adding noise helps us to do data. And adding noise helps us to do
that. It helps us to have the gradient that. It … |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 41:
41 | strength of low strength also. And once strength of low strength also. And once
we train the network to understand all we train the network to understand all we train the network to understand all
the… |
diffusion-principles-vizuara |
L4: Lecture 7 - Noise Conditional Score Networks | Principles of Diffusion Models | 42:
36 | images, videos or audio. As an homework, images, videos or audio. As an homework,
I want you to play around with this I want you to play around with this I want you to play around with this
notebook. … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 00:
03 | Hello everyone, welcome to the next Hello everyone, welcome to the next
lecture of the course principles of lecture of the course principles of lecture of the course principles of
diffusion models. di… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 01:
15 | in space and you go in that space with a
compass which tells you which direction compass which tells you which direction compass which tells you which direction
should I move so that I move closer to … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 02:
45 | probability distribution. Right? So we
do not know this term. do not know this term. do not know this term.
In supervised machine learning, we often In supervised machine learning, we often In supervi… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 03:
54 | requirement of this entire probability requirement of this entire probability
distribution which we have no idea distribution which we have no idea distribution which we have no idea
about. about. abo… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 05:
07 | data which is available to us which is
what makes it incredibly powerful. So we are grateful to this author to So we are grateful to this author to
have made the stream of score matching have made the… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 06:
08 | is close to the sink is pulled inwards. is close to the sink is pulled inwards.
It's like a black hole. If you go near a It's like a black hole. If you go near a It's like a black hole. If you go near… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 06:
51 | Now assume that you you are close to the
data. Assume that you are the data data. Assume that you are the data data. Assume that you are the data
itself. Then you don't need to move itself. Then you d… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 07:
48 | for this training data at every single for this training data at every single
point in the space. You can see the point in the space. You can see the point in the space. You can see the
score function… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 08:
50 | So just if you look at this visually you
can see that wherever I start in the can see that wherever I start in the can see that wherever I start in the
field and I follow the score function I field an… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 10:
02 | function? So and even we have looked at
one example where uh the loss function one example where uh the loss function one example where uh the loss function
actually works. actually works. actually wo… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 11:
19 | the diagonal elements. You can't the diagonal elements. You can't
individually calculate the diagonal individually calculate the diagonal individually calculate the diagonal
elements. elements. elemen… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 12:
37 | in in real life use cases. And that is
why for a lot of for many years actually why for a lot of for many years actually why for a lot of for many years actually
uh uh uh
the score matching technique … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 14:
00 | the same word dn noising right and in
diffusion that made sense because we diffusion that made sense because we diffusion that made sense because we
started from a noisy image and then we started from… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 15:
10 | These three papers are going to form the
foundation of the theory of denoising foundation of the theory of denoising foundation of the theory of denoising
score matching. Okay. So this technique was i… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 16:
22 | formulation which was trackable. But
that did not make sense because the that did not make sense because the that did not make sense because the
complexity of computations squared complexity of comput… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 17:
42 | and you do have access to some of them.
So let's say you have access to 15 or 20 So let's say you have access to 15 or 20 So let's say you have access to 15 or 20
of these magnets but many others are … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 18:
48 | nearest magnet is pulling me. If I take
a point here I could say that this is a point here I could say that this is a point here I could say that this is
the direction in which the nearest the directi… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 19:
45 | uh I will I will superimpose the uh I will I will superimpose the
magnetic fields for all these different magnetic fields for all these different magnetic fields for all these different
magnets. magne… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 20:
51 | We take any magnet on the table and we We take any magnet on the table and we
flick it with a hand just like this. I'm flick it with a hand just like this. I'm flick it with a hand just like this. I'm… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 21:
49 | the student, look, this is the ball's
new location. new location. new location.
and we hide the original uh magnet and we hide the original uh magnet and we hide the original uh magnet
location. So we… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 23:
02 | student has no idea where this
new ball position came from. But we know new ball position came from. But we know new ball position came from. But we know
where it came from. where it came from. where … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 24:
02 | balls, all these magnets that you know balls, all these magnets that you know
that the magnets are placed there. You that the magnets are placed there. You that the magnets are placed there. You
place… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 25:
02 | this point should be should move so that
it goes back to the original position it goes back to the original position it goes back to the original position
and that is exactly the direction of the and … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 26:
19 | to a new location. to a new location.
So the noisy new spot is represented as So the noisy new spot is represented as So the noisy new spot is represented as
xhat. xhat. xhat.
Uh this is I have actual… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 27:
22 | score does exactly the same thing. The
score tells you the direction score tells you the direction score tells you the direction
to the actual data in the space. And our to the actual data in the spac… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 28:
42 | of mistakes but the feedback is given of mistakes but the feedback is given
from this. This is the true direction from this. This is the true direction from this. This is the true direction
where whic… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 29:
59 | It is not like your true It is not like your true
probability distribution of data which probability distribution of data which probability distribution of data which
is not tractable but it is tracta… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 31:
00 | given any point in space
how can I pull this point so that it how can I pull this point so that it how can I pull this point so that it
becomes as close as possible to your becomes as close as possibl… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 32:
02 | really making sense. I have an excellent really making sense. I have an excellent
explanation to that which will come soon explanation to that which will come soon explanation to that which will come … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 33:
05 | perturvation to the data. We can
simplify the score function which we simplify the score function which we simplify the score function which we
want to learn as follows. want to learn as follows. want… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 34:
14 | uh the mean of this pixel is centered at uh the mean of this pixel is centered at
0.5 because the value is 0.5 but then you add a small noise with a but then you add a small noise with a
standard devi… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 35:
36 | And uh in fact
this can be represented this can be represented this can be represented
using a standard normal distribution using a standard normal distribution using a standard normal distribution
li… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 36:
55 | expression for this probability
distribution. distribution. distribution.
And uh because it is tractable once we And uh because it is tractable once we And uh because it is tractable once we
know the … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 37:
56 | anything is whatever is there in the
exponential itself. So you get this at exponential itself. So you get this at exponential itself. So you get this at
the bottom and then when you take the bottom a… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 39:
07 | you you you might have thought why
didn't I guess it before because it it didn't I guess it before because it it didn't I guess it before because it it
should be the vector which points from should be… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 40:
18 | predict we are trying to match the noise
the noise vector which has been added in the noise vector which has been added in the noise vector which has been added in
the to the clean data. the to the cl… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 41:
22 | which means that the density is maximum
near somewhere around this maybe minus4 near somewhere around this maybe minus4 near somewhere around this maybe minus4
and the density is maximum somewhere and… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 42:
55 | So this is the neural network that we So this is the neural network that we
define and the loss function is the main define and the loss function is the main define and the loss function is the main
s… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 43:
51 | Okay. And now what is the target score?
The target score is something which The target score is something which The target score is something which
you're trying to estimate which is minus you're tryi… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 44:
59 | And uh finally you see the learned score And uh finally you see the learned score
function which looks something like function which looks something like function which looks something like
this. Now … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 45:
56 | magnet. magnet.
Similarly, if you are somewhere on the Similarly, if you are somewhere on the Similarly, if you are somewhere on the
right, uh the score is negative because right, uh the score is nega… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 46:
42 | left you reach magnet number one. If you
are on the far right, you reach magnet are on the far right, you reach magnet are on the far right, you reach magnet
number two. And if you are anywhere in num… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 47:
32 | We first add a noise and then we try to We first add a noise and then we try to
predict that noise. predict that noise. predict that noise.
And in that process of predicting the And in that process of… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 48:
34 | us away from the local minima so that we
explore the global minimas as well. explore the global minimas as well. explore the global minimas as well.
So uh this is what we get So uh this is what we get… |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 49:
35 | much of a difference. Something might be much of a difference. Something might be
happening in the 3D space right that happening in the 3D space right that happening in the 3D space right that
adding … |
diffusion-principles-vizuara |
L5: Lecture 6 - Denoising Score Matching | Principles of Diffusion Models | 50:
26 | Okay. So this is it for dnoising score
matching and uh in the next part matching and uh in the next part matching and uh in the next part
uh I will explain about noise condition uh I will explain abou… |