diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 00:
03 | So in attention as I mentioned the main So in attention as I mentioned the main
idea is to have these token embeddings idea is to have these token embeddings idea is to have these token embeddings
or … |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 00:
52 | to capture
if I'm looking at a particular token if I'm looking at a particular token if I'm looking at a particular token
like brightite, how much attention I want to capture all of that I want to cap… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 01:
50 | vectors, value vectors, etc. So what is vectors, value vectors, etc. So what is
done is that we have this input matrix, done is that we have this input matrix, done is that we have this input matrix,
… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 02:
33 | with query key and values in the
attention block. Right? attention block. Right? attention block. Right?
Once we have the queries and the keys Once we have the queries and the keys Once we have the qu… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 03:
16 | by 5 right 1 2 3 4 5 and let me
duplicate this duplicate this duplicate this
1 2 3 1 2 3 1 2 3
4 and 5. 4 and 5. 4 and 5.
And And And
this is the this is the this is the
next next next
day is bright. … |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 04:
19 | statements like if I'm looking at
bright, bright, bright,
I want to make statements like give 10% I want to make statements like give 10% I want to make statements like give 10%
attention to the, give… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 05:
10 | to or give 50% importance to bright.
Um there is also the scaling by square Um there is also the scaling by square Um there is also the scaling by square
root of this square root of keys root of this … |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 05:
48 | multiply it with the value matrix and
this gives me my context vector matrix. this gives me my context vector matrix. this gives me my context vector matrix.
That's my final output from the That's my … |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 06:
45 | So what I'll do is that I'll take the So what I'll do is that I'll take the
values vector for each of these and values vector for each of these and values vector for each of these and
multiply it with… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 07:
37 | tokens. But now what I'm saying is that
when we look at next I have information when we look at next I have information when we look at next I have information
of how next relates to the other tokens.… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 08:
30 | Hogwarts and he Hogwarts and he
now I have encoded how he relates to now I have encoded how he relates to now I have encoded how he relates to
these different tokens. these different tokens. these dif… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 09:
30 | So, we are doing the next token
prediction task here. Um, so if next is prediction task here. Um, so if next is prediction task here. Um, so if next is
the input, I want to predict what comes the inpu… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 10:
11 | is no need of finding the attention
score between is and bright. But for score between is and bright. But for score between is and bright. But for
bright it's fine because all the other bright it's fi… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 10:
59 | essentially, I only need to find the
attention scores of what comes in this attention scores of what comes in this attention scores of what comes in this
triangle. I don't need these attention triangl… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 11:
42 | these attention score values are these attention score values are
essentially set to zero because we don't essentially set to zero because we don't essentially set to zero because we don't
need them a… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 12:
28 | attention mass does not exist for attention mass does not exist for
diffusion models. In multi head diffusion models. In multi head diffusion models. In multi head
attention what's actually done is th… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 13:
19 | that causal attention mask here right we
don't need these attention score so we don't need these attention score so we don't need these attention score so we
put them to zero. That's why it's called p… |
diffusion-lm-vizuara |
L10: Lecture 9: Auto Regressive Models (ARM) Attention Block | 14:
01 | explanation which I've tried to give explanation which I've tried to give
you. If you want to go and drill down you. If you want to go and drill down you. If you want to go and drill down
into the det… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 00:
03 | So once we come out of the attention So once we come out of the attention
block, we have a dropout layer. Uh let block, we have a dropout layer. Uh let block, we have a dropout layer. Uh let
me me me
… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 00:
55 | these vectors to zero. And the reason
dropout is implemented is to improve the dropout is implemented is to improve the dropout is implemented is to improve the
generalization performance. I always ge… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 01:
37 | the generalization performance, right?
U so after dropout we again have these U so after dropout we again have these U so after dropout we again have these
context vectors. Now take a look at the cont… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 02:
26 | we want to avoid the problem of
vanishing gradients. vanishing gradients. vanishing gradients.
The thing is if multiple gradients The thing is if multiple gradients The thing is if multiple gradients
… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 03:
23 | So we have to go from here to now here So we have to go from here to now here
which is the second layer normalization. which is the second layer normalization. which is the second layer normalization.… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 04:
08 | The activation layer here which is
generally preferred is JLU activation. generally preferred is JLU activation. generally preferred is JLU activation.
The way it differs from RLU is that uh The way i… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 04:
58 | been borrowed and which show up
repeatedly. So which are these elements? repeatedly. So which are these elements? repeatedly. So which are these elements?
You'll see that the dropout layer is You'll s… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 05:
44 | output to the transformer block. The
output of the transformer block is this. output of the transformer block is this. output of the transformer block is this.
Again, we have these tokens with the Aga… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 06:
55 | build neural networks from scratch, I build neural networks from scratch, I
think. Let's see. Or build LLM from think. Let's see. Or build LLM from think. Let's see. Or build LLM from
scratch new, I t… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 08:
24 | block goes to the second the output of
the second goes to the third. The output the second goes to the third. The output the second goes to the third. The output
of the third goes to the fourth to the… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 09:
05 | say transformer block output it's the say transformer block output it's the
output from those 12 24 or 96 output from those 12 24 or 96 output from those 12 24 or 96
transformer blocks that output goe… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 10:
04 | moves you right. This has as we have
seen this is one input sequence but it seen this is one input sequence but it seen this is one input sequence but it
actually has four input output actually has fo… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 11:
01 | training. Right? Initially I would have
some random random guesses. So this is some random random guesses. So this is some random random guesses. So this is
the target and I need to have the target an… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 11:
47 | and the predictions. Okay. So, how do we
get to the predictions? Currently we get to the predictions? Currently we get to the predictions? Currently we
just have uh the output of this just have uh the… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 12:
34 | that's the vocabulary size which will be that's the vocabulary size which will be
the number of columns. the number of columns. the number of columns.
So you have something like every effort
moves U a… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 13:
41 | vector it will make sure that each value vector it will make sure that each value
lies between 0 to one and they sum up to lies between 0 to one and they sum up to lies between 0 to one and they sum u… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 14:
36 | maybe token ID number 40,000
which is a boy So these are my predictions now and then So these are my predictions now and then
I'll find the loss function between the I'll find the loss function betwee… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 15:
34 | each other. As the target and the
prediction start to approximate each prediction start to approximate each prediction start to approximate each
other, what will happen is that the other, what will ha… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 16:
27 | model is trained properly the model is trained properly the
probability at token ID number three probability at token ID number three probability at token ID number three
should be equal to one should… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 17:
29 | my loss will be zero which is what we
want. But in this case the loss will be want. But in this case the loss will be want. But in this case the loss will be
negative log of2. negative log of2. negati… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 19:
33 | probability which should actually be one
but maybe it's.3 if every effort moves but maybe it's.3 if every effort moves but maybe it's.3 if every effort moves
is the input U should be the output. is th… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 21:
26 | if we take negative of this it will be if we take negative of this it will be
equal to 2.619 619 that's the loss for equal to 2.619 619 that's the loss for equal to 2.619 619 that's the loss for
this … |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 22:
14 | models are trained. I hope all of you
now see the end to end details of why now see the end to end details of why now see the end to end details of why
they are called as auto reggressive they are cal… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 22:
55 | All right. So if you take a look at the
parameters now I hope all of you can parameters now I hope all of you can parameters now I hope all of you can
visualize where all there are parameters visualiz… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 23:
50 | is the positional embeddings also a is the positional embeddings also a
parameter? Because we have the token parameter? Because we have the token parameter? Because we have the token
embedding matrix … |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 24:
49 | attention mechanism. So that's also a
parameter. If you scroll about to the parameter. If you scroll about to the parameter. If you scroll about to the
attention mechanism, you'll see uh WQ, attention… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 25:
56 | this is the feed forward neural network.
So we have 768 dimensions going to 4 So we have 768 dimensions going to 4 So we have 768 dimensions going to 4
into 768 and then that again going to into 768 a… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 26:
44 | parameters. We have parameters in the
final layer norm. And in the output final layer norm. And in the output final layer norm. And in the output
layer, why do we have parameters here? layer, why do w… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 27:
46 | contributes to the parameters in the
output layer. So all of these parameters output layer. So all of these parameters output layer. So all of these parameters
termed as P are the trainable parameters… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 28:
31 | start of this that to truly understand start of this that to truly understand
diffusion language models, we really diffusion language models, we really diffusion language models, we really
need to fir… |
diffusion-lm-vizuara |
L11: Lecture 10: Auto Regressive Models (ARM) Output and Loss Blocks | 29:
20 | we'll see how the ARM architecture is
the backbone of this predicting noise the backbone of this predicting noise the backbone of this predicting noise
step. So let's start looking at that step. So le… |
diffusion-lm-vizuara |
L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference | 00:
02 | Before moving on to the fusion model Before moving on to the fusion model
architecture architecture architecture
and uh assembling it from scratch, and uh assembling it from scratch, and uh assembling… |
diffusion-lm-vizuara |
L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference | 00:
59 | continues. That's pre-training of continues. That's pre-training of
language models or traditional language language models or traditional language language models or traditional language
model pre-tr… |
diffusion-lm-vizuara |
L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference | 01:
50 | we get the token embeddings we get the we get the token embeddings we get the
position embeddings we add the token position embeddings we add the token position embeddings we add the token
embeddings … |
diffusion-lm-vizuara |
L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference | 02:
40 | function like the way we have um looked
at on the whiteboard at on the whiteboard at on the whiteboard
right we saw that once we pass the input right we saw that once we pass the input right we saw th… |
diffusion-lm-vizuara |
L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference | 03:
24 | essentially the context length or the
sequence length. That's the number of sequence length. That's the number of sequence length. That's the number of
tokens in one tokens in one tokens in one
input … |
diffusion-lm-vizuara |
L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference | 04:
17 | not. Right? The whole idea when we
started this exercise was started this exercise was started this exercise was
um we wanted to train the model so that um we wanted to train the model so that um we w… |
diffusion-lm-vizuara |
L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference | 04:
58 | completely correct here. There are some
mistakes here and there, but overall the mistakes here and there, but overall the mistakes here and there, but overall the
grammar makes sense. The meaning also… |
diffusion-lm-vizuara |
L12: Lecture 11: Auto Regressive Models (ARM) Code: Pre training and Inference | 05:
40 | exactly, but mostly it's pretty cool. So
auto reggressive models definitely work auto reggressive models definitely work auto reggressive models definitely work
and they give good results. Right now w… |
diffusion-lm-vizuara |
L2: Lecture 1: Introduction to Diffusion Language Models | 00:
03 | Hello everyone. This is Dr. Raj Dandeek Hello everyone. This is Dr. Raj Dandeek
and uh today I'm very excited to present and uh today I'm very excited to present and uh today I'm very excited to prese… |
diffusion-lm-vizuara |
L2: Lecture 1: Introduction to Diffusion Language Models | 01:
01 | models from scratch. But that's not how models from scratch. But that's not how
I want this course or this learning I want this course or this learning I want this course or this learning
material to … |
diffusion-lm-vizuara |
L2: Lecture 1: Introduction to Diffusion Language Models | 01:
49 | scared of them, then how I learned the scared of them, then how I learned the
subject, how I mastered the subject now subject, how I mastered the subject now subject, how I mastered the subject now
so… |
diffusion-lm-vizuara |
L2: Lecture 1: Introduction to Diffusion Language Models | 02:
27 | model fully from scratch. You'll compare
it with the auto reggressive language it with the auto reggressive language it with the auto reggressive language
models. So it's an amazing exercise or models… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 00:
02 | So I hope all of us are on the same page So I hope all of us are on the same page
now with respect to the motivation now with respect to the motivation now with respect to the motivation
behind this l… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 00:
50 | questions which researchers should ask
that leads to the best research. that leads to the best research. that leads to the best research.
If you ask this question, you'll slowly If you ask this questi… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 01:
49 | models. This so this paper I believe
is a very important one mask diffusion is a very important one mask diffusion is a very important one mask diffusion
language models this is where many language mo… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 02:
51 | introduction section itself
uh that why should images and video uh that why should images and video uh that why should images and video
generation models have all the fun why generation models have al… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 03:
39 | images might lie and then sampling from
it. So why can't this same thing be it. So why can't this same thing be it. So why can't this same thing be
applied to text as well? applied to text as well? ap… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 04:
58 | when we talked about images, first we
broke down the images into pixels, broke down the images into pixels, broke down the images into pixels,
right? Similarly, text can be broken right? Similarly, te… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 05:
54 | So one way to look at generating text is So one way to look at generating text is
just one word at a time. Another way of just one word at a time. Another way of just one word at a time. Another way o… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 06:
57 | in high dimensional spaces? What does it
mean probability distribution of mean probability distribution of mean probability distribution of
meaningful text? Let's see. So text is trickier than images.… |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 08:
12 | sequence of words based on two things. sequence of words based on two things.
Grammar and logic both need to be Grammar and logic both need to be Grammar and logic both need to be
correct, right? And … |
diffusion-lm-vizuara |
L6: Lecture 5: Motivation behind Language Diffusion Models | 09:
12 | can be carried forward to languages. can be carried forward to languages.
What these people suggested is auto What these people suggested is auto What these people suggested is auto
reggressive models… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 00:
03 | To understand the concept of auto To understand the concept of auto
reggressive models, reggressive models, reggressive models,
let me go to the whiteboard and uh let me go to the whiteboard and uh le… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 01:
37 | upon a time and I want the language
model to complete this prompt. model to complete this prompt. model to complete this prompt.
What it will do is that it will not What it will do is that it will not… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 02:
23 | sequence then the next token is
predicted this is during the training predicted this is during the training predicted this is during the training
during inference also one new token is during inferenc… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 03:
06 | see multiple tokens here, it's one token
being generated at a time and the next being generated at a time and the next being generated at a time and the next
token which is generated depends on the to… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 03:
58 | one thing which works. But why can't
diffusion be the default option? This is diffusion be the default option? This is diffusion be the default option? This is
what Andre Karpati also mentions here, w… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 04:
46 | predicts one token at a time. So if you
see to generate the same amount of text see to generate the same amount of text see to generate the same amount of text
the right hand side which is diffusion t… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 05:
34 | it's fine because it's one new token
generated at a time but this reverse generated at a time but this reverse generated at a time but this reverse
reasoning becomes very difficult for the reasoning b… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 06:
21 | mentioned is that if the true language
distribution lies somewhere here can we distribution lies somewhere here can we distribution lies somewhere here can we
find the model can we make a diffusion fi… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 07:
13 | what's the need of auto reggressive what's the need of auto reggressive
models to achieve this to make the models to achieve this to make the models to achieve this to make the
probability distributio… |
diffusion-lm-vizuara |
L7: Lecture 6: Auto Regressive Models (ARMs) Introduction | 08:
02 | ure of diffusion models differs from ure of diffusion models differs from
auto reggressive models and then we are auto reggressive models and then we are auto reggressive models and then we are
going … |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 00:
02 | To truly understand To truly understand
diffusion language models, we actually diffusion language models, we actually diffusion language models, we actually
really need to understand how auto really n… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 01:
07 | small aspect like the feed forward small aspect like the feed forward
neural network block here etc. So this neural network block here etc. So this neural network block here etc. So this
is going to b… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 02:
05 | model. This data set essentially
consists of around 2 million stories consists of around 2 million stories consists of around 2 million stories
which are such that 3 to four year old which are such th… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 02:
51 | so that I can predict or I can generate
language. language. language.
And what does what does it mean? So the And what does what does it mean? So the And what does what does it mean? So the
goal is to… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 03:
58 | want to do. And what does it mean learn
patterns from the data? Well, the model patterns from the data? Well, the model patterns from the data? Well, the model
needs to understand about the English ne… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 04:
46 | the kicked. That means it has not
understand not understood the correct understand not understood the correct understand not understood the correct
grammatical constructions. And the model grammatical… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 05:
40 | Yeah. So
how do we get started with this task? how do we get started with this task? how do we get started with this task?
The first step is to look at our data The first step is to look at our data T… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 06:
49 | size. It defines how many tokens or how size. It defines how many tokens or how
many words the model can pay attention many words the model can pay attention many words the model can pay attention
to … |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 07:
38 | if this is my first input sequence,
right? I shift it to the right hand side right? I shift it to the right hand side right? I shift it to the right hand side
by one. If this is my second input by one… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 08:
38 | model to learn that given any token it model to learn that given any token it
has to predict the next token. So if one has to predict the next token. So if one has to predict the next token. So if one… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 09:
24 | input, little will be the output. And if
one day a little is the input, girl will one day a little is the input, girl will one day a little is the input, girl will
be the output. be the output. be the… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 10:
24 | so imagine the huge data set which we
had 2 million rows that's but that's had 2 million rows that's but that's had 2 million rows that's but that's
coupled into in let's say input sequence coupled in… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 11:
19 | and the true answer, I have my loss
function. Right? Once I have my loss function. Right? Once I have my loss function. Right? Once I have my loss
function, I find the gradient of the function, I find… |
diffusion-lm-vizuara |
L8: Lecture 7: Auto Regressive Models (ARM) Architecture Intro | 12:
10 | Um, then what I'll do is that I go
through my second batch, I get the loss, through my second batch, I get the loss, through my second batch, I get the loss,
I update the parameters, then I go I updat… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 00:
02 | So the way a batch is constructed is as So the way a batch is constructed is as
follows. Let's say for now we just follows. Let's say for now we just follows. Let's say for now we just
assume the batc… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 00:
50 | the third stage is the output stage.
When we look at diffusion language When we look at diffusion language When we look at diffusion language
models, there are changes in all of models, there are chan… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 01:
29 | first is that since the model cannot
understand text these tokens these are understand text these tokens these are understand text these tokens these are
called as tokens they are converted called as … |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 02:
27 | is given a token ID of 1 then this is 12 is given a token ID of 1 then this is 12
then this is 50 and this is 80. then this is 50 and this is 80. then this is 50 and this is 80.
These token ids are th… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 03:
12 | model is not trained the vectors for cat
and kitten might lie further apart from and kitten might lie further apart from and kitten might lie further apart from
each other. But once the language model… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 04:
01 | This is my whole vocabulary set. I
assign token ids to each of them. Let's assign token ids to each of them. Let's assign token ids to each of them. Let's
say this goes up till 100,000. This is say th… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 04:
55 | If I want to retrieve the embedding
vectors for these, I just go to my token vectors for these, I just go to my token vectors for these, I just go to my token
embedding matrix. I look at that embeddin… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 05:
52 | the same words. So their token embedding
vectors will be the same but they come vectors will be the same but they come vectors will be the same but they come
at different positions, right? So their at… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 06:
45 | length. So this goes from one to right
up to the sequence length. And I have a up to the sequence length. And I have a up to the sequence length. And I have a
768 dimensional vector for my position 76… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 07:
44 | corresponding to position one this is
the vector vector corresponding to the vector vector corresponding to the vector vector corresponding to
position two, vector corresponding to position two, vecto… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 08:
28 | block is where all the magic happens. So
initially we started out by uh thinking initially we started out by uh thinking initially we started out by uh thinking
that how will the model learn the that … |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 09:
13 | are also calling processor here. are also calling processor here.
This transformer block consists of all This transformer block consists of all This transformer block consists of all
of these componen… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 09:
58 | by another layer normalization followed
by feed forward followed by dropout. The by feed forward followed by dropout. The by feed forward followed by dropout. The
good thing is that the transformer go… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 10:
46 | that so now the input tokens let's say
so ideally we should we should have so ideally we should we should have so ideally we should we should have
continued with the same one day a little continued wi… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 11:
28 | root of variance. Why do we do this? root of variance. Why do we do this?
Because normalization really helps the Because normalization really helps the Because normalization really helps the
back prop… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 12:
17 | normalization. The second block which I normalization. The second block which I
want to emphasize a lot over here. This want to emphasize a lot over here. This want to emphasize a lot over here. This
… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 13:
11 | Harry Potter daughter
or Harry lived or Harry lived or Harry lived
at Hogwarts and he so let's say this is my input and he so let's say this is my input
sequence if I want to complete this sequence if… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 14:
05 | have not linked one token to another have not linked one token to another
token anywhere. Even this uh layer token anywhere. Even this uh layer token anywhere. Even this uh layer
normalization is an o… |
diffusion-lm-vizuara |
L9: Lecture 8: Auto Regressive Models (ARM) Architecture Data-Transformer Blocks | 14:
58 | the mask diffusion models as well. So
let's understand this attention let's understand this attention let's understand this attention
mechanism and what it's actually doing mechanism and what it's act… |