L19: Lecture 18: Diffusion LLM Series Conclusion
Transcript
-
All right, everyone. So, this brings us All right, everyone. So, this brings us to the end of this diffusion language to the end of this diffusion language to the end of this diffusion language model learning journey model learning journey model learning journey which I went on. which I went on. which I went on. And ultimately, we built a diffusion And ultimately, we built a diffusion And ultimately, we built a diffusion language model fully from scratch. We language model fully from scratch. We language model fully from scratch. We understood the nuts and bolts of how the understood the nuts and bolts of how the understood the nuts and bolts of how the architecture is assembled. architecture is assembled. architecture is assembled. The idea was not to make a 5-minute or The idea was not to make a 5-minute or The idea was not to make a 5-minute or 10-minute tutorial, but to show you 10-minute tutorial, but to show you 10-minute tutorial, but to show you everything from scratch to open the everything from scratch to open the everything from scratch to open the black box so that you can do research on black box so that you can do research on black box so that you can do research on top of this, which means that you can top of this, which means that you can top of this, which means that you can vary different hyper parameters, run vary different hyper parameters, run vary different hyper parameters, run ablation studies, maybe build your first ablation studies, maybe build your first ablation studies, maybe build your first open-source diffusion language model, open-source diffusion language model, open-source diffusion language model, which is as good as Gwen. which is as good as Gwen. which is as good as Gwen. All of these things are possible now All of these things are possible now All of these things are possible now once this black box is opened and you once this black box is opened and you once this black box is opened and you realize realize realize exactly how diffusion models work from exactly how diffusion models work from exactly how diffusion models work from scratch. Okay. I know this tutorial is scratch. Okay. I know this tutorial is scratch. Okay. I know this tutorial is not completed. not completed. not completed. I've not shown supervised fine-tuning,
-
I've not shown supervised fine-tuning, I've not shown supervised fine-tuning, which you can do on your own since at which you can do on your own since at which you can do on your own since at different sections of this video, I've different sections of this video, I've different sections of this video, I've also shown you how to learn things on also shown you how to learn things on also shown you how to learn things on your own. your own. your own. So, SFT is a part which you can do on So, SFT is a part which you can do on So, SFT is a part which you can do on your own. your own. And I hope hope that I will cover some And I hope hope that I will cover some And I hope hope that I will cover some more things which I did not get more things which I did not get more things which I did not get time to cover. I hope that I will cover time to cover. I hope that I will cover time to cover. I hope that I will cover some more mathematical aspects, some more mathematical aspects, some more mathematical aspects, but I could not fit it in this tutorial. but I could not fit it in this tutorial. but I could not fit it in this tutorial. So, that is also something which someone So, that is also something which someone So, that is also something which someone wants to dive deeper mathematically, you wants to dive deeper mathematically, you wants to dive deeper mathematically, you can feel free to do that, of course. can feel free to do that, of course. can feel free to do that, of course. I also wanted to uh take I also wanted to uh take I also wanted to uh take for example, this fine-web-edu fine-web-edu dataset and train a fine-web-edu dataset and train a diffusion model on top of it. I don't diffusion model on top of it. I don't diffusion model on top of it. I don't think I've not seen this being done in think I've not seen this being done in think I've not seen this being done in an open-source way. So, you can be the an open-source way. So, you can be the
-
an open-source way. So, you can be the first one to do this. The same pipeline first one to do this. The same pipeline first one to do this. The same pipeline which we have built for small language which we have built for small language which we have built for small language models for the tiny stories dataset, you models for the tiny stories dataset, you models for the tiny stories dataset, you can do it for the fine-web-edu dataset can do it for the fine-web-edu dataset can do it for the fine-web-edu dataset and that will be an amazing and an and that will be an amazing and an and that will be an amazing and an awesome project. awesome project. awesome project. So, the goal at Visual AI is always to So, the goal at Visual AI is always to So, the goal at Visual AI is always to show you things from scratch, um make show you things from scratch, um make show you things from scratch, um make sure foundations are built and at the sure foundations are built and at the sure foundations are built and at the same time inculcate a research mindset. same time inculcate a research mindset. same time inculcate a research mindset. Thanks a lot, everyone. I hope you found Thanks a lot, everyone. I hope you found Thanks a lot, everyone. I hope you found this tutorial useful. this tutorial useful. this tutorial useful. And I hope many of you get into And I hope many of you get into And I hope many of you get into diffusion language models as a research diffusion language models as a research diffusion language models as a research topic because this can be the next big topic because this can be the next big topic because this can be the next big thing thing thing apart from autoregressive language apart from autoregressive language apart from autoregressive language models. Thanks.