Train Your Own LLM – Tutorial

This course is designed to help beginners learn how to train a language model from start to finish. Imad will guide you through the whole process, using Moroccan Darija as an example. In this course, you will learn: How to load text data How to train a tokenizer from scratch using the Byte Pair Encoding (BPE) method How to use the tokenizer to encode text data How the Transformer architecture works in language models How to pre-train a model How to create a supervised fine-tuning dataset How to fine-tune the model and build an AI assistant that you can chat with You can find the slides, notebook, and scripts in this GitHub repository: https://github.com/ImadSaddik/Train_Y... The supervised fine-tuning dataset is available here: https://github.com/ImadSaddik/BoDmagh... https://huggingface.co/datasets/ImadS... The tokenizers trained on AtlaSet can be found here: https://github.com/ImadSaddik/DarijaT... You can access the AtlaSet on HuggingFace here: https://huggingface.co/datasets/atlas... To connect with Imad Saddik, check out his social accounts: LinkedIn: LinkedIn: imadsaddik YouTube: @3codecampers Discord: imad_saddik ❤️ Support for this channel comes from our friends at Scrimba – the coding platform that's reinvented interactive learning: https://scrimba.com/freecodecamp ⭐️ Course Contents ⭐️ (0:00:00) About the Course (0:03:03) Introduction (0:07:24) Training Data (0:15:33) Tokenization (0:29:00) The Transformer Architecture (0:52:21) Pre-training (1:24:46) Fine-tuning Dataset (1:33:05) Instruction Fine-tuning (2:06:17) Fine-tuning with LoRA (2:20:39) Let's Scale Everything (3:09:40) Bonus (3:27:10) Conclusion 🎉 Thanks to our Champion and Sponsor supporters: 👾 Drake Milly 👾 Ulises Moralez 👾 Goddard Tan 👾 David MG 👾 Matthew Springman 👾 Claudio 👾 Oscar R. 👾 jedi-or-sith 👾 Nattira Maneerat 👾 Justin Hual -- Learn to code for free and get a developer job: https://www.freecodecamp.org Read hundreds of articles on programming: https://freecodecamp.org/news