Stanford CS336: Language Modeling from Scratch | Spring 2026
Percy Liang and Tatsunori Hashimoto
Stanford’s course on building a language model from start to finish. Its 18 lectures go through tokenization, Transformer architectures and attention alternatives, GPUs and custom kernels with Triton, parallel training, scaling laws, inference and evaluation, then pretraining data and post-training with RL and alignment. It is a graduate course and expects solid Python, PyTorch and machine learning.
What you'll learn
- Tokenization and Transformer architecture choices
- Writing GPU kernels with Triton
- Parallelizing training across GPUs
- Scaling laws and compute budgets
- Collecting and filtering pretraining data
- Post-training with reinforcement learning and alignment
