Assignments

• Organize into project groups of 4-5 by 9/14
• HW 0 available here, due 9/16
• Project proposal template available here, due 10/9
• Exam 1 scheduled for 10/19 in-class
• HW 1 available here, due 11/11
• HW 2 due 12/2
• Exam 2 scheduled for 12/9 in-class
• Project report due 12/18

Schedule

Make sure to reload this page to ensure you're seeing the latest version. Readings should be done before watching the corresponding lecture videos. See this page for materials (videos / slides / reading) from the Fall 2025 offering.

Week 1 (8/31-9/2): introduction, language modeling

• • Course introduction // [video] // [notes]
  • No associated readings!
  • HW 0 released here, due 9/16
  • Final projects:
  • • organize into groups of 4-5 by 9/14
    • proposal due 10/9 on Gradescope, use this Overleaf template


• • Language modeling // [video] // [notes]
  • [reading] Jurafsky & Martin, 3.1-3.5 (language modeling)
  • [reading] Jurafsky & Martin, 7 (neural language models)

Week 2 (9/9): neural language models

• • No class 9/7 (Labor Day)


• • Neural language models // [video] // [notes]
  • [reading] Neural language models (Bengio 2003)

Week 3 (9/14-16): Neural language models, backpropagation

• • Neural language models (cont.): decoding and composition // [video] // [notes]
  • same readings as last class


• • Neural language models (cont.): backpropagation // [video] // [notes]
  • [optional reading] Andrej Karpathy's coding-based backpropagation post
  • [optional reading] Pascanu et al., ICML 2013 (vanishing gradients in RNNs)
  • See F2020 video timestamped here for derivation of gradient through linear layer

Week 4 (9/21-23): Transformer language models

• • Attention mechanisms & self-attention // [video] // [notes]
  • [reading] Bahdanau et al., ICLR 2015 (paper that introduced attention)
  • [optional reading] An easy-to-read blog post on attention


• • Transformer language models // [video] // [notes]
  • [reading] Vaswani et al., NeurIPS 2017 (paper that introduced Transformers)
  • [reading] An easy-to-read blog post on Transformer language models

Week 5 (9/28-30): Mixture of experts, training stages

• • Mixture of experts models // [video] // [notes]
  • [reading] Switch Transformers (Fedus et al., 2021)


• • Pretraining vs. post-training // [video] // [notes]
  • [reading] Instruction tuning (Wei et al., 2022, FLAN)
  • [reading] Nathan Lambert's slides on post-training

Week 6 (10/5-7): Tokenization, Transformer configurations

• • Tokenization & Transformer configurations // [video] // [notes]
  • [reading] Neural Machine Translation... with Subword Units (Sennrich et al., ACL 2016)
  • [reading] Catherine Arnett's blogpost on tokenizers
  • [optional reading] Byte Latent Transformer: Patches Scale Better Than Tokens (Pagnoni et al., 2025)


• • No class 10/7 (Mohit traveling to COLM)

Week 7 (10/14): Scaling laws, exam review

• • No class 10/12 (Fall break)


• • Scaling laws of LLMs + exam 1 review // [video] // [notes]
  • [reading] Scaling Laws for Neural Language Models (Kaplan et al., 2020)
  • [reading] Training Compute-Optimal Large Language Models (Chinchilla, Hoffmann et al., 2022)

Week 8 (10/19-21): In-class exam, in-context learning

• • 10/19: In-class exam 1


• • In-context learning / instruction tuning // [video] // [notes]
  • [reading] Instruction tuning (Wei et al., 2022, FLAN)

Week 9 (10/26-28): Parameter-efficient fine-tuning, on-policy distillation

• • Parameter-efficient fine-tuning // [video] // [notes]
  • [reading] LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
  • [optional reading] Power of Scale for Prompt Tuning (Lester et al., EMNLP 2021)


• • On-policy distillation // [video]
  • [reading] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (Agarwal et al., ICLR 2024)

Week 10 (11/2-4): On-policy distillation

• • On-policy distillation (cont.) // [video] // [notes]
  • same reading as last class