Schedule

Lectures are Tuesday and Thursday, 9:30–10:59 AM in The Gateway Building 1210. Discussion sections and homework deadlines are both on Friday, with homework due at 10:59 PM. Lecture pages unlock on the day of the lecture.

The topic schedule is tentative and will be adjusted during the semester based on student and instructor feedback, as well as developments in the field.

Week 0: Introduction

Tue, Aug 25
Thu, Aug 27	Lecture 0 Intros, Logistics, high-level themes/goals
Fri, Aug 28

Week 1: Basic Principles and Optimization

Tue, Sep 01	Lecture 1 Basic Principles
Thu, Sep 03	Lecture 2 Optimization: Implicit Regularization, SGD, and Momentum
Fri, Sep 04	Discussion 1Visualizing ReLU, Loss Funcs and SGD Homework 0 DueVector calculus, ridge regularization, basic coding of fully connected networks

Week 2: Adam, Local Linearity, Initialization

Tue, Sep 08	Lecture 3 Adam, Local Linear Perspective, Initialization
Thu, Sep 10	Lecture 4 RMS RMS norm, Maximal update parameterization (μP)
Fri, Sep 11	Discussion 2Initializations and different optimizers Homework 1 DueLearning rates, stepsizes, Momentum, SGD, Adam, and more basics.

Week 3: Maximal Update Parameterization

Tue, Sep 15	Lecture 5 MuP
Thu, Sep 17	Lecture 6 Guest lecture
Fri, Sep 18	Discussion 3Topic to be announced Homework 2 DueXavier/He initialization and RMS Norm, Features and Taylor Expansions, Adam as signSGD and how it changes what we converge to, Demonstrating why muP is needed, induced norms and why Adam and Shampoo fit into this framework.

Week 4: Convolutional Neural Networks

Tue, Sep 22	Lecture 7 Convolutional neural nets
Thu, Sep 24	Lecture 8 Convolutional neural nets (Continued)
Fri, Sep 25	Discussion 4Topic to be announced Homework 3 DueBatchnorm, Dropout and Implicit Regularization

Week 5: Convolutional Neural Networks

Tue, Sep 29	Lecture 9 Convolutional neural nets (Continued)
Thu, Oct 01	Lecture 10 Finish Neural Nets
Fri, Oct 02	Discussion 5Topic to be announced Homework 4 DueConvolution, Filters, ConvNets and Inductive Bias, CNN dimensions

Week 6: Graph Neural Networks

Tue, Oct 06	Lecture 11 Graph Neural Nets
Thu, Oct 08	Lecture 12 Finish GNN: Pooling, Start RNN
Fri, Oct 09	Discussion 6Topic to be announced Homework 5 DueConv-nets continued

Week 7: Recurrent Networks and Self-Supervision

Tue, Oct 13	Lecture 13 RNNs, Self-supervision
Thu, Oct 15	Lecture 14 Self-supervision, State-space models
Fri, Oct 16	Discussion 7Topic to be announced Homework 6 DueMore Resnet and Graph Neural Networks

Week 8: State-Space Models

Tue, Oct 20	Lecture 15 State-space models
Thu, Oct 22	Lecture 16 State space models Continued
Fri, Oct 23	Discussion 8Topic to be announced Homework 7 DueTopics to be announced

Week 9: Attention and Transformers

Tue, Oct 27	Lecture 17 Attention
Thu, Oct 29	Lecture 18 Attention recap, Transformer architecture, Embedding layer, Positional encoding
Fri, Oct 30	Discussion 9Topic to be announced Homework 8 DueRecurrent Neural Networks, LSTM, Seq-to-Seq Models

Week 10: Transformer Architectures

Tue, Nov 03	Lecture 19 Position Encoding, GPT-3, BERT, Other architectures
Thu, Nov 05	Lecture 20 Guest lecture
Fri, Nov 06	Discussion 10Topic to be announced Homework 9 DueSelf-supervision, Autoencoders, Attention

Week 11: Embeddings, Soft Prompting, LoRA

Tue, Nov 10
Thu, Nov 12	Lecture 21 Embeddings, soft prompting, LoRA
Fri, Nov 13	Discussion 11Topic to be announced Homework 10 DueCoding Transformers, Masked Auto-encoders

Week 12: Meta-Learning and Transfer Learning

Tue, Nov 17	Lecture 22 Meta-Learning, Transfer Learning
Thu, Nov 19	Lecture 23 Generative Models
Fri, Nov 20	Discussion 12Topic to be announced Homework 11 DueMeta-Learning, Finetuning, Transfer

Week 13: Generative Models

Tue, Nov 24	Lecture 24 Generative Models
Thu, Nov 26	Holiday (Thanksgiving)
Fri, Nov 27	Thanksgiving holiday — campus closed, no discussion sections Homework 12 DueGenerative Models, Prompt Engineering

Week 14: Generative Models

Tue, Dec 01	Lecture 25 Generative Models
Thu, Dec 03	Lecture 26 Generative Models
Fri, Dec 04	Discussion 13Topic to be announced Homework 13 DueDiffusion models

Week 15: RRR Week

Tue, Dec 08	Project Poster Session
Thu, Dec 10	Review session
Fri, Dec 11	Homework 14 DueAdvanced Topics

Week 16: Finals Week

Tue, Dec 15
Thu, Dec 17	Exam Final Exam — exact slot within finals week to be confirmed by the registrar
Fri, Dec 18