Systems for Machine Learning (CMSC828G)

Lecture Paper Readings

There will be two assigned readings for each lecture (starting on Feb 25). All students are supposed to read all the assigned papers before the lecture. Your class participation grade for the course will be determined by:

• Questions about the reading: Every student will submit 2-3 questions or discussion topics on one of the readings (If your last name starts with A-N: Reading 1; if your last name starts with P-Z: Reading 2). These will be due 9 AM on the date of the lecture. Name the PDF that you will submit on gradescope as follows: MMDD-LastName-FirstName.pdf where MM and DD are the month and date of the lecture.
• Short presentation on the reading: Once in the semester, each student (paired with another student) will do a short 5-minute presentation on one of the assigned readings. You will upload a PDF of the presentation by 9 AM on the date of the lecture. Name the PDF that you will submit on gradescope as follows: MMDD-LastName-FirstName.pdf where MM and DD are the month and date of the lecture. These presentations should be 4-5 minutes (in total including both students' parts) and follow the provided format (PDF, PPTX).

Resource on how to read a scientific paper.

Lecture Slides

No.	Date	Topic and Slides	Reading 1	Presenters	Reading 2	Presenters	Additional Reading
1	Jan 28	Course Introduction and Overview
2	Jan 30	Introduction to HPC / Systems
	Feb 4	(contd.)
3	Feb 6	Introduction to GPU Programming
4	Feb 11	Introduction to Triton Programming
5	Feb 13	Introduction to Deep Learning
6	Feb 18	Transformers and Performance Modeling	Attention 2017
	Feb 20	No class
7	Feb 25	Challenges in High Performance DL	COTS HPC 2013		Extra-Deep 2023
8	Feb 27	Parallel Training	PyTorch DDP 2020	UB, KB	PyTorch FSDP 2023	AB, ZCh
	Mar 4	(contd.)	Megatron-LM 2019	PD, CDz	AxoNN 2024	Guest - Siddharth Singh
	Mar 6	Pipeline and Hybrid Parallel Training	GPipe 2018	DE, AH	Hybrid Parallelism 2021	Guest - Deepak Narayanan [video]
10	Mar 11	Optimizing GPU Kernels	Sputnik 2020	HH, LH	Flash Attention 2022	Guest - Tri Dao [video]
	Mar 13	Deep Learning Compilers	TVM 2018	DJ, DK	TorchDynamo 2024	Guest - Jason Ansel [video]
	Mar 18	Spring Break
	Mar 20	Spring Break
	Mar 25	Optimizers	HyLo 2022	BM, AN	Distributed Shampoo 2023	Guest - Shi & Iwasaki [video]
	Mar 27	Sparsity in Training	MoE 2017	SP, SS	MegaBlocks 2022	Guest - Trevor Gale [video]
	Apr 1	Memory offload	vDNN 2016	IR, MS	ZeRO-Infinity 2021
	Apr 3	Introduction to Inference	Transformers 2022	XT, PU	vLLM 2023
	Apr 8	Approximating Attention	Top-k 2021	CU, WW	H2O 2023
	April 10	Midterm Exam (during class)
	Apr 15	Long context optimizations	LongBench 2023	CW, YW	RingAttention 2023
	Apr 17	Quantization	LLM.int8() 2022	SY, NY	AWQ 2024		GPTQ 2022
	Apr 22	Optimizing Data Movement	PCCL 2025	YZ, ZZ	DataStates-LLM 2024	Guest - Avinash Maurya [video]
	Apr 24	Hardware-software Co-design	Cerebras 2023	RQ, YB	DLRM 2021
	Apr 29	Specific DL Models	CAGNET 2020	ZCa, PC	DLRM 2020	Guest - Bilge Acun [video]
	May 1	No class
	May 6	Project Presentations
	May 8	Project Presentations
	May 13	Project Presentations
	May 15	Final Project Due