CMSC 828W: Foundations of Deep Learning

Welcome to the CMSC 828W course on Foundations of Deep Learning for Fall 2022.

CMSC 828W is currently being taught by Soheil Feizi.

Announcements

• Lecture 1 (8/30): DL basics: ERM and basic architectures (MLP, ResNet and CNNs), overview of SGD, ADAMs, Momentum, etc.
  Readings:

  • The Deep Learning Book

• Lecture 2 (9/1): Neural tangent kernels
  Readings:

  • Neural Tangent Kernel: Convergence and Generalization in Neural Networks

  • A Note on Lazy Training in Supervised Differentiable Programming

• Lecture 3 (9/6): Guest Lecture by Prof. Misha Belkin (UCSD)
  Remarkable phenomena of deep learning through the prism of interpolation.
  Video Lecture
  Readings:

  • (main) Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation

  • (additional) A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors

  • (additional) On the Universality of the Double Descent Peak in Ridgeless Regression

• Lecture 4 (9/8): Guest Lecture by Prof. Andrew Wilson (NYU)
  How do we build models that learn and generalize?
  Video Lecture
  Readings:

  • (main) Bayesian Deep Learning and a Probabilistic Perspective of Generalization

  • (main) Bayesian Model Selection, the Marginal Likelihood, and Generalization

  • (additional) Dangers of Bayesian Model Averaging under Covariate Shift

  • (additional) Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

  • (additional) Residual Pathway Priors for Soft Equivariance Constraints

• Lecture 5 (9/13): Guest Lecture by Prof. Simon Du (University of Washington)
  Passive and Active Multi-Task Representation Learning
  Video Lecture
  Readings:

  • (main) Active Multi-Task Representation Learning

  • (main) Few-Shot Learning via Learning the Representation, Provably

• Lecture 6 (9/15): Self-supervised DL
  Readings:

  • A Simple Framework for Contrastive Learning of Visual Representations

  • Emerging Properties in Self-Supervised Vision Transformers

  • Bootstrap your own latent: A new approach to self-supervised Learning

  • Emerging Properties in Self-Supervised Vision Transformers

  • Exploring Simple Siamese Representation Learning

• Lecture 7 (9/20): Guest Lecture by Dr. Balaji Lakshminarayanan (Google Brain)
  Plex: Towards Reliability using Pretrained Large Model Extensions
  Video Lecture
  Readings:

  • Plex: Towards Reliability using Pretrained Large Model Extensions, Read Blog

  • Practical Uncertainty Estimation & Out-of-Distribution Robustness in Deep Learning

  • A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness

  • Deep Ensembles: A Loss Landscape Perspective

• Lecture 8 (9/22): Guest Lecture by Prof. Daniel Roy and Mufan (Bill) Li (U Toronto)
  The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
  Video Lecture
  Readings:

  • The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization

• Lecture 9 (9/27): Deep generative models: VAEs.
  Readings:

  • Auto-Encoding Variational Bayes

  • β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

  • Neural Discrete Representation Learning

  • Generating Diverse High-Fidelity Images with VQ-VAE-2

  • NVAE: A Deep Hierarchical Variational Autoencoder

• Lecture 10 (9/29): Guest Lecture by Prof. Jonathan Frankle (Harvard)
  Faster Neural Network Training, Algorithmically
  Video Lecture
  Readings:

  • ResNet strikes back: An improved training procedure in timm

  • Mosaic ResNet Recipe Overview

  • Mosaic ResNet Recipe Deep Dive

  • Mosaic Research Methodology

• Lecture 11 (10/4): Guest Lecture by Prof. Mahdi Soltanolkotabi (USC)
  Feature learning via gradient descent beyond the NTK lazy regime and deep learning for inverse problems
  Video Lecture
  Readings:

  • Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction

  • Neural networks can Learn Representations with Gradient Descent

  • HUMUS-NET: Hybrid unrolled multi-scale network architecture for accelerated MRI reconstruction

  • Data augmentation for deep learning based accelerated MRI reconstruction with limited data

• Lecture 12 (10/6): Guest Lecture by Prof. Constantinos Daskalakis (MIT)
  Equilibrium Complexity and Deep Learning
  Video Lecture
  Readings:

  • The Complexity of Constrained Min-Max Optimization

  • The Complexity of Markov Equilibrium in Stochastic Games

• Lecture 13 (10/11): Guest Lecture by Dr. Xinyun Chen (Google)
  Learning-Based Program Synthesis
  Video Lecture
  Readings:

  • SpreadsheetCoder: Formula Prediction from Semi-structured Context

  • Execution-Guided Neural Program Synthesis

  • Compositional Generalization via Neural-Symbolic Stack Machines

  • Competition-Level Code Generation with AlphaCode

  • Compositional Semantic Parsing with Large Language Models

• Lecture 14 (10/13): Deep generative models: GANs.
  Readings:

  • Generative Adversarial Nets

  • Wasserstein GAN

  • Spectral Normalization for Generative Adversarial Networks

  • Large Scale GAN Training for High Fidelity Natural Image Synthesis

  • Robust Optimal Transport with Applications in Generative Modeling and Domain Adaptation

  • Understanding GANs in the LQG setting

• Lecture 15 (10/18): Guest Lecture by Prof. Dorsa Sadigh (Stanford)
  Learning Robot Policies from Non-Traditional Sources of Human Data
  Video Lecture
  Readings:

  • PLATO: Predicting Latent Affordances Through Object-Centric Play

  • Imitation Learning by Estimating Expertise of Demonstrators

  • LILA: Language-Informed Latent Actions

  • Controlling Assistive Robots with Learned Latent Actions

• Lecture 16 (10/20): Domain adaptation and generalization.
  Readings:

  • Analysis of Representations for Domain Adaptation

  • Unsupervised Domain Adaptation by Backpropagation

  • CYCADA: Cyle-Consistent Adversarial Domain Adaptation

  • In Search of Lost Domain Generalization

  • The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

• Lecture 17 (10/25): Midpoint Presentations

• Lecture 18 (10/27): Midpoint Presentations

• Lecture 19 (11/1): Guest Lecture by Amanda Coston (CMU)
  Principled machine learning for societally consequential decision making
  Readings:

  • Counterfactual risk assessments, evaluation, and fairness

  • Counterfactual Predictions under Runtime Confounding

• Lecture 20 (11/3): Flow-based models.
  Readings:

  • Density estimation using Real NVP

  • Glow: Generative Flow with Invertible 1×1 Convolutions

  • Do Deep Generative Models Know What They Don't Know?

  • Adversarial Robustness of Flow-Based Generative Models

• Lecture 21 (11/8): Meta-learning
  Readings:

  • Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

  • Meta-Learning with Differentiable Convex Optimization

  • Prototypical Networks for Few-shot Learning

  • On First-Order Meta-Learning Algorithms

• Lecture 22 (11/10): Guest Lecture by Kristy Choi (Stanford)
  Harnessing Creative Machines with Controllable Generation
  Readings:

  • Density Ratio Estimation via Infinitesimal Classification

  • Featurized Density Ratio Estimation

  • Fair Generative Modeling via Weak Supervision

  • Encoding Musical Style with Transformer Autoencoders

• Lecture 23 (11/13): Adversarial Robustness
  Readings:

  • Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

  • Perceptual Adversarial Robustness: Defense Against Unseen Threat Models

  • Improved, Deterministic Smoothing for L1 Certified Robustness

  • Are adversarial examples inevitable?

  • Explicit Tradeoffs between Adversarial and Natural Distributional Robustness

• Lecture 24 (11/17): Guest Lecture by Arash Vahdat (Nvidia Research)
  Denoising Diffusion Models: the Generative Learning Champion of the 2020s
  Readings:

  • Score-based Generative Modeling in Latent Space

  • Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

  • GENIE: Higher-Order Denoising Diffusion Solvers

• Lecture 25 (11/22): Poisoning Robustness
  Readings:

  • Clean-Label Backdoor Attacks

  • Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks

  • Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks

  • Deep Partition Aggregation: Provable Defense against General Poisoning Attacks

  • Improved Certified Defenses against Data Poisoning with (Deterministic) Finite Aggregation

  • Lethal Dose Conjecture on Data Poisoning

• Lecture 26 (11/24): Thanksgiving

• Lecture 27 (11/29): Final Project Presentations

• Lecture 28 (12/1): Final Project Presentations

• Lecture 29 (12/6): Final Project Presentations

• Lecture 30 (12/8): Final Exam