CMSC 828W: Foundations of Deep Learning

Welcome to the CMSC 828W course on Foundations of Deep Learning for Fall 2020.

CMSC 828W is currently being taught by Soheil Feizi.

This course will be taught virtually.

Announcements

• Lecture 1 (9/1): DL basics: ERM and basic architectures (MLP, ResNet and CNNs), overview of SGD, ADAMs, Momentum, etc.
  Video: Click here
  Readings:

  • The Deep Learning Book

• Lecture 2 (9/3): DL optimization: SGD convergence and its implicit bias.
  Video: Click here
  Readings:

  • Toward a theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning

  • Gradient Descent Provably Optimizes Over-parameterized Neural Networks

  • The Implicit Bias of Gradient Descent on Separable Data

• Lecture 3 (9/8): Cont. DL optimization
  Video: Click here
  

• Lecture 4 (9/10): DL optimization, Implicit Bias of GD
  Video: Click here
  

• Lecture 5 (9/15): DL generalization: Re-thinking generalization measures, over-parameterization and double descent.
  Video: Click here
  Readings:

  • Understanding deep learning requires rethinking generalization

  • Reconciling modern machine learning practice and the bias-variance trade-off

  • Fantastic Generalization Measures and Where to Find Them

  • Two models of double descent for weak features

  • Fantastic Generalization Measures and Where to Find Them

• Lecture 6 (9/17): Cont. DL generalization
  Video: Click here

• Lecture 7 (9/22): Neural tangent kernels
  Video: Click here
  Readings:

  • Neural Tangent Kernel: Convergence and Generalization in Neural Networks

  • A Note on Lazy Training in Supervised Differentiable Programming

• Lecture 8 (9/24): DL robustness: Adversarial attacks and defenses.
  Video: Click here
  Readings:

  • Towards Deep Learning Models Resistant to Adversarial Attacks

  • Towards Evaluating the Robustness of Neural Networks

• Lecture 9 (9/29): DL robustness: Gradient Masking + Are adversarial examples inevitable?
  Video: Click here
  Readings:

  • Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

  • Are adversarial examples inevitable?

• Lecture 10 (10/1): DL robustness: Provable and Generalizable Defenses
  Video: Click here
  Readings:

  • Certified Adversarial Robustness via Randomized Smoothing

  • Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness

  • Robustness Certificates for Sparse Adversarial Attacks by Randomized Ablation

  • Perceptual Adversarial Robustness: Defense Against Unseen Threat Models

• Lecture 11 (10/6): DL Robustness: Adversarial Poisoning Attacks and Defenses
  Video: Click here
  Readings:

  • Clean-Label Backdoor Attacks

  • Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks

  • Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks

  • Deep Partition Aggregation: Provable Defense against General Poisoning Attacks

• Lecture 12 (10/8): Deep generative models: VAEs.
  Video: Click here
  Readings:

  • Auto-Encoding Variational Bayes

  • β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

  • Neural Discrete Representation Learning

  • Generating Diverse High-Fidelity Images with VQ-VAE-2

  • NVAE: A Deep Hierarchical Variational Autoencoder

• Lecture 13 (10/13): Deep generative models: GANs.
  Video: Click here
  Readings:

  • Generative Adversarial Nets

  • Wasserstein GAN

  • Spectral Normalization for Generative Adversarial Networks

  • Large Scale GAN Training for High Fidelity Natural Image Synthesis

  • GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

  • Robust Optimal Transport with Applications in Generative Modeling and Domain Adaptation

  • Understanding GANs in the LQG setting

• Lecture 14 (10/15): DL optimization: Min-Max (Convex-Concave).
  Video: Click here
  Readings:

  • The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization

  • The Numerics of GANs

  • The Complexity of Constrained Min-Max Optimization

• Lecture 15 (10/20): DL optimization: Min-Max (NonConvex-NonConcave).
  Video: Click here

• Lecture 16 (10/22): Deep generative models: Flow-based models.
  Video: Click here
  Readings:

  • Density estimation using Real NVP

  • Glow: Generative Flow with Invertible 1×1 Convolutions

  • Do Deep Generative Models Know What They Don't Know?

  • Why Normalizing Flows Fail to Detect Out-of-Distribution Data?

  • Adversarial Robustness of Flow-Based Generative Models

• Lecture 17 (10/27): Relaxing i.i.d.: Domain adaptation (part I).
  Video: click here
  Readings:

  • Analysis of Representations for Domain Adaptation

  • Unsupervised Domain Adaptation by Backpropagation

  • CYCADA: Cyle-Consistent Adversarial Domain Adaptation

  • Impossibility Theorems for Domain Adaptation

• Lecture 18 (10/29): Relaxing i.i.d.: Domain adaptation (part II).
  Video: click here

• Lecture 19 (11/3): No Lecture

• Lecture 20 (11/5): Relaxing i.i.d.: Domain generalization.
  Video: Click here
  Readings:

  • In Search of Lost Domain Generalization

  • The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

  • The Risks of Invariant Risk Minimization

  • Learning to Generalize: Meta-Learning for Domain Generalization

• Lecture 21 (11/10): Contrastive, self-supervised DL
  Video: Click here
  Readings:

  • A Simple Framework for Contrastive Learning of Visual Representations

  • A Theoretical Analysis of Contrastive Unsupervised Representation Learning

  • Contrastive learning, multi-view redundancy, and linear models

• Lecture 22 (11/12): Contrastive, self-supervised DL
  Video: Click here

• Lecture 23 (11/17): + Meta-learning
  Video: Click here
  Readings:

  • Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

  • Meta-Learning with Differentiable Convex Optimization

  • Prototypical Networks for Few-shot Learning

  • On First-Order Meta-Learning Algorithms

• Lecture 24 (11/22): LSTMs and transformers
  Video: Click here
  Readings:

  • Long Short-Term Memory

  • Bidirectional recurrent neural networks

  • Attention Is All You Need

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• Lecture 25 (11/24): Explainable DL: Interpretation methods.
  Video: Click here
  Readings:

  • How to Explain Individual Classification Decisions

  • A Unified Approach to Interpreting Model Predictions

  • Axiomatic Attribution for Deep Networks

  • SmoothGrad: removing noise by adding noise

  • Sanity Checks for Saliency Maps

  • Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation

• Lecture 26 (11/26): Thanksgiving

• Lecture 27 (12/1): Deep RL
  Video: Click here
  Readings:

  • Proximal Policy Optimization Algorithms

  • Deep Reinforcement Learning with Double Q-learning

  • Policy Gradient Methods for Reinforcement Learning with Function Approximation

  • Human-level control through deep reinforcement learning

  • Deep Reinforcement Learning from Self-Play in Imperfect-Information Games

• Lecture 28 (12/3): Cont. Deep RL
  Video: Click here

• Lecture 29 (12/8): Cont. Deep RL
  Video: Click here

• Lecture 30 (12/10): Course summary + Final Exam
  Video: Click here