CMSC 720: Foundations of Deep Learning

Welcome to the CMSC 720 course (previously 828W) on Foundations of Deep Learning for Spring 2024.

CMSC 720 is currently being taught by Soheil Feizi.

Announcements

• Video Lectures

• Lecture 1: Deep Classification Basics: ERM and basic architectures (MLP, ResNet and CNNs), overview of SGD, ADAMs, Momentum, etc.
  Readings:

  • The Deep Learning Book

• Lecture Block 2: Transformers and Vision Transformers
  Readings:

  • Attention is all you need

  • Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Deep Generative Models for Images

• Lecture Block 3: Variational AutoEncoders (VAEs)
  Readings:

  • Auto-Encoding Variational Bayes

  • β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

  • Neural Discrete Representation Learning

  • Generating Diverse High-Fidelity Images with VQ-VAE-2

  • NVAE: A Deep Hierarchical Variational Autoencoder

• Lecture Block 4: Generative Adversarial Networks (GANs)
  Readings:

  • Generative Adversarial Nets

  • Wasserstein GAN

  • Spectral Normalization for Generative Adversarial Networks

  • Large Scale GAN Training for High Fidelity Natural Image Synthesis

  • GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

  • Robust Optimal Transport with Applications in Generative Modeling and Domain Adaptation

  • Understanding GANs in the LQG setting

• Lecture Block 5: Normalizing Flows
  Readings:

  • Density estimation using Real NVP

  • Glow: Generative Flow with Invertible 1×1 Convolutions

  • Do Deep Generative Models Know What They Don't Know?

  • Why Normalizing Flows Fail to Detect Out-of-Distribution Data?

  • Adversarial Robustness of Flow-Based Generative Models

• Lecture Block 6: Diffusion Models
  Readings:

  • Denoising Diffusion Probabilistic Models

  • High-Resolution Image Synthesis with Latent Diffusion Models

  • Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation

  • Understanding Diffusion Models: A Unified Perspective

  • Score-Based Generative Modeling through Stochastic Differential Equations

  • Adding Conditional Control to Text-to-Image Diffusion Models

Large Language Models (LLMs)

• Lecture Block 7: Encoder/Decoder Models
  Readings:

  • T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  • GPT3: Language Models are Few-Shot Learners

• Lecture Block 8: Training/Fine-tuning Language Models
  Readings:

  • Llama 2: Open Foundation and Fine-Tuned Chat Models

  • LoRA: Low-Rank Adaptation of Large Language Models

  • Prefix-Tuning: Optimizing Continuous Prompts for Generation

  • Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

• Lecture Block 9: In-Context Learning, Chain of Thoughts, Reasoning
  Readings:

  • Large Language Models are Zero-Shot Reasoners

  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

  • An Explanation of In-context Learning as Implicit Bayesian Inference

  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Multi-Modal Models

• Lecture Block 10: Vision Language Models
  Readings:

  • CLIP: Learning Transferable Visual Models From Natural Language Supervision

  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

  • SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

  • VirTex: Learning Visual Representations from Textual Annotations

  • Flamingo: a Visual Language Model for Few-Shot Learning

  • VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

  • LLaVA: Visual Instruction Tuning

  • VisualBERT: A Simple and Performant Baseline for Vision and Language

  • FLAVA: A Foundational Language And Vision Alignment Model

• Lecture Block 11: Compositionality Issues in Vision Language Models
  Readings:

  • When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?

  • Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

  • Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality

  • COLA: A Benchmark for Compositional Text-to-image Retrieval

• Lecture Block 12: Spurious Correlation Issues in Deep Models
  Readings:

  • Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases

  • Salient ImageNet: How to discover spurious features in Deep Learning?

  • Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

  • Explicit Tradeoffs between Adversarial and Natural Distributional Robustness

Adversarial Robustness

• Lecture Block 13: Adversarial Attacks and Defenses on Vision Models
  Readings:

  • Towards Deep Learning Models Resistant to Adversarial Attacks

  • Towards Evaluating the Robustness of Neural Networks

  • Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

  • Diffpure: Diffusion Models for Adversarial Purification

  • Instruct2Attack: Language-Guided Semantic Adversarial Attacks

  • Are adversarial examples inevitable?

  • Certified Adversarial Robustness via Randomized Smoothing

  • Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness

  • Improved, Deterministic Smoothing for L1 Certified Robustness

  • Robustness Certificates for Sparse Adversarial Attacks by Randomized Ablation

  • Perceptual Adversarial Robustness: Defense Against Unseen Threat Models

• Lecture Block 14: Adversarial Attacks and Defenses on Large Language Models
  Readings:

  • Universal and Transferable Adversarial Attacks on Aligned Language Models

  • AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

  • Jailbreaking Black Box Large Language Models in Twenty Queries

  • MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

  • Baseline Defenses for Adversarial Attacks Against Aligned Language Models

  • Certifying LLM Safety against Adversarial Prompting

• Lecture Block 15: Poisoning Attacks and Defenses on Vision Models
  Readings:

  • Clean-Label Backdoor Attacks

  • Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks

  • Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks

  • Deep Partition Aggregation: Provable Defense against General Poisoning Attacks

  • Run-Off Election: Improved Provable Defense against Data Poisoning Attacks

• Lecture Block 16: Poisoning Attacks and Defenses on Large Language Models
  Readings:

  • Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

  • Poisoning Language Models During Instruction Tuning (Note: This link appears to be a duplicate of the first one)

  • BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT

  • Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attacks

Detection of AI-Generated Content

• Lecture Block 17: Detection of AI-generated Text
  Readings:

  • TweepFake: about Detecting Deepfake Tweets

  • DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature

  • A Watermark for Large Language Models

  • Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense

  • Can AI-Generated Text be Reliably Detected?

• Lecture Block 18: Detection of AI-generated Images
  Readings:

  • Robust Invisible Video Watermarking with Attention

  • StegaStamp: Invisible Hyperlinks in Physical Photographs

  • Digital Watermarking and Steganography

  • A Recipe for Watermarking Diffusion Models

  • Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust

  • FaceForensics: Learning to Detect Manipulated Facial Images

  • Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks

Data Extraction and Model Editing

• Lecture Block 19: Data Extraction from Vision/Language Models
  Readings:

  • Extracting Training Data from Diffusion Models

  • Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models

  • Extracting Training Data from Large Language Models

  • Scalable Extraction of Training Data from (Production) Language Models

• Lecture Block 20: Model Editing
  Readings:

  • Model Editing for Classification

  • Model Editing for LLMs

  • Scalable Model Editing for LLMs

  • Fast Model Editing for Diffusion Models

  • Fine-tuning based Model Editing for Diffusion Models

  • Localizing and Editing Knowledge in Text-to-Image Generative Models

Additional Topics

• Lecture Block 21: Neural tangent kernels
  Readings:

  • Neural Tangent Kernel: Convergence and Generalization in Neural Networks

  • A Note on Lazy Training in Supervised Differentiable Programming

• Lecture Block 22: Self-Supervised and Meta Learning
  

  • A Simple Framework for Contrastive Learning of Visual Representations

  • Emerging Properties in Self-Supervised Vision Transformers

  • Bootstrap your own latent: A new approach to self-supervised Learning

  • Emerging Properties in Self-Supervised Vision Transformers

  • Exploring Simple Siamese Representation Learning

  • Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

  • Meta-Learning with Differentiable Convex Optimization

  • Prototypical Networks for Few-shot Learning

  • On First-Order Meta-Learning Algorithms

• Lecture Block 23: Deep Learning Explainabality
  

  • How to Explain Individual Classification Decisions

  • A Unified Approach to Interpreting Model Predictions

  • Axiomatic Attribution for Deep Networks

  • SmoothGrad: removing noise by adding noise

  • Sanity Checks for Saliency Maps

  • Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation