Department of Computer Science
College of Computer, Mathematical, and Natural Sciences
The University of Maryland at College Park

CMSC 838B / 498Z (Fall 2023): Differentiable Programming


CMSC 838B / 498Z (Fall 2023): Differentiable Programming

Instructor: Ming C. Lin


Time and Place: Tues/Thur 12:30pm - 1:45pm, CSI 2120
Office Hours: Tues/Thur After Class
Prerequisites:CMSC 330, CMSC 351 AND CMSC 422 OR Instructor's approval
Breadth Areas: AREA 2 - Information Processing (AI/ML/CV/CG) and AREA 3 - Software Engineering & Programming Languages (SE/PL/HCI)
Textbook: Course Notes and Electronic Handouts
TA: Sanghyun Son (IRB 5108, shh1295@umd.edu; Office Hours: By Appointment)
================================================================================================================================================================================================================================================================================================================

  • Course Overview
  • Lectures and Approximate Schedule
  • Course Reading Materials
  • Assignments and Projects
  • Course Syllabus
  • ================================================================================================================================================================================================================================================================================================================

    COURSE OVERVIEW:

    Differentiable Programming has the potential to impact many different fields in upcoming years. With recent advances in general-purpose GPU (GPGPU), cloud computing, data-driven modeling and deep learning, some of the areas, where differentiable programming has shown success, include computer vision, physics-based simulation, speech analysis to natural language processing. "Differentiable Programming" can be broadly defined as a novel programming paradigm, where programs are treated as compositions of differentiable operations that can then be optimized to fit data -- with tremendous possibility to transform how we can utilize computers to perform calculations on "big data".

    This course will examine at how differentiable programming works, from theoretical foundations, practical design and consideration, to system implementation. The course will cover popular concepts like automatic differentiation, models to be integrated with deep learning (including convolutional and recurrent neural networks), as well as some specific instances in differentiable vision, differentiable rendering, differentiable physics, etc. The students are free to choose application of differentiable programming in various topic areas, including Quantum Computing, Genomics, Robotics and AI, Computational Biology and Neuroscience, etc.

    The overall goal of this course is to survey some of the most important recent development in differentiable programming, and not to teach you how to train pre-existing models (although you may learn to do that if you choose to do a course project). The aim of this course is to equip you with the broad knowledge and basic skills, so you can understand the literature and implement models and novel differentiable programming concepts. The intent is that after this course, you will develop the ability to critically evaluate and appreciate new system ideas, constructively dissect and criticize, and skillfully identify limitations and potential of various differentiable programming approaches.

    Pre-requisite: The predominant focus is on looking at models that can be optimized via gradient-based methods. So, you are expected to have a good grasp of linear (matrix) algebra and matrix calculus, as well as the fundamentals of machine learning, probability and statistics. You should also be comfortable with Python programming and the use of numeric/matrix libraries such as pyTorch, if you choose the 3-credit option (vs. 1-credit for only attending and participation) to do a final course project with significant programming and software components.

    ================================================================================================================================================================================================================================================================================================================

    LECTURES AND APPROXIMATE SCHEDULE

    There will be lectures each week. The lectures will all take place either in-person or online, with materials made available through links placed on the course page. By taking part in this course at the minimum of 1-credit option, we expect you to actively participate, attend most lectures, and get involved -- asking thoughtful questions, partaking in stimulating discussion – either in person or via text chat, etc.

    A tentative class schedule, illustrating the planned topic and course materials, is posted below. This schedule will evolve as the class progresses throughout the semester. Many of the lectures are coupled with assigned reading. Whenever possible, students are strongly encouraged to read before the lecture takes place. Advanced reading will enhance understanding of the topic, while contributing to much more lively discussion in class. Some of the lectures will be invited guest lectures given by well-known researchers in differentiable programming.

  • Overview & Introduction (Tues, Aug 29, 2023)
  • Background and Reviews (Thur, Aug 31, 2023)
  • Differentiation (Tues, Sept 5, 2023)
  • Backpropagation (Thurs, Sept 7, 2023)
  • Backpropagation Examples (Tues, Sept 12, 2023)
  • Backpropagation Examples (Thur, Sept. 14, 2023)
  • Optimization (Tues, Sept 19, 2023)
  • Differentiable Imaging and Beyond and Wolfgang Heidrich (Thurs, Sept 21, 2023)
  • Differentiable Physics (Tues, Sept 26, 2023)
  • Differentiable Physics (Thur, Sept 28, 2023)
  • Differentiable Physics (Tues, Oct 3, 2023)
  • Midterm Exam (Thur, Oct 5, 2023)
  • Computing in Miniature by Bo Zhu (Tues, Oct 10, 2023)
  • Differentiable MPM by Chenfanfu Jiang (Thur, Oct 12, 2023)
  • Project Proposal (Tues, Oct 17, 2023)
  • Project Proposal (Thur, Oct 19, 2023)
  • Neural Radiance Fields & (Tues, Oct 24, 2023)
  • Leveraging NeRF to Dynamic Contents (Thurs, Oct 26, 2023)
  • Differentiable Navigation & Path Planning (Tues, Oct 31, 2023)
  • Enhancing Sampling Efficiency in RL (Tues/Thur, Oct 31-Nov 2, 2023)
  • reinforcement learning from human feedback (RLHF) (Thur, Nov 2, 2023)
  • Physics Informed Neural Networks for Fluid Simulations (Nov 7, 2023)
  • Differentiable Fluid Simulation (Nov 7-9, 2023)
  • Learning Vector Fields for Dynamical Systems (Nov 9, 2023)
  • Project Progress Report (Tues, Nov 14, 2023)
  • Project Progress Report (Thurs, Nov 16, 2023)
  • NO CLASS: THANKSGIVING BREAK (Nov 21-23, 2023)
  • NL-to-SQL Generation Models (Tues, Nov 28, 2023)
  • Differentiable Circuit Simulation and Optimization (Tues, Nov 30, 2023)
  • Differentiable Digital Signal Processing and JAX (Thurs, Nov 30, 2023)
  • 3D Pose Estimation Using Audiovisual Learning (Thurs, Nov 30-Dec 5, 2023)
  • Hand Manipulation (Tues, Dec 5, 2023)
  • Differentiable Optics (Tues, Dec 5, 2023)
  • Differentiable Autonomous Driving (Thur, Dec 7, 2023)
  • Differentiable Learning for Geometry Processing (Thurs, Dec 7, 2023)
  • ================================================================================================================================================================================================================================================================================================================

    COURSE READING MATERIALS

  • Supplementary Materials and Reading List for the Class
  • ================================================================================================================================================================================================================================================================================================================

    ASSIGNMENTS AND PROJECTS

    The class grade of each student taking 3-credits is determined by

  • In-Class Exams (30%)
  • Class Presentation (20%)
  • Final Project (50%)
  • ================================================================================================================================================================================================================================================================================================================

    For more information, contact Ming C. Lin, lin@cs.umd.edu.

    ================================================================================================================================================================================================================================================================================================================

    Copyright 1999-2023. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the author.

    This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

    Web Accessibility