Logistics

When & Where

CISC 2120
Tuesday, Thursday 3:30pm - 4:45pm

Instructor

Abhinav Shrivastava

3209 AV Williams
abhinav@cs.umd.edu
Office hours: email for appointments (‘CMSC828I’ in the subject line)

Teaching Assistant

Alex Hanson

4420 AV Williams
hanson@cs.umd.edu
Office hours: email for appointments (‘CMSC828I’ in the subject line)

Overview

Summary

A graduate-level course in computer vision, with an emphasis on high-level recognition tasks. We will read an eclectic mix of classic and contemporary papers on a wide-range of topics. The course structure will combine lectures, student presentations, in-class discussions, and a course project. The goal of this course is to:

Become knowledgeable about how a particular sub-topic evolved, the state-of-the-art, and the challenges that need to be addressed.
Develop skills to critically read, analyze, write-up, and present a body of research.
Learn to draw connections between different works and engage in productive discussions.

Prerequisites

While there are no formal prerequisites for this course, familiarity with introductory courses in computer vision (CMSC426 or similar) and machine learning (CMSC422 or similar) is assumed. If you have not taken courses covering this material, consult with the instructor. Note that a basic knowledge of linear algebra, probability, and calculus is required.

Awards

At the end of the course, we will have prizes (and bonus credits) for:

Best paper/sub-topic review (5x)
Best Project
Best Presentation

Class Presentation

Every student will give one presentation in the course. One week before the presentation date, schedule a meeting with the instructor for discussion. Two days before the presentation, present an initial draft to the instructor.

Paper/Sub-topic Review or Assignments

Paper reviews and assignments, their format, and submission will be announced in the class.

Late Days

You get 5 late days (to be used in 24-hour blocks) that can be used throughout the course. Late days may be used for paper reviews and project proposal submission. Late days may not be used towards the presentations, exams, or final project report (see ‘Accommodations and Policies’ below for exceptions).

Project Report Format and Deadline

Final project reports (5 pages in NeurIPS final format) are due on Dec 15, 2018 at Midnight. Submit via email with subject “[CMSC828i] Final project report” to the instructor.

Grading

Participation: 10%
Presentation: 10%
Paper/Sub-topic Reviews: 20%
Research Project: 30%
Exams (1 mid-term, 1 final): 30%

Schedule

Presenter(s)	Date	Lecture topics*	Papers	Slides/Notes
Abhinav	Aug 28	Introduction to the class		01-Intro.pdf
Abhinav	Aug 30	Introduction to data		02-Data-01.pdf
Abhinav	Sep 4	Introduction to Data-driven methods in vision		03-Data-02.pdf
Abhinav	Sep 4	Introduction to ConvNet Architecures		03-ConvNets.pdf
Alex	Sep 6	Debugging Deep Networks + Adversarial Examples		04-Viz.pdf
Abhinav	Sep 11	Introduction to ConvNet Architecures (cont.)		03-ConvNets.pdf Assignment 0
Abhinav	Sep 13	Introduction to Data-driven methods (cont.)		06-Data-03.pdf
Abhinav	Sep 18	Object Detection (Part I)	Background: Histograms of Oriented Gradients for Human Detection; CVPR 2015 Object Detection with Discriminatively Trained Part-Based Models; IEEE TPAMI 2009 Single-stage object detection: OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks; ICLR 2014 SSD: Single Shot MultiBox Detector; ECCV 2016 You Only Look Once: Unified, Real-Time Object Detection; CVPR 2016	07-ObjDet-01.pdf
Abhinav	Sep 20	Object Detection (Part I) (cont.)		07-ObjDet-01.pdf
Abhinav	Sep 20	Image Segmentation (+ other pixel labeling tasks)	Background: TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context; IJCV 2007 Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials; NIPS 2011 Superpixels and Supervoxels in an Energy Optimization Framework; ECCV 2010 Semantic Segmentation: Fully Convolutional Networks for Semantic Segmentation; ICCV 2015 Hypercolumns for Object Segmentation and Fine-grained Localization; CVPR 2015 Learning Deconvolution Network for Semantic Segmentation; ICCV 2015 DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs; TPAMI 2017 Object Proposals: Selective Search for Object Recognition; IJCV 2013 What makes for effective detection proposals?; TPAMI 2015	08-Seg+ObjDet-2.pdf
Abhinav	Sep 25	Image Segmentation (cont.)		08-Seg+ObjDet-2.pdf
Abhinav	Sep 25	Object Detection (Part II)	Multi-stage object detection: Region-based Convolutional Networks for Accurate Object Detection and Semantic Segmentation; TPAMI 2015 Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition; TPAMI 2015 Fast R-CNN; ICCV 2015 Faster R-CNN; NIPS 2015 Multi-stage object detection & instance segmentation: Simultaneous Detection and Segmentation; ECCV 2014 R-FCN: Object Detection via Region-based Fully Convolutional Networks; NIPS 2016 Instance-aware Semantic Segmentation via Multi-task Network Cascades; CVPR 2016 Mask R-CNN; ICCV 2017 Architectures: Aggregated Residual Transformations for Deep Neural Networks; CVPR 2017 Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks; CVPR 2016 Beyond Skip Connections: Top-Down Modulation for Object Detection; arxiv 2017 Feature Pyramid Networks for Object Detection; CVPR 2017 Analysis and diagnosis: Deformable Part Models are Convolutional Neural Networks; CVPR 2015 Speed/accuracy trade-offs for modern convolutional object detectors; CVPR 2017 Diagnosing Error in Object Detectors; ECCV 2012 Training: [OHEM] Training region-based object detectors with online hard example mining; CVPR 2016	09-ObjDet-3.pdf
Abhinav	Sep 27	Object Detection (Part II) (cont.)		09-ObjDet-3.pdf
Kamal Gupta	Sep 27	Human Pose Estimation	Background: Articulated pose estimation with flexible mixtures-of-parts; CVPR 2011 Readings: Pose Machines: Articulated Pose Estimation via Inference Machines; ECCV 2014 Convolutional Pose Machines; CVPR 2016 Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields; CVPR 2017 Towards Accurate Multi-person Pose Estimation in the Wild; CVPR 2017 Stacked hourglass models for pose detection; ECCV 2016	Review 1 (Action) 10-Pose-Kamal.pdf
Abhinav Jack Wang	Oct 2	Action Recognition	Background: Action Recognition by Dense Trajectories; CVPR 2011 Video Representation Large-scale Video Classification with Convolutional Neural Networks; CVPR 2014 Two-Stream Convolutional Networks for Action Recognition in Videos; NIPS 2014 Learning Spatiotemporal Features with 3D Convolutional Networks; ICCV 2015 Dynamic Image Networks for Action Recognition; CVPR 2016 Rethinking Spatiotemporal Feature Learning For Video Understanding; ECCV 2018 Action Classification Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors; CVPR 2015 ActionVLAD: Learning spatio-temporal aggregation for action classification; CVPR 2017 Temporal Action Detection R-C3D: Region Convolutional 3D Network for Temporal Activity Detection; ICCV 2017 SST: Single-Stream Temporal Action Proposals; CVPR 2017 Spatio-temporal Action Detection Action Tubelet Detector for Spatio-Temporal Action Localization; ICCV 2017 Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos; ICCV 2017	Review 2 (Fine-graned and Attributes) 11-Videos.pdf 11-Videos-Jack.pdf
Abhinav Yu Shen Luyu Yang	Oct 4	Attributes and Fine-grained Recognition	Attributes (background and readings): Describing Objects by their Attributes; CVPR 2009 Learning To Detect Unseen Object Classes by Between-Class Attribute Transfer; CVPR 2009 Relative Attributes; ICCV 2011 Understanding Objects in Detail with Fine-grained Attributes; CVPR 2014 Large-Scale Object Classification Using Label Relation Graphs; ECCV 2014 COCO Attributes: Attributes for People, Animals, and Objects. ; ECCV 2016 Discovering States and Transformations in Image Collections; CVPR 2015 From Red Wine to Red Tomato: Composition with Context; CVPR 2017 Fine-grained Recognition (background and readings): Fine-Grained Visual Categorization Workshop/challenge (latest, first) Bird Species Categorization Using Pose Normalized Deep Convolutional Nets; ECCV 2014 Multiple Granularity Descriptors for Fine-grained Categorization; ICCV 2015 Fine-grained Image Classification by Exploring Bipartite-Graph Labels; CVPR 2016 Bilinear CNNs for Fine-grained Visual Recognition; TPAMI 2017 Kernel Pooling for Convolutional Neural Networks; CVPR 2017	12-Attributes-and-FG.pdf 12-FG-Yu.pdf 12-Attributes-Luyu.pdf
Abhinav Jane Tsai Kuan-Ho Lao	Oct 9	3D Scene Understanding - Primitives and Reasoning	Background: Recovering Intrinsic Scene Characteristics From Images; Computer Vision Systems, 1978 Data-Driven 3D Primitives for Single Image Understanding; ICCV 2013 Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture; ICCV 2015 Primitives (required reading) Learning Ordinal Relationships for Mid-Level Vision; ICCV 2015 Designing Deep Networks for Surface Normal Estimation; CVPR 2015 Reasoning (required reading) Blocks World Revisited; ECCV 2010 Holistic Scene Understanding for 3D Object Detection with RGBD cameras; ICCV 2013 Optional readings: Unfolding an Indoor Origami World; ECCV 2014 Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene; CVPR 2018	13-3D-1.pdf 13-3D-Jane.pdf 13-3D-KuanHo.pdf
Invited Talk Debadeepta Dey	Oct 11	Reinforcement Learning Primer; Neural Architecture Search	RL Primer: Reinforcement Learning: An Introduction; (website, pdf) (skim the first two chapters) NAS Papers: Neural Architecture Search with Reinforcement Learning; ICLR 2017 Efficient Neural Architecture Search via Parameter Sharing; ICML 2018 Progressive Neural Architecture Search; arXiv 2017 DARTS: Differentiable Neural Architecture Search; arXiv 2018 Bonus papers: Regularized Evolution for Image Classifier Architecture Search; arXiv 2018 Neural Architecture Search: A Survey; arXiv 2018	14-NAS-Dey.pdf 14-POMDP-Geoff.pdf
Abhinav Haoying Jun Wang	Oct 16	Objects + 3D	Required readings: 3D ShapeNets: A Deep Representation for Volumetric Shapes; CVPR 2015 Learning a Predictable and Generative Vector Representation for Objects; ECCV 2016 Learning Shape Abstractions by Assembling Volumetric Primitives; CVPR 2017 Category-Specific Object Reconstruction from a Single Image; CVPR 2015 Additional and background readings: Learning Category-Specific Mesh Reconstruction from Image Collections; ECCV 2018 ObjectNet3D: A Large Scale Database for 3D Object Recognition; ECCV 2016 ShapeNet: An Information-Rich 3D Model Repository; arXiv 2015 Beyond PASCAL: A Benchmark for 3D Object Detection in the Wild; WACV 2014 3D2PM – 3D Deformable Part Models; ECCV 2012	15-3D-2.pdf 15-3D-Haoying.pdf 15-3D-Jun.pdf
Abhinav Lillian Anurag	Oct 18	Context Reasoning	Required readings: Auto-context and Its Application to High-level Vision Tasks and 3D Brain Image Segmentation; TPAMI 2010 Efficient Object Category Recognition Using Classeme; ECCV 2010 An Empirical Study of Context in Object Detection; CVPR 2009 Contextual Priming and Feedback for Faster R-CNN; ECCV 2016 Additional readings: Objects in Context; ICCV 2007 Putting Objects in Perspective; CVPR 2006	16-context.pdf 16-context-Lillian.pdf 16-context-Anurag.pdf
	Oct 23	Mid-term presentations
	Oct 25	Mid-term exam
Abhinav Ankan Saketh	Oct 30	Weakly, Semi Supervised Learning	Semi-supervised Learning (required readings): Constrained Semi-Supervised Learning using Attributes and Comparative Attributes; ECCV 2012 Semi-Supervised Deep Learning with Memory; ECCV 2018 Weakly-supervised Learning (required readings): Is object localization for free?-weakly-supervised learning with convolutional neural networks; CVPR 2015 Constrained convolutional neural networks for weakly supervised segmentation; ICCV 2015 Additional readings: Semi-supervised Learning in Gigantic Image Collections; NIPS 2009 Data Distillation: Towards Omni-Supervised Learning; CVPR 2018 Weakly Supervised Deep Detection Networks; CVPR 2016 ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization; ECCV 2016 Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation; ICCV 2015	17-Weakly-Saketh.pdf 17-SSL-Ankan.pdf
Abhinav Anshul Ketul	Nov 1	Learning from the Web	NEIL: Extracting Visual Knowledge from Web Data; ICCV 2013 Learning Everything about Anything: Webly-Supervised Visual Concept Learning; CVPR 2014 Enriching Visual Knowledge Bases via Object Discovery and Segmentation; CVPR 2014 Webly Supervised Learning of Convolutional Networks; ICCV 2015 Learning Visual Features from Large Weakly Supervised Data; ECCV 2016 Revisiting Unreasonable Effectiveness of Data in Deep Learning Era; ICCV 2017	18-SSL+Web.pdf 18-web-Anshul.pdf 18-web-Ketul.pdf
Abhinav Sahil	Nov 6	Visual data mining and discovery	Required readings: Unsupervised Discovery of Mid-Level Discriminative Patches; ECCV 2012 What Makes Paris Look like Paris?; SIGGRAPH 2012 Additional readings: Style-aware Mid-level Representation for Discovering Visual Connections in Space and Time; ICCV 2013 Where have all the people gone? Enhancing global conservation using night lights and social media; Ecological Applications 2015 Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States; PNAS 2017 Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images; ICCV 2017 Visualizing Brand Associations from Web Community Photos; ACM WSDM 2014	19-midlevel+mining.pdf 19-mining-Sahil.pdf
Abhinav Yixuan	Nov 8	Generative Models	GANs: Generative Adversarial Nets (primary reading) Tutorial on GANs (slides) (primary reading) VAEs: Tutorial on VAEs (primary reading) Auto-Encoding Variational Bayes (additional reading) Stochastic Backpropagation and Approximate Inference in Deep Generative Models (additional reading) Additional recommended readings: Unsupervised representation learning with deep convolutional generative adversarial networks; ICLR 2016 Deep Convolutional Inverse Graphics Network; NIPS 2015 Image-to-Image Translation with Conditional Adversarial Nets; CVPR 2017 Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks; ICCV 2017 Semi-parametric Image Synthesis; CVPR 2018 PixelNN: Example-based Image Synthesis; ICLR 2018 Pixel Recurrent Neural Networks; ICML 2016 Learning to Generate Chairs, Tables and Cars with Convolutional Networks; PAMI 2017	20-26-gen+unsup+self.pdf 20-gen-Yixuan.pdf 21-gen-Bo.pdf 21-gen-Yue.pdf 21-gen-Hao.pdf
Abhinav Bo He Yue Jiang Hao Chen	Nov 13	Generative Models
	Nov 15	No Class	CVPR deadline
Abhinav	Nov 20	Misc.		(see above)
	Nov 22	Thanksgiving Break
Abhinav	Nov 27	Misc.		(see above)
Ruoxi Sweta	Nov 29	Self-Supervised Learning	Context from images: Unsupervised Visual Representation Learning by Context Prediction; ICCV 2015 Context Encoders: Feature Learning by Inpainting; CVPR 2016 Context from videos: Learning Features by Watching Objects Move; CVPR 2017 Transitive Invariance for Self-supervised Visual Representation Learning; ICCV 2017 Additional readings: follow the references of the listed papers.	24-self-Sweta.pdf 24-self-Ruoxi.pdf
Mingfei Kyungjun	Nov 29	Few-shot (low-shot), Zero-shot Recognition	Few-shot (low-shot) Recognition readings: Low-shot Visual Recognition by Shrinking and Hallucinating Features; ICCV 2017 Predicting Visual Exemplars of Unseen Classes for Zero-Shot Learning; ICCV 2017 Zero-shot Recognition (using KG) readings: Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs; CVPR 2018 The More You Know: Using Knowledge Graphs for Image Classification; CVPR 2017 Additional readings: Zero-Shot Learning - The Good, the Bad and the Ugly; CVPR 2017 Learning a Deep Embedding Model for Zero-Shot Learning; CVPR 2017 Matching Networks for One Shot Learning; NIPS 2016 An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild; ECCV 2016	24-fsl-Kyungjun.pdf 24-zsl-Mingfei.pdf
Abhinav Shi Mozhi Pedro	Dec 4	Vision + Language/text	Easier tasks: Baby Talk: Understanding and Generating Image Descriptions; CVPR 2011 Show and Tell: A Neural Image Caption Generator; CVPR 2015 Referitgame: Referring to objects in photographs of natural scenes; EMNLP 2014 Tasks: VQA: Visual Question Answering; ICCV 2015 MovieQA: Understanding Stories in Movies through Question-Answering; CVPR 2016 Visual Madlibs; ICCV 2015 Sampling of methods: Simple Baseline for Visual Question Answering; 2015 Neural Module Networks; CVPR 2016 Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data; CVPR 2016	Review 3 (group) 25-text-Mozhi.pdf 25-text-Pedro.pdf 25-text-Shi.pdf
Abhinav Jack Uttaran Rohan	Dec 6	Embodied visual perception (vision + action)	Primary readings: AI2-THOR: An interactive 3d environment for visual AI; 2017 IQA: Visual question answering in interactive environments; CVPR 2018 Embodied question answering; CVPR 2018 Neural Modular Control for Embodied Question Answering; ECCV 2018 Target-driven visual navigation in indoor scenes using deep reinforcement learning; ICRA 2017 Unifying Map and Landmark Based Representations for Visual Navigation; 2017 Cognitive mapping and planning for visual navigation; CVPR 2017 Additional readings: VirtualHome: Simulating Household Activities via Programs; CVPR 2018 Building generalizable agents with a realistic and rich 3D environment; 2018 Visual Semantic Planning using Deep Successor Representations; ICCV 2017	26-action-Uttaran.pdf 26-action-Rohan.pdf 26-action-Jack.pdf
	Dec 13	Final Presentations	12:00 - 4:00 PM Location: CSIC 2120 Format: 10 min per group (+2 minutes for questions)

	Dec 18	Final exam	Time: 10:30am-12:30pm Location: CSIC 2120

*Note that the topic schedule is subject to change.

Additional topics we can cover if the schedule changes:

Intuitive physics
Hyperparameter optimization and architecture search (AutoML, HyperOpt, TPE, SMAC, RoBO, etc.)
Information theoretic interpretation of deep networks
Training with synthetic data
Neuroscience study example
?

Accommodations and Policies

Academic Integrity

Note that academic dishonesty includes not only cheating, fabrication, and plagiarism, but also includes helping other students commit acts of academic dishonesty by allowing them to obtain copies of your work. In short, all submitted work must be your own. Cases of academic dishonesty will be pursued to the fullest extent possible as stipulated by the Office of Student Conduct. It is very important for you to be aware of the consequences of cheating, fabrication, facilitation, and plagiarism. For more information on the Code of Academic Integrity or the Student Honor Council, please visit http://www.shc.umd.edu.

Excused Absence and Academic Accommodations

Any student who needs to be excused for an absence from a single lecture, recitation, or lab due to a medically necessitated absence shall:

Make a reasonable attempt to inform the instructor of his/her illness prior to the class.
Upon returning to the class, present their instructor with a self-signed note attesting to the date of their illness. Each note must contain an acknowledgment by the student that the information provided is true and correct. Providing false information to University officials is prohibited under Part 9(i) of the Code of Student Conduct (V-1.00(B) University of Maryland Code of Student Conduct) and may result in disciplinary action.
This self-documentation may not be used for the Major Scheduled Grading Events as defined below.

Any student who needs to be excused for a Major Scheduled Grading Event, must provide written documentation of the illness from the Health Center or from an outside health care provider. This documentation must verify dates of treatment and indicate the time frame that the student was unable to meet academic responsibilities. No diagnostic information shall be given. The Major Scheduled Grading Events for this course include midterm and final exam. For class presentations, the instructor will help the student swap their presentation slot with other students.

It is also the student's responsibility to inform the instructor of any intended absences from exams and class presentations for religious observances in advance. Notice should be provided as soon as possible, but no later than the Monday prior to the the midterm exam, the class presentation date, and the final exam.

Any student eligible for and requesting reasonable academic accommodations due to a disability is requested to provide a letter of accommodation from the Office of Disability Support Services within the first three weeks of the semester.

Other Accommodations and Policies

You can find the university’s course policies here.