A Neuromorphic Approach to Visual Motion and Action

Neuromorphic Computing is a vertical approach of computer engineering of both hardware and software design that seeks to emulate principles of the human brain and nervous system for efficient, low-power, and robust computation, and in recent years various individual concepts have been adopted in main-stream engineering. In this talk I will describe my work on neuromorphic visual motion analysis. Many real-world AI applications, including self-driving cars, robotics, augmented reality, and human motion analysis are based on visual motion. Yet most approaches treat motion as extension of static images by matching features in consecutive video frames. Inspired by biological vision, we use as input to our computational methods, spatiotemporal filters and events from neuromorphic dynamic vision sensors that simulate the transient response in biology. I will describe a bio-inspired pipeline for the processing underlying the navigation tasks, and present algorithms on 3D motion estimation and foreground-background segmentation. The design of these algorithms is guided by a) questions about where to best use geometric constraints in machine learning, and b) experiments with visual motion illusions to gain insight into computational limitations. Lastly, I will discuss the advantages of event-based vision for action understanding and describe recent work on action interpretation for AI in music education.