Generative Computer Vision for the Physical World

Generative models are revolutionizing our world, with the ability to generate photorealistic visual content that are indistinguishable from reality. Despite their overwhelming presence in the cyber world, they haven’t been very useful in the physical world that we live in. In this talk, I will present how the rich priors learned by large-scale generative models—ranging from shape and geometry to motion and dynamics—can be harnessed for real-world perception and interaction tasks. I will showcase how these models can facilitate tasks like 3D reconstruction and robotic manipulation by incorporating the structure of the physical world. Moreover, I will discuss methods to further refine and adapt these systems through self-learning, enabling machines to continually improve as they explore new scenarios and environments. Together, these breakthroughs build the foundation for my vision of creating self-supervised machines that can perceive and interact with the physical world.