Visual Imagination in the 3D Physical World

My long-term research goal is to enable computers to construct highly realistic and easily authored 3D digital twins of the physical world and to produce new, photorealistic, and physics-informed visual content, often referred to as “imagination”. This talk will summarize two recent directions our group has taken toward this goal. Firstly, I will discuss the generation of realistic movies that depict physical phenomena in real-world scenes captured on video. For instance, visualize how the local children’s park might appear during a flood or after a heavy snowstorm. Current generative models face challenges in such scenarios due to inconsistencies and a lack of physical feasibility. We believe that integrating physics-informed simulations with neural scene representations is the answer. I will showcase how to combine these techniques to simulate coherent, realistic, and physically plausible extreme climate effects and relighting effects for visual scenes. Next, I will introduce a novel robot imagination paradigm that can seamlessly merge the physical world with simulation. I’ll present “Sim-on-Wheels,” a safe and realistic vehicle-in-loop simulation framework. Sim-on-Wheels operates on a self-driving vehicle in the real world, generating virtual traffic participants exhibiting risky behaviors. It then seamlessly integrates these virtual events into images captured from the physical environment in real-time. I will explain how these manipulated images from Sim-on-Wheels are fed into an autonomous system in real time, enabling the evaluation of autonomous vehicle performance under safety-critical scenarios in the real world. Finally, I will provide a brief personal perspective on open research topics related to photorealistic 3D modeling and visual content creation.