PhD Proposal: Generating Novel Synthetic Photorealistic Data For Dynamic UAV Scenes Using Neural Radiance Fields

https://umd.zoom.us/j/6054713798

Abstract:

Although there have been many advancements in perception algorithms for a variety of computing platforms, training said algorithms still requires a large amount of data. In order for these algorithms to perform in esoteric domains, such as search and rescue imagery from Unmanned Aerial Vehicles (UAVs), appropriate datasets are needed for training. Within said domains, there is a dearth of available datasets due to reasons including scene novelty, flight regulations, and difficulty in collecting varied data overall.In order to account for gaps in available UAV datasets, our research focuses on synthetic data generation to augment real world data when training perception algorithms. In particular, we focus on Neural Radiance Fields (NeRF) to capture the 3-dimensional structure of a scene and render novel views with photorealistic fidelity. Accomplishments include developing a pipeline for rendering novel data as well as ground truth labels such as bounding boxes, extending existing state of the art dynamic NeRF algorithms to handle difficult UAV scenes, and developing a new NeRF technique, Tiered K-planes, to increase the fidelity of small dynamic portions of a scene when compared with previous state of the art neural rendering methods. Ongoing and future work includes a NeRF algorithm that incorporates “shared feature vectors” in order to leverage mutual information within a scene and extrapolate on viable novel view imagery from available training camera pose trajectories as well as a “codex” of feature vectors amongst independent scenes that can accelerate the training of new scenes and further extrapolate on the novelty of rendered camera poses from scenes with extremely limited training trajectories, e.g. fixed camera positions.