Efficient and High Quality Space Robotic VR Teleoperation through Neural Scene and Object Reconstruction

We aim to significantly improve robotic teleoperation efficiency and quality for applications in space, such as reconstruction of the lunar environment around a rover for remote controlled sorties, or robotic manipulation of objects within the lunar environment. The objective is to allow the human operator to naturally reason about the detailed 3D shape of the environment and objects around a robot, to make declarative control instructions and predict the outcome of actions within a 3D environment, and for the robot to perform more complex semi-automatic control operations from these declarative instructions. Meeting this objective will require 1) the development of core science in new methods of visual computing, AI, and human-robot interaction (HRI) called ‘neural fields’, in which the PIs are leading experts, and 2) specific consideration of how neural field methods could apply to lunar robotic teleoperation needs as part of the NASA Artemis program. This science and application merits study because, in the last two years alone, neural field techniques have dramatically improved the detail and accuracy of the reconstruction of 3D geometry and appearance for real-world scenes and objects, exceeding the capabilities of classical photogrammetry and allowing photorealistic reproductions from arbitrary angles. Neural field techniques have also increased the compactness in bytes of the recovered scenes and objects, producing 3D representations that need fewer bytes than their equivalent 2D images. This makes them potentially well suited to the space teleoperation scenario: instead of transmitting image and video feeds, neural reconstruction can form compact representations of the world around a robot, transmit this, then recreate a photorealistic image for the human operator. This can be thought of as `scene compression’ rather than classical `image compression’. At the same time, both the human and the robot can exploit the knowledge of the recovered geometry for analysis and planning, leading potentially to better human awareness when making control decisions and to better execution of those control decisions by the robot. This is especially true when using immersive view and control interfaces, such as virtual reality (VR), that allow an operator to act as if they were in the same space as the robot via these neural field reconstructions. The proposal will culminate in a teleoperation demonstration on tasks co-designed with NASA, backed by evaluation via experimental evidence through user studies. We will also evaluate the impact of the neural scene reconstruction methods in 2D and VR settings when compared to traditional image and video feeds and in their ability to accurately reconstruct scene and object representations, and their impact upon semi-autonomous control. In sum, we hypothesize that more efficient but still high-quality teleoperation can be achieved by investigating the core science and applications to space of new neural field scene and object reconstruction methods. We will work with NASA Goddard Space Flight Center, where AR/VR and robotic teleoperation are of particular interest for the Artemis program to the Moon, via NASA engineers and scientists Marc Lupisella and Brent Garry.

This is a mature technology (TRL 7+) — the realistic path in is usually NASA's Technology Transfer Program: licensing an existing NASA patent, or a Space Act Agreement to use NASA facilities/expertise directly. NASA also runs a startup licensing program with no upfront fee for companies formed to commercialize a specific NASA technology.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.