GUARDIAN: Grounding visual data with actions for task verification

To support NASA missions that expand to the Moon, Mars and deeper into space, robots must be capable of increased levels of autonomous operation, such that they can provide effective support on ground and launch activities where human presence is either not viable, such as in hazardous environments, or in remote locations where human interaction is expected to be through remote, limited supervision. In this proposal, we focus on improving the state estimation capabilities of a robot system, such that during the execution of a long-horizon task, the robot agent will be capable of: (1) Descriptively assess its current state, by grounding visual data into predicates; and (2) Use this discrete representation to execute robot plans with closed-loop monitoring, where the visual feedback state is compared to the expected planned state, and if an anomaly is detected, the system is notified so appropriate action is taken. To provide robotic systems with the ability to ground visual feedback into symbolic states for long horizon robot planning to support sustained operation in space, TRACLabs proposes to develop GUARDIAN, a framework that allows a robot system to analyze RGB images, and infer predicates that describe its state, enabling it to perform close-loop task verification. GUARDIAN will be a system that bridges visual feedback information and transforms it into a symbolic state that will allow a robot to plan complex, multi-step tasks. For this, GUARDIAN will be implemented as a representation network that receives as input an image, and - after being trained in synthetic data labeled with efficient, weak supervision - outputs a set of predicates that describe the current state of the system, which can then be compared with the expected state, and thus anomalous situations can be timely detected.

A number of near-term NASA missions could benefit from the advances developed during this project. Some of the applications may include IVR caretakers such as the Astrobee robot in the ISS, or more dexterous robots in deep space uncrewed spacecrafts. This work is also applicable to lunar surface robots for activities such as sample retrieval. Future missions could leverage the technology delivered by this project in more advanced applications involving robot manipulation, such as Commercial Lunar Payload Services (CLPS) and Mars sample return.

The technology developed in this project is applicable to any application involving robot task execution with continuous failure detection. As such, commercial applications including the automotive and aerospace sectors can be served with the increased robustness provided by this technology.

This is early/mid-stage (TRL 3) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.