An Affordance Driven, Human-in-the-Loop Perception Framework

Despite continued advances in robotics, it has been difficult to deploy robots for non-trivial tasks in unstructured environments in a fully autonomous way. As a result, for such tasks there is often a human in the loop to create supervised autonomy. We are proposing to develop a flexible software-based solution that enables operators to easily command robots by leveraging more computer vision. The key innovation of this proposal is to connect a perception pipeline that recognizes instances of known classes of objects with affordances: a description of how an object should be inspected or manipulated. For example, given a depth image of a scene, the system will recognize a hatch door handle and its hinge and annotate the scene to automatically inform the user how a robot could open the hatch. The user can then confirm the desired action to have a robot open the hatch door autonomously. The proposed work will result in a software tool that enables the training of perception pipelines using domain randomization. The perception capabilities will be demonstrated in inspection and manipulation tasks that are relevant to IVR scenarios such as inspection of hatch seals, and manipulation of buttons, switches, and handrails. Experiments will be performed in simulation and on hardware using a UR5e and the Astrobee platform. This proposal focuses on visual perception for supervised autonomy robotics applications. Through the proposed work we will make it easier to create perception pipelines for given objects of interest and automate inspection of such objects. A unique feature of our work is the use of affordance templates to characterize motion constraints, which can be used to capture both physically constrained interactions and non-contact visual inspection tasks. The proposed work builds on the results of Phase I, which led to a prototype implementation of a system for domain randomization (a method for producing a broad range of realistic environments within specified parameters) in a photorealistic physics simulation environment. In Phase II we will further expand on this and make it a generally useful robot software development tool that will be demonstrated on a number of manipulation and inspection tasks using both a manipulator and the Astrobee robots. Astrobee experiments will be performed at the NASA Ames Granite Lab and, as time and resources permit, aboard the ISS next.   The main objectives are to (1) build tools for robot software developers that facilitate robust recognition of known classes of objects in a user-specified range of environments, and (2) implement software for scripting inspection tasks using cameras mounted on either robotic manipulators or free flying robots. The proposed system will be integrated with MoveIt Studio, a robot software framework that provides supervised autonomy. The deliverables will consist of (1) a computer vision pipeline, (2) an improved version of the MoveIt Studio software with new perception and inspection capabilities, (3) integration of selected MoveIt Studio software components into NASA Ames’ software stack for Astrobee, and (4) test reports that demonstrate MoveIt Studio capabilities for a variety of NASA and non-NASA use cases.

The initial focus is on enabling higher levels of autonomy for IVA such as inspection, cargo unloading and science experiment tending. The technology may also be applicable to EVA and ISAM-related robotic activities. In the long run, we also envision applications in, e.g., construction and assembly on the moon and other planets. The same applications that are of interest to NASA are also of interest to the rapidly expanding commercial space industry. Terrestrial applications that are enabled by the proposed work include inspection and maintenance of industrial sites and offshore platforms.

This is early/mid-stage (TRL 4) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.