An Affordance Driven, Human-in-the-Loop Perception Framework

Despite all the advances in robotics, it has been difficult to deploy robots for non-trivial tasks in unstructured environments in a fully autonomous way. As a result, for such tasks there is often a human in the loop to create supervised autonomy. We are proposing to develop a software framework that enables operators to more easily command robots by leveraging more computer vision. The key innovation of this proposal is to connect a perception pipeline that recognizes instances of known classes of objects with affordances: a description of how an object should be manipulated. For example, given a depth image of a scene, the system will recognize door handles and hinges and annotate the scene to automatically inform the user how a robot could open the corresponding doors. Our work combines automatic recognition of affordances with a user interface (UI) that enables an operator to fine tune affordances if needed. The operator feedback will be used to retrain the system to further improve system performance and decrease the amount of operator input required. This is expected to reduce the cognitive load on robot operators by allowing them to focus on high level tasks including appropriate actions on objects that are relevant to a given task.

The initial focus is on enabling higher levels of autonomy for IVA such as cargo unloading and science experiment tending. The technology may also be applicable to EVA and OSAM-related robotic activities. In the long run, we also envision applications in, e.g., construction and assembly on the moon and other planets.

The same applications that are of interest to NASA are also of interest to the rapidly expanding commercial space industry. Terrestrial applications that are enabled by the proposed work include inspection and maintenance of industrial sites and offshore platforms.

This is early/mid-stage (TRL 5) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.