Person Aware Liaison (PAL)

Collaborative human–robot teaming behaviors use the complementary strengths of humans and robots and are at their most successful when the human trusts in their robot companion. However, despite contemporary approaches to make robots more transparent in their behaviors and safer in their operation, humans still do not trust robots the same way they do a human partner. The inability of current systems to observe all communications from their partner, draw usable inference from the contextual clues of the interaction, and respond in a courteous manner may prohibit humans from placing full trust in their robot collaborators. To that end, we propose Person Aware Liaison (PAL) as an intermediary interface between the human and the robot as they complete a collaborative task. PAL generates multimodal observations (speech, gestures, and where the human’s gaze lingers) during the execution of the task and fuses the information into a singular natural language transcription of events. This transcript is queried by a Generative Pre-trained Transformer (GPT) architecture, a state-of-the-art text-based inference model to understand the human’s intent. Finally, PAL acts upon the extrapolated intent by developing safe, dexterous motion plans for an articulated robot arm and executing them in a courteous manner that avoids interrupting the human user while they are focused on completing task subgoals. PAL is task agnostic and can use modular task knowledge to quickly adapt to new applications. We will demonstrate the capabilities of PAL both in simulation and in a physical setting using a collaborative welding task.

The innovations proposed under PAL will support NASA’s strategic goals toward the development of autonomous systems, specifically with a focus toward Behavior and Intent Prediction (TX 10.3.2) and Motion Planning (TX 10.2.3). Additionally, the expected increase in robots in human spaces planned under ARTEMIS has highlighted a need for systems that “develop integrated human and robotic systems with inter-relationships” for Lunar and Martian missions (Moon to Mars Objectives, 2022, TH-9 and TH-10).

Improving the trust between human and robot collaborators has demonstrable improvements to both efficiency and morale. These benefits are of imminent interest to industries such as manufacturing and assembly, which are expected to see an increase in the adoption of collaborative robots within the next decade.

This is early/mid-stage (TRL 4) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.