Person Aware Liaison (PAL)

Collaborative humanrobot teaming behaviors use the complementary strengths of humans and robots and are at their most successful when the human trusts in their robot companion. However, despite contemporary approaches to make robots more transparent in their behaviors and safer in their operation, humans still do not trust robots the same way they do a human partner. Humans may feel prohibited from placing full trust in their robot collaborators due to current robots inability to observe all communications from their partner, draw usable inference from the contextual clues of the interaction, and respond in a courteous manner. To that end, we propose Person Aware Liaison (PAL) as an intermediary interface between the human and the robot as they complete a collaborative task. PAL generates multimodal observations (speech, gestures, where the humans gaze lingers, body posture, and robot registered events) during the execution of the task and fuses the information into a singular natural language transcription of events. This transcript is queried by a large language model (LLM), a state-of-the-art text-based inference model to understand the humans intent. Finally, PAL acts upon the extrapolated intent by developing safe, dexterous motion plans for an articulated robot arm and executing the necessary actions in a courteous manner that avoids interrupting the human user while they are focused on completing task subgoals. PAL is task agnostic and can use modular task knowledge to quickly adapt to new applications. We will demonstrate the component capabilities of PAL in simulation and the complete system in a physical, representative manufacturing setting. Human–robot teaming is most effective when the human can place their trust in their robot companion as they would in another human. However, limitations of contemporary systems, including the modalities investigated, the inability to correctly infer intent from an interaction, and the lack of consideration for the human collaborator’s current occupation when executing, prohibit complete adoption of robotic partners by human collaborators. Person Aware Liaison (PAL) addresses these shortcomings by capturing multimodal inputs and fusing these sensor measurements into a singular text transcript of the interaction. PAL then uses the embeddings present in a Large Language Model to extrapolate the human collaborator’s intent from contextual clues in the text transcript. Finally, PAL executes dexterous actions according to the extrapolated intent that are both safe (i.e., avoiding the human) and nonintrusive. PAL is designed to be task agnostic but will be evaluated in both a simulation and physical setting on collaborative manufacturing tasks. Work on PAL will manifest three contributions: (1) development and fusion of a multimodal suite of sensors to capture speech, gesture, and gaze fixation, among other passive observations into a singular text representation; (2) refinement of a Large Language Model toward the learning of human intent from the text transcript of a collaborative task; and (3) safe, dexterous, and conscientious execution of robotic actions. We will develop these three contributions in parallel, with most of our focus attributed to the latter two attributes as they demonstrate the novel innovations of PAL. We will demonstrate the capabilities of PAL in both simulated validation of component parts and in physical demonstration of the system in representative, collaborative manufacturing tasks. Throughout the duration of the project timely reports and briefings will be conducted to appraise NASA stakeholders on the progress of PAL development that include both quantitative metrics and video demonstrations of the physical system in action.  

The innovations proposed under PAL support NASA’s strategic goals toward the development of autonomous systems, specifically with a focus toward Multi-Modal and Proximate Interaction (TX04.4.1), Behavior and Intent Prediction (TX 10.3.2) and Motion Planning (TX 10.2.3). Additionally, the expected increase in robots in human spaces planned under ARTEMIS has highlighted a need for systems that “develop integrated human and robotic systems with inter-relationships” for Lunar and Martian missions (Moon to Mars Objectives, 2022, TH-9 and TH-10). Improving the trust between human and robot collaborators has demonstrable improvements to both efficiency and morale. These benefits are of imminent interest to industries such as manufacturing and assembly, which are expected to see an increase in the adoption of collaborative robots within the next decade.  

This is early/mid-stage (TRL 4) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.