HCAAM VNSCOR: Enhancing Situation Awareness of Automated Procedures Using Adaptive Multimodal Augmented Reality Displays

This task is part of the Human Capabilities Assessments for Autonomous Missions (HCAAM) Virtual NASA Specialized Center of Research (VNSCOR).

Future deep space missions will present new challenges for crew, and increased risks to human performance due to the stress, fatigue, radiation exposure, and isolation that characterizes these missions. In addition, crew will no longer be able to depend on timely support from Mission Control due to distance from the Earth, but will have to work autonomously, while maintaining high performance. Mission Controllers may not be available to answer questions, check system status, assist with procedures, monitor for errors, or troubleshoot problems. Greater crew autonomy will increase dependence on automated systems, and design of these automated systems must be driven by sound human-system integration standards and guidelines in order to ensure mission success. Historically, crew have had very limited dependence on automated systems, thus crew will be faced with a new way of working that may put situation awareness (SA) at risk. We must develop methods for promoting good situation awareness in the automated systems that will most certainly be part of future deep space vehicles and habitats.

Procedure automation is a promising technology for reducing crew workload. We define procedure automation as technology that automates the selection or execution of procedural tasks. Structuring the work of automation according to human procedures should improve the transparency of automation actions. This approach provides a means for establishing common ground about ongoing tasks to improve operator understanding of automation behavior.

New technologies such as adaptive, multimodal, augmented reality displays can offer the benefits of information presentation tailored to meet the needs of each crewmember, taking into consideration the current state of that crewmember (e.g., sleep-deprived, high workload), as well as the current state of his/her environment and ongoing activities (e.g., emergency situation, time-critical operations).

We propose to combine technology for procedure automation with technology for augmented reality multi-modal (ARMM) user interfaces using Microsoft Hololens head-mounted display to provide a virtual task assistant to assist crew in performing procedural work. This virtual task assistant will be capable of identifying which procedures should be performed, performing actions in crew procedures, and summarizing actions taken by the human-automation team to assist crew in preparing for tasks and taking over tasks from other team members.

Four studies are planned to evaluate the effects of a virtual task assistant combining procedure automation with augmented reality multi-modal (AARM) user interfaces on human task performance. These studies will achieve the following aims:

Aim 1. Determine best methods to improve situation awareness and improve crew autonomy when using a virtual task assistant to prepare for and perform manual maintenance.

Aim 2. Determine best methods to improve situation awareness and reduce workload when a virtual task assistant is used to handover maintenance tasks between users.

Aim 3. Determine best methods to improve situation awareness and reduce workload when using a virtual task assistant to help manage concurrent manual and automated tasks.

Aim 4. Determine best methods for display of procedural work to improve crew autonomy and satisfaction

The proposed work addresses a number of gaps in the Human Research Program Human Factors and Behavioral Performance risks. This project will provide guidelines for designing effective human-automation systems and evaluate human-automation performance for exemplar procedure automation systems. This project also will provide guidance for the application of multi-modal and adaptive displays and control to Human-Computer Interaction (HCI) design for long duration operations.

Technologies for virtual task assistance are increasingly available in everyday life. One of the most common is voice enabled assistance, like Siri and Alexa, that aid some activities of daily living. And augmented and virtual reality technologies are becoming mainstream, with the introduction of new devices such as Microsoft HoloLens 2, and improved standards such as the WebXR standards ( https://www.w3.org/TR/webxr/ ) for accessing virtual and augmented reality devices.

The Virtual Intelligent Task Assistant (VITA) project is leveraging augmented reality platforms and new WebXR standards to develop a virtual task assistant that can be used to assist users with procedural task work on the job. Our technical approach is innovative in that new procedural tasks can be supported without custom software development. Our experimental research is distinguished by investigating effective task assistance for maintenance or assembly tasks where hands-free operation of task assistance is beneficial. For the first year we are investigating best techniques for using augmented reality task assistance when assembling small devices that are held in the hands during assembly.

This technology and associated research findings have potential benefit to NASA for the assembly, maintenance, and repair of aircraft, spacecraft, habitats, and robotics. This technology and associated research findings also have broader potential benefit for any organization performing assembly and maintenance procedural work. This includes assembly and maintenance of drilling equipment for the oil and gas industry, equipment used in chemical processing plants, and maintenance and repair of commercial aircraft.

This is early/mid-stage (TRL 5) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.