Context-Sensitive Augmented Reality for Mission Operations

Current NASA missions to the International Space Station (ISS) are heavily dependent upon ground controllers to assist crew members in performing routine operations and maintenance as well as responses to off-nominal situations. Performing these procedures often requires close collaboration between ground controllers who have deep knowledge of the spacecraft's systems. This collaboration becomes more difficult in extended missions and crew members will need to have more autonomy. Augmented and virtual reality technology can help replace some of the guidance that ground controllers offer to crew members during procedure execution. Our proposed approach focuses on integrating augmented and virtual reality technologies into the same tools that mission operations uses today, allowing for augmented and virtual reality assistance to be built and updated by flight controllers and other mission specialists just as they do for their other work products. In Phase I, TRACLabs integrated its PRIDE electronic procedure platform with augmented and virtual reality systems such as the HTC Vive. In Phase II we will extend the PRIDE platform to augmented reality devices such as the Microsoft HoloLens. TRACLabs is also partnered with the Georgia Institute of Technology's Augmented Environments Laboratory who will develop algorithms that dynamically adjust the augmented reality cues (both their position, orientation, and scale, along with the specific graphical elements used) such that they align with the necessary parts of the world without obscuring important parts of the world.

This research will be immediately useful to the NASA JSC Hybrid Reality Laboratory (HRL), which will use our software to guide users through their ISS models and train users on interacting with these models. There are also applications for augmented reality on-board ISS, including applications of the HoloLens and tablet browser to procedure execution. This research can have immediate application to ISS operations because there are several iPads already on ISS and a Microsoft HoloLens. We are also integrating with NASA?s Dynamic On-board Ubiquitous Graphics (DOUG) system to assist in training for EVA and ROBO operations. PRIDE is being evaluated for use by ground operators for the Resource Prospector robotic mission to the moon being jointly developed by NASA JSC and ARC. Finally, we will work with NASA's Human Research Program (HRP) to identify applications for this work including analog test environments.

TRACLabs is already selling PRIDE as a commercial product with a major oil field services company as a launch customer. PRIDE is being field-tested at several sites world-wide before deployment in actual operations in mid-2017. Augmented reality would immediately increase the effectiveness of the PRIDE software by providing hands-free assistance in performing complex and dangerous procedures. TRACLabs expects additional customers in the oil and gas industry will deploy PRIDE once it has been proven effective. A major chemical manufacturer is also performing a pilot project in their plants using PRIDE for complex operations. In these cases, a heads-up display like the HoloLens or the DAQRI could be very useful in field environments In all of these cases, we will offer augmented reality systems as an ``add-on'' to the existing PRIDE software we deliver. Thus, we can immediately move this research out into industry by leveraging our existing PRIDE user base.

This is early/mid-stage (TRL 6) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.