Integration of Notification with 3D Visualization of Rover Operations

Future NASA exploration missions will rely on remote operation of robots. As human explorers move further away from Earth, robotic precursors will scout destinations and robotic assistants will perform tasks to reduce astronaut risk and workload. 3D visualization is a key component of how humans will interact with robots for these missions. When the operator engages a robot using visualization, there is a risk that he or she will become too focused on what is happening now in the vicinity of the robot and will not be aware of other important events that are not apparent in the field of view. This risk only increases when operations involve multiple robots. It is essential to ensure that the user does not miss important events that do not manifest in the vicinity of the robot. TRACLabs, Carnegie Mellon University (CMU), and Stinger Ghaffarian Technologies (SGT) propose to develop software for notifying users of 3D visualization about important notices without distracting users unnecessarily or adding to the visual clutter around the robot avatar. This software will monitor events from the robot or user, identify which events should be brought to the user's attention, and alert users in the 3D pane. The appearance of alerts is altered to shift a user's attention to new notices based on an assessment of the importance and urgency of the notice specific to the user. Thus the same notice may be presented to different users in different ways. Because notices are anchored to a screen overlay, they are visible regardless of what location the user is viewing in the 3D space. In Phase II we will implement this software and evaluate its effectiveness for NASA missions.

The United States Department of Defense (DOD) is making significant investment in military robots, including unmanned air vehicles for surveillance and search and rescue, unmanned ground vehicles for logistics, and surface robots for reconnaissance. 3D visualization of robots for military operations includes supervising multiple robots operating concurrently to accomplish a military objective. In such multi-robot operations, it is essential to shift the user's focus when another robot outside the current field of view requires attention. Thus the proposed technology for notification should be directly applicable to remote supervision of military robots. Since the military is actively funding research on human attention shifting during complex operations and visual de-cluttering of geospatial displays, the proposed approach should be both interesting to a variety of DOD customers and compatible with military robotic operations.

The notification software produced in Phase II will be immediately useful for NASA's future analog robotic field tests where VERVE provides 3D robot visualization. NASA's Robonaut 2 mission to ISS in FY11 is another near-term mission that could benefit from the proposed notification technology by alerting the human operator to important events that occur while he or she is interacting with the robot. The modularization of notice construction from notice presentation should ease integration of the notification software with other 3D visualization software that NASA might use in the future. Longer term, the techniques for determining when to shift the user's attention from the visualization and how to notify without increasing visual clutter should be useful when specifying flight support software for future exploration missions, including robotic precursors for human exploration, ranging from missions to Near Earth Asteroids (NEA) to Mars.

This is early/mid-stage (TRL 6) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.