Sensory Manipulation as a Countermeasure to Robot Teleoperation Delays

Currently, most interactions with robots in space exploration are achieved through teleoperations. During future space teleoperations, communicating time delays associated with long distances may negatively affect performance if operators do not calibrate to it. The goal of this research is to test if sensory manipulation, especially providing virtual force cues via haptic device-generated feelings of touch and resistance (paired with delayed visual cues), can help mitigate the negative influence of teleoperation delays measured by perceived presence, neural efficiency, and task performance.

This research aims to test the following hypothesis: Modifying haptic sensation alleviates the subjective perception of time delays and expedites operator’s adaptation to stochastic delays in robot teleoperations. Human sensorimotor controls rely on multimodal sensory feedback, such as the visual, auditory, and tactile cues, to make sense of the consequence of the initiated action. Any latency between the action and the consequence creates a mismatch in motor perception and thus leads to perceptual-motor dysfunction. Literature has already found that sensory manipulation, i.e., providing additional sensory modalities as reinforcement cues, can modulate the effectiveness of motor learning and rehabilitation. The rationale of the proposed approach is that by simulating virtual force of physical interactions on the operator end, the delayed visual cues of teleoperation are reinforced by multimodal sensory feedback, mitigating the perception of time delays and improving performance.

This research originally proposed three aims. The fourth aim was added for exploring longer delays:

Aim 1: Develop a haptics-based sensory augmentation system for robot teleoperation with varying delays. It develops a Virtual Reality (VR) and haptics-based simulator to support robot teleoperation with varying levels of delays (i.e., teleoperation latency). The system is expected as a potential human-robot interface in the future missions, as well as a testbed for supported the proposed human-subject experiments.

Aim 2: Perform Experiment I to explore the impact of the sensory manipulation system up to 1s delay. It collects experimental data how modified haptic stimulation expedites operator’s adaptation to varying delays in teleoperation up to 1s. The haptic simulation refers to reproducing the contact dynamics of the remote robotic system (e.g., resistance, torque, and nominal weight etc.) for operator via haptic devices.

Aim 3: Data analysis to quantify the impacts of the proposed sensory manipulation method on teleoperation performance and human function. This aim proposes to analyze the experiment data to better understand if the proposed system can improve teleoperation performance while reducing the perceived delays.

Aim 4 (added to the original proposal): Perform Experiment II to explore the impact of the proposed sensory augmentation system up to 5s delay. This aim proposes to collect experimental data how modified haptic stimulation expedites operator’s adaptation to varying delays in teleoperation up to 5s.

The deliverables of this research include: (1) proof of concept evidence about the use of sensory manipulation in reducing the sense of time delays and expediting human adaptation to time-delayed robot teleoperations; (2) multimodal sensory feedback system design suggestions for human-robot interaction (HRI) in time-delayed teleoperations; and (3) quantitative models of functional and performance improvements in a variety of delay scenarios.

This research proposes an innovative sensory manipulation approach to help reduce risks related to teleoperation delays. The neural, perception, and performance evidence contributes to the formulation of effective space teleoperation designs. The quantitative human models of perceptual and performance provide predictive models for NASA to perform risk and opportunity assessment for yet-to start missions that involve robot teleoperations. Lessons learned in this research will also inform a new training paradigm for both crewmembers and ground supports as for adapting to the changing environments in future deep space exploration with adaptive and assistive sensory augmentation. The data can also be transferred to other domains such as aviation and manufacturing industry with automation controls.

This research project directly contributes to the HRR Human Factors and Behavioral Performance (HFBP) element, by narrowing the gap due to the inadequate design of human and robotic integration, via a new method and corresponding evidence pertaining to the design guidelines that take into account human capabilities and limitations with regards to management of automation or robotic asset(s) under time-varying communication latencies. Specifically, it proposes and tests an innovative, aggressive, while still technically feasible method of induced human adaptation to varying robot teleoperation latencies by sensory manipulation, i.e., modifying sensory stimulation paired with the motor actions in a way that (1) alleviates the subjective feeling of time delays and (2) expedites cognitive and behavioral adaptation to the delayed teleoperation. If warranted, the proposed method provides NASA with a new dimension of human-automation-robot-interaction (HARI) for time-varying communication latencies that differs from previous mitigation methods based on automation system design and training. The expected benefits include improved teleoperation performance, perceived higher comfort level and quality, with reduced training needs and simplified automation/robotic designs.

This is a mature technology (TRL 7+) — the realistic path in is usually NASA's Technology Transfer Program: licensing an existing NASA patent, or a Space Act Agreement to use NASA facilities/expertise directly. NASA also runs a startup licensing program with no upfront fee for companies formed to commercialize a specific NASA technology.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.