← Back to NASA Technology Projects
Completed TRL 5 (started at 4, targeting 5)
Effectively controlling a high degree-of-freedom robot is a challenging and often safety-critical task. As these systems approach human-like dexterity, the body of the operator becomes the most natural and effective control device. In addition, accurate tracking of the operator’s pose enables the rendering of a more immersive virtual representation of the robot’s environment. Recent advances in filmmaking motion capture systems and new motion-based controllers for video games have produced sensors that will enable rapid progress towards these goals.
This task will use body-tracking input devices (i.e. Microsoft Xbox Kinect and accelerometer gloves) to immerse an operator in an accurate virtual model of the robot’s environment, capture the intent of the operator, and safely execute mobility and manipulation tasks suitable for platforms such as Robonaut 2. Initially, the operator’s head position will be tracked in order to render an appropriate point of view in the virtual environment. Next, model-based recognizers will be developed and trained to detect gestures by the human operator and trigger autonomous behaviors on the robotic system. Initial efforts will use the Kinect sensor, with additional potential investigations into other similar or complementary sensors.
In FY14, development will focus on further extending our natural user interface system to address the concurrent operation of manipulation and mobility aspects of hybrid robotic systems such as Robonaut 2 with legs (ground only in FY14) or an ATHLETE robot driving while manipulating a payload.
The team is developing software that will speed up task development and execution both in-space and on the ground for Robonaut. The JPL “Scaffolding for Manipulation” software creates an interface that allows task developers to quickly provide intelligent constraints to Robonaut reducing the task space and adding manipulation aids to the visual scene. For example, a longitudinal axis is added to a tool in the robot’s world model and the robot can easily taps its storehouse of task primitives to select a proper grasp and plan the path to execute the grasp. The JPL scaffolding software will be used first in simulation to verify integration interfaces and then will be integrated with a ground based R2 unit and evaluated during ISS IVA task development. The software is expected to speed up both ground based and teleoperated task execution.
This task will use body-tracking input devices (i.e. Microsoft Xbox Kinect and accelerometer gloves) to immerse an operator in an accurate virtual model of the robot’s environment, capture the intent of the operator, and safely execute mobility and manipulation tasks suitable for platforms such as Robonaut 2.
Once fully developed, this technology could be used by the space program to control exploratory robots visiting an asteroid parked in near-Earth space, on the surface of the moon or IVA/EVA robots on ISS. In addition, it could be used by astronauts on board a spacecraft orbiting the moon or Mars or other bodies, to control robots on the surface or to control an EVA robot to perform maintenance or repairs.
Additional potential terrestrial applications abound, such as: use by search and rescue personnel to cooperatively control and monitor humanoid search and rescue robots; and use by industry to control humanoid work robots in dangerous environments.
Listed on TechPort itself — the most direct way to ask about this specific project.
This is early/mid-stage (TRL 5) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.
None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.