Autonomous Guidance for Medical Procedures

Problem: Mars missions will not have real-time communications with Mission Control Center (MCC), and correspondingly limited access to supervision for complex medical scenarios that lie outside the skill set of crew members. Thus we need solutions that can provide just-in-time training, monitoring, and autonomous guidance of medical procedures, to make the crew independent of MCC. Solution: We propose a system for automatically building computational models of a complex physical task, such as a medical procedure performed by humans, given only a handful of recorded expert demonstrations of the task. Once such a model is built, our system can finely analyze the same task being performed in live video, to provide measurements and analytics, improve efficiency, guide a crew member through the task, or provide just-in-time training. We combine recent advances in machine learning and computer vision, including our own prior work, in human pose estimation, 3D object estimation, action classification, and long-term causal reasoning to build novel systems that can understand goal-driven multi-step activities in live video feed. Existing commercial solutions: Some AI platforms offer capabilities to estimate human skeletal poses, locate objects, as well as classify simple actions in video. However in order to understand a certain multi-step activity (e.g. a medical procedure), a solution provider still needs a team of computer vision or IoT engineers, who write customized computer code to represent that specific activity, relating human pose changes with object movements over time, i.e. building this temporal causation structure on top of the capabilities provided by the existing AI platforms. In contrast, our solution is able to learn complex activities that combine human-object and human-human interaction over time merely from example demonstrations of the activity. Our system does not require customization at the level of new programming to model a new activity and scenario.

Mars missions face the challenge of significant communication delays with Mission Control, while complexity of operations keeps increasing. Thus, just-in-time training and autonomous guidance and monitoring solutions are valuable for medical operations and beyond. Our solution packages an "extra pair of trained eyes" (in the form of cameras and artificial intelligence software) to assist the crew, and we predict the solution has the potential on some missions to reduce crew headcount by 1 or more, which will mean enormous savings for NASA.

Medical learners need attending physicians or expert nurses to provide them with feedback when learning a procedure such as Lumbar Puncture on a medical simulator. Unfortunately, expert time in healthcare is incredibly valuable and also experts are not geographically scalable. Our solution replaces the need for expert feedback at medical simulation centers saving them millions of dollars each year.

This is early/mid-stage (TRL 4) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.