Enhancing Operational Imagery and Video for Human Exploration

This project takes a multifaceted approach toward enhancing imagery and video analysis capabilities through automated techniques, including artificial intelligence and machine learning. There has been a recent uptick in activity around machine learning powered visual data analysis and processing, and we aim to leverage this toward NASA’s unique use cases. The main aspects of the project are to

• enhance metadata through machine learning-based labeling including object detection, segmentation, text detection+recognition (Fig 5).
  • train models to detect NASA data (Fig 1)
  • provide a fine-grained search tool for imagery and video (Fig 2)
  • build out a framework for ML training with few training samples, through techniques like copy and paste data augmentation and few shot learning (Fig 3)
  • enable video analysis through tracking objects across frames (Fig 4)
• reduce downlink burden through machine learning-based compression and reconstruction (Fig 6)
• examine image quality metrics – can we determine relative quality rankings of images? (Fig 7)
• explore anomaly detection via ML
• examine the portability and use of these models to flight-like hardware (Fig 8)

(See figures in library item under “Project Description Figures")

These advancements in ML-enabled processing will increase efficiency across imagery analysis workflows. In particular, ML metadata like “what objects are in this frame”, “what text is detected in this photo” can help flight controllers and mission operations personnel quickly query the most recent time that a tool was used by a crew member, the last time a tagged handrail or module was visible in an image, or gather all of the imagery containing “angled tongs” from specific date ranges or EVAs. Additionally, this metadata can be added in near-realtime and can reduce the burden on human catalogers.

Our exploration of techniques to train on NASA specific, rare (few training samples) data will enable the preparation of models with pre-flight imagery/video, for immediate use once hardware is in space. This capability is important to ensure that models are constant up to date with the latest hardware on station or other missions, and ensures that there is fast access to searchable object imagery, without having to wait to train a model on flight data.

Experiments porting ML models to flight-like hardware paves the way for running the models developed in this and other projects on future missions with already flight certified and tested hardware. This will enable future ML processing to have in-situ, which will help with data prioritization for downlink.

Regarding the data downlink bottleneck, our compression and quality ranking experiments lay a roadmap for increasing the quantity of imagery downlinked during deep space missions. Only 14% of Artemis I imagery was downlinked during flight (the rest was saved onboard and recovered post-flight), limiting the types of analysis that could be conducted in mission. The proposed compression and quality prioritization pipeline can increase this percentage dramatically, while still maintaining most of the image detail that is critical for analysis.

This is early/mid-stage (TRL 5) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.