SLICE: Semi-supervised Learning from Images of a Changing Earth (SLICE)

The field of computer vision (CV) is advancing rapidly, enabling significant accuracy improvements for image classification, segmentation, object detection problems. In particular, multiple self- and semi-supervised learning approaches have been published in recent years: • SimCLR (v2) from Google (A Simple Framework for Contrastive Learning of Visual Representations), •. FLASH from University of Pennsylvania and Georgia Tech (Fast Learning via Auxiliary signals, Structured knowledge, and Human expertise) •. DINO / PAWS from Facebook (Self-Supervised Vision Transformers with DINO / Predicting View-Assignments with Support Samples), •. EsViT from Microsoft (Efficient Self-supervised Vision Transformers for Representation Learning). We propose to investigate and characterize the efficacy of multiple SSL techniques for representative image problems on Earth imagery, and then select the best for further infusion into mission and science workflows. The challenge for assessing climate change is to build a flexible Cloud-based system, with additional GPU training on supercomputers, that can apply cutting-edge computer vision techniques at scale on years of satellite-based Earth and ocean imagery. Therefore, we propose to build a Cloud and supercomputing-based platform for cutting-edge computer vision at scale. The three top-level goals of the SLICE system, "Semi-supervised Learning from Images of a Changing Earth" are: 1. Establish the SLICE framework and platform for applying scalable semi-supervised computer vision models to Earth imagery, running in AWS Cloud and supercomputing environments, that can be easily adopted as a reusable platform by NASA data centers, mission science teams and NASA PIs. 2. Investigate and characterize the accuracy of multiple SSL models (i.e. SimCLRv2, DINO, EsViT) on a variety of relevant remote sensing tasks with minimum labels (here ocean phenomena). 3. Build and publish self- and semi-supervised learning models with a focus on the upper ocean small-scale processes in anticipation of several on-going and upcoming NASA missions (i.e. SWOT, WaCM, and PACE). Ocean eddy properties and derived heat flux will be modeled and predicted from SST, SSH, and SAR data. The proposed work is directly traceable to AIST Objective O2 by developing a task-agnostic deep learning framework that facilitates large-scale image analytics on disparate, multi-domain datasets. The development of a framework and platform to train SOTA deep learning models with limited labels is responsive to the Analytic Collaborative Frameworks (ACF) thrust of the AIST element because such models can be used as individual, high-performing building blocks of a unified ACF, or for that matter surrogate models in an ESDT. The SLICE platform will provide correct example ML workflows, parallel image tile preparation, best of breed data augmentation and training frameworks that do distributed training on multiple GPU's, and publish multiple tuned SOTA SSL and vision transformer models for reuse. The pretrained SSL models will give scientists a headstart in that they can be immediately applied and finetuned on the target problem, with less training time required. By providing models pretrained on physics model outputs (not ImageNet), the starting model weights can be "physics informed" for the geophysical system being studied. Since all of the algorithms will be "pluggable", they can be replaced with modified data preprocessing & augmentation, or the latest cutting-edge DL approach and network architecture. The CV field applied to Earth imagery is growing rapidly and is overdue for a standard platform that can evolve rapidly, so that the science applications stay on the cutting edge of ML. SLICE is a first step at standardization and spreading SOTA semi-supervised learning approaches to CV in all science areas.

Advance Earth system science knowledge through the Identification, develop, and demonstrate innovative information systems technologies

This is early/mid-stage (TRL 4) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.