Time series multi-modal foundation model for near-real-time land surface dynamics characterization in support of ESDT

The soil moisture and fuel moisture content play critical roles in wildfire prediction, fire behavior simulation, emergency response, and air pollution estimation. However, current remote sensing algorithms for these parameters fall short of meeting the accuracy and near-real-time requirements of Earth System Digital Twins (ESDT) due to insufficient training samples and limited spatial-temporal resolution. Existing algorithms combining multi-modal observations often result in reduced temporal resolution because they can only retrieve moisture or other surface parameters at the multiple sensor contemporaneous acquisition dates. This limitation reduces the usable data significantly since many satellites, particularly multi-modal sensors like Sentinel-1 C-band SAR, Landsat 8/9, Sentinel-2 optical data, and the upcoming NISAR L-band SAR, are not coordinated to acquire data simultaneously. Deep learning foundation models offer an avenue to overcome the challenge of limited training data and integrate multi-modal data for reliable near-real-time land surface characterization. Building on the success of models like ChatGPT and Masked Autoencoders (MAE), foundation models have become a cornerstone in various fields, including the interpretation of earth observation data. These models are trained through self-supervised tasks and fine-tuned using domain-specific training samples. However, existing foundation models like NASA-IBM Prithvi often view earth observations as individual static images rather than time series data, relying solely on selected cloud-free images while discarding valuable good-quality observations from partially cloud-covered images. Moreover, these models tend to overlook surface seasonal dynamics, such as phenology, which are essential for dynamic soil and fuel moisture retrieval. This proposal aims to (i) develop a foundation model to fuse time series multi-modal data, including Sentinel-1 C-band SAR, NISAR L-band SAR, and Harmonized Landsat Sentinel-2 (HLS) optical reflectance data; (ii) fine-tune the foundation model for near-real-time mapping of soil moisture and live and dead fuel moisture content at any satellite data acquisition dates; and (iii) interpret the models in terms of multi-modal data fusion efficacy and input predictor importance. The proposed method utilizes a year of multi-modal satellite data to estimate soil moisture and fuel moisture content at any satellite acquisition date to utilize seasonal dynamics information. The proposed model builds on the Transformer model which is known for its excellence in time series modeling (e.g., ChatGPT), and uses a novel cascade Transformer structure to handle the uncoordinated acquisition dates of different satellites. The foundation model will be pre-trained using a masked mechanism which has been shown perform well on HLS data. It will be fine-tuned using publicly available training samples (e.g., International Soil Moisture Network, National Fuel Moisture Database) across conterminous United States, alongside additional data collected by the research team, and evaluated by comparing with existing soil moisture and fuel moisture datasets. The proposed research is responsive to the AIST Early-Stage Technology by using "Transformers and foundation models" and by seeking to "understand" the models. It could "maximize science mission return" by effectively fusing daily uncoordinated multi-modal satellite acquisitions. It could be a game-changer for accurate and near-real-time soil moisture and fuel moisture content retrieval in support of ESDT by leveraging foundation models. Furthermore, the foundation model is expected to work for other land surface parameters. The proposal has Technology Readiness Level (TRL) entry of 1 and exit of 3.

This is early/mid-stage (TRL 2) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.