Pix4DCloud: A Suite of Physics-Constrained Transformer Models to Retrieve 4D Clouds in Real World and Digital Twins (Pix4DCloud)

Atmospheric clouds exhibit both vertical and horizontal structures. The 3D structure of clouds exerts over-arching impacts on the top-of-atmosphere (TOA) radiation budget and surface precipitation characteristics. Moreover, the vertical cloud structure also impacts quality of downstream tasks (e.g., aerosol, fire or ocean color retrievals). It is however extremely challenging to retrieve cloud vertical structures from spaceborne wide-swath passive sensors using physics-based models because of the high computational cost and large uncertainties involved in the radiative transfer calculation. On the modeling side, clouds usually form as an "ad-hoc" process depending on fixed thresholds calculated from atmospheric fields, which does not necessarily represent the weather-dependent physical processes in nature. Many recent works, including ours, explored a variety of machine learning (ML) approaches to predict cloud vertical structures using passive sensor measurements trained on "truths" from active sensors. These works still employ traditional ML methods for each individual instrument, while temporal contextual information from adjacent overpasses from multiple similar instruments are not utilized thus far. This proposal aims to introduce and evaluate the transformer neural network architecture (often used in "foundation models" as the core for the large language models) to overcome the common hurdle when applying the ML/AI to satellite data: cross-time and cross-mission knowledge transfer. We target at both improving the observational 3D cloud retrievals and creating an ESDT subcomponent cloud generator. Through creating and utilizing the multi-timestep, multi-spectral and multi-instrument pre-trained all-sky radiance foundation models, we will for the first time quantify (1) the merits of using a foundation model to improve the 3D cloud mask and type retrieval from Advanced Baseline Imager (ABI) by leveraging available temporal and spatial contextual information; (2) the advantage of using CloudSat radar observations and ABI radiance-based foundation model to accurately represent 3D cloud fields in global models. Moreover, as deep convective systems (DCs) are the prominent source of extreme precipitation events, we will (3) generate A 3D Vertical Object-oriented deep Convective systems And their DevelOpment stages model (A3DVOCADO) that auto-identifies and predicts the development stages and lifespan of the DCs from the 3D cloud fields in both retrieved and DT-clouds, and we will (4) investigate whether the DT-clouds respond to inter-annual variabilities the same way clouds in nature do. "Pix4Dcloud" as called employs temporal information to predict 3D clouds. This project enters with TRL 3 and is expected to exit at TRL 4 or 5, which fits the AET category. Being one of the pioneer efforts at introducing the transformer/foundation model to NASA atmospheric science applications and the first comprehensive evaluation of its merits against both physics and traditional ML models, this proposal perfectly fits the core scope of the AIST program for "novel computer science technologies expected to be needed by the Earth Science Division in the 5-10-year timeframe". It specifically responds to both O3 and O2 as PIX4DCloud contains a suite of ML models to allow flexibilities in generating 3D cloud structures or further detecting and tagging DC development stage in observations or model simulations. DCs not only fall into one of the 5 NOS categories (water cycle), but are also one of the 8 objectives of the upcoming decadal survey AOS mission. Therefore, outcomes from this AIST project will bring broader benefits to NASA's future earth science missions by potentially transforming the retrieval and flying strategy for NASA's next generation instruments. Last but not the least, this project will deliver a foundation model to the public that can bring direct benefits to a wide variety of atmospheric science applications.

Expand current definitions of modeling and leverage state-of-the-art computer and information science for innovating advanced modeling techniques as well as new technologies and frameworks that will be essential in the development of Earth System Digital Twins

This is early/mid-stage (TRL 3) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.