Kernel Flows: emulating complex models for massive data sets

Inference about atmospheric and surface phenomena from remote sensing data often requires computationally expensive empirical or physical models, and always requires uncertainty quantification (UQ). Running these models to predict or retrieve geophysical quantities for very large data sets is prohibitive, and Monte Carlo experiments for UQ, which involve rerunning these models many times, are out of the question except possibly for small case studies. These problems can be overcome with emulators: machine learning models that ''emulate" physical models. Emulators are trained on carefully selected examples of inputs and outputs generated either by pairs of inputs and outputs acquired directly from observations, or by a physical model under specific conditions that are representative of the problem space. Then, the emulator is applied to new inputs, and produces estimates of corresponding outputs, ideally with uncertainties due to the emulation itself. The latter are crucial for interpreting emulator output, and must be included in the total uncertainty ascertained from Monte Carlo-based UQ experiments. We propose a general-purpose, versatile emulation tool that (1) provides fast, accurate emulation with little tuning, (2) scales up to very large training sets, (3) provides uncertainties associated with outputs, and (4) is open source. This tool set will facilitate large-scale implementation of forward modeling and retrievals, and of UQ at production scales. We choose two science application areas to showcase these capabilities: (A) nowcasting the evolution of convective storms; an example of empirical modeling, and (B) radiative transfer for Earth remote sensing; an example of physical modeling. Our methodology is based on Gaussian Processes and cross-validation. These are combined in an algorithm called Kernel Flows, hereafter KF (Owhadi and Yoo, 2019). KF has been used in a variety of settings with excellent results, including climate model emulation (Hamzi, Maulik, and Owhadi, 2021). Preliminary results applying KF to radiative transfer problems for OCO-2, MLS, and imaging spectroscopy show that KF is well-suited to high-dimensional emulation required in the types of problem represented by our applications. Our method is general enough to apply to a wide range of analysis and prediction problems, and will enable agile science investigations as called for in Objective O2 of the Notice of Funding Opportunity. Software interfaces will be lightweight, simple, and general, to enable easy integration with different data sources. The two science applications involve running models on very large data sets, the need to do it faster than is currently possible, and to quantify uncertainties that result from the emulation process. In the nowcasting example, the model to be emulated is the relationship between vertical structure of storm clouds and convective storm formation. In the other application, it is a radiative transfer model. The Gaussian Process underlying our method is based on a rigorous probabilistic model that can be used to generate Monte Carlo replicates of the predicted fields. This enables forward UQ experiments to derive uncertainties on predicted quantities, including emulator uncertainty.

Advance Earth system science knowledge through the Identification, develop, and demonstrate innovative information systems technologies

This is early/mid-stage (TRL 5) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.