Hybrid Neural Scene Representations for Earth Observation: Unifying Physics-Based Models, Heterogeneous Datasets, and Radiance Fields

We will develop, assess, and validate novel continuous scene representations from heterogeneous Earth-observation data sets. Our goal is to better model complex terrestrial landscapes under challenging circumstances, such as changing lighting conditions or rapidly evolving terrain due to natural or man-made disasters. We will create a physics-based continuous scene representation that encodes diverse physical and biophysical information from diverse data sets, such as NASA GEDI's ground elevation and canopy top height products (L2A), Leaf Area Index (LAI) and LAI profile products (L2B), ICESat-2's Land Water Vegetation elevation products (ATL08), and multi-view surface reflectance products by Landsat 8/9, Sentinel 2A and 2B, or Harmonized Landsat Sentinel-2 (HLS). The scene representation will be implemented using Neural Radiance Fields (NeRFs), which are a cutting-edge technology in Computer Vision (CV) that capture reality by encoding detailed information about an environment within a bounded volume and show considerable promise for encoding a diversity of Earth-observation data. Incorporating such heterogeneous Earth-observation products from NASA sensors in a novel NeRF pipeline will allow us to produce accurate Digital Surface Models (DSMs), Digital Terrain Models (DTMs), and Canopy Height Models (CHMs), as described by the Surface Topology and Vegetation (STV) mission incubation study. We will develop techniques for continually monitoring our NeRF-derived terrain to detect and identify semantically significant temporal alterations, such as freeze and thaw cycles in cold climates, changes in surface elevations, changes in vegetation/forest structures, short-term changes in weather patterns before and after data acquisition, and other land-use/land-cover changes. For this purpose, we will employ change-detection algorithms on NeRF products and design techniques that can identify semantically meaningful changes. The final output of these algorithms will provide both a class and a change label for each pixel in a series of repeated images. For rigid surfaces (e.g., terrain deformation), the algorithm will provide an image-to-image motion estimate and for entities undergoing arbitrary changes (e.g., seasonal foliage), image-to-image deformation will be estimated. The output products and images can be used for a variety of on-the-ground planning, such as developing view-shed perspectives and path-planning decisions. It is due to the novel encoding of high dimensional continuous scene geometry that allows neural scene representations to contain such a variety and density of data. Our Early-Stage Technology (EST) applies to several sub-elements including, but not limited to: "Beyond Deep Learning", "Physics-based and Hyperdimensional AI", "Validation assessment and understanding of AI/ML", and "Algorithm performance optimization via GPU and brain-inspired neuromorphic". Our algorithms will be tested on a plethora of platforms and datasets . At a high level, we want to develop a novel mapping system that can help inform the development of a science traceability matrix for a large mission concept, such as STV. In fact, the STV incubation study states "when stereo imaging is acquired , the acquisition parameters vary widely (e.g., illumination conditions, view angles, season) making systematic use of the data for height models very challenging. […] An existing need is to study the best acquisition strategy for Stereo Photogrammetry" We claim that we can overcome these challenges with our proposed novel technology.

This is early/mid-stage (TRL 2) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.