Open-Source Pipeline for Large-Scale Data Processing, Analysis and Collaboration

NASA's observational and modeled data products encompass petabytes of earth science data available for analysis, analytics, and exploitation. Unfortunately, these data are underutilized due to vast computational resource requirements; disparate formats, projections, and resolutions that hinder data fusion and integrated analyses; complex and disjoint data access and retrieval protocols; and task-specific and non-reusable code development processes that hinder algorithm sharing and collaboration. Due to these limitations, recent advances in unsupervised machine learning using deep neural nets (DNNs) have not been widely adopted for applications such as pixel-based classification, image preprocessing, feature recognition, and scene understanding. Creare proposes to develop an open-source, standards-based Python software framework that removes major barriers to widespread exploitation of geospatial earth science data. This will be achieved through development of PODPAC (Pipeline for Observational Data Processing, Analysis, and Collaboration), a pipeline-based architecture that (1) enables multi-scale and multi-windowed access, exploration, and integration of available earth science data sets to support both analysis and analytics; (2) automatically accounts for differences in underlying geospatial data formats, projections, and resolutions; (3) simplifies implementation and parallelization of geospatial data processing routines; (4) seamlessly integrates with DNN machine learning frameworks; and (5) unifies access, processing, and sharing of data and algorithms via interfaces to existing NASA repositories. To demonstrate the impact of these innovations, we will use PODPAC to derive an on-demand, high-resolution global soil moisture data product from the Soil Moisture Active/Passive (SMAP) satellite radiometer observational data to support applications in hydrology, agriculture, and humanitarian response missions involving flooding, drought, and water resources.

Through close involvement with the SMAP and AirMOSS programs during Phase I and II of this effort, we are targeting these NASA programs for initial PODPAC commercialization. We envision that infusion of PODPAC into these programs will enhance publishing, access, analysis, and analytics of ongoing observational data products from these programs, while also generating new model-generated data products such as the global high-resolution soil moisture data sets developed under this effort. Subsequently, we will also target other NASA earth science observational programs seeking increased exploitation of their products, such as AIRS, AMSR-E, AMSR2, GMI, MODIS, and VIIRS for further program infusion and commercialization of PODPAC. PODPAC will also be made available as open source software so that any NASA scientists involved in the analysis, exploration, integration, and fusion of earth science data sets can benefit.

We envision the primary non-NASA applications for high-resolution soil moisture prediction and data analytics are in the areas of agriculture, forestry, construction, and recreation. In particular, the agriculture industry would greatly benefit from detailed knowledge of near surface and root zone soil moisture conditions by enabling improved irrigation and fertilization efficiencies.

This is early/mid-stage (TRL 4) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.