Rapid Data Analytics Platform using Machine Learning

NASAs vast collection of simulated, experimental, and observational datasets has paved the way for groundbreaking advancements in addressing real-world physical challenges. In particular, NASA has made significant investments in developing state-of-the-art simulation software for climate modeling, aerodynamics, space vehicle launch environments, etc. These software tools along with large computing resources available to NASA scientists enable highly detailed simulations of physical phenomena. There is a need to address the big data bottleneck so that these data can be efficiently stored, shared, and analyzed while providing accessibility to a range of users, from the scientists with large HPC resources at their disposal to downstream users lacking the said resources. In this Phase II SBIR project, RNET and FAU will develop an advanced data management and analytics platform that is designed specifically to address the big data challenges associated with high-fidelity scientific simulation data hosted by NASA and other agencies. We employ a reduce-then-analyze strategy. Our approach combines modern machine learning and traditional computational mechanics based discretization to provide robust and interpretable compression. The use of a computational mechanics framework makes the compressed data amenable to accelerated data analytics without the need to first decompress. The platform aims to provide best-in-class compression-to-error ratios, exceptionally fast point-wise decompression, and a geometry-preserving data format to significantly reduce computational, storage, and bandwidth costs. Seamlessly assimilated into existing scientific data analytics pipelines (e.g., NetCDF, HDF5, PANGEO) and engineered to operate on a wide range of compute resources, this powerful platform will ensure that scientists across all levels of expertise can harness the transformative capabilities embedded in highly detailed simulation datasets. NASA has made significant investments in developing state-of-the-art simulation software for climate modeling, aerodynamics, space vehicle launch environments, etc. These software tools along with access to large computing resources enable highly detailed simulations of physical phenomena. There is a need to address the big data bottleneck so that these data can be efficiently stored, shared, and analyzed while providing accessibility to a range of users. In this project, RNET and FAU will develop an advanced data analytics platform. Anchored by a state-of-the-art data reduction algorithm, this platform will use best-in-class compression-to-error ratios, fast point-wise decompression, and a geometry-preserving data format to significantly reduce the computational, storage, and bandwidth costs typical of high-fidelity data analytics.  Seamlessly assimilated into existing data analytics pipelines, and engineered to operate on a wide range of compute resources, this platform will ensure that users across all levels can harness the transformative capabilities of high-fidelity datasets. The Phase II effort will focus on three technical objectives; 1) further optimization of the climate modeling (for example, GEOS-5 data) compressor; 2) tight integration of the developed platform with the existing data management and analytics workflows for seamless operation and adaptability our tool; and 3) demonstrate the potential of our DLS rapid analytics platform in fields outside of climate modeling.  The goal of the first technical objective is to increase the compression ratios achieved using the Phase I GEOS-5 compressor. Increased compression ratios will be achieved through 3D compression, Bit Grooming, and adaptive enrichment selection. To achieve the second objective, the team will develop a DLS HDF5 filter plugin; a Python-based, PANGEO compatible, rapid analytics package; and a rapid analytics command line tool. For the final objective, the project team will develop a DLS rapid analytics pipeline for fast, simple, and accurate processing of FUN3D simulation data.   

The platform will have an immediate impact on applications that deal with huge volumes of scientific data. NASA programs that would see a near-instant impact from this platform include the Earth Information System (EIS), the Earth Science Data and Information System (ESDIS) Project, the EarthData Citizen Science for Earth Systems Program (CSESP), the Making Earth System Data Records for Use in Research Environments (MEaSUREs), and the ESDS MultiMission Data Processing System Study.  Beyond NASA, the proposed platform will have an immediate impact on industries that use high-fidelity simulation data as part of an informed decision-making process, including the Automotive industry, the Aerospace industry, and the Oil and Gas industry.

This is early/mid-stage (TRL 3) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.