Framework for Mining and Analysis of Petabyte-size Time-series on the NASA Earth Exchange (NEX)

Time-series analysis is key to understanding and uncovering changes in the Earth system. However many currently available geospatial tools only provide easy access to the spatial rather than the temporal component. Therefore the burden is on the researchers to correctly extract the time-series from multiple files for further analysis. While inconvenient, this is often achievable on a small scale, but to search for trends across millions of time-series, quickly becomes a huge undertaking for individual researchers, because apart from scaling the analysis algorithm itself it requires much effort in large-scale data processing, metadata and data management. Additionally, for most researchers in Earth sciences, there are almost no tools that would enable easy time-series access, search and analysis. Finally, there are limited places where algorithms supporting novel time-series approaches can be tested and evaluated at scale. Given the importance of time-series analysis to Earth sciences, we view it as an opportunity to engage and bring together Earth science, machine learning and data mining communities - an important goal of the NASA Earth Exchange (NEX) project. The overall goal of the proposed effort is to develop a platform for fast and efficient analysis of time-series data from NASA's satellite and derived datasets at large scale (billions of time-series from 100's of terabytes to petabytes of data) that would be deployed on NEX and accessible in both supercomputing and cloud environments. While the initial focus will be on deploying the technology to support NEX and NASA users, the overall system will be developed as a flexible framework that can easily accommodate any user's time-series data and codes and will be deployable outside NEX using Docker containers. The project will significantly enhance the scale of state-of-the-art in time-series analysis, currently several orders of magnitude below the needs of the Earth science community. To accomplish this goal, we will develop time-series indexing and search components based on the Symbolic Aggregate approXimation (SAX/iSAX) that will be able to extract and index billions of time-series from satellite, model and climate data, giving both science and application users an important analysis tool and lower a major barrier in Earth science research. Finally, as time-series analysis is very active field of research, the platform will be developed as a plug-in framework and will be able to accommodate new improvements in time-series analysis, such as different space reduction methods that are first step in the indexing process. Apart from production use on the NEX system, we will deploy the system as a test-bed for users that will drive advancements in time-series analysis research, while providing unified access to billions of time-series. Because of the symbolic nature of the SAX representation, it is possible to deploy a number of algorithms from text mining, deep learning and bioinformatics that will provide giant leap in our ability to analyze time-series data and are already showing good results in other fields. In terms of the current NRA, this project is proposing to develop a data-centric technology that will significantly reduce development time of Earth science research and increase accessibility and utility of NASA data. In terms of specific technology areas outlined in the NRA, the proposed project will provide new big data analytics capability, as well as tools for scalable data mining and machine learning. Through the use of flexible container-based architecture and building upon existing capabilities of NEX and OpenNEX, the system will be demonstrated in both high-performance computing (HPC) as well as cloud environment on AWS. Period of performance for the proposed project is 2 years. The entry TRL is 3 and exit TRL is 6. While this seems like a big TRL jump in two years, given our understanding of the underlying technologies, we believe it is realistic.

Advance Earth system science knowledge through the identification, development, and demonstration of innovative information systems technologies

This is early/mid-stage (TRL 3) — the most realistic path in is NASA SBIR/STTR, which funds small businesses and research institutions to develop technology aligned with NASA's needs (equity-free, phased funding). Check whether a current SBIR/STTR solicitation topic overlaps with this project's technology area, or contact the project directly (above) to ask.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.