HY-LaTiS: Evolving the functional data model through creation of a tool set for hyperspectral image analysis

Researchers in the Earth atmosphere remote sensing community need tools to be able to effectively handle the hyperspectral data that is now becoming available. The massive data volumes of these datasets, demanding up to an estimated 100 TB of storage for a set of analysis activities to be completed, is a barrier to their use. We propose to develop a tool, HY-LaTiS, to facilitate the use of hyperspectral data (HD) by providing convenient access to and operations over the massive data volumes they present. The data will be stored in the cloud where server-side computations can be performed. Users will be able to analyze data in the cloud using a notebook approach that enables processing to occur in the cloud without requiring dataset download to user workstations. They will also be able to reduce HD and stream it into local memory for the purpose of integrating hyperspectral information into existing sequential analyses. Overall, HD will be easier to access and use, with better interactivity in analyses than is available today. The will be achieved by putting a functional data processing layer over the Spark big data processing engine, leveraging the existing LaTiS data access server and framework as the functional data processing layer. LaTiS has been in use at the Laboratory for Atmospheric and Space Physics (LASP) for many years, with about 20 instances currently running, serving a wide variety of datasets from a wide variety of domains, formats, and locations. We estimate LaTiS to be at an entry assessment of TRL 4 due to its demonstrated success within the lab. This project expands the Spark interface to offer not only the relational data model interface that it does now, but also a functional interface in the functional programming sense. This is accomplished by implementing the LaTiS functional data model and associated Hyperspectral Imaging Analysis (HIA) operations with a functional style, while leveraging the power of Spark underneath. We argue that a functional approach towards data and processing provides a flexibility of representation coupled with the right set of semantics to be a very good approach for scientific analysis. This two-year project will start with the technical team working with our science Co-Is, who are our subject matter experts (SMEs), to develop a core hyperspectral image analysis (HIA) domain tool set that supports HD analysis in the code and streaming into memory reduced volume HD. The tool set will provide an intuitive interface designed specifically at the level of HIA. The first operations to implement will be access operations, such as subsetting on geolocation, wavelength, or pixel. The second set of operations will be server side analyses, such as various forms of integration to be applied to reduce the data volume to a lower resolution. After this work is underway, and depending on the timing of the needs of our SMEs, new analyses and underlying operations will be developed that leverage HY-LaTiS operations via notebook technologies, such as Jupyter notebook or JupyterLab. This is where the true interactive experience will lie. Instead of the batch oriented HIA practices of today, this approach will save time by reducing or eliminating the tedious download-extract-operate processing now required. Once the capabilities of subsetting, interpolating, and performing a few other essential operations are operational in a notebook environment, the HY-LaTiS development team will hold a workshop to demonstrate the tool to the Earth atmosphere remote sensing community. There they will be able to test the tool and provide feedback. Our success will be determined by the adoption and usage of the tools by our SMEs and in performing science research operations that involve these hyperspectral datasets.

Advance Earth system science knowledge through the identification, development, and demonstration of innovative information systems technologies

This is a mature technology (TRL 7) — the realistic path in is usually NASA's Technology Transfer Program: licensing an existing NASA patent, or a Space Act Agreement to use NASA facilities/expertise directly. NASA also runs a startup licensing program with no upfront fee for companies formed to commercialize a specific NASA technology.

None of these are guaranteed paths for this specific project — TechPort itself doesn't have an "apply" button. Reaching out to the contact(s) above with a specific question is usually the fastest way to find out what's actually open.