Research and Development Office

Evaluation of file formats for storage and transfer of large datasets in the RISE platform

Project ID: 22041
Principal Investigator: Drew Loney
Research Topic: Water Resource Data Analysis
Funded Fiscal Years: 2022
Keywords: None

Research Question

This effort proposes to identify large datasets within Reclamation and evaluate formats for efficiently handling the datasets within RISE. The effort will therefore focus on five core questions:

1) Who is generating large datasets within Reclamation?
2) What open source file formats are available to archive large datasets?
3) What are the positives and negatives of each format for storage, processing, and transfer?
4) How challenging is it for users to prepare data in each available or proposed format?
5) What modifications to RISE would be required to accommodate each file format?

The results of these questions will yield recommended file formats and processing steps to that format. Subsequent efforts will build upon the proposed work to create capabilities in RISE and perform data migration into RISE.

Need and Benefit

The proposed effort addresses the open data research area within the water operations planning need. The proposed investigation will lay the groundwork to expand the RISE platform to efficiently handle large datasets. This is a requirement for RISE operations to scale to the amount of data being produced by Reclamation observation and modeling efforts. This proposed project has been coordinated with the RISE lead and includes RISE team members as members of the proposed effort.

Contributing Partners

Contact the Principal Investigator for information about partners.

Research Products

Please contact research@usbr.gov about research products related to this project.

---

Return to Research Projects