NARA - Phase 1

High Performance Storage and Access to Electronic Federal Records with scientifc and Engineering Data with Electronic Records

Phase One of our research investigated the attributes of a record format that make it more or less suitable for long-term archiving, with an emphasis on the HDF5 meta-file format and formats based on HDF5, such as HDF-EOS. Much of the phase I research was done in collaboration with the Illinois State Geological Survey (ISGS).

• The group investigated appropriateness of scientific data formats for long term archiving.
  
  The results are presented in the technical report:
  "Attributes of File Formats for Long-Term Preservation of Scientific and Engineering Data in Digital Libraries [pdf]" by Mike Folk and Bruce Barkstrom.


• The HDF Team selected geospatial data as its initial test case in its investigations of the suitability of HDF5 and HDF-EOS 5 for storing geospatial data. A number of geospatial raster data types were converted to HDF5, including digital elevation models (DEM), color digital orthophoto quadrangles (DOQ), and 2D and 3D spatial geologic grids. HDF-EOS was found to be good data model for certain geospatial data types, adding performance and scalability capabilities, as well as standardization advantages not available in other formats. 2D and 3D geologic grids were mapped and converted with scientific formats, showing significant gains in storage size and I/O capability and no loss of information.


• In addition, the group studied vector data storage, including a study of the SDTS vector data format for storing digital line graph (DLG) datasets, and Shapefiles for other vector data. For this data, HDF5 was found to be a suitable format in terms of storage size and I/O capabilities for large datasets.
  
  The results of this work are presented in the technical report:
  Scientific formats for geospatial data preservation -- A study of suitability and performance by Mike Folk and Vailin Choi.


• A third project, in collaboration with the ISGS, studied the development and use of subsetting capabilities and visualization tools for geologic data in HDF5, particularly visualization of 3 dimensional volumes of geologic data.
  
  This work is described in the poster:
  HDF5, HDF-EOS and Geospatial Data Archives by Don Keefer and Mike Folk.