Conversion of from HDF4 to HDF5:
"Hybrid" HDF-EOS Files

Robert E. McGrath
Kent Yang
National Center for Supercomputing Applications
University of Illinois, Urbana Champaign

January 18, 2002

Contents

1 Introduction
2. Method
3. Results
4. Conclusions
Acknowledgments
References
Appendix 1
Appendix 2
Appendix 3

1. Introduction

Version 2.6 and earlier of HDF-EOS were built on top of the HDF4 library [2].  Version 5 of HDF-EOS is built on top of HDF5 [3]. Files created with HDF-EOS2.x cannot be read with HDF-EOS5 and vice versa.  However, the HDF-EOS objects are equivalent, so files can be translated.

The heconvert program [4] converts an HDF-EOS 2.6 file to an equivalent HDF-EOS 5 file. All the EOS objects--the Grid, Swath, and Point objects, and associated metadata--are read from the HDF-EOS2.x (HDF4) file and written to an equivalent HDF-EOS5 (HDF5) file.

Most HDF-EOS data products contain standard HDF objects as well as the HDF-EOS objects. These may include:

These HDF objects are created through calls to the HDF4 library, they are not managed by the HDF-EOS library.

The heconvert utility converts only the objects and metadata managed by the HDF-EOS library, and consequently cannot convert the other HDF objects. The result is that the file created by heconvert may omit some of the objects and metadata form the original.

The NCSA library, libh4toh5 [5], is a library of routines that move individual HDF objects from an HDF4 file to an HDF5 file, performing a default translation [6].  This library can be used by C applications to create a custom conversion for specific data products.

In this experiment, the heconvert utility is augmented using libh4toh5. In addition to the standard conversion of HDF-EOS objects, The experimental program  identifies and converts non-EOS objects in an EOS dataset, creating a more complete HDF-EOS5 file.

2. Method

A sample HDF-EOS file was selected from the EOS-SAMPLER [7].  The selected file is a CERES file: 
CER_ES8_Terra-FM2_Test_SCF_016011.20000830.subset_70_20_-140_-40.20001012_204110Z.hdf
(For brevity, this file is denoted as 'ceres.hdf' below.)

The heosls utility [2] shows the contents of ceres.hdf  (slightly edited for space): 

FILE NAME: ./ceres.hdf                
NCSA HDF Version 4.1 Release 2, March 1998                                    
HDF-EOS Version: HDFEOS_V2.6                                                  
"CERES_ES8_subset"  SWATH                                                     
"StructMetadata.0"  Global Attribute                                          
"coremetadata"  Global Attribute                                              
"archivemetadata"  Global Attribute                                           
"SubsetMetadata.0"  Global Attribute                                          
VDATA   "CERES_metadata"  (CERES)
This file was written with HDF-EOS 2.6.  It contains one Swath, and the corresponding StructMetadata.0.  The HDF file also contains four important HDF4 objects: These objects are not managed by the HDF-EOS library. They are product-specific, created by the data processing software and written directly using the HDF4 library.

Thus, this file is an example of a 'hybrid', which contains both HDF-EOS and regular HDF objects.

In the control condition, the test file was converted with the heconvert utility from [4]. In the experimental condition, the heconvert utility was modified to call the H4toh5 library to convert additional HDF4 objects. The Software configuration is summarized in Table 1.
 

Table 1. Summary of Configuration
heconvert
HDF-EOS (4)  V. 2.6
HDF4  4.1.R5
HDF-EOS 5 5.1
libh4toh5 1.0 beta (pre release)
Sun SPARC 5 Solaris 2.7

3. Results

3.1 Control Condition

The heconvert utility was run on the test input file: 
heconvert -i ceres.hdf -o ceres-cont.he5
The conversion ran in approximately 1 minute 41 seconds (input and output files were on a network mounted disk.)

The output file was successfully created, size 77,127,264 bytes. The file contains the HDF5 objects for the Swath, and the HDF-EOS metadata (HDFEOSVersion and StructMetadata.0).  Appendix 1 shows a summary of the HDF-5 file.

As expected, the product specific objects were not copied by heconvert, and consequently they are missing from the output file.

3.2. Experimental Condition

The source code to heconvert was modified to add calls to the libh4toh5 [5] to copy additional elements from the HDF4 to the HDF5.  The experimental code identifies some of the regular HDF4 objects that may exist in an HDF-EOS and converts them to equivalent HDF5 objects.  The code handles: The experimental code is called when the '-hybrid' option is selected. A sketch of this code is given in Appendix 3.

The program was rebuilt on the same system with the same configuration.  A prerelease of libh4toh5 1.0 Beta was used.

The revised program was run on the experimental input file: 

heconvert -hybrid -i ceres.hdf -o ceres-exper.he5
The conversion ran in approximately 1 minute 44 seconds.

The output file was successfully created, size 77,181,076 bytes.  The file contains the same HDF5 objects for the Swath and the HDF-EOS metadata as the control. In addition, the product specific annotations are copied to the output file (as attributes of "/") and the Vdata is created (as a Compound Dataset under "/"). Appendix 2 shows a summary of the HDF-5 file.

Note that the objects have been created in a default location (under "/"), and the attributes have default names:  "coremetadata_GLOSDS", etc.   For a realistic conversion, it is likely that these would be put in more appropriate locations in the output file.  Also, the default conversions might not be desired.  For example, the "coremetadata" attribute in HDF4 might better be a Dataset in HDF5 (as is done with "StructMetadata.0"), and all the product-specific objects would probably be stored in a single group, not under "/".

4. Conclusions

This experiment demonstrates the effective use of the libh4toh5 to construct a customized conversion utility for "hybrid" HDF-EOS files.  The standard heconvert utility manages the HDF-EOS objects.  It was extended with approximately fifty lines of code to handle several classes of standard HDF4 files. The conversion accurately detects the additional objects and performs a complete conversion into the HDF5 file.  The running time is essentially identical to the original utility.

This experiment shows that the libh4toh5 provides a toolkit to more easily construct conversion utilities.

This toolkit might be used to create a standard converter for standardized data products that are defined in both HDF4 and HDF5. It might also be used for more ad hoc conversions, e.g., for a small science team that needs to convert HDF output from a program to be read using HDF5.

It should be noted that the libh4toh5 performs a default conversion, assigning default names to objects and placing them in default locations in the output file. The HDF5 version of a standardized data product will be specifically designed for HDF5, and should not be expected to conform to the default mapping in [6]. Therefore, a customized conversion utility would not want to call the libh4toh5 with the default settings, and for some cases might need to create its own conversion library.

Acknowledgments

This report is based upon work supported in part by a Cooperative Agreement with NASA under NASA grants NAG 5-2040 and NCC5-599. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration.

Other support provided by NCSA and other sponsors and agencies [9].

References

1. HDF, http://hdf.ncsa.uiuc.edu

2. HDF-EOS 4 (HE4), http://hdfeos.gsfc.nasa.gov/hdfeos/he4.cfm

3. HDF-EOS 5 (HE5), http://hdfeos.gsfc.nasa.gov/hdfeos/he5.cfm

4.  heconvert, http://hdfeos.gsfc.nasa.gov/hdfeos/SoftwareDist.html#three

5. HDF (4.x) and HDF5, http://hdf.ncsa.uiuc.edu/h4toh5/

6. Mapping HDF4 Objects to HDF5 Objects, http://hdf.ncsa.uiuc.edu/HDF5/doc/ADGuide/H4toH5Mapping.pdf

7. "EOSDIS Terra Data Sampler  #1", 2000, http://ivanova.gsfc.nasa.gov/outreach/Terra_CD-01/start.htm

8. "HDFEOS ASCII dumper", http://hdfeos.gsfc.nasa.gov/hdfeos/viewall_1.cfm

9. http://hdf.ncsa.uiuc.edu/acknowledgement.html


 

Appendix 1.  Summary of HDF-5 file, control condition


Output of h5dump -H, edited for space. 

HDF5 "ceres-control.he5" {
GROUP "/" {
   GROUP "HDFEOS" {
      GROUP "ADDITIONAL" {
         GROUP "FILE_ATTRIBUTES" {
         } 
      } 
      GROUP "SWATHS" {
         GROUP "CERES_ES8_subset" {
            GROUP "Data Fields" {
               DATASET "CERES LW flux at TOA" {} 
               DATASET "CERES LW unfiltered radiance" {} 
               DATASET "CERES SW filtered radiance" {} 
               DATASET "CERES SW flux at TOA" {} 
               DATASET "CERES SW unfiltered radiance" {} 
               DATASET "CERES TOT filtered radiance" {} 
               DATASET "CERES WN filtered radiance" {} 
               DATASET "CERES WN unfiltered radiance" {} 
               DATASET "CERES relative azimuth at TOA" {} 
               DATASET "CERES solar zenith at TOA" {} 
               DATASET "CERES viewing zenith at TOA" {} 
               DATASET "Colatitude of Sun at observation" {} 
               DATASET "Colatitude of satellite nadir at record end" {} 
               DATASET "Colatitude of satellite nadir at record start" {} 
               DATASET "ERBE scene identification at observation" {} 
               DATASET "Earth-Sun distance at record start" {} 
               DATASET "Longitude of Sun at observation" {} 
               DATASET "Longitude of satellite nadir at record end" {} 
               DATASET "Longitude of satellite nadir at record start" {} 
               DATASET "Rapid retrace flag words" {} 
               DATASET "SW channel flag words" {} 
               DATASET "Scanner FOV flag words" {} 
               DATASET "TOT channel flag words" {} 
               DATASET "WN channel flag words" {} 
               DATASET "X component of satellite position at record end" {} 
               DATASET "X component of satellite position at record start" {} 
               DATASET "X component of satellite velocity at record end" {} 
               DATASET "X component of satellite velocity at record start" {} 
               DATASET "Y component of satellite position at record end" {} 
               DATASET "Y component of satellite position at record start" {} 
               DATASET "Y component of satellite velocity at record end" {} 
               DATASET "Y component of satellite velocity at record start" {} 
               DATASET "Z component of satellite position at record end" {} 
               DATASET "Z component of satellite position at record start" {} 
               DATASET "Z component of satellite velocity at record end" {} 
               DATASET "Z component of satellite velocity at record start" {} 
            } 
            GROUP "Geolocation Fields" {
               DATASET "Colatitude of CERES FOV at TOA" {} 
               DATASET "Longitude of CERES FOV at TOA" {} 
               DATASET "Time of observation" {} 
            } 
            GROUP "Profile Fields" {
            } 
         } 
      } 
   } 
   GROUP "HDFEOS INFORMATION" {
      ATTRIBUTE "HDFEOSVersion" { 
      } 
      DATASET "StructMetadata.0" {
      } 
   } 
} 
}

Appendix 2.  Description of HDF5 file for experimental condition.

Output of h5dump -H, edited for space. 
HDF5 "ceres-exper.he5" {
GROUP "/" {
   ATTRIBUTE "coremetadata_GLOSDS" {
   } 
   ATTRIBUTE "archivemetadata_GLOSDS" { 
   } 
   ATTRIBUTE "SubsetMetadata.0_GLOSDS" {
   } 
   DATASET "CERES_metadata" {
      DATATYPE  H5T_COMPOUND {
         "SHORTNAME";
         "RANGEBEGINNINGDATE";
         "RANGEBEGINNINGTIME";
         "RANGEENDINGDATE";
         "RANGEENDINGTIME";
         "AUTOMATICQUALITYFLAG";
         "AUTOMATICQUALITYFLAGEXPLANATION";
         "ASSOCIATEDPLATFORMSHORTNAME";
         "ASSOCIATEDINSTRUMENTSHORTNAME";
         "LOCALGRANULEID";
         "LOCALVERSIONID";
         "CERPRODUCTIONDATETIME";
         "NUMBEROFRECORDS";
         "PRODUCTGENERATIONLOC";
      }        
      ATTRIBUTE "HDF4_OBJECT_TYPE" {
      } 
      ATTRIBUTE "HDF4_OBJECT_NAME" {
      } 
      ATTRIBUTE "HDF4_REF_NUM" {} 
   } 
   GROUP "HDFEOS" {
      GROUP "ADDITIONAL" {

         GROUP "FILE_ATTRIBUTES" {
         } 
      } 
      GROUP "SWATHS" {
         GROUP "CERES_ES8_subset" {
            GROUP "Data Fields" {
               DATASET "CERES LW flux at TOA" {} 
               DATASET "CERES LW unfiltered radiance" {} 
               DATASET "CERES SW filtered radiance" {} 
               DATASET "CERES SW flux at TOA" {} 
               DATASET "CERES SW unfiltered radiance" {} 
               DATASET "CERES TOT filtered radiance" {} 
               DATASET "CERES WN filtered radiance" {} 
               DATASET "CERES WN unfiltered radiance" {} 
               DATASET "CERES relative azimuth at TOA" {} 
               DATASET "CERES solar zenith at TOA" {} 
               DATASET "CERES viewing zenith at TOA" {} 
               DATASET "Colatitude of Sun at observation" {} 
               DATASET "Colatitude of satellite nadir at record end" {} 
               DATASET "Colatitude of satellite nadir at record start" {} 
               DATASET "ERBE scene identification at observation" {} 
               DATASET "Earth-Sun distance at record start" {} 
               DATASET "Longitude of Sun at observation" {} 
               DATASET "Longitude of satellite nadir at record end" {} 
               DATASET "Longitude of satellite nadir at record start" {} 
               DATASET "Rapid retrace flag words" {} 
               DATASET "SW channel flag words" {} 
               DATASET "Scanner FOV flag words" {} 
               DATASET "TOT channel flag words" {} 
               DATASET "WN channel flag words" {} 
               DATASET "X component of satellite position at record end" {} 
               DATASET "X component of satellite position at record start" {} 
               DATASET "X component of satellite velocity at record end" {} 
               DATASET "X component of satellite velocity at record start" {} 
               DATASET "Y component of satellite position at record end" {} 
               DATASET "Y component of satellite position at record start" {} 
               DATASET "Y component of satellite velocity at record end" {} 
               DATASET "Y component of satellite velocity at record start" {} 
               DATASET "Z component of satellite position at record end" {} 
               DATASET "Z component of satellite position at record start" {} 
               DATASET "Z component of satellite velocity at record end" {} 
               DATASET "Z component of satellite velocity at record start" {} 
            } 
            GROUP "Geolocation Fields" {
               DATASET "Colatitude of CERES FOV at TOA" {} 
               DATASET "Longitude of CERES FOV at TOA" {} 
               DATASET "Time of observation" {} 
            } 
            GROUP "Profile Fields" {
            } 
         } 
      } 
   } 
   GROUP "HDFEOS INFORMATION" {
      ATTRIBUTE "HDFEOSVersion" {
      } 
      DATASET "StructMetadata.0" {} 
   } 
} 
}

Appendix 3.  Sketch of the demonstration code.

void get_the_rest(char * h4file, char *h5file);

int main (int argc, char *argv[])
{
 /* ... */
        status = DoSwathConversion(hdf4Info);
 /* ... */
        status = DoGridConversion(hdf4Info);
 /* ... */
        status = DoPointConversion(hdf4Info);

/***************************************************************************
  All EOS objects have been recreated in the output file.
  Now pick up at least some of the regular HDF objects, i.e.,
  not managed by HDF-EOS.
***************************************************************************/
    if (convertHybrid == CONVERT_TRUE) {
        get_the_rest(inNameGlobal, outNameGlobal);
        /* check for errors ... */
    }

    return 0;
}

/*
 *  Demonstration of the use of libh4toh5
 *
 *  This routine finds some of the HDF objects that are not managed
 *  by the HDF-EOS library and does a default conversion.
 *
 *  This is a generic routine, you would write a customized 
 *  version for specific data products.
 */
void get_the_rest(char * h4file, char *h5file)
{
hid_t h5fid;
hid_t h5gid;
hid_t h5aid;
hid_t h45id;
herr_t res;
int32 nvd;

    h45id = H4toh5open(h4file, h5file, H425_OPEN);

    /*
     *  HDF-EOS doesn't use file annotations, so any we find
     *  should be converted.
     */
    H4toh5annofil_alldescs(h45id);
    H4toh5annofil_alllabels(h45id);

    /*
     *  HDF-EOS uses SDS Global attributes for 'StructMetadata.0',
     *  and HDFEOSVersion.
     *
     *  Other SDS global attributes have important product 
     *  metadata but are not managed by HDF-EOS.
     *
     *  This section moves them all to default HDF-5 attributes,
     *  and then get rid of the ones moved by heconvert.
     */
    H4toh5_glosdsattr(h45id); /* this moves everything */

    /* Find duplicate HDFEOSVersion, StructMetadata.0 and delete
     *  from output file.
     */
    h5fid = H5Fopen(h5file,H5F_ACC_RDWR, H5P_DEFAULT);
    if (h5fid >=0 ) {
        h5gid = H5Gopen(h5fid,"/");
        res = H5Adelete(h5gid, "HDFEOSVersion_GLOSDS" ) ;
        res = H5Adelete(h5gid, "StructMetadata.0_GLOSDS" ) ;
        H5Gclose(h5gid);
        H5Fclose(h5fid);
    }

   /*
    *  The file may have images, e.g. a browse image.
    *
    *  Find them and move to HDF5.
    */
    res = H4toh5allloneimage(h45id, "/", NULL, H425_ALLATTRS, H425_PAL);

   /*
    *  The file may have Vdata tables.
    *
    *  Find them and move to HDF5.
    */

    nvd = H4toh5alllonevdata(h45id, "/", H425_ALLATTRS);
   /*
    *  The file may have SDS datasets.
    *
    *  Find them and move to HDF5.
    */

    res = H4toh5alllonesds(h45id, "/", NULL, H425_DIMSCALE, H425_ALLATTRS);
   /*
    *  ... there may be other objects as well, which need to be located
    *    and converted.
    */

    H4toh5close(h45id);
}

 

Appendix 4. Configuration and build

When compiling, it is necessary to compile/link with HDF4, HDF-EOS2.x, HDF5, HDF-EOS5.x, and libh4toh5.  The following gives a sketch of a Unix makefile. 
#
#  Need paths to everything!
#

# HDF4 and HDFEOS2.x
HDF=/usr/local/hdf
HE2=/usr/local/hdfeos

# HDF5 and HDFEOS5.x
HE5=/usr/local/hdfeos5
HDF5=/usr/local/hdf5

# The libh4toh5 installed
H45=/usr/local/h4toh5

CFLAGS = -I$(HDF5)/include -I$(HE2)/include -I$(HE5)/include -I$(HDF)/include -I$(H45)/include

LIBS=-L$(H45)/lib -L$(HDF5)/lib -L$(HDF)/lib -L$(HE2)/lib/sun5 -L$(HE5)/lib/sun5 -L$(H45LIB)/lib -lh4toh5 -lhdfeos -lhe5_hdfeos -lhdf5 -lmfhdf -ldf -lz -ljpeg -lGctp -lnsl -lm -lsocket -lnsl

heconvert: convert.o
        $(CC) -o heconvert convert.o $(LIBS)

convert.o: convert.c
For other build environments, such as Windows or Macintosh, equivalent include and library paths are needed.