HDF5 Tools

HDF5 Tool Interfaces

HDF5-related tools are available to assist the user in a variety of activities, including examining or managing HDF5 files, converting raw data between HDF5 and other special-purpose formats, moving data and files between the HDF4 and HDF5 formats, measuring HDF5 library performance, and managing HDF5 library and application compilation, installation and configuration. Unless otherwise specified below, these tools are distributed and installed with HDF5.

• User utilities:
  • h5dump -- Enables a user to examine the contents of an HDF5 file and dump those contents to an ASCII file.
  • h5ls -- Lists specified features of HDF5 file contents.
  • h5diff and ph5diff -- Compare two HDF5 files and report the differences.
  • h5repack -- Copies an HDF5 file to a new file with or without compression and/or chunking.
  • h5repart -- Repartitions a file, creating a family of files.
  • h5jam -- Adds a user block to the front of an HDF5 file .
  • h5unjam -- Splits an existing user block from an HDF5 file, placing it in a separate file.
  • h5copy -- Copies HDF5 objects from a file to a new file
  • h5mkgrp -- Creates a new HDF5 group in a file
  • h5stat -- Reports statistics regarding an HDF5 file and the objects in the file.
  • h5perf -- Measures Parallel HDF5 performance.
  • h5perf_serial -- Measures HDF5 serial performance.
    
    
• Configuration and library management utilities:
  • h5redeploy -- Updates HDF5 compiler tools after an HDF5 software installation in a new location.
  • h5cc and h5pcc -- Simplify the compilation of HDF5 programs written in C.
  • h5fc and h5pfc -- Simplify the compilation of HDF5 programs written in Fortran90.
  • h5c++ -- Simplifies the compilation of HDF5 programs written in C++.
    
    
• Java-based tools for HDF5 for viewing, manipulating, and generating HDF4 and HDF5 files:
  (Distributed separately; external link is http://www.hdfgroup.org/hdf-java-html/)
  • HDFview -- a browser that works with both HDF4 and HDF5 files and can be used to transfer data between the two formats
  • Java interfaces for both the HDF4 and HDF5 libraries
  • Other HDF4- and HDF5-related products
    
    
• Data conversion utilities:
  • h5import -- Imports data into an existing or new HDF5 file.
  • gif2h5 -- Converts a GIF file to an HDF5 file.
  • h52gif -- Converts images in an HDF5 file to a GIF file.
    
    
• HDF5/HDF4 conversion tools:
  (Distributed separately; external link is http://www.hdfgroup.org/h4toh5/.)
  • H4toH5 Conversion Library -- Provides APIs for use in tools that perform customized conversions of HDF4 files to HDF5 files.
  • h5toh4 -- Converts an HDF5 file to an HDF4 file.
  • h4toh5 -- Converts an HDF4 file to an HDF5 file.
    
    
• Other tools, including third-party and commercial utilities and applications:
  (Distributed separately; external link is http://www.hdfgroup.org/tools5.html.)

Tool Name: h5dump

Syntax:

h5dump [OPTIONS] file

Purpose:

Displays HDF5 file contents.

Description:

h5dump enables the user to examine the contents of an HDF5 file and dump those contents, in human readable form, to an ASCII file.

h5dump dumps HDF5 file content to standard output. It can display the contents of the entire HDF5 file or selected objects, which can be groups, datasets, a subset of a dataset, links, attributes, or datatypes.

The --header option displays object header information only.

Names are the absolute names of the objects. h5dump displays objects in the order same as the command order. If a name does not start with a slash, h5dump begins searching for the specified object starting at the root group.

If an object is hard linked with multiple names, h5dump displays the content of the object in the first occurrence. Only the link information is displayed in later occurrences.

h5dump assigns a name for any unnamed datatype in the form of #oid1:oid2, where oid1 and oid2 are the object identifiers assigned by the library. The unnamed types are displayed within the root group.

Datatypes are displayed with standard type names. For example, if a dataset is created with H5T_NATIVE_INT type and the standard type name for integer on that machine is H5T_STD_I32BE, h5dump displays H5T_STD_I32BE as the type of the dataset.

h5dump can also dump a subset of a dataset. This feature operates in much the same way as hyperslabs in HDF5; the parameters specified on the command line are passed to the function H5Sselect_hyperslab and the resulting selection is displayed.

The h5dump output is described in detail in the DDL for HDF5, the Data Description Language document.

Note: It is not permissible to specify multiple attributes, datasets, datatypes, groups, or soft links with one flag. For example, one may not issue the command
         WRONG:   h5dump -a /attr1 /attr2 foo.h5
to display both /attr1 and /attr2. One must issue the following command:
         CORRECT:   h5dump -a /attr1 -a /attr2 foo.h5

It's possible to select the file driver with which to open the HDF5 file by using the --filedriver (-f) command-line option. Acceptable values for the --filedriver option are: "sec2", "family", "split", and "multi". If the file driver flag isn't specified, then the file will be opened with each driver in turn and in the order specified above until one driver succeeds in opening the file.

One byte integer type data is displayed in decimal by default. When displayed in ASCII, a non-printable code is displayed in 3 octal digits preceeded by a back-slash unless there is a C language escape sequence for it. For example, CR and LF are printed as \r and \n. Though the NUL code is represented as \0 in C, it is printed as \000 to avoid ambiguity as illustrated in the following 1 byte char data (since this is not a string, embedded NUL is possible).

	141 142 143 000 060 061 062 012
	  a   b   c  \0   0   1   2  \n 

h5dump prints them as "abc\000012\n". But if h5dump prints NUL as \0, the output is "abc\0012\n" which is ambiguous.

XML Output:

With the --xml option, h5dump generates XML output. This output contains a complete description of the file, marked up in XML. The XML conforms to the HDF5 Document Type Definition (DTD) available at http://www.hdfgroup.org/DTDs/HDF5-File.dtd.

The XML output is suitable for use with other tools, including the HDF5 Java Tools.

Options and Parameters:

-h   or   --help	Print a usage message and exit.
-n   or   --contents	Print a list of the file contents and exit.
-B   or   --bootblock	Print the content of the boot block.
-H   or   --header	Print the header only; no data is displayed.
-A   or   --onlyattr	Print the header and value of attributes; data of datasets is not displayed.
-i   or   --object-ids	Print the object ids.
-r   or   --string	Print 1-byte integer datasets as ASCII.
-e   or   --escape	Escape non-printing characters.
-V   or   --version	Print version number and exit.
-a P  or  --attribute=P	Print the specified attribute.
-d P   or   --dataset=P	Print the specified dataset.
-y   or   --noindex	Do not print array indices with data.
-p   or   --properties	Print information regarding dataset properties, including filters, storage layout, fill value, and allocation time. The filter output lists any filters used with a dataset, including the type of filter, its name, and any filter parameters. The storage layout output specifies the dataset layout (chunked, compact, or contiguous), the size in bytes of the dataset on disk, and, if a compression filter is associated with the dataset, the compression ratio. The compression ratio is computed as (uncompressed size)/(compressed size). The fill value output includes the fill value datatype and value. The allocation time output displays the allocation time as specified with H5Pset_alloc_time.
-f D  or  --filedriver=D	Specify which driver to open the file with.
-g P   or   --group=P	Print the specified group and all members.
-l P  or  --soft-link=P	Print the value(s) of the specified soft link.
-o F   or   --output=F	Output raw data into file F.
-b B   or   --binary=B	Output dataset to a binary file using the datatype specified by B. B must have one of the following values:      LE        Little-endian      BE        Big-endian      MEMORY    Memory datatype      FILE      File datatype Recommended usage is with the -d and -o options.
-t P   or   --datatype=P	Print the specified named datatype.
-w N   or   --width=N	Set the number of columns of output.
-m T   or   --format=T	Set the floating point output format. T is a string defining the floating point format, e.g., '%.3f'.
-q Q   or   --sort_by=Q	Sort groups and attributes by the specified index type, Q. Valid values of Q are as follows:      name              Alpha-numeric index by name (Default)      creation_order    Index by creation order
-z Z   or   --sort_order=Z	Sort groups and attributes in the specified order, Z. Valid values of Z are as follows:      ascending    Sort in ascending order (Default)      descending   Sort in descending order
-x   or   --xml	Output XML using XML schema (default) instead of DDL.
-u   or   --use-dtd	Output XML using XML DTD instead of DDL.
-D U   or   --xml-dtd=U	In XML output, refer to the DTD or schema at U instead of the default schema/DTD.
-X S   or   --xml-dns=S	In XML output, (XML Schema) use qualified names in the XML:       ":": no namespace, default: "hdf5:"
-s L   or   --start=L	Offset of start of subsetting selection. Default: the beginning of the dataset.
-S L   or   --stride=L	Hyperslab stride. Default: 1 in all dimensions.
-c L   or   --count=L	Number of blocks to include in the selection.
-k L   or   --block=L	Size of block in hyperslab. Default: 1 in all dimensions.
--	Indicates that the following argument is not an option. E.g., to dump a file called `-f', use h5dump -- -f.     (This option is necessary only when the name of the file to be examined starts with a dash (-), which could confuse the tool’s command-line parser.)
file	The file to be examined.

The option parameters listed above are defined as follows:

D	which file driver to use in opening the file. Acceptable values are "sec2", "family", "split", and "multi". Without the file driver flag, the file will be opened with each driver in turn and in the order specified above until one driver succeeds in opening the file.
P	The full path from the root group to the object
F	A filename
N	An integer greater than 1
L	A list of integers, the number of which is equal to the number of dimensions in the dataspace being queried
U	A URI (as defined in [IETF RFC 2396], updated by [IETF RFC 2732]) that refers to the DTD to be used to validate the XML
B	The form of binary output:     MEMORY    for a memory type     FILE      for the file type     LE or BE   for pre-existing little- or big-endian types

Subsetting parameters can also be expressed in a convenient compact form, as follows:
         --dataset="/foo/mydataset[START;STRIDE;COUNT;BLOCK]"
All of the semicolons (;) are required, even when a parameter value is not specified. When not specified, default parameter values are used.

Examples:

Dump the group /GroupFoo/GroupBar in the file quux.h5:      h5dump -g /GroupFoo/GroupBar quux.h5 Dump the dataset Fnord, which is in the group /GroupFoo/GroupBar in the file quux.h5:      h5dump -d /GroupFoo/GroupBar/Fnord quux.h5 Dump the attribute metadata of the dataset Fnord, which is in the group /GroupFoo/GroupBar in the file quux.h5:      h5dump -a /GroupFoo/GroupBar/Fnord/metadata quux.h5 Dump the attribute metadata which is an attribute of the root group in the file quux.h5:      h5dump -a /metadata quux.h5 Produce an XML listing of the file bobo.h5, saving the listing in the file bobo.h5.xml:      h5dump --xml bobo.h5 > bobo.h5.xml Dump a subset of the dataset /GroupFoo/databar/ in the file quux.h5:      h5dump -d /GroupFoo/databar --start="1,1" --stride="2,3"          --count="3,19" --block="1,1" quux.h5 The same example, using the short form to specify the subsetting parameters:      h5dump -d "/GroupFoo/databar[1,1;2,3;3,19;1,1]" quux.h5 Dump a binary copy of the dataset /GroupD/FreshData/ in the file quux.h5, with data written in little-endian form, to the output file FreshDataD.bin:      h5dump -d "/GroupD/FreshData" -b LE          -o "FreshDataD.bin" quux.h5

Current Status:

The current version of h5dump displays the following information:

• Group
  • group attribute (see Attribute)
  • group member
• Dataset
  • dataset attribute (see Attribute)
  • dataset type (see Datatype)
  • dataset space (see Dataspace)
  • dataset data
• Attribute
  • attribute type (see Datatype)
  • attribute space (see Dataspace)
  • attribute data
• Datatype
  • integer type
    - H5T_STD_I8BE, H5T_STD_I8LE, H5T_STD_I16BE, ...
  • floating point type
    - H5T_IEEE_F32BE, H5T_IEEE_F32LE, H5T_IEEE_F64BE, ...
  • string type
  • compound type
    - named, unnamed and transient compound type
    - integer, floating or string type member
  • opaque types
  • reference type
    - object references
    - data regions
  • enum type
  • variable-length datatypes
    - atomic types only
    - scalar or single dimensional array of variable-length types supported
• Dataspace
  • scalar and simple space
• Soft link
• Hard link
• Loop detection

See Also:

HDF5 Data Description Language syntax at DDL for HDF5 HDF5 XML Schema at http://www.hdfgroup.org/DTDs/HDF5-File.xsd HDF5 XML information at http://www.hdfgroup.org/HDF5/XML/

History:

Release	Command Line Tool
1.6.5	The following options added in this release: -n or --contents -e or --escape -y or --noindex -p or --properties -b or --binary
1.8.0	The following options added in this release: -q or --sort_by -z or --sort_order
1.8.1	Compression ratio added to -p option in this release.

Tool Name: h5ls

Syntax:

h5ls [OPTIONS] file [OBJECTS...]

Purpose:

Prints information about a file or dataset.

Description:

h5ls prints selected information about file objects in the specified format.

Options and Parameters:

-h  or  -?  or   --help	Print a usage message and exit.
-a   or   --address	Print addresses for raw data.
-d   or   --data	Print the values of datasets.
-e   or   --errors	Show all HDF5 error reporting.
-f   or   --full	Print full path names instead of base names.
-g   or   --group	Show information about a group, not its contents.
-l   or   --label	Label members of compound datasets.
-r   or  --recursive	List all groups recursively, avoiding cycles.
-s   or   --string	Print 1-bytes integer datasets as ASCII.
-S   or   --simple	Use a machine-readable output format.
-wN   or   --width=N	Set the number of columns of output.
-v   or   --verbose	Generate more verbose output.
-V   or   --version	Print version number and exit.
-x   or   --hexdump	Show raw data in hexadecimal format.
file	The file name may include a printf(3C) integer format such as %%05d to open a file family.
objects	Each object consists of an HDF5 file name optionally followed by a slash and an object name within the file (if no object is specified within the file then the contents of the root group are displayed). The file name may include a printf(3C) integer format such as "%05d" to open a file family.

Tool Name: h5diff    

Syntax:

h5diff [OPTIONS] file1 file2 [object1 [object2 ] ]

ph5diff [OPTIONS] file1 file2 [object1 [object2 ] ]

Purpose:

Compare two HDF5 files and report the differences.

Description:

h5diff and ph5diff are command line tools that compare two HDF5 files, file1 and file2, and report the differences between them. h5diff is for serial use while ph5diff is for use in parallel environments.

Optionally, h5diff and ph5diff will compare two objects within these files. If only one object, object1, is specified, h5diff will compare object1 in file1 with object1 in file2. If two objects, object1 and object2, are specified, h5diff will compare object1 in file1 with object2 in file2.

object1 and object2 can be groups, datasets, named datatypes, or links and must be expressed as absolute paths from the respective file’s root group.

• If these objects are groups, the name string is compared.
• If these objects are datasets, array rank and dimensions, datatypes, and data values are compared.
• If these objects are named datatypes, the comparison is based on the return value of H5Tequal.
• If these objects are links, the link value is compared.
(The link value is the string that is often referred to as the name of the linked object.)

h5diff and ph5diff have the following output modes:

Normal mode	Prints the number of differences found and where they occurred.
Report mode (-r)	Prints the above plus the differences.
Verbose mode (-v)	Prints all of the above plus a list of objects and warnings.
Quiet mode (-q)	Prints no output. (h5diff always returns an exit code of 1 when differences are found.)

h5diff and NaNs:
h5diff detects when a value in a dataset is a NaN (a "not a number" value), but does not differentiate among various types of NaNs. Thus, when one NaN is compared with another NaN, h5diff treats them as equal; when a NaN is compared with a valid number, h5diff treats them as not equal.

Difference between h5diff and ph5diff:
With the following exception, h5diff and ph5diff behave identically. With ph5diff, the comparison of objects is shared across multiple processors, with the comparison of each pair of objects assigned to a single processor. I.e., the comparison of a single object, even a very large dataset, in each file is not shared.

Options and Parameters:

-h   or  --help	Print help message.
-V   or  --version	Print version number and exit.
-r   or  --report	Report mode — Print the differences.
-v   or  --verbose	Verbose mode — Print the differences, a list of objects, and warnings.
-q   or  --quiet	Quiet mode — Do not print output.
-n count   or --count=count	Print difference up to count differences, then stop. count must be a positive integer.
-d delta   or --delta=delta	Print only differences that are greater than the limit delta. delta must be a positive number. The comparison criterion is whether the absolute value of the difference of two corresponding values is greater than delta (e.g.,  |a–b| > delta, where a is a value in file1 and b is a value in file2).
-p relative   or --relative=relative	Print only differences that are greater than a relative error. relative must be a positive number. The comparison criterion is whether the absolute value of the difference 1 and the ratio of two corresponding values is greater than relative (e.g., |1–(b/a)| > relative where a is a value in file1 and b is a value in file2).
file1    file2	The HDF5 files to be compared.
object1    object2	Specific object(s) within the files to be compared.

Returns:

Returns 1 (one) if differences are found, 0 (zero) if no differences are found, and 2 (two) if the call fails.

Examples:

The following h5diff call compares the object /a/b in file1 with the object /a/c in file2:
    h5diff file1 file2 /a/b /a/c

This h5diff call compares the object /a/b in file1 with the same object in file2:
    h5diff file1 file2 /a/b

And this h5diff call compares all objects in both files:
    h5diff file1 file2

History:

Release	Change
1.6.0	h5diff introduced in this release.
1.8.0	ph5diff introduced in this release. h5diff command line syntax changed in this release.
1.8.2 and   1.6.8	Return value on failure changed in this release.