Files

Tracking Free Space in HDF5 Files

Problem: You sometimes delete objects in HDF5 files and don't have new content to use the free space, but would like to reuse it in the future.

Solution: At file creation time, set the file space management strategy and persistence of free space tracking information via H5Pset_file_space_strategy().

Note: This feature is only supported in HDF5 1.10.1+.

     {
         __label__ fail_fcpl, fail_fapl, fail_file;
         hid_t fcpl, fapl, file;
  
         if ((fcpl = H5Pcreate(H5P_FILE_CREATE)) < 0) {
             ret_val = EXIT_FAILURE;
             goto fail_fcpl;
         }
 #if H5_VERSION_GE(1, 10, 1)
         if (H5Pset_file_space_strategy(fcpl, H5F_FSPACE_STRATEGY_FSM_AGGR, 1, 4096) < 0) {
 #else
 #error HDF5 1.10.1+ required
 #endif
             ret_val = EXIT_FAILURE;
             goto fail_fapl;
         }
  
         if ((fapl = H5Pcreate(H5P_FILE_ACCESS)) < 0) {
             ret_val = EXIT_FAILURE;
             goto fail_fapl;
         }
 #if H5_VERSION_GE(1, 10, 1)
         if (H5Pset_libver_bounds(fapl, H5F_LIBVER_V110, H5F_LIBVER_LATEST) < 0) {
 #else
 #error HDF5 1.10.x+ required
 #endif
             ret_val = EXIT_FAILURE;
             goto fail_file;
         }
  
         if ((file = H5Fcreate("free_space.h5", H5F_ACC_TRUNC, fcpl, fapl)) < 0) {
             ret_val = EXIT_FAILURE;
             goto fail_file;
         }
         H5Fclose(file);
  
 fail_file:
         H5Pclose(fapl);
 fail_fapl:
         H5Pclose(fcpl);
 fail_fcpl:;
     }

Discussion

Free space tracking is supported only in HDF5 versions 1.10.x and higher. This has implications for the accessibility of your HDF5 files and should be considered carefully. If compatibility with previous versions of HDF5 must be maintained, space reclamation via h5repack might be an option.
The file space strategy H5F_FSPACE_STRATEGY_FSM_AGGR is not the only option that supports free-space tracking. H5F_FSPACE_STRATEGY_PAGE is another option, which adds paged allocation and is used most effectively with page buffering.
For an in-depth account of HDF5 file space management, paged-allocation, and page buffering, see the following documents:

See Also: See Maintaining Compatibility with other HDF5 Library Versions for HDF5 compatibility implications.

Removing Unused Space from HDF5 Files

Problem: Based on estimates or h5stat output you know that a large portion of an HDF5 file consists of free or unaccounted space, and you would like to remove it.

Creating an HDF5 File User Block

Problem: You would like to include certain ancillary, non-HDF5 content in an HDF5 file such that it can be accessed without the HDF5 library.

Solution: Use a file creation property list in which the user block size is set via H5Pset_userblock(). In the example below, we create an 8 MiB user block.
60 {

61 __label__ fail_fcpl, fail_file;

62 hid_t fcpl, file;

63

64 if ((fcpl = H5Pcreate(H5P_FILE_CREATE)) < 0) {

65 ret_val = EXIT_FAILURE;

66 goto fail_fcpl;

67 }

68 if (H5Pset_userblock(fcpl, 8192 * 1024) < 0) {

69 ret_val = EXIT_FAILURE;

70 goto fail_file;

71 }

72 if ((file = H5Fcreate("userblock.h5", H5F_ACC_TRUNC, fcpl, H5P_DEFAULT)) < 0) {

73 ret_val = EXIT_FAILURE;

74 goto fail_file;

75 }

76 H5Fclose(file);

77

78 fail_file:

79 H5Pclose(fcpl);

80 fail_fcpl:;

81 }

Discussion: The user block begins at offset 0 and must be at least 512 bytes and a power of 2. The HDF5 library ignores any content between the beginning of the file and the end of the user block.
You can add or strip a user block to/from an existing HDF5 file with the h5jam/h5unjam tool, respectively.

Warning: If you try to embed content into the user block for use by other applications, pay close attention to how they handle space beyond the last used byte in the user block or the user block in general. In the worst case, applications might try to truncate the rest of the file and destroy the HDF5 portion of the file.

See Also: References to related recipes