H5FDsubfiling.h File Reference

Data Structures
struct	H5FD_subfiling_params_t
	Subfiling parameter structure that is shared between the H5FD_SUBFILING and H5FD_IOC drivers. More...
struct	H5FD_subfiling_config_t
	Configuration structure for H5Pset_fapl_subfiling() / H5Pget_fapl_subfiling() More...

Macros
#define	H5FD_SUBFILING   (H5FDperform_init(H5FD_subfiling_init))
#define	H5FD_SUBFILING_NAME   "subfiling"
#define	H5FD_SUBFILING_CURR_FAPL_VERSION   1
#define	H5FD_SUBFILING_FAPL_MAGIC   0xFED01331
#define	H5FD_SUBFILING_DEFAULT_STRIPE_SIZE   (32 * 1024 * 1024)
#define	H5FD_SUBFILING_DEFAULT_STRIPE_COUNT   -1
#define	H5FD_SUBFILING_FILENAME_TEMPLATE   "%s.subfile_%" PRIu64 "_%0*d_of_%d"
#define	H5FD_SUBFILING_CONFIG_FILENAME_TEMPLATE   "%s.subfile_%" PRIu64 ".config"
#define	H5FD_SUBFILING_STRIPE_SIZE   "H5FD_SUBFILING_STRIPE_SIZE"
#define	H5FD_SUBFILING_IOC_PER_NODE   "H5FD_SUBFILING_IOC_PER_NODE"
#define	H5FD_SUBFILING_IOC_SELECTION_CRITERIA   "H5FD_SUBFILING_IOC_SELECTION_CRITERIA"
#define	H5FD_SUBFILING_SUBFILE_PREFIX   "H5FD_SUBFILING_SUBFILE_PREFIX"
#define	H5FD_SUBFILING_CONFIG_FILE_PREFIX   "H5FD_SUBFILING_CONFIG_FILE_PREFIX"

Enumerations
enum	H5FD_subfiling_ioc_select_t {   SELECT_IOC_ONE_PER_NODE = 0, SELECT_IOC_EVERY_NTH_RANK, SELECT_IOC_WITH_CONFIG, SELECT_IOC_TOTAL,   ioc_selection_options }

Functions
H5_DLL hid_t	H5FD_subfiling_init (void)
	Internal routine to initialize H5FD_SUBFILING driver. Not meant to be called directly by an HDF5 application. More...
H5_DLL herr_t	H5Pset_fapl_subfiling (hid_t fapl_id, const H5FD_subfiling_config_t *vfd_config)
	Modifies the specified File Access Property List to use the H5FD_SUBFILING driver. More...
H5_DLL herr_t	H5Pget_fapl_subfiling (hid_t fapl_id, H5FD_subfiling_config_t *config_out)
	Queries a File Access Property List for H5FD_SUBFILING file driver properties. More...

Macro Definition Documentation

H5FD_SUBFILING

#define H5FD_SUBFILING   (H5FDperform_init(H5FD_subfiling_init))

Macro that returns the identifier for the H5FD_SUBFILING driver. Returns a file driver identifier if successful; otherwise returns H5I_INVALID_HID.

H5FD_SUBFILING_CONFIG_FILE_PREFIX

#define H5FD_SUBFILING_CONFIG_FILE_PREFIX   "H5FD_SUBFILING_CONFIG_FILE_PREFIX"

Macro for name of the environment variable that specifies a prefix to apply to the subfiling configuration filename. Useful for cases where the application wants to place the configuration file in a different directory than the default of putting it alongside the generated subfiles. For example, when writing to node-local storage one may wish to place the configuration file on a scratch file system readable by all nodes, while the subfiles are initially written to the node-local storage.

The value set for this environment variable is interpreted as a pathname that must already exist.

NOTE: As this prefix string will be encoded in the driver info message that gets written to the file, there is an upper limit of about ~900 single-byte characters for this string, though possibly less due to other information the driver may encode. Avoid long prefix names where possible.

H5FD_SUBFILING_CONFIG_FILENAME_TEMPLATE

#define H5FD_SUBFILING_CONFIG_FILENAME_TEMPLATE   "%s.subfile_%" PRIu64 ".config"

The basic printf-style template for a H5FD_SUBFILING driver configuration filename. The format specifiers correspond to:

%s

base filename, e.g. "file.h5"

%PRIu64

file inode, e.g. 11273556

yielding a filename such as:

file.h5.subfile_11273556.config

H5FD_SUBFILING_CURR_FAPL_VERSION

#define H5FD_SUBFILING_CURR_FAPL_VERSION   1

The version number of the H5FD_subfiling_config_t configuration structure for the H5FD_SUBFILING driver

H5FD_SUBFILING_DEFAULT_STRIPE_COUNT

#define H5FD_SUBFILING_DEFAULT_STRIPE_COUNT   -1

Macro for the default Subfiling stripe count value. The default is currently to use one subfile per node.

H5FD_SUBFILING_DEFAULT_STRIPE_SIZE

#define H5FD_SUBFILING_DEFAULT_STRIPE_SIZE   (32 * 1024 * 1024)

The default stripe size (in bytes) for data stripes in subfiles

H5FD_SUBFILING_FAPL_MAGIC

#define H5FD_SUBFILING_FAPL_MAGIC   0xFED01331

Unique number used to distinguish the H5FD_SUBFILING driver from other HDF5 file drivers

H5FD_SUBFILING_FILENAME_TEMPLATE

#define H5FD_SUBFILING_FILENAME_TEMPLATE   "%s.subfile_%" PRIu64 "_%0*d_of_%d"

The basic printf-style template for a H5FD_SUBFILING driver subfile filename. The format specifiers correspond to:

%s

base filename, e.g. "file.h5"

%PRIu64

file inode, e.g. 11273556

%0*d

number (starting at 1) signifying the Nth (out of total number of subfiles) subfile. Zero-padded according to the number of digits in the number of subfiles (calculated by log10(num_subfiles) + 1)

%d

number of subfiles

yielding filenames such as:

file.h5.subfile_11273556_01_of_10
file.h5.subfile_11273556_02_of_10
file.h5.subfile_11273556_10_of_10

H5FD_SUBFILING_IOC_PER_NODE

#define H5FD_SUBFILING_IOC_PER_NODE   "H5FD_SUBFILING_IOC_PER_NODE"

Macro for name of the environment variable that specifies the number of MPI ranks per node to use as I/O concentrators

The value set for this environment variable is interpreted as a long value and must be > 0.

H5FD_SUBFILING_IOC_SELECTION_CRITERIA

#define H5FD_SUBFILING_IOC_SELECTION_CRITERIA   "H5FD_SUBFILING_IOC_SELECTION_CRITERIA"

Macro for name of the environment variable that provides information for selection MPI ranks as I/O concentrators

The value set for this environment variable is interpreted differently, depending on the IOC selection method chosen.

For SELECT_IOC_ONE_PER_NODE, this value is ignored.

For SELECT_IOC_EVERY_NTH_RANK, this value is interpreted as a long value and must be > 0. The value will correspond to the N value when selecting every N-th MPI rank as an I/O concentrator.

For SELECT_IOC_WITH_CONFIG, this value is ignored as that particular IOC selection method is not currently supported.

For SELECT_IOC_TOTAL, this value is interpreted as a long value and must be > 0. The value will correspond to the total number of I/O concentrators to be used.

H5FD_SUBFILING_NAME

#define H5FD_SUBFILING_NAME   "subfiling"

The canonical name for the H5FD_SUBFILING driver

H5FD_SUBFILING_STRIPE_SIZE

#define H5FD_SUBFILING_STRIPE_SIZE   "H5FD_SUBFILING_STRIPE_SIZE"

Macro for name of the environment variable that specifies the size (in bytes) for data stripes in subfiles

The value set for this environment variable is interpreted as a long long value and must be > 0.

H5FD_SUBFILING_SUBFILE_PREFIX

#define H5FD_SUBFILING_SUBFILE_PREFIX   "H5FD_SUBFILING_SUBFILE_PREFIX"

Macro for name of the environment variable that specifies a prefix to apply to the filenames generated for subfiles

The value set for this environment variable is interpreted as a pathname.

Enumeration Type Documentation

H5FD_subfiling_ioc_select_t

enum H5FD_subfiling_ioc_select_t

This enum defines the various constants to allow different allocations of MPI ranks as I/O concentrators.

Enumerator
SELECT_IOC_ONE_PER_NODE	Default selection method. One MPI rank per node is used as an I/O concentrator. If this selection method is used, the number of I/O concentrators per node can be adjusted with the H5FD_SUBFILING_IOC_PER_NODE environment variable.
SELECT_IOC_EVERY_NTH_RANK	Starting with MPI rank 0, a stride of 'N' is applied to the MPI rank values to determine the next I/O concentrator. The H5FD_SUBFILING_IOC_SELECTION_CRITERIA environment variable must be set to the value desired for 'N'.
SELECT_IOC_WITH_CONFIG	Currently unsupported. Use a configuration file to determine the mapping from MPI ranks to I/O concentrators. The H5FD_SUBFILING_IOC_SELECTION_CRITERIA environment variable must be set to the path to the configuration file.
SELECT_IOC_TOTAL	Specifies that a total of 'N' I/O concentrators should be used. Starting with MPI rank 0, a stride of 'MPI comm size' / 'N' is applied to the MPI rank values to determine the next I/O concentrator. The H5FD_SUBFILING_IOC_SELECTION_CRITERIA environment variable must be set to the value desired for 'N'.
ioc_selection_options	Unused. Sentinel value

Function Documentation

H5FD_subfiling_init()

H5_DLL hid_t H5FD_subfiling_init

(

void

)

Internal routine to initialize H5FD_SUBFILING driver. Not meant to be called directly by an HDF5 application.