Space can be allocated for dataset elements when a dataset is created, or when the dimensions of an existing dataset with unlimited maximum dimensions are extended. When using a non-MPI VFD, the space allocation can be deferred until elements are actually written to the dataset’s storage, but because there is no central coordinating agent for allocating space in HDF5 files, all space for storing data elements in dataset must be allocated collectively in parallel applications, so that the space allocation information on all processes is kept synchronized. Additionally, metadata in the file is also created or modified during these dataset operations, which also requires them to be collective operations.
Allocating space involves different algorithms for each type of dataset storage:
- Contiguous Storage: (H5D_contig_alloc() in src/H5Dcontig.c)
- Dataset elements are stored in one large, contiguous block in the HDF5 file, so all processes collectively call the space allocation routine within the HDF5 library and retrieve the same address for storing the dataset elements (because the space allocation information is implicitly synchronized on all processes).
- Chunked Storage:
- Dataset elements are stored in one or more blocks within the HDF5 file, called "chunks". Access to the chunks for a dataset is through an index which maps the coordinates of the elements in the chunk to an address of the chunk in the file. Because independent access to data elements is possible later, all the chunks for a dataset must be allocated and inserted into the index at this point.
- <create the index data structure> (H5D_chunk_create() in src/H5Dchunk.c)
- <allocate space for all chunks and insert them in the chunk index> (H5D_chunk_allocate() in src/H5Dchunk.c)
- <loop over all chunks in the dataset>
- <collectively insert chunk in the index, which allocates space for the chunk in the file>
- <if there's a fill value for the dataset, write it out from process 0>
- <if there's a fill value for the dataset, MPI_Barrier() to make certain all chunks have been initialized in the file>
- Compact Storage:
- Dataset elements are stored in the object header for the dataset and space was allocated for them when the dataset's object header was created, so no MPI actions are taken here.
Additionally, when space is allocated for dataset elements, the application may request writing a fill value to each element, which involves different algorithms for each type of dataset storage:
- Contiguous Storage: (H5D_contig_fill() in src/H5Dcontig.c)
- <initialize block of elements to fill value>
- <loop over all the elements in the dataset>
- <write fill value block to dataset from process 0>
- MPI_Barrier() to make certain all elements have been initialized in the file
- Chunked Storage:
- Writing fill values occurred when chunks were allocated, so no action here
- Compact Storage:
- Fill values are written to memory buffer, on each process, but no actual MPI actions are taken here.