Programming Model For Fletcher32 Checksum
By Raymond Lu
Feb 14, 2003
The checksum filter is an HDF5 filter that computes a Fletcher32 checksum for HDF5 datasets. At this time, the checksum is only available for chunked datasets, and only computes a checksum on the raw data. No checksum is computed for non-chunked (“contiguous”) datasets, nor is on metadata. For more background on error detecting codes in HDF5, please see the RFC documents at http://hdf.ncsa.uiuc.edu/RFC/H5checksum/.
There are two ways to set up Fletcher32 checksum filter for dataset creation property list:
· Through the general filter setting function H5Pset_filter
· By using the function H5Pset_fletcher32
herr_t
H5Pset_filter(hid_t dcpl, H5Z_filter_t filter, unsigned int flags, size_t cd_nelmts, const unsigned int cd_values[])
By passing the value H5Z_FILTER_FLETCHER32
to parameter filter, the Fletcher32 checksum is
enabled. The dcpl has to
be a dataset creation property list for chunked dataset. No additional data is needed for cd_values.
herr_t
H5Pset_fletcher32(hid_t dcpl)
Again, the dcpl has to be a dataset creation property list for chunked dataset.
The general filter functions H5Pget_nfilters and H5Pget_filter can be used to query whether the Fletcher32 checksum is enabled. Please refer to the HDF5 User’s Guide and Reference Manual on how to use them.
Sometimes, it is desirable to skip checksumming during data reading in order to get good performance. There is a function H5Pset_edc_check to offer this option:
herr_t
H5Pset_edc_check(hid_t dxpl, H5P_EDC_t check)
By passing H5P_ENABLE_EDC or H5P_DISALBE_EDC to check, this function decides whether to enable an error-detection code (EDC) for data reading process. This error-detecting algorithm is whichever user chooses earlier (At this moment, there is only Fletcher32 checksum available). This function cannot disable or enable error-detection for data writing process. Please note the dxpl has to be a dataset transfer property list. The default configuration is EDC enabled.
To check whether an EDC is enabled for dataset reading, use the function H5Pget_edc_check:
H5P_EDC_t
H5Pget_edc_check(hid_t dxpl)
The return value can be H5P_ENABLE_EDC or H5P_DISABLE_EDC which indicate whether EDC enabled or disabled for dataset reading. The dxpl has to be a dataset transfer property list.
When Fletcher32 checksum is enabled and there is error detected by this checksum during data reading, the default behavior of the library is returning an error message through the error stack. A general filter function, H5Pset_filter_callback, can give application program options to control checksum failure:
herr_t
H5Pset_filter_callback(hid_t dxpl, H5Z_filter_func_t
func,
void* op_data)
During data reading, if there is checksum failure, the user’s callback function
func is called and user’s op_data is passed back into this callback function.
The dxpl has to be a dataset transfer property list.
This callback function is defined as
typedef H5Z_cb_return_t
(H5Z_filter_func_t)(H5Z_filter_t filter,
void* buf, size_t
buf_size, void* op_data)
Receiving the type of filter through filter, data through buf, data size through buf_size, and user’s operation data back through op_data, user can decide what to do when there is a filter failure. The return value of this function, H5Z_CB_FAIL and H5Z_CB_CONT, tell the library whether to fail the process or continue reading ignoring the failure.
Below is a programming example of how to enable Fletcher32 checksum for dataset creation and how to disable an EDC for dataset reading:
/* Enable
Fletcher32 checksum for dataset creation property list */
if((dcpl =
H5Pcreate(H5P_DATASET_CREATE))<0) goto error;
if (H5Pset_chunk (dc, 2,
chunk_size)<0) goto error;
if (H5Pset_filter
(dc,H5Z_FILTER_FLETCHER32,0,0,NULL)<0)
goto error;
/* Create the dataset */
if ((dataset = H5Dcreate(file_id, name, H5T_NATIVE_INT, space_id, dcpl))<0) goto error;
/* Write
the data */
if (H5Dwrite(dataset, H5T_NATIVE_INT,
H5S_ALL, H5S_ALL, H5P_DEFAULT, points)<0) goto error;
/*
Disable Fletcher32 checksum to speed up reading */
if((dxpl
= H5Pcreate(H5P_DATASET_XFER))<0) goto error;
if(H5Pset_edc_check(dxpl, H5Z_DISABLE_EDC)<0) goto error;
/* Read
data without Fletcher32 checksum */
if (H5Dread (dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, dxpl,
check)<0) goto error;
The following program example demonstrates how to set up and use the callback function. In the callback function, it chooses to continue to read data even if the checksum fails. The H5Dread will fail and return error stack if there is other failure.
H5Z_cb_return_t filter_cb_cont(H5Z_filter_t filter, void* UNUSED
buf,
size_t UNUSED buf_size, void* UNUSED op_data)
{
/* If
failure is from checksum, do something and continue
* the reading; otherwise, let the program
fail. */
if(H5Z_FILTER_FLETCHER32==filter)
{
:
:
return H5Z_CB_CONT;
} else
return
H5Z_CB_FAIL;
}
herr_t
test_fletcher(hid_t file)
{
:
:
/* Enable
Fletcher32 checksum for dataset creation property list */
if((dcpl =
H5Pcreate(H5P_DATASET_CREATE))<0) goto error;
if (H5Pset_chunk (dc, 2,
chunk_size)<0) goto error;
if (H5Pset_fletcher32 (dc)<0)
goto error;
/* Create the dataset */
if ((dataset = H5Dcreate(file_id, name, H5T_NATIVE_INT, space_id, dcpl))<0) goto error;
/* Write
the data */
if (H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, points)<0) goto error;
/* Set
callback function,let read continue even if checksum fails*/
if((dxpl
= H5Pcreate(H5P_DATASET_XFER))<0) goto error;
if(H5Pset_filter_callback(dxpl,
filter_cb_cont, NULL)<0)
goto error;
/* Read
data */
if (H5Dread (dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, dxpl, check)<0) goto error;
:
:
}
In the program below, a flag variable is passed into the filter callback function. There, it is set and returned back to the program. The program then decides whatever it wants to do when this happens.
H5Z_cb_return_t filter_cb_cont(H5Z_filter_t filter, void* UNUSED
buf,
size_t UNUSED buf_size, void* op_data)
{
/* If
failure is from checksum, continue the reading and set
* flag to indicate failure; otherwise, let the program fail. */
if(H5Z_FILTER_FLETCHER32==filter)
{
*(int*)op_data
= FAIL;
return H5Z_CB_CONT;
} else
return
H5Z_CB_FAIL;
}
herr_t
test_fletcher(hid_t file)
{
herr_t
flag = SUCCEED;
:
:
/* Enable
Fletcher32 checksum for dataset creation property list */
if((dcpl =
H5Pcreate(H5P_DATASET_CREATE))<0) goto error;
if (H5Pset_chunk (dc, 2,
chunk_size)<0) goto error;
if (H5Pset_fletcher32 (dc)<0)
goto error;
/* Create the dataset */
if ((dataset = H5Dcreate(file_id, name, H5T_NATIVE_INT, space_id, dcpl))<0) goto error;
/* Write
the data */
if
(H5Dwrite(dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT, points)<0)
goto error;
/* Set
callback function,let read continue even if checksum fails*/
if((dxpl
= H5Pcreate(H5P_DATASET_XFER))<0) goto error;
if(H5Pset_filter_callback(dxpl,
filter_cb_cont, &flag)<0)
goto error;
/* Read
data */
if (H5Dread (dataset, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, dxpl, check)<0) goto error;
/* Decide what to do if checksum fails */
if(flag == FAIL)
:
:
}