HDF5 File Remains In Memory After Closing (Linux) ================================================= Question: I am having performance problems. The memory runs down to where it requires swapping, and then the system is very slow. Using "top" I can see that the memory does not get released after an HDF5 file is closed. It stays in memory until the file is deleted. Answer: The memory usage shown by "top" means total memory used by the system, both kernel and users. The Linux OS, contrary to many other OSs, does not impose an upper limit on Kernel memory. If the kernel needs more memory, it grabs as much as there is physical memory. Output to disk is buffered in kernel memory first, before making its way slowly to disk. If a program is output intensive, it can quickly use up all physical memory. At that point, the whole system is memory starved. Not much gets done until the output data is written to the disks and frees up memory. Therefore, a Linux system can become very inefficient by just one write-intensive application. This problem is not limited to HDF5 programs. Any write-intensive program can exhibit the same phenomenon. For example, keep "top" running in one window and do the following in another window. You will see the same behavior, assuming you have less than 2GB of memory. If you have more memory, increase the value of count=200 to exceed it. % dd if=/dev/zero of=junk bs=10MB count=200 #generate a 2000MB file % # by now top will show not much free memory left. % rm junk You will then see a big jump of free memory because the data in the dirty buffers is no longer valid and the memory is free for other use. Question: I believe it's a system problem, not the HDF5 library. I'm just looking for a manual flush that would alleviate my problem. I appreciate the help, and If you all do find a way to manually flush the buffers, let me know. Answer: "Flushing" data means an application is very paranoid of the data integrity and is willing to initiate a real disk write and WAIT until it is finished. It is a requirement of data integrity at the cost of response speed. You can keep telling the system to flush the data out but the data can only move at the limit of the I/O channel. If you want your memory to be freed up sooner, you would have to employ one or more of the following: 1. faster disks and faster I/O channel; 2. more memory; 3. more disks and use stripping;