I/O Backends

By default, on Windows, a buffered strategy is chosen, whereas Linux and Mac OS use unbuffered memory-mapped I/O. You can pass an instance of IOBackend as the io_backend parameter of load() to use a different backend. This allows you to override the default backend choice, or set parameters for the backend.

Note that some file formats can’t support different I/O backends, such as HDF5 or SER, because they are implemented using third-party reading libraries which perform their own I/O.

Available I/O backends

BufferedBackend

class libertem.io.dataset.base.BufferedBackend(max_buffer_size=16777216)[source]

I/O backend using a buffered reading strategy. Useful for slower media like HDDs, where seeks cause performance drops. Used by default on Windows.

This does not perform optimally on SSDs under all circumstances, for better best-case performance, try using MMapBackend instead.

Parameters:: max_buffer_size (int) – Maximum buffer size, in bytes. This is passed to the tileshape negotiation to select the right depth.

MMapBackend

class libertem.io.dataset.base.MMapBackend(enable_readahead_hints=False)[source]

I/O backend using memory mapped files. Used by default on non-Windows systems.

Parameters:: enable_readahead_hints (bool) – Linux only. Try to influence readahead behavior (experimental).

DirectBackend

class libertem.io.dataset.base.DirectBackend(max_buffer_size=16777216)[source]

I/O backend using a direct I/O reading strategy. This currently works on Linux and Windows, Mac OS X is not yet supported.

Use this backend if your data is much larger than your RAM, and you have fast enough storage (NVMe RAID, for example). In these cases, the MMapBackend or BufferedBackend is not efficient, as the system is constantly under memory pressure. In that case, this backend can perform much better.

Parameters:: max_buffer_size (int) – Maximum buffer size, in bytes. This is passed to the tileshape negotiation to select the right depth.