Libdeflate

About

Libdeflate is a library for fast, whole-buffer DEFLATE-based compression and decompression. It depends on some of the Zlib methods and functions. It is widely used in various applications, including data compression, data archiving, and data management. For more information, visit the official Libdeflate website.

Note

On Discoverer we use libdeflate as main productive DEFLATE-based compression and decompression library for CPU-based applications.

Supported versions

In case your applications requite the use of another DEFLATE-based compression and decompression library, you can install it by yourself, but those builds will not be supported by the Discoverer HPC team.

To check which Libdeflate versions and build types are currently supported on Discoverer, execute on the login node:

module avail libdeflate

Compiler support

Warning

Currently, we support only LLVM builds of Libdeflate. No other builds will be officially supported. If you still see some environment modules for other compiler sets, they are not supported and will be retired soon.

Loading

If you need to access the latest version of the Libdeflate library, load the module without specifying the version:

module load libdeflate

In case you need to access a specific version of the Libdeflate library, you can specify the version:

module load libdeflate/<version>

Gzip and gunzip utilities

As part of the Libdeflate library installation, we provide highly productive gzip and gunzip utilities that are linked to the optimized libdeflate versions installed in our software repository. For most tasks, they seem to perform faster than the gzip and gunzip utilities that come with the Linux distribution installed on our compute nodes. If it is necessary to rely on those optimized tools, first load the appropriate libdeflate environment modules, as previously mentioned, and then execute the executables under their respective names:

libdeflate-gzip

or

libdeflate-gunzip

Warning

When large archives should be processed, use the Slurm batch job to execute the libdeflate-gzip and libdeflate-gunzip utilities.

Build recipes and tests

We use the recent LLVM Compiler Infrastructure to build the Libdeflate library code. We do not support any other compilers for that purpose. The LLVM.org compilers are the default compilers on Discoverer Petascale Supercomputer. Since the code of the library does not include Fortran, the compiled binaries are not Fortran-compatible and can be used with any other compiler collection. That means our build of libdeflate is optimised for producing the fastest possible binaries for CPU-based applications.

For those who are interested in the process of compiling and optimizing the code of Libdeflate we follow, there is public access to our build recipes at:

https://gitlab.discoverer.bg/vkolev/recipes/-/tree/main/libdeflate/

There are also logs that were collected during the compilation and testing processes.

Getting help

See Getting help