Libdeflate

About

Libdeflate is a library for fast, whole-buffer DEFLATE-based compression and decompression. It depends on some of the Zlib methods and functions.

Supported versions

To check which Libdeflate versions and build types are currently supported on Discoverer, execute on the login node:

module avail libdeflate

Loading/Compiler affinity

Each version of Libdeflate installed in the public software repository is separately built against four compiler sets: (i) Intel oneAPI, (ii) AMD Optimized Compiler Collection (AOCC), (iii) LLVM Compiler Infrastructure, (iv) GNU Compiler Collection (GCC). The name of the environmental module hints the provided compiler affinity.

Note

In the majority of instances, the users have the option to adhere to the GCC baseline build of the library. Please utilize the alternative build solely when you are fully cognizant of the advantages it will bring to your computations.

Intel oneAPI build

module avail libdeflate/*/*intel

NVIDIA HPC SDK build

Warning

This compiler collection is not capable of compiling the code of libdeflate.

LLVM build

module avail libdeflate/*/*llvm

AMD AOCC build

module avail libdeflate/*/*aocc

GNU GCC build

module avail libdeflate/*/*gcc

Gzip and gunzip utilities

We provide highly productive gzip and gunzip utilities that are linked to the optimized libdeflate versions installed in our software repository. For most tasks, they seem to perform faster than the gzip and gunzip utilities that come with the Linux distribution installed on our compute nodes. If it is necessary to rely on those optimized tools, first load the appropriate libdeflate environment modules, as previously mentioned, and then execute the executables under their respective names:

libdeflate-gzip

or

libdeflate-gunzip

Warning

When large archives should be processed, use the Slurm batch job to execute the gzip and gunzip utilities.

Build recipes and tests

For those who are interested in the process of compiling and optimizing the code of Libdeflate we follow, there is public access to our build recipes at:

https://gitlab.discoverer.bg/vkolev/recipes/-/tree/main/libdeflate/

There are also logs that were collected during the compilation and testing processes.

Getting help

See Getting help