Zstandard/zstd¶
Fast real-time compression with better ratios than zlib
Overview¶
Zstandard (zstd) is a fast lossless compression algorithm designed for:
- Real-time compression scenarios
- Better compression ratios than zlib
- Fast entropy stage through Huff0 and FSE library
Note
We provide Zstandard installation that is faster and newer than the one provided by the system-wide package manager (e.g. /usr/lib64/libzstd.so brought to the system by the Red Hat Enterprise Linux package manager). Therefore, we recommend using our installation instead of the system-wide one (see below).
Available versions¶
In case your applications requite the use of another Zstandard library, you can install it by yourself, but those builds will not be supported by the Discoverer HPC team.
To check which Zstandard versions and build types are currently supported on Discoverer, execute on the login node:
module avail zstd
Compiler support¶
Warning
Currently, we support only LLVM builds of Zstandard. No other builds will be officially supported. If you still see some environment modules for other compiler sets, they are not supported and will be retired soon.
Loading¶
If you need to access the latest version of the Zstandard library, load the module without specifying the version:
module load zstd # LLVM build (recommended, will be supported)
In case you need to access a specific version of the Zstandard library, you can specify the version:
module load zstd/<version>
Build recipes and tests¶
We use the recent LLVM Compiler Infrastructure to build the Zstandard library code. We do not support any other compilers for that purpose. The LLVM.org compilers are the default compilers on Discoverer Petascale Supercomputer. Since the code of the library does not include Fortran, the compiled binaries are not Fortran-compatible and can be used with any other compiler collection. That means our build of zstd is optimised for producing the fastest possible binaries for CPU-based applications.
For those who are interested in the process of compiling and optimizing the code of Zstandard we follow, there is public access to our build recipes at:
https://gitlab.discoverer.bg/vkolev/recipes/-/tree/main/zstd
There are also logs that were collected during the compilation and testing processes.