LLVM Compiler Infrastructure¶
Warning
The package provided under the name “llvm” in the software repository of Discoverer HPC adopts the programming code of LLVM Compiler Infrastructure project without applying any modifications to that code. In case you need specific LLVM build provided and supported by other vendors of compiler packages, check Intel oneAPI, AMD Optimized Compiler Collection (AOCC), and NVIDIA HPC SDK (former PGI Compilers).
Supported versions¶
To check which versions of LLVM Compiler Infrastructure are currently supported on Discoverer, execute on the login node:
module avail llvm
The LLVM build, available in the software repository, is supported by HPC Discoverer team. Below is the complete list of LLVM projects and runtimes included:
- Projects
- bolt
- clang
- clang-tools-extra
- flang
- libclc
- lld
- lldb
- mlir
- openmp
- polly
- pstl
- Runtimes
- libcxx
- libcxxabi
- libunwind
- compiler-rt
- libc
The LLVM compilers are equipped with internal code generators for producing code optimized for running on major NVIDIA, Intel, and AMD GPU accelerators with HPC compute capabilities. While such GPU accelerators are not yet available on Discoverer’s compute nodes, a new, brand-new GPU partition is on the way.
Important
Our LLVM builds are compiling time self-sufficient. For each new version of the vanilla LLVM code, we use GCC to initially compile that code, and then we use the compilers and libraries from that compilation to compile the code again. It is important to declare here that the LLVM compilers, tools, and libraries we host depend only on the system libraries that come with the Linux distribution (which includes libstdc++). The only exception is to load an alternative version of binutils to support the LLVMgold plugin. But that binutils package also falls back to the system libraries.
The implemented compilation recipe is publicly available at https://gitlab.discoverer.bg/vkolev/recipes/-/tree/main/llvm
Loading¶
To access the latest LLVM compilers, load the environment module llvm/latest:
module load llvm/latest
or select a particular version.
LLVM compilers¶
The LLVM build includes the following C, C++, and Fortran compiler executables:
clang
clang++
flang
Warning
Contrary to the approach adoped in AMD AOCC package, we do not symlink flang to clang. Based on our type of installation flang is a symlink to flang-new.
Notes about storing raw profiles (Source-based Code Coverage)¶
Note
If you are developer who requires the generation and further use of raw profiles, always set the appropriate LLVM_PROFILE_FILE value before running any binary code compiled by passing -fprofile-instr-generate and -fcoverage-mapping to the clang compiler.
Shortly, when -fprofile-instr-generate and -fcoverage-mapping flags are passed to clang during the compilation process, the created binaries (when invokved) will start storing the corresoinding raw profiles as files on the disk. If that is the case, the location of the raw profile file should be set in advance as the only value of the environment variable LLVM_PROFILE_FILE. For example:
export LLVM_PROFILE_FILE=/discofs/${USER}/tmp/application.profraw
Warning
The environment modules handling access to the LLVM installation on Discoverer (when involved) set LLVM_PROFILE_FILE=/dev/null to avoid the generation of large files in the user directories, without being noticed. That is just a workaround that reduce the disk space utilization. Always consider setting LLVM_PROFILE_FILE` to a real file location that matches your goals.
If the use of raw profiles is imposed on the binary code at compile time, but the variable LLVM_PROFILE_FILE is not set, the running code will generate “permission deined” kind of fatal error and exit:
LLVM Profile Error: Failed to write file "/project/soft-raid1-admin/build/vkolev/compile/llvm-project-llvmorg-19.1.5/build-llvm/profiles/7577661472985617262_2.profraw": Permission denied
Blame the clang compiler for getting that, for it hardcodes inside the produced binary code a raw profile file location that falls inside the folder used to build the code of the LLVM compiler infrastructure. That folder is not writable by all users, and it is usually located on the build host, which makes the error unrecoverable if LLVM_PROFILE_FILE is not set by the user running the code.
Warning
The generated raw profile files may occupy a considerable amount of disk space. To save disk space, always delete the raw profile files that are not needed any more.
Compiler optimization flags for AMD Zen2 CPU¶
Note
The compute nodes of Discoverer HPC are equipped with AMD EPYC 7H12 64-Core processors, which implies AMD Zen2 CPU architecture, whereupon AXV 256 is the highest SIMD instruction (no AVX 512 is available).
Sticking to the following compiler flags can be profitable during the compile-time optimization on AMD Zen2:
-march=znver2 -mtune=native
and it is up on you to add -mfma to that set.
For example:
clang -march=znver2 -mtune=native ...
clang++ -march=znver2 -mtune=native ...
flang -march=znver2 -mtune=native ...
More on the supported compiler flags and types of optimizations: LLVM User Guides / Optimizations
Interaction with CMake¶
It is recommended to specify explicitly the compiler executables:
-DCMAKE_C_COMPILER=clang
-DCMAKE_CXX_COMPILER=clang++
-DCMAKE_Fortran_COMPILER=flang
The corresponding optimization compiler flags can be passed to cmake the usual way:
-DCMAKE_C_FLAGS="-march=znver2 -mtune=native"
-DCMAKE_CXX_FLAGS="-march=znver2 -mtune=native"
-DCMAKE_Fortran_FLAGS="-march=znver2 -mtune=native"
Notes on using llvm-bolt¶
The following presentation illustrates the benefits of using LLVM Bolt:
https://llvm.org/devmtg/2024-03/slides/practical-use-of-bolt.pdf
Notes on linking¶
We employ ld.gold as default linker (through Binutuls). If you want to rely on different linker you have to specify its executable explicitly:
https://cmake.org/cmake/help/latest/variable/CMAKE_LINKER_TYPE.html
https://www.gnu.org/software/automake/manual/html_node/How-the-Linker-is-Chosen.html
Getting help¶
See Getting help