ScaLAPACK

About

ScaLAPACK is a LAPACK replacement/enhancement that runs on parallel distributed memory machines (using MPI).

Supported versions

To check which ScaLAPACK versions and build types are currently supported on Discoverer, execute on the login node:

module avail scalapack

Note

Currently, we support only GCC build that works in conjunction with Open MPI 4.1.6 build. All other builds should be considered non-supported or retired.

The recipe used for compiling the programming code of ScaLAPACK on Discoverer, as well as the logs collected during the building process, are publicly available at:

https://gitlab.discoverer.bg/vkolev/recipes/-/tree/main/scalapack

Warning

Intel oneAPI comes with its own ScaLAPACK library, which is included in MKL. We do not recommend the use of that ScaLAPACK library installation. There is a general problem with using Intel MPI library on Discoverer compute nodes, which affects the MPI-based MKL subroutines.

Loading/Compiler affinity (32-bit API, LP)

To obtain access to the latest LAPACK load the environment module that matches your compiler requirements and MPI library affinity:

Intel oneAPI builds (retired)

  • Open MPI
module load scalapack/2/latest-intel-openmpi
  • MPICH
module load scalapack/2/latest-intel-mpich

NVIDIA HPC SDK build (retired)

  • Open MPI
module load scalapack/2/latest-nvidia-openmpi
  • MPICH
module load scalapack/2/latest-nvidia-mpich

AMD AOCC build (retired)

  • Open MPI

Warning

Switch to AMD Optimizing CPU Libraries (AOCL).

module load scalapack/2/latest-aocc-openmpi
  • MPICH
module load scalapack/2/latest-aocc-mpich

Notes about the availability of 64-bit API (ILP)

Note that the official ScaLAPACK code does not officially support 64-bit API (yet).

Warning

If your code requires linking to ScaLAPACK with ILP support, stay away from Math Kernel Library (MKL) on Discoverer (MKL does not perform as expected on AMD Zen2 microarchitecture).

But you may fully rely on the ScaLAPACK installation that ships with AMD Optimizing CPU Libraries (AOCL), in conjunction with Open MPI (see MPI). Just append the full path to the folder containing libscalapack.so and libscalapack.a to LDFLAGS or pass it directly to the linker. For instance:

export LDFLAGS+=" -L/opt/software/amd/aocl/4.2.0/aocc/lib_ILP64"

or

export LDFLAGS+=" -L/opt/software/amd/aocl/4.2.0/aocc/lib_ILP64 -lscalapack"

Getting help

See Getting help