ScaLAPACK

About

ScaLAPACK is a LAPACK replacement/enhancement that runs on parallel distributed memory machines (using MPI).

LP64 vs ILP64 integer models

On Discoverer, ScaLAPACK is available in two integer model variants:

LP64 (Long and Pointer 64-bit):

  • Standard integer model where long and pointers are 64-bit, while int remains 32-bit
  • Default integer size: 32-bit (int in C, INTEGER in Fortran)
  • Maximum array index: ~2.1 billion (2³¹ - 1)
  • Use case: Standard applications, smaller to medium-sized matrices, compatibility with most existing code
  • Recommended for most users

ILP64 (Integer, Long, and Pointer 64-bit):

  • Integer model where int, long, and pointers are all 64-bit
  • Default integer size: 64-bit (int in C, INTEGER in Fortran)
  • Maximum array index: ~9.2 quintillion (2⁶³ - 1)
  • Use case: Very large problems where array dimensions exceed 2.1 billion elements, high-performance computing with massive matrices
  • Requires all dependent libraries and application code to also use 64-bit integers

Note

If you are unsure which version to use, start with LP64. It is the standard model and compatible with most existing code. Only use ILP64 if you specifically need to handle arrays larger than 2.1 billion elements.

Supported versions

ScaLAPACK is compiled with GCC and Open MPI 4.1.6. To check which ScaLAPACK versions are currently available, execute:

Important

ScaLAPACK installations are compiler agnostic. The dynamic libraries (libscalapack.so) provided by these modules can be linked to any application code, regardless of the compiler used to compile your application. You can use ScaLAPACK with GCC, LLVM/clang, Intel oneAPI, or any other compatible compiler that can link against the shared libraries. The only requirement is that your application code and ScaLAPACK must be compatible with the same system ABI (Application Binary Interface).

For LP64 (32-bit integers):

module avail scalapack/lp64

For ILP64 (64-bit integers):

module avail scalapack/ilp64

The recipe used for compiling ScaLAPACK on Discoverer, as well as the logs collected during the building process, are publicly available at:

https://gitlab.discoverer.bg/vkolev/recipes/-/tree/main/scalapack

Warning

Intel oneAPI comes with its own ScaLAPACK library, which is included in MKL. We do not recommend the use of that ScaLAPACK library installation. There is a general problem with using Intel MPI library on Discoverer compute nodes, which affects the MPI-based MKL subroutines.

Loading ScaLAPACK modules

LP64 modules (32-bit integers)

LP64 modules are located under scalapack/lp64/2/ and provide the standard 32-bit integer interface.

For listing available versions, execute:

module avail scalapack/lp64/

For loading a specific version, execute:

module load scalapack/lp64/2/2.2.2

The LP64 modules automatically load the required OpenBLAS dependency (openblas/0/0.3.28-gcc or openblas/0/0.3.29-gcc) if it is not already loaded.

ILP64 modules (64-bit integers)

ILP64 modules are located under scalapack/ilp64/2/ and provide the 64-bit integer interface for very large problems.

For listing available versions, execute:

module avail scalapack/ilp64/

For loading a specific version, execute:

module load scalapack/ilp64/2/2.2.2

The ILP64 modules automatically load the required OpenBLAS ILP64 dependency (openblas/0/0.3.28-gcc-int64 or openblas/0/0.3.29-gcc-int64) if it is not already loaded.

Important

LP64 and ILP64 modules conflict with each other. You cannot load both simultaneously. Choose the appropriate version based on your problem size requirements.

Module environment variables

When a ScaLAPACK module is loaded, the following environment variables are set:

  • SCALAPACK_ROOT: Root installation directory
  • SCALAPACK_VERSION: Version number (e.g., “2.2.2”)
  • SCALAPACK_INTEGER_MODEL: Integer model (“LP64” or “ILP64”)

The module also sets up:

  • Library paths (LD_LIBRARY_PATH, LIBRARY_PATH)
  • Include paths (CPATH)
  • CMake module paths (CMAKE_MODULE_PATH)
  • pkg-config paths (PKG_CONFIG_PATH)

Open MPI dependency

ScaLAPACK libraries are compiled with RPATH that hardcodes the path to the Open MPI 4.1.6 installation used during compilation. This means:

  • The Open MPI module is not required to be loaded for ScaLAPACK to function
  • MPI libraries are resolved automatically via RPATH
  • The exact Open MPI version used during compilation is used at runtime

Note

While the Open MPI module is not required for ScaLAPACK itself, you may still want to load it if your application code directly uses MPI functions, as it provides the necessary compiler wrappers (mpicc, mpif90, etc.) and environment setup.

Using ScaLAPACK in your code

CMake:

The ScaLAPACK modules set up CMAKE_MODULE_PATH, so you can use:

find_package(ScaLAPACK REQUIRED)
target_link_libraries(your_target ScaLAPACK::ScaLAPACK)

pkg-config:

pkg-config --cflags --libs scalapack

Direct linking:

gfortran your_code.f90 -lscalapack -L/path/to/scalapack/lib -I/path/to/scalapack/include

Getting Help

See Getting help