ScaLAPACK
=========

.. toctree::
   :maxdepth: 1
   :caption: Contents:

About
-----

`ScaLAPACK`_ is a :doc:`lapack` replacement/enhancement that runs on parallel distributed memory machines (using MPI).

Supported versions
------------------

To check which ScaLAPACK versions and build types are currently supported on Discoverer, execute on the login node:

.. code-block:: bash

 module avail scalapack

.. note:: Currently, we support only GCC build that works in conjunction with Open MPI 4.1.6 build. All other builds should be considered non-supported or retired.

The recipe used for compiling the programming code of ScaLAPACK on Discoverer, as well as the logs collected during the building process, are publicly available at:

https://gitlab.discoverer.bg/vkolev/recipes/-/tree/main/scalapack

.. warning:: :doc:`oneapi` comes with its own ScaLAPACK library, which is included in MKL. We do not recommend the use of that ScaLAPACK library installation. There is a general problem with using Intel MPI library on Discoverer compute nodes, which affects the MPI-based MKL subroutines.

Loading/Compiler affinity (32-bit API, LP)
------------------------------------------

To obtain access to the latest LAPACK load the environment module that matches your compiler requirements and MPI library affinity:

GNU GCC build (Recommended)
...........................

* Open MPI

.. code:: bash

 module load scalapack/2/latest-gcc-openmpi

Intel oneAPI builds (retired)
.............................

* Open MPI

.. code:: bash

 module load scalapack/2/latest-intel-openmpi

* MPICH

.. code:: bash

 module load scalapack/2/latest-intel-mpich

NVIDIA HPC SDK build (retired)
..............................

* Open MPI

.. code:: bash

 module load scalapack/2/latest-nvidia-openmpi

* MPICH

.. code:: bash

 module load scalapack/2/latest-nvidia-mpich


AMD AOCC build (retired)
........................

* Open MPI

.. warning:: Switch to :doc:`aocl`.

.. code:: bash

 module load scalapack/2/latest-aocc-openmpi

* MPICH

.. code:: bash

 module load scalapack/2/latest-aocc-mpich


Notes about the availability of 64-bit API (ILP)
------------------------------------------------

Note that the official `ScaLAPACK`_ code does not officially support 64-bit API (yet).

.. warning:: If your code requires linking to ScaLAPACK with ILP support, stay away from :doc:`mkl` on Discoverer (MKL does not perform as expected on AMD Zen2 microarchitecture).

But you may fully rely on the ScaLAPACK installation that ships with :doc:`aocl`, in conjunction with Open MPI (see :doc:`mpi`). Just append the full path to the folder containing ``libscalapack.so`` and ``libscalapack.a`` to ``LDFLAGS`` or pass it directly to the linker. For instance:

.. code:: bash

   export LDFLAGS+=" -L/opt/software/amd/aocl/4.2.0/aocc/lib_ILP64"

or

.. code:: bash

   export LDFLAGS+=" -L/opt/software/amd/aocl/4.2.0/aocc/lib_ILP64 -lscalapack"


Getting help
------------

See :doc:`help`

.. _`ScaLAPACK`: https://netlib.org/scalapack/