OpenFOAM Foundation (CPU)
=========================

.. toctree::
   :maxdepth: 1
   :caption: Contents:

This page documents the OpenFOAM Foundation builds on Discoverer
(integer major versions such as OpenFOAM 11 and OpenFOAM 13).
Those packages are distinct from the OpenFOAM.com (OpenCFD / ESI) line.
For OpenFOAM.com modules (``v2312``, ``v2606``, and similar), see
:doc:`openfoam.com`.

About this lineage
------------------

OpenFOAM (Open Field Operation And Manipulation) is a C++ toolbox for
continuum mechanics, used most often for computational fluid dynamics.
The Foundation distribution provides finite-volume discretisation on
unstructured meshes, solvers and utilities, turbulence and multiphase
models, mesh tools, and parallel execution through MPI.

On Discoverer, Foundation installs use:

* environment modules under ``openfoam/``
* system OpenMPI with the shim name ``openmpi-system``
* a chosen label size (``WM_LABEL_SIZE`` 32 or 64), reflected in the
  module name as ``int32`` or ``int64``

Foundation and OpenFOAM.com share ancestry but are not drop-in
replacements. Do not mix binaries, case dictionaries, or modules from
one line with the other without checking compatibility.

Versions available
------------------

Supported versions
..................

.. note:: The versions of `OpenFOAM`_ installed in the software
   repository are built and supported by the Discoverer HPC team.

To list Foundation modules on the login node:

.. code-block:: bash

   module avail openfoam

Module names follow the pattern:

.. code-block:: bash

   openfoam/<major>/<major>-<compiler>-openmpi-int32
   openfoam/<major>/<major>-<compiler>-openmpi-int64

Examples for OpenFOAM 13:

.. code-block:: bash

   module load openfoam/13/13-gcc-openmpi-int32
   module load openfoam/13/13-gcc-openmpi-int64
   module load openfoam/13/13-llvm-openmpi-int32
   module load openfoam/13/13-llvm-openmpi-int64

After loading a module, confirm the environment:

.. code-block:: bash

   echo "$WM_PROJECT_VERSION" "$WM_OPTIONS" "$WM_LABEL_SIZE"
   which foamDictionary

Build recipes used by the Discoverer HPC team are published at:

https://gitlab.discoverer.bg/vkolev/recipes/-/tree/main/openfoam

Compiler notes
..............

Tests on OpenFOAM 10 and 11 showed that Intel oneAPI compilers
(``icx``/``icpx`` or ``icc``/``icpc``) did not speed up the produced
executables compared with GCC or LLVM for the same purpose. Prefer
:doc:`llvm` (version 16 or higher) or GCC for Foundation builds on
this site. LLVM modules still use Clang for C/C++; a GCC module may be
loaded only for Fortran / runtime support.

User-supported versions
.......................

Users may bring or compile their own OpenFOAM builds, but those builds
are not supported by the Discoverer HPC team. Public recipes (see the
link above) may be useful as a starting point.

Running OpenFOAM
----------------

.. warning:: You MUST NOT execute OpenFOAM calculations directly on the
   login node (login.discoverer.bg). Run calculations only as Slurm jobs.

.. warning:: Write results only inside your :doc:`scratchfolder`. Do not
   use your :doc:`homefolder` for result data under any circumstances.

Example: ``wave`` tutorial (OpenFOAM 13)
........................................

The following example runs the Foundation ``wave`` tutorial as a Slurm
batch job. On OpenFOAM 13 the case lives under
``tutorials/incompressibleVoF/wave`` (the older
``multiphase/interFoam/laminar/wave`` path does not apply to this
release). The case ``decomposeParDict`` requests 6 subdomains.

First, load a Foundation module and copy the tutorial to scratch:

.. code:: bash

   module load openfoam/13/13-llvm-openmpi-int32
   mkdir -p /discofs/$USER/run_openfoam
   cd /discofs/$USER/run_openfoam
   cp -pr $FOAM_TUTORIALS/incompressibleVoF/wave .

Create ``/discofs/$USER/run_openfoam/wave/run_openfoam_mpi.batch`` with
content similar to:

.. code:: bash

   #!/bin/bash

   #SBATCH --partition=cn
   #SBATCH --job-name=openfoam_wave
   #SBATCH --time=04:00:00

   #SBATCH --account=your_slurm_account_name
   #SBATCH --qos=your_slurm_qos_name

   #SBATCH --nodes           1
   #SBATCH --ntasks-per-node 6
   #SBATCH --ntasks-per-core 1

   #SBATCH -o slurm.%j.out
   #SBATCH -e slurm.%j.err

   module purge
   module load openfoam/13/13-llvm-openmpi-int32

   . $WM_PROJECT_DIR/bin/tools/RunFunctions

   cd $SLURM_SUBMIT_DIR

   runApplication blockMesh
   runApplication extrudeMesh
   runApplication refineMesh
   runApplication setWaves
   runApplication decomposePar

   runParallel foamRun

   runApplication reconstructPar

.. warning:: For this tutorial, keep ``--ntasks-per-node`` equal to
   ``numberOfSubdomains`` in ``system/decomposeParDict`` (here: 6).
   For other cases, always match the Slurm task count to the
   decomposition defined for that case.

Submit the job:

.. code:: bash

   cd /discofs/$USER/run_openfoam/wave
   sbatch run_openfoam_mpi.batch

Slurm writes standard output to ``slurm.%j.out`` and standard error to
``slurm.%j.err`` in the submit directory (``%j`` is the job ID). Tail
``slurm.%j.out`` to follow progress.

Getting help
------------

See :doc:`help`.

Related pages
-------------

* :doc:`openfoam.com` — OpenFOAM.com (OpenCFD / ESI) builds on Discoverer
* OpenFOAM Foundation: https://openfoam.org/
* Foundation documentation: https://doc.openfoam.org/

.. _`OpenFOAM`: https://openfoam.org/