NWChem
======

.. toctree::
   :maxdepth: 1
   :caption: Contents:

Versions avaiable
-----------------

Supported versions
..................

.. note:: The versions of `NWChem`_ installed in the software repository are built and supported by the Discoverer HPC team.

To check which `NWChem`_ versions are currently supported on Discoverer, execute on the login node:

.. code-block:: bash

 module avail

and grep the output for "nwchem".

User-supported versions
.......................

Users are welcome to bring, or compile, and use their own builds of NWChem, but **those builds will not be supported by the Discoverer HPC team.**

Pre-prosessing
--------------

If you need to convert certain AMBER topology into the format of one used by NWChem, you can run the tool ``amber-helper``, which is built by the Discoverer HPC team. Just load the environment module:

.. code-block:: bash

 nwchem/latest-intelmpi

and after that ``amber-helper`` can be invoked directly (without specifying the path to the executable):

.. code-block:: bash

 amber-helper

Running simulations
-------------------

.. warning:: **You MUST NOT execute simulations directly upon the login node (login.discoverer.bg).** You have to run your simulations as Slurm jobs only.

.. warning:: Write your trajectories and results of analysis only inside your :doc:`scratchfolder` and DO NOT use for that purpose (under any circumstances) your :doc:`homefolder`!

Use of OpenMP (SM)
..................

.. important:: Be aware that not all simulation protocols and methods included in NWChem support shared-memory parallelism, based on the OpenMP library. The use of OpenMP promoted threading should be in sync with the input configuration. Check the NWChem documentation.

Slurm batch job script
......................

The following template can be useful for creating a Slurm batch job script to submit to the queue:

.. code:: bash

 #!/bin/bash
 #
 #SBATCH --partition=cn         # Partition (you may need to change this, ask the support team)
 #SBATCH --job-name=nwchem
 #SBATCH --time=01:00:00        # WallTime - set it accordningly

 #SBATCH --nodes           1    # Number of node employed
 #SBATCH --ntasks-per-node 128  # Use all 128 CPU cores on each node
 #SBATCH --ntasks-per-core 1    # Run only one MPI process per CPU core
 #SBATCH --cpus-per-task   1    # Number of OpenMP threads per MPI process
                                # Read the note about the use of OpenMP in NWChem

 #SBATCH -o slurm.%j.out        # STDOUT
 #SBATCH -e slurm.%j.err        # STDERR

 ulimit -Hs unlimited
 ulimit -Ss unlimited

 module purge
 module load nwchem/7/latest-intelmpi

 export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
 export OMP_PLACES=cores
 export OMP_PROC_BIND=spread
 export I_MPI_PIN=1
 export I_MPI_DEBUG=8
 export I_MPI_PIN_DOMAIN=omp:platform
 export FI_PROVIDER=verbs
 export UCX_NET_DEVICES=mlx5_0:1
 export UCX_POSIX_USE_PROC_LINK=n

 cd $SLURM_SUBMIT_DIR

 mpirun nwchem input.nw

Specify the parmeters and resources required for successfully running and completing the job:

 - Slurm partition of compute nodes, based on your project resource reservation (``--partition``)
 - job name, under which the job will be seen in the queue (``--job-name``)
 - wall time for running the job (``--time``)
 - number of occupied compute nodes (``--nodes``)
 - number of MPI proccesses per node (``--ntasks-per-node``)
 - number of threads (OpenMP threads) per MPI process (``--cpus-per-task``)
 - version of NWChem to run after ``module load`` (see `Supported versions`_)

Save the complete Slurm job description as a file, storing it inside the folder with the input configuration, for example ``/discofs/$USER/run_nwchem/run.batch``, and submit it to the queue afterwards

.. code:: bash

 cd /discofs/$USER/run_nwchem/
 sbatch run.batch

Upon successful submission, the standard output will be directed by Slurm into the file ``/discofs/$USER/run_nwchem/slurm.%j.out`` (where ``%j`` stands for the Slurm job ID), while the standard error output will be stored in ``/discofs/$USER/run_nwchem/slurm.%j.err``.

Check the provided working example (see below) to find more details about how to create a complete Slurm batch job script for running NWChem.

Working example
...............

The goal of this working example is to show one possible way NWChem can run on Discoverer HPC through a Slurm batch job. Running the example is simple - just execute on the login node (login.discoverer.bg):

.. code:: bash

 cd /discofs/$USER
 sbatch /opt/software/WRF/4/4.4-nvidia-openmpi/examples/1.batch

Once started successfully by Slurm, that job will create first a directory under your :doc:`scratchfolder`. The name of the directory will be similar to this one: ``nwchem_2022-06-21-22-21-06-52.1655836192`` (numbers should be different in your case). You may check the advance of the simulation by executing:

.. code:: bash

 cd /discofs/$USER
 tail -f slurm.XXXXX.out

where ``XXXXX`` stands for the Slurm ID of the running job.

Getting help
------------

See :doc:`help`


.. _`NWChem`: https://www.nwchem-sw.org/