Per-project scratch and storage folder ====================================== About ----- .. warning:: The per-project scratch and storage folder is *not* intended to be used as a storage space for keeping an archive of files. **Its purpose is to store operational data — files retained for a brief duration.** Furthermore, Discoverer Petascale Supercomputer reserves the right to remove any of the files stored therein that have not been accessed within the last 61 days. Allocations on ``/valhalla`` ............................ └ Available on clusters: Discoverer CPU, Discoverer GPU └ Total storage capacity: 5.7 TB └ Per-project allocation: 50 TB and 50000000 files .. note:: That is the default for regular projects. Each project that is onboarded on Discoverer since February 10, 2025, receives a project folder on our Cray ClusterStor E1000 storage cluster (HPE part number S-9100, based on NVMe), mounted on all nodes as ``/valhalla``, as `Luster file system`_. The total storage capacity of that file system is 5.7 PB. Depending on the requirements of the projects, they should receive on ``/valhalla`` a project folder with a storage capacity that does not exceed 50 TB. Additional space on ``/valhalla`` may be provided as an exception only after approval on our side, based on the appropriate justification received in advanced from the project principal (PI). Allocations on ``/disco2fs`` ............................ └ Available on clusters: Discoverer CPU └ Total storage capacity: 29 TB └ Per-project allocation: 50 GB and 5000000 files .. note:: That is the default for benchmark projects. This is a Lustre file system provided by DDN storage cluster solution, based on NVMe storage devices. Its total capacity is 29 TB, which limits the number of projects we can host on simultaneously. On that storage location, we used to provide per-project scratch folders with capacity that does not exceed 50 GB. However, due to the temporary presence of that storage system in the Discoverer's infrastructure, it cannot be guaranteed that ``/disco2fs`` will remain available for an extended period. Furthermore, effective February 10, 2025, we will solely allocate new allocations on ``/disco2fs`` from benchmark projects. Allocations on ``/discofs`` ........................... └ Available on clusters: Discoverer CPU └ Total storage capacity: 2.1 PB └ Per-project allocation: 50 TB and 50000000 files .. note:: Legacy, limited use for scratching. Those allocations are to support projects on-boarded on Discoverer before February 10, 2025. Here, the Lustre file system is provided based on a relatively old DDN storage solution, which was designed and deployed in 2020, and is entirely based on hard disk drives. The latter makes ``/discofs`` not very suitable to handle the size of the most parallel jobs running on Discoverer at the moment. The latter makes ``/discofs`` not very suitable to handle the size of the most parallel jobs running on Discoverer nowadays. For that reason, we consider ``/discofs`` as a secondary for supporting scratching. We may occasionally provide new per-project or per-user allocations on ``/discofs``, but that will be done on a limited basis and for jobs that are of no critical importance. The storage capacity of single new allocation here will not exceed 50 TB. Each project is assigned its own scratch and storage folder, designed to provide a shared space where all users who contribute to a certain project can create, store, or process files related to the project. That folder helps to keep track of the amount of disk space each project takes up on the storage system of Discoverer. The folder is located at ``/discofs/projectname``, where ``projectname`` is your Discoverer HPC project name. Physically, that folder lives on `Luster file system`_, which is able to handle high I/O loads. Your running jobs should be configured to write there their output files. Project folder location and naming convention --------------------------------------------- The naming convention delineates the methodology employed to identify the project folder. ``/valhalla`` ............. The project scratch folder location is: .. code-block:: bash /valhalla/projects/project_name Where ``project_name`` is the Slurm account name assigned to your project (for example: ``ehpc-reg-2025d0-000``). ``/disco2fs`` ............. The project scratch folder location is: .. code-block:: bash /disco2fs/projects/project_name Where ``project_name`` is the Slurm account name assigned to your project (for example: ``ehpc-reg-2025d0-000``). .. note:: If the folder is owned by a project onboarded before February 10, 2025 then the location follows the pattern ``/disco2fs/projects/project_name``. ``/discofs`` ............ The project scratch folder location is: .. code-block:: bash /discofs/project_name Where ``project_name`` is the Slurm account name assigned to your project (for example: ``ehpc-reg-2025d0-000``). .. note:: If the folder is owned by a project onboarded before February 10, 2025 then the location follows the pattern ``/discofs/projects/project_name``. Size/Quota ---------- The amount of disk space and number of files allocated for the project scratch and storage folder depends on the class of the project (one of benchmark, development, regular) and the initial negotiation between the applicant and Discoverer. Here we only show how to find the quota limits and current utilization. For the projects on-boarded after February 10, 2025 ................................................... Initially, it is necessary to obtain the numerical identification number (ID) of the project. Here is an example of how to do that for a project with name of the Slurm account ``ehpc-reg-2025d0-000`` hosted on ``/valhalla`` (the same applies for the other storage locations): .. code-block:: bash lfs project -d /valhalla/projects/ehpc-reg-2025d0-000 or (if your project is assigned a folder under ``/disco2fs``) .. code-block:: bash lfs project -d /disco2fs/ehpc-reg-2025d0-000 When that command is successfully executed, a similar outcome will be displayed: .. code-block:: bash 901 P /valhalla/projects/ehpc-reg-2025d0-000 The number that the output starts with is the ID of the project — in the example above, that is 901. Afterwards, that number should be passed to the ``lfs`` tool in order to display the quota limits and current utilization of the project folder. .. code-block:: bash lfs quota -p 901 /valhalla/projects/ehpc-reg-2025d0-000 For the projects on-boarded before February 10, 2025 .................................................... The utilization of the project disk space on ``/discofs`` is estimated as the sum of the size of all files on that file system that belong to the POSIX group created for supporting the project. All users who contribute to a project are designated as members of the project's POSIX group. However, the group ownership is enforced only for files created under the ``/discofs/projectname`` folder. This means that the disk quota established for the project is enforced only for files created under ``/discofs/projectname``. .. warning:: Carefully study the document :doc:`storage_moving_files_overquota` to learn about how to properly transfer files from or to the project folder to comply with the proper calculation of the quota in the project folder. To check the current project disc space utilization on ``/discofs``, execute the following command line on the login node (replace projectname with the actual name of the project): .. code-block:: bash lfs quota -g projectname /discofs The document :doc:`calculating_the_disk_usage_basics` explains how to interpret the numeric information provided by the quota check. .. important:: If a file or folder is moved from another location into ``/discofs/projectname``, it will preserve its original group ownership, and therefore its size will not be added to the total disk space size occupied by the project on ``/discofs/projectname``. In case you need to move or copy a file from your user folder into ``/discofs/projectname`` and add its size to the project disk space size, you have to change the group ownership of the file. For example: .. code-block:: bash mv /discofs/username/file1 /discofs/projectname/ chown :projectname /discofs/projectname/file1 Data retention policy --------------------- The files stored in the project folder are retained for 45 days after the project has been deactivated. Afterwards, they undergo a complete and irreversible wipe out. Help ---- If you experience issues with accessing your per-project scratch folder and the content stored there, contact the Discoverer HPC support team (see :doc:`help`). .. _`Luster file system`: https://en.wikipedia.org/wiki/Lustre_(file_system)