Per-project scratch and storage folder¶
About¶
Warning
The per-project scratch and storage folder is not intended to be used as a storage space for keeping an archive of files. Its purpose is to store operational data — files retained for a brief duration. Furthermore, Discoverer Petascale Supercomputer reserves the right to remove any of the files stored therein that have not been accessed within the last 61 days.
Allocations on /valhalla¶
- └ Available on clusters: Discoverer CPU, Discoverer GPU
- └ Total storage capacity: 5.7 TB
- └ Per-project allocation: 50 TB and 50000000 files
Note
That is the default for regular projects.
Each project that is onboarded on Discoverer since February 10, 2025, receives a project folder on our Cray ClusterStor E1000 storage cluster (HPE part number S-9100, based on NVMe), mounted on all nodes as /valhalla, as Luster file system. The total storage capacity of that file system is 5.7 PB. Depending on the requirements of the projects, they should receive on /valhalla a project folder with a storage capacity that does not exceed 50 TB. Additional space on /valhalla may be provided as an exception only after approval on our side, based on the appropriate justification received in advanced from the project principal (PI).
Allocations on /disco2fs¶
- └ Available on clusters: Discoverer CPU
- └ Total storage capacity: 29 TB
- └ Per-project allocation: 50 GB and 5000000 files
Note
That is the default for benchmark projects.
This is a Lustre file system provided by DDN storage cluster solution, based on NVMe storage devices. Its total capacity is 29 TB, which limits the number of projects we can host on simultaneously. On that storage location, we used to provide per-project scratch folders with capacity that does not exceed 50 GB. However, due to the temporary presence of that storage system in the Discoverer’s infrastructure, it cannot be guaranteed that /disco2fs will remain available for an extended period. Furthermore, effective February 10, 2025, we will solely allocate new allocations on /disco2fs from benchmark projects.
Allocations on /discofs¶
- └ Available on clusters: Discoverer CPU
- └ Total storage capacity: 2.1 PB
- └ Per-project allocation: 50 TB and 50000000 files
Note
Legacy, limited use for scratching.
Those allocations are to support projects on-boarded on Discoverer before February 10, 2025. Here, the Lustre file system is provided based on a relatively old DDN storage solution, which was designed and deployed in 2020, and is entirely based on hard disk drives. The latter makes /discofs not very suitable to handle the size of the most parallel jobs running on Discoverer at the moment. The latter makes /discofs not very suitable to handle the size of the most parallel jobs running on Discoverer nowadays. For that reason, we consider /discofs as a secondary for supporting scratching. We may occasionally provide new per-project or per-user allocations on /discofs, but that will be done on a limited basis and for jobs that are of no critical importance. The storage capacity of single new allocation here will not exceed 50 TB.
Each project is assigned its own scratch and storage folder, designed to provide a shared space where all users who contribute to a certain project can create, store, or process files related to the project. That folder helps to keep track of the amount of disk space each project takes up on the storage system of Discoverer. The folder is located at /discofs/projectname, where projectname is your Discoverer HPC project name. Physically, that folder lives on Luster file system, which is able to handle high I/O loads. Your running jobs should be configured to write there their output files.
Project folder location and naming convention¶
The naming convention delineates the methodology employed to identify the project folder.
/valhalla¶
The project scratch folder location is:
/valhalla/projects/project_name
Where project_name is the Slurm account name assigned to your project (for example: ehpc-reg-2025d0-000).
/disco2fs¶
The project scratch folder location is:
/disco2fs/projects/project_name
Where project_name is the Slurm account name assigned to your project (for example: ehpc-reg-2025d0-000).
Note
If the folder is owned by a project onboarded before February 10, 2025 then the location follows the pattern /disco2fs/projects/project_name.
/discofs¶
The project scratch folder location is:
/discofs/project_name
Where project_name is the Slurm account name assigned to your project (for example: ehpc-reg-2025d0-000).
Note
If the folder is owned by a project onboarded before February 10, 2025 then the location follows the pattern /discofs/projects/project_name.
Size/Quota¶
The amount of disk space and number of files allocated for the project scratch and storage folder depends on the class of the project (one of benchmark, development, regular) and the initial negotiation between the applicant and Discoverer. Here we only show how to find the quota limits and current utilization.
For the projects on-boarded after February 10, 2025¶
Initially, it is necessary to obtain the numerical identification number (ID) of the project. Here is an example of how to do that for a project with name of the Slurm account ehpc-reg-2025d0-000 hosted on /valhalla (the same applies for the other storage locations):
lfs project -d /valhalla/projects/ehpc-reg-2025d0-000
or (if your project is assigned a folder under /disco2fs)
lfs project -d /disco2fs/ehpc-reg-2025d0-000
When that command is successfully executed, a similar outcome will be displayed:
901 P /valhalla/projects/ehpc-reg-2025d0-000
The number that the output starts with is the ID of the project — in the example above, that is 901. Afterwards, that number should be passed to the lfs tool in order to display the quota limits and current utilization of the project folder.
lfs quota -p 901 /valhalla/projects/ehpc-reg-2025d0-000
For the projects on-boarded before February 10, 2025¶
The utilization of the project disk space on /discofs is estimated as the sum of the size of all files on that file system that belong to the POSIX group created for supporting the project. All users who contribute to a project are designated as members of the project’s POSIX group. However, the group ownership is enforced only for files created under the /discofs/projectname folder. This means that the disk quota established for the project is enforced only for files created under /discofs/projectname.
Warning
Carefully study the document Move and copy files between folders to learn about how to properly transfer files from or to the project folder to comply with the proper calculation of the quota in the project folder.
To check the current project disc space utilization on /discofs, execute the following command line on the login node (replace projectname with the actual name of the project):
lfs quota -g projectname /discofs
The document Calculation of the storage space utilization explains how to interpret the numeric information provided by the quota check.
Important
If a file or folder is moved from another location into /discofs/projectname, it will preserve its original group ownership, and therefore its size will not be added to the total disk space size occupied by the project on /discofs/projectname. In case you need to move or copy a file from your user folder into /discofs/projectname and add its size to the project disk space size, you have to change the group ownership of the file. For example:
mv /discofs/username/file1 /discofs/projectname/
chown :projectname /discofs/projectname/file1
Data retention policy¶
The files stored in the project folder are retained for 45 days after the project has been deactivated. Afterwards, they undergo a complete and irreversible wipe out.
Help¶
If you experience issues with accessing your per-project scratch folder and the content stored there, contact the Discoverer HPC support team (see Getting help).