Personal scratch and storage folder =================================== Discoverer does not provide a dedicated personal scratch area on a separate high-performance storage system (for example an individual Lustre project quota or Weka folder tied only to you). Scratch and bulk I/O space are allocated per project, not per user. If you need somewhere to keep large temporary files, checkpoints, or job output, use space under your project's storage directory or your home directory, as described below. .. warning:: We do not offer a personal scratch or storage folder on a separate storage backend. All large working data must fit within your project's allocated quota on ``/valhalla`` and, where applicable, ``/weka``, or within your home-directory quota on ext4. What storage is available ------------------------- +----------------------------+------------------------------------------------------------+--------------------------------------------------------------------------+ | Area | Typical path | Quota | +============================+============================================================+==========================================================================+ | Home directory | ``$HOME`` | Per-user UID quota on ext4 (2GiB, small) | +----------------------------+------------------------------------------------------------+--------------------------------------------------------------------------+ | Project storage (Lustre) | ``/valhalla/projects//`` | Shared project group quota on LustreFS (5TiB) | +----------------------------+------------------------------------------------------------+--------------------------------------------------------------------------+ | Project storage (Weka) | Folder under ``/weka`` (if allocated) | Comersial projects receive a dedicated project folder quota on WekaFS | +----------------------------+------------------------------------------------------------+--------------------------------------------------------------------------+ Here ```` is your project's SLURM account name (the same identifier used in job submission and in onboarding material). See :doc:`storage` for mount points, performance notes, and current limits. Home directories are intended for personal settings, SSH client configuration, small scripts, and modest personal files — not for large-scale scratch or parallel I/O. The `Discoverer Use Policy (DUP) `_ (§7.1–§7.2) describes how quotas differ between home and project storage. Creating a personal scratch folder under your project ----------------------------------------------------- When you need a private working area for large files, create a subdirectory inside your project tree. Usage still counts against the single shared project quota; PetaSC does not subdivide project storage per user. Replace ```` with your project SLURM account ID (for example the value of ``$SLURM_ACCOUNT`` in a job, or the project ID from your onboarding letter): .. code-block:: bash mkdir -p /valhalla/projects//scratch/$USER chmod 700 /valhalla/projects//scratch/$USER Example for user ``jsmith`` on project ``ehpc-2024-001``: .. code-block:: bash mkdir -p /valhalla/projects/ehpc-2024-001/scratch/jsmith You may choose any subdirectory name (``myscratch``, ``tmp``, ``work``, and so on). Coordinate with your Principal Investigator and teammates so everyone understands how shared project space is used; the PI is responsible for staying within the fixed project quota (DUP §7.2). Permissions and group quota ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Project directories on ``/valhalla`` are owned by the project POSIX group. New files and directories should remain group-accessible so that Lustre group quota accrues correctly to the project (DUP §7.4). Do not change ownership of the project root or remove group access in ways that would expose data outside the project team. If you create a folder only for yourself, restrictive permissions (for example ``700`` on your scratch subdirectory) are acceptable provided the parent project tree remains correctly group-owned. Batch jobs and ``TMPDIR`` ------------------------- For large temporary files inside batch jobs, point ``TMPDIR`` at project storage rather than node-local ``/tmp`` (DUP §7.5). Node-local storage is ephemeral and not backed up. .. code-block:: bash export TMPDIR=/valhalla/projects//scratch/$USER/tmp mkdir -p "$TMPDIR" Include ``export TMPDIR=...`` in your Slurm batch script or job prologue when jobs write substantial temporary data. Weka projects ------------- If your allocation includes Weka storage, the same principle applies: create a working subdirectory under your project's Weka folder. There is still no separate personal Weka allocation. Paths and folder names are given at onboarding; see :doc:`storage`. What not to use --------------- - Do not store large datasets or scratch files only in ``$HOME`` — home quota is limited and not suited to parallel I/O. - Do not assume a path such as ``/valhalla//`` without the ``projects/`` prefix; project data lives under ``/valhalla/projects//``. - Do not rely on compute-node ``/tmp`` for data you need after the job ends. Retention --------- Project data, including scratch you create under the project path, is subject to project retention rules when the allocation ends (export within 30 days; see the DUP §8.4). Home-directory retention applies separately when you leave all projects (DUP §8.5). Help ---- If you are unsure of your project account name, quota, or whether Weka was allocated, check your onboarding letter or contact support via :doc:`help`. See also -------- - :doc:`storage` — filesystem layout and performance guidance - :doc:`running_jobs` — submitting jobs and environment variables - :doc:`help` — support contact - `Lustre file system`_ — background on the LustreFS backend used for ``/valhalla`` project storage - `Discoverer Use Policy (DUP) `_ — §7 (storage) and §8 (retention) .. _`Lustre file system`: https://en.wikipedia.org/wiki/Lustre_(file_system)