Storage and Filesystem#
Our HPC systems provide several storage options for different stages of a research workflow. Use your home directory for persistent files, shared scratch storage for datasets and job input/output, and temporary storage for data that only needs to exist while a job is running.
Storage quotas and paths differ between systems. Scratch and temporary storage are not backed up, so keep an independent copy of any data you cannot afford to lose.
Storage at a glance#
Aire#
Storage type |
Path |
Environment variable |
Quota or capacity |
Backed up? |
Automatically deleted? |
Recommended use |
|---|---|---|---|---|---|---|
Home directory |
|
|
65 GB and 1.5 million files per user |
Yes |
No |
Scripts, notes, documentation and configuration files |
Scratch on Lustre (disk-based) |
|
|
1 TB and 1.5 million files per user |
No |
No |
Large datasets and active project data |
Flash on Lustre (NVMe-based) |
|
|
1 TB and 1.5 million files per job |
No |
Yes, at job completion |
I/O-intensive work that benefits from high-performance shared storage |
Scratch on compute nodes |
|
|
No fixed quota; limited by available space on the node |
No |
Yes, at job completion |
Fast, node-local temporary data for single-node jobs |
Calder#
Storage type |
Path |
Environment variable |
Quota or capacity |
Backed up? |
Automatically deleted? |
Recommended use |
|---|---|---|---|---|---|---|
Home directory |
|
|
Shared with Aire |
Yes |
No |
Scripts, notes, documentation and configuration files |
Scratch on Lustre (disk-based) |
|
Not currently defined |
1 TB and 1.5 million inodes per project |
No |
No |
Large datasets and active project data |
Flash on Lustre (NVMe-based) |
|
Not currently defined |
1 TB and 1.5 million inodes per project |
No |
Confirm the lifetime of files for your job before relying on them |
I/O-intensive work that benefits from high-performance shared storage |
On Calder, quotas for both scratch filesystems are applied per Slurm project. The current limits are 1 TB and 1.5 million inodes per project on each filesystem. The $SCRATCH and $TMP_SHARED environment variables are not currently defined on Calder, so use the paths listed above. Aire 2.0 will also use project-based quotas; the current Aire system uses per-user quotas. For instructions on checking your quota and usage, see Filesystem Quotas. If you are unsure which project quota applies to your work, contact the Research Computing team.
The home directory is shared between Aire and Calder, so files stored in /users/<username> are available on both systems. This does not mean that the scratch filesystems are shared: use the appropriate path for the system on which you are working.
Total filesystem capacity#
The figures below describe the overall capacity of each filesystem, not the quota available to an individual user or project.
Aire#
Filesystem |
Total space |
Total inodes |
|---|---|---|
Home directory |
106 TB |
2,269,138,752 |
Scratch on Lustre |
3.7 PB |
2,997,485,568 |
Flash on Lustre |
139 TB |
293,022,729 |
Scratch on compute nodes |
372 GB per node |
24,838,144 per node |
Calder#
Filesystem |
Total space |
|---|---|
Home directory |
Shared with Aire |
Scratch on Lustre ( |
Approximately 6.3 PB |
Flash on Lustre ( |
Approximately 246 TB |
When Aire 2.0 launches, Aire’s scratch and flash paths are expected to change to /scratch-aire and /flash-aire respectively, distinguishing them from Calder’s filesystems. Check the updated documentation when the Aire 2.0 upgrade is complete.
Choosing and managing storage#
Use your home directory for persistent, relatively small files. It is backed up, but is not intended for high-I/O workloads.
Use disk-based scratch for large datasets and active work. It is not backed up and files are not automatically deleted, so remove data you no longer need and keep important data elsewhere.
Use flash scratch for jobs with demanding I/O requirements. It uses NVMe-based storage and is not backed up. Treat it as temporary job storage and check the relevant job environment and file lifetime before using it.
Use compute-node scratch for node-local temporary files. It is not shared between nodes, so a file written to one node’s
/tmpis not automatically available on another node.Do not treat any scratch area as your only copy of important data. Copy important results to backed-up storage or an appropriate external storage location.
Monitor both space and file counts. A quota can be reached by using too much storage or by creating too many files.
For more guidance, see File and Data Management, including Best Practices and Filesystem Quotas.
When a filesystem becomes full#
Per-user or per-project quotas help manage individual usage, but they do not guarantee that the filesystem as a whole will have free space. Quotas may be oversubscribed because not every user or project uses its full allowance at the same time. As a result, a filesystem can become critically full even when some users are within their quotas.
As filesystem usage approaches capacity, performance and reliability can be affected. At high utilisation, users may experience slower jobs, failed writes or incomplete output. If a filesystem reaches 100% usage, write operations may fail with No space left on device, jobs may fail, and files being written may be truncated or corrupted.
If a filesystem becomes critically full, Research IT may need to take urgent action to recover space and protect the service. This may include:
Temporarily suspending job scheduling so that additional jobs do not increase the pressure on the filesystem.
Emailing users to request urgent data cleanup.
Removing files to recover space, potentially without individual warning in an emergency.
Rebooting or otherwise recovering affected services if necessary.
Warning
Emergency recovery actions may temporarily interrupt access to the system. In exceptional circumstances, immediate action may be necessary to protect the wider service, and files may be removed without warning. Keep independent copies of important data and respond promptly to Research IT notices.