File Systems#
| File system | Purpose | Features | Access | Cost | Backup | Data Retention Policy |
|---|---|---|---|---|---|---|
/home |
Secure file system for the user's critical information. | Boasts data replication and robust backup mechanisms. | Accessible from all SCITAS clusters. Limited to 100 GB per user. | Free of charge. | Yes | Data will be erased after two years of inactivity or 6 months after leaving EPFL, whichever occurs first. |
/work |
Team storage, ideal for users needing substantial storage capacity, particularly for research data retention over several years. | Efficient file system with significant storage capacity. Option for data backup available upon request. | Accessible from all SCITAS clusters. Quota: 20 TB, may be increased upon request. | Pay-per-use model | Available upon request | Data will be erased 6 months after the cessation of payment. |
/export |
Provides exported spaces extending accessibility beyond the clusters. | Efficient file system with significant storage capacity. Versatile options with specific protocols for seamless access and data sharing. Option for data backup available upon request. | These file systems offer seamless integration, being not only accessible from all SCITAS clusters but also available to any computer connected to the EPFL network. They support a range of protocols including NFS, SMB, and S3. | Pay-per-use model | Available upon request | Data will be erased 6 months after the cessation of payment. |
/transfer |
Temporary storage during data transfers between SCITAS and other global HPC centers. | Optimized to quickly process a large volume of inodes, enhancing the speed of data transfers. | Accessible from the front-end nodes and the specialized machine fdata1.epfl.ch |
Usage is included in the compute services. | No | Files older than 15 days will be automatically removed without notice. |
/archive |
Preservation of cold data and very long-term storage exceeding 10 years, ideal for research data that does not require rapid restoration. | Slow tape-based file system to offer extensive storage capacity. For maximum security, data is automatically duplicated across two sites. | Accessible from all SCITAS clusters. | Pay-per-use model | No | Data will be erased 6 months after the cessation of payment. |
/scratch |
Temporary storage needs and high-speed data processing. | Offers high-performance storage with faster read/write capabilities, suitable for computation-intensive tasks. | Each /scratch space is cluster-specific (/scratch on cluster A is not accessible from cluster B, and vice versa), optimized for swift data processing and temporary storage. |
Usage is included in the compute services. | No | Files older than 30 days are automatically removed without notice. |
/tmp/${SLURM_JOB_ID} |
Local temporary storage needs for ultra high-speed data processing tasks. | This high-performance storage solution offers fast read/write speeds, perfect for sequential jobs on a single compute node. | Exclusively available from the compute node itself. It is not accessible from login nodes. | Usage is included in the compute services. | No | Data will be automatically erased at the end of the job. |
Security of user data#
The contents of the /home file system are backed-up on a daily basis with a
six month retention period. The backed-up data are held at a separate physical
location to the original data.
The contents of the /work file system are not backed up by default.
Users can easily enable backup services by creating a .backup file in the
main directory. For example, to initiate backup for SCITAS data, users should
execute the command:
This action will ensure that the entire /work/scitas directory is backed up
starting from the day following the .backup file's creation.
The /scratch file systems are not backed up under any circumstances. It is
the user's responsibility to transfer important data to a more permanent storage
space. The /scratch file systems are only for short-lived files and, in the
case of insufficient free space, files older than two weeks will be deleted
without notice in order to ensure the usability of the cluster (see Scratch
automatic cleanup).
Scratch automatic cleanup#
If the /scratch file system gets full, Slurm automatically drains all the
running jobs. To prevent this, we have established:
- A 30-day file deletion policy.
- Occupancy limits that trigger a series of automated actions.
- Policy compliance rules to ensure fair resource sharing.
30-day file deletion policy#
Files older than 30 days are automatically removed daily. Please do not use the
/scratch file system for long-term storage.
Please transfer regularly your important data to a more permanent location, for
example, to the /work file system or to your personal storage.
Occupancy limits#
- At 60% occupancy, we urge immediate action to delete unnecessary data.
- If occupancy reaches 80%, we will take proactive measures to reduce it. This will involve removing files and folders, and prioritizing the oldest ones first, regardless of their age. We will continue this process until occupancy falls below 80%.
- Please note, this may include the removal of files less than 30 days old.
Policy compliance#
Use of the touch command or similar techniques to prevent file deletion is
strictly prohibited. This action contravenes our community's
resource-sharing principles.