File Systems#
| File system | Purpose | Features | Access | Cost | Backup | Data Retention Policy |
|---|---|---|---|---|---|---|
/home |
Secure file system for the user's critical information. | Boasts data replication and robust backup mechanisms. | Accessible from all SCITAS clusters. Limited to 100 GB per user. | Free of charge. | Yes | Data will be erased after two years of inactivity or 6 months after leaving EPFL, whichever occurs first. |
/work |
Team storage, ideal for users needing substantial storage capacity, particularly for research data retention over several years. | Efficient file system with significant storage capacity. Option for data backup available upon request. | Accessible from all SCITAS clusters. | Pay-per-use model | Available upon request | Data will be erased 6 months after the cessation of payment. |
/export |
Provides exported spaces extending accessibility beyond the clusters. | Efficient file system with significant storage capacity. Versatile options with specific protocols for seamless access and data sharing. Option for data backup available upon request. | These file systems offer seamless integration, being not only accessible from all SCITAS clusters but also available to any computer connected to the EPFL network. They support a range of protocols including NFS, SMB, and S3. | Pay-per-use model | Available upon request | Data will be erased 6 months after the cessation of payment. |
/transfer |
Temporary storage during data transfers between SCITAS and other global HPC centers. | Optimized to quickly process a large volume of inodes, enhancing the speed of data transfers. | Accessible from the front-end nodes and the specialized machine fdata1.epfl.ch |
Usage is included in the compute services. | No | Files older than 15 days will be automatically removed without notice. |
/archive |
Preservation of cold data and very long-term storage exceeding 10 years, ideal for research data that does not require rapid restoration. | Slow tape-based file system to offer extensive storage capacity. For maximum security, data is automatically duplicated across two sites. | Accessible from all SCITAS clusters. | Pay-per-use model | No | Data will be erased 6 months after the cessation of payment. |
/scratch |
Temporary storage needs and high-speed data processing. | Offers high-performance storage with faster read/write capabilities, suitable for computation-intensive tasks. | Each /scratch space is cluster-specific (/scratch on cluster A is not accessible from cluster B, and vice versa), optimized for swift data processing and temporary storage. |
Usage is included in the compute services. | No | Files older than 15 days will be automatically removed without notice. |
/tmp/${SLURM_JOB_ID} |
Local temporary storage needs for ultra high-speed data processing tasks. | This high-performance storage solution offers fast read/write speeds, perfect for sequential jobs on a single compute node. | Exclusively available from the compute node itself. It is not accessible from login nodes. | Usage is included in the compute services. | No | Data will be automatically erased at the end of the job. |
Security of user data#
The contents of the /home file system are backed-up on a daily basis with a
six month retention period. The backed-up data are held at a separate physical
location to the original data.
The contents of the /work file system are not backed up by default.
Users can easily enable backup services by creating a .backup file in the
main directory. For example, to initiate backup for SCITAS data, users should
execute the command:
This action will ensure that the entire /work/scitas directory is backed up
starting from the day following the .backup file's creation.
The /scratch file systems are not backed up under any circumstances. It is
the user's responsibility to transfer important data to a more permanent storage
space. The /scratch file systems are only for short-lived files and, in the
case of insufficient free space, files older than two weeks will be deleted
without notice in order to ensure the usability of the cluster (see Scratch
automatic cleanup).
Scratch automatic cleanup#
If the /scratch file system gets full, Slurm will automatically drains all the
running jobs. To prevent this, when the /scratch reaches 90% of used space, an
automatic cleanup procedure is activated. Starting with the oldest files
present, deletion takes place until the occupancy has been reduced to 70%. Only
files less than 2 weeks old are sure not to be deleted by the cleanup
mechanism.
Make sure to transfer your important data to a more permanent location, e.g.
/work or your personal storage.