How to use the file systems#

As detailed in the file systems documentation, there are six main file systems that you can use for different purposes: /home, /work, /export, /transfer, /archive and /scratch.

The following sections illustrate how to manage your data in different scenarios.

Transferring data#

The first case is for transferring data between two machines. We need to distinguish two particular situations that are treated differently, mainly for security reasons. The first scenario is when you want to transfer some data between two computers that are on the EPFL network, e.g. between your laptop (connected to the EPFL WiFi or via the VPN) and one of the clusters. For this, you can follow the simple procedure detailed in the Transferring data documentation.

If the second computer to/from which you want to send/receive the data is not on the EPFL network, e.g. a cluster from another university, you need to use the /transfer file system and a special data-transfer node, fdata1.epfl.ch, that is the only one exposed to the outside of the EPFL network.

The /transfer file system is a special file system optimized for data transfers. For security reasons it has a data retention policy of only 15 days (files older than 15 days will be automatically removed without notice). Furthermore, the /transfer file system is the only one that is mounted on all the SCITAS clusters and the fdata1.epfl.ch transfer node.

The procedure to transfer data externally to EPFL is the following:

1. From one of the clusters, place the data you want to send in the /transfer/<username> folder, where <username> is the GASPAR ID, to make it available on the transfer node.
2. Log into fdata1.epfl.ch and transfer your data in /transfer/<username> using your favorite protocol, e.g. scp or rsync.

You can also copy data to the SCITAS clusters using the same procedure, but in reverse order. First transfer the data to fdata1.epfl.ch and then copy it back where you would like to store them from one of the clusters.

Storing data#

Numerical simulations often lead to some substantial amount of data results that need to be stored on different time scales. For example, you may need to store large restart files for a few days in order the continue a simulation or you may need to archive simulation results for decades in order to publish your results. For these different scenarios, we provide access to different file systems.

Short-term storage#

For short-term storage you can use either of the /scratch, /home, or /work file systems. They each have different properties that you need to take into account:

• The /home file system is your personal space with a free quota of 100GB that is backed up. It is mainly used to store small files such as source code or input files.
• The /scratch file system is a personal space that is mainly used to temporarily store your simulation results. Because of its high-performance storage capacities, your simulations should used it as an I/O space. It is not charged and has no quota, but a retention policy of 15 days. All the files older than 15 days will be deleted without notice.
• The /work file system is a collaborative space that you can use to store the large files and share them with your colleagues. There is no quota as it is a pay-per-use system (you will be billed for the space you actually use) and it can easily enable backup services by creating a .backup file in the main directory.

Note that the first two spaces, /home and /scratch, are personal. You must not change the access rights in order to open them to your colleagues. If you need to share your files, please use the /work file system.

Medium-term storage#

For medium-term storage you can use either your /home or the /work file systems depending mainly on the size of your files and if you need to share them with your colleagues.

Long-term storage or archiving#

Currently, the only possibilities for long-term storage are your /home and /work. Please, note that apart from the size of your data, you must also consider the time frame of your storage. Indeed, your /home will only be kept as long as your are an active user and registered to EPFL. If you need to store data for a longer period, we strongly advise that you use the /work file system.

In the future, we will also activate the /archive file system that allows you to store your data on magnetic tapes. The main advantages are that the price per TB is lower than hard drives and magnetic tapes can store your data for decades.