Skip to content

Transferring data#

To transfer data from or to a SCITAS cluster, you may use either rsync or scp. While they both allow for a secure transfer of your data over the SSH protocol, we encourage you to use rsync for the reasons presented below.

Note

In both cases, the data retrieving command must be executed from your computer.

MS Windows and Apple macOS users wishing to use a GUI application, may have a look at:

Using rsync#

Remote synchronization (or rsync) is a utility for synchronizing files between a source and a destination. Data can be compressed before synchronization and rsync works using an incremental protocol. This means that:

  • It will only synchronize data that has changed, thus reducing the network usage and transfer time
  • It can resume the synchronization from where it was in case of an interruption.

The basic usage for rsync is the following

rsync [options] <src> <dest>

The typical options you can use are the following:

  • -a: archive mode. Among other things, it copies recursively, keeps symlinks, preserves file permissions and modification times.
  • -z: use data compression.
  • -P: show progress

To transfer data from your computer to a SCITAS machine, use the following command:

rsync -azP /path/to/src <GASPAR>@<machine>.hpc.epfl.ch:/path/to/dest
where <GASPAR> is your GASPAR username and <machine> is the machine name.

To do the inverse (from a SCITAS machine to your computer), do the following:

rsync -azP <GASPAR>@<machine>.hpc.epfl.ch:/path/to/src /path/to/dest

Trailing /

rsync makes a difference if you put a trailing / at the end of the source path. For example, if you want to transfer a folder data located at /home/user/data, the following two commands will act differently:

  • rsync -azP /home/user/data <dest>: will transfer the folder data into <dest>, meaning that you will end up with <dest>/data
  • rsync -azP /home/user/data/ <dest>: will transfer the content of data into <dest>, meaning that you will end up with <dest>/<data_content>

rsync for local copy

rsync is not limited to synchronizing files over a network. You can also use it to synchronize files locally. This and its options make it very convenient as a backup tool.

Using scp#

Secure copy (or scp) is a tool used to copy files between hosts over a network. It acts very similarly as the usual cp command. Contrarily to rsync, scp will always transfer the whole data and cannot resume in case of an interuption.

The syntax is the following: 

scp [options] <src> <dest>
as for cp, the main option you may use is -r for a recursive copy. Other than that, the usage is similar as rsync presented adove.

Problem that may occur with scp

Should you have to copy a directory from your local computer to the remote server, you may encounter the following error:
scp: realpath ./[Your_local_dir_name]: No such file
scp: upload "./[Your_local_dir_name]": path canonicalization failed
scp: failed to upload directory [Your_local_dir_name] to .

Try with the -O option: 

$ scp -r -O My_dir username@jed.hpc.epfl.ch: