S3 bucket storage#

https://en.wikipedia.org/wiki/Amazon_S3

Get an S3 account#

Create a request at 1234@epfl.ch with SCITAS S3 in the title, including the lab name you want to associate your account with. Having the lab name is important for billing purpose.

Prices are the same as /work, see Prices.

We will then create your S3 account and copy your credentials in a file named s3_cred.txt at the root of your /home/<user> directory on clusters.

This file will contain:

access_key_id = <account_access_key>
secret_access_key = <account_secret_access_key>

Those keys are personal and mandatory for your S3 client configuration. Once your client configuration is done, we advise to delete it from your /home on the clusters.

We support 3 ways to connect to your S3 buckets:

rclone (cli/available on SCITAS clusters frontend nodes)
s3cmd (cli)
Cyberduck (graphic interface)

rclone#

rclone configuration#

To use rclone with our S3 service, you need to create a configuration file at ~/.config/rclone/rclone.conf on your personal device or on a SCITAS cluster in your home directory:

vim ~/.config/rclone/rclone.conf

Then copy-paste this block in it:

[scitas-object]
type = s3
provider = Other
access_key_id = <account_access_key>
secret_access_key = <account_secret_access_key>
endpoint = https://scitas-object.epfl.ch:443
acl = private

access_key_id and secret_access_key are parameters that will be given to you in your /home/<user> at your S3 account creation.

rclone from clusters#

rclone is available through a module on our compute clusters.

To be able to use it just activate the module on any frontend node:

module load rclone

Then you can check if it has been effectively activated:

$ module list

Currently Loaded Modules:
  1) rclone/1.65.2

rclone basic commands#

Create a bucket with rclone#

This is the first thing you will have to do before anything: $ rclone mkdir scitas-object:<bucket>
Example:

$ rclone mkdir scitas-object:test-bucket2
$ rclone lsd scitas-object:
          -1 2025-01-22 13:32:17        -1 test-bucket
          -1 2025-01-28 15:01:28        -1 test-bucket2
          -1 2025-01-23 14:31:24        -1 test-server

List your buckets with rclone#

Next, verify by using $ rclone lsd scitas-object: to list your buckets like this:
Example:

$ rclone lsd scitas-object:
    -1 2025-01-20 10:35:27        -1 test-bucket
    -1 2025-01-22 10:57:56        -1 test-bucket2

List files in bucket with rclone#

Get a list of file in a specific bucket: $ rclone ls scitas-object:<bucket>
Example:

$ rclone ls scitas-object:test-bucket
  2425 rack.drawio

Upload a file in your bucket with rclone#

Use this command to copy file to your bucket
$ rclone copy <file> scitas-object:<bucket>
Example:

$ rclone copy awscliv2.zip scitas-object:test-bucket2
$ rclone ls scitas-object:test-bucket2
 67605287 awscliv2.zip

Download a file from a bucket with rclone#

Use this command:
$ rclone copy scitas-object:<bucket>/<path_to_file> <destination_path>

$ rclone ls scitas-object:test-bucket
104857600 test1
 10485760 test2
 20971520 test3
$ rclone copy scitas-object:test-bucket/test1 .
$ ls -lh test1
total 100M
-rw-r--r-- 1 fgagnepain fgagnepain 100M 22 janv. 11:26 test1

Delete a file in a bucket with rclone#

Use this command: $ rclone delete scitas-object:<bucket>/<file>
Example:

$ rclone ls scitas-object:test-bucket
104857600 test1
 10485760 test2
 20971520 test3
$ rclone delete scitas-object:test-bucket/test1
$ rclone ls scitas-object:test-bucket
 10485760 test2
 20971520 test3

Synchronize a folder in a bucket with rclone#

Upload#

Use this command to synchronise a folder from your local source towards your bucket:
$ rclone sync /path/to/folder scitas-object:<bucket>/path/to/destination
Example:

$ ls -lh stuff/
total 130M
-rw-r--r-- 1 fgagnepain fgagnepain 100M 22 janv. 11:26 test1
-rw-r--r-- 1 fgagnepain fgagnepain  10M 22 janv. 11:25 test2
-rw-r--r-- 1 fgagnepain fgagnepain  20M 22 janv. 11:26 test3

$ rclone sync /home/fgagnepain/stuff scitas-object:test-bucket/stuff
$ rclone ls scitas-object:test-bucket
104857600 stuff/test1
 10485760 stuff/test2
 20971520 stuff/test3

Download#

Use this command to synchronise a folder from your local source towards your bucket:
$ rclone sync /path/to/folder scitas-object:<bucket>/path/to/folder /your/local/path
Example:

$ rclone sync scitas-object:test-bucket/stuff /home/fgagnepain/tata
$ ls -lh /home/fgagnepain/tata
total 130M
-rw-r--r-- 1 fgagnepain fgagnepain 100M 22 janv. 11:26 test1
-rw-r--r-- 1 fgagnepain fgagnepain  10M 22 janv. 11:25 test2
-rw-r--r-- 1 fgagnepain fgagnepain  20M 22 janv. 11:26 test3

s3cmd#

s3cmd configuration#

Run s3cmd --configure You will be asked to enter a number of parameters.

Here is a example of what you should answer:

$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3.
Leave them empty for using the env variables.
Access Key: <your_access_key>
Secret Key: <your_secret_key>
Default Region [US]: EU

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: scitas-object.epfl.ch

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3.
"%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing
a bucket [%(bucket)s.s3.amazonaws.com]: 443

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: <your_access_key>
  Secret Key: <your_secret_key>
  Default Region: CH
  S3 Endpoint: scitas-object.epfl.ch
  DNS-style bucket+hostname:port template for accessing a bucket: 443
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/home/<user>/.s3cfg'

If nothing is provided in our example, just press enter.

s3cmd basic commands#

List your buckets witch s3cmd#

Use s3cmd ls

$ s3cmd ls
2025-01-22 12:32  s3://test-bucket

Create a bucket with s3cmd#

Use s3cmd mb s3://<bucket>

$ s3cmd mb s3://test-bucket2
Bucket 's3://test-bucket2/' created
$ s3cmd ls
2025-01-22 12:32  s3://test-bucket
2025-01-23 12:28  s3://test-bucket2

Delete a bucket with s3cmd#

Bucket must be empty before being deleted
Use s3cmd rb s3://<bucket>

$ s3cmd rb s3://test-bucket2
Bucket 's3://test-bucket2/' removed
$ s3cmd ls
2025-01-22 12:32  s3://test-bucket

List the contents of the bucket with s3cmd#

Use s3cmd ls s3://<bucket>

$ s3cmd ls s3://test-bucket
2025-01-22 12:35    104857600  s3://test-bucket/test1
2025-01-22 12:35     10485760  s3://test-bucket/test2
2025-01-22 12:35     20971520  s3://test-bucket/test3

Upload a file into a bucket with s3cmd#

Run s3cmd put <file> s3://test-bucket/<file>

$ s3cmd put test_file s3://test-bucket/test_file
upload: 'test_file' -> 's3://test-bucket/test_file'  [part 1 of 7, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    64.28 MB/s  done
upload: 'test_file' -> 's3://test-bucket/test_file'  [part 2 of 7, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    77.38 MB/s  done
upload: 'test_file' -> 's3://test-bucket/test_file'  [part 3 of 7, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    84.84 MB/s  done
upload: 'test_file' -> 's3://test-bucket/test_file'  [part 4 of 7, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    78.20 MB/s  done
upload: 'test_file' -> 's3://test-bucket/test_file'  [part 5 of 7, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    85.90 MB/s  done
upload: 'test_file' -> 's3://test-bucket/test_file'  [part 6 of 7, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    81.55 MB/s  done
upload: 'test_file' -> 's3://test-bucket/test_file'  [part 7 of 7, 10MB] [1 of 1]
 10485760 of 10485760   100% in    0s    78.54 MB/s  done

NB: you can change the name of the destination file in the command.

Download a file from the bucket with s3cmd#

Retrieve the file back with s3cmd get s3://logix.cz-test/<file> <file>

$ s3cmd get s3://test-bucket/test_file test_file_back
download: 's3://test-bucket/test_file' -> 'test_file_back'  [1 of 1]
 104857600 of 104857600   100% in    1s    97.72 MB/s  done

NB: Here I changed the destination file name in the command.

Cyberduck#

Activate deprecated S3 profile#

To connect to your S3 account with Cyberduck, you will have to activate the "S3 (Depcrated path style requests)" profile.

Go to Preferences > Profiles and search for "Deprecated".

Then select "S3 (Depcrated path style requests)":

Create a bookmark to access your scitas S3 account rapidly#

Go to Bookmark > Right click > New bookmark and fill the fields like in the screen shot below:

Use the "S3 (Depcrated path style requests)" profile you just enabled before
Fill Server field with : scitas-object.epfl.ch
Port: 443
Use the Access Key and Secret Key provided at your account creation in your /home/<user> directory on clusters.

Create a bucket with Cyberduck#

To create a new bucket for your account, browse to the root and choose: File > New Folder

You can choose the bucket location in Preferences > S3

When creating the bucket, enter the name of your folder and choose region EU (Zürich)

Other services#

Central EPFL IT also provide an S3 service