DartFS archive example to Google Cloud Platform

Overview

The following document outline steps a researcher can take to archive their DartFS storage to Google Cloud Platform (GCP) storage.

Configuring using SSH Tunnel

Open up a terminal window to allow for authorization of rclone to GCP

​​​​​​ssh -L localhost:53682:localhost:53682 YOUR_NETID@andes

Configure rclone integration for Google Cloud Platform

Here is an example of how to make a rclone remote called gcp. The initial setup involves authorizing access to the cloud storage from your browser. rclone config walks you through the process.
The setting choices for object_acl Option bucket_acl and storage_class>are dependent upon the options you chose when setting up your bucket.

​​​​​rclone config
​n/s/q> n
name> gcp (name can be whatever you wish)

Type of storage to configure.
XX / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
storage > google cloud storage
client_id> <ENTER>
client_secret> <ENTER>
This section you will need to get your project number from the GCP console
project_number>  <YOUR_PROJECT_NUMBER>
user_project> <ENTER>
service_account_file> <ENTER>
anonymous> <ENTER>
object_acl> 2
bucket_acl> 2
bucket_policy_only> <ENTER>
location> us-east1
storage_class> NEARLINE
env_auth> <ENTER>
Edit advanced config? n
Use web browser to automatically authenticate rclone with remote?
y/n> y

Then copy and paste the auth url http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx to the browser on your local machine, complete the auth steps below and it is done.

Allow rclone to access your account 

Success!

Test

See all the buckets in your project: Note you must have provided a project number during config to display this information 

rclone lsd gcp:gcp
-1 1999-12-31 19:00:00        -1 bucketname

References

• https://rclone.org/googlecloudstorage/
• https://rclone.org/install/
• https://rclone.org/remote_setup/