rclone is an alternative of rsync for cloud storage. It can be applied to remote servers, OneDrive, Google Drive, Box, … (almost all existing cloud storage). The supported protocols:
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others
\ (s3)
5 / Backblaze B2
\ (b2)
6 / Better checksums for other remotes
\ (hasher)
7 / Box
\ (box)
8 / Cache a remote
\ (cache)
9 / Citrix Sharefile
\ (sharefile)
10 / Combine several remotes into one
\ (combine)
11 / Compress a remote
\ (compress)
12 / Dropbox
\ (dropbox)
13 / Encrypt/Decrypt a remote
\ (crypt)
14 / Enterprise File Fabric
\ (filefabric)
15 / FTP
\ (ftp)
16 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
17 / Google Drive
\ (drive)
18 / Google Photos
\ (google photos)
19 / HTTP
\ (http)
20 / Hadoop distributed file system
\ (hdfs)
21 / HiDrive
\ (hidrive)
22 / ImageKit.io
\ (imagekit)
23 / In memory object storage system.
\ (memory)
24 / Internet Archive
\ (internetarchive)
25 / Jottacloud
\ (jottacloud)
26 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ (koofr)
27 / Linkbox
\ (linkbox)
28 / Local Disk
\ (local)
29 / Mail.ru Cloud
\ (mailru)
30 / Mega
\ (mega)
31 / Microsoft Azure Blob Storage
\ (azureblob)
32 / Microsoft Azure Files
\ (azurefiles)
33 / Microsoft OneDrive
\ (onedrive)
34 / OpenDrive
\ (opendrive)
35 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)
\ (swift)
36 / Oracle Cloud Infrastructure Object Storage
\ (oracleobjectstorage)
37 / Pcloud
\ (pcloud)
38 / PikPak
\ (pikpak)
39 / Proton Drive
\ (protondrive)
40 / Put.io
\ (putio)
41 / QingCloud Object Storage
\ (qingstor)
42 / Quatrix by Maytech
\ (quatrix)
43 / SMB / CIFS
\ (smb)
44 / SSH/SFTP
\ (sftp)
45 / Sia Decentralized Cloud
\ (sia)
46 / Storj Decentralized Cloud Storage
\ (storj)
47 / Sugarsync
\ (sugarsync)
48 / Transparently chunk/split large files
\ (chunker)
49 / Uloz.to
\ (ulozto)
50 / Union merges the contents of several upstream fs
\ (union)
51 / Uptobox
\ (uptobox)
52 / WebDAV
\ (webdav)
53 / Yandex Disk
\ (yandex)
54 / Zoho
\ (zoho)
55 / premiumize.me
\ (premiumizeme)
56 / seafile
\ (seafile)
I’ve used this to rsync data files between my Azure (in Microsoft Intranet) and my School Server (in school intranet that requires VPN) using OneDrive as the exchange. I’ll use another blog to talk about this.
Recently, our lab purchases Surgery NAS. And I need to move some of my data from remote server to the NAS. Currently, I cannot directly mount the NAS to the remote server. So I try the alternative way, i.e. mounting NAS on my Mac and use rsync to transfer.
However, I find the parallelism of rsync is too complicated after I searched for some blogs. Luckily, I find rclone can support multiple connections in a very easy way. So I config the rclone on my Mac and use it for transfer.
Config
There is a menuconfig UI by typing in rclone config after installation. We can also directly edit ~/.config/rclone/rclone.conf for configuration. We use sftp/ssh for remote server and here is one example
[hku-a100]
type = sftp
host = YOUR_HOST
user = YOUR_USR_NAME
key_file = YOUR_PATH2PRIVATE_KEY
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum
However, our remote server requires a ProxyJumper when connected. The key is to add --sftp-ssh to specify the connection explicitly. And the overall ⌘ is
rclone --sftp-ssh "ssh YOUR_HOST" copy YOUR_HOST:PATH LOCAL_PATH --ignore-existing --transfers=32 --checkers=8 --multi-thread-streams=4 -P
We use ignore-existing to omit the existing files transferred. The --transfers denotes how many files to transfer at a time, and --checkers denotes how many checks to be used. --multi-thread-streams is used to split them into multiple streams, allowing different parts of the same file to be downloaded/uploaded at once.
Note: the destination is a bit different. In rsync, if we want to copy .../A/folder to .../B/folder, the destination should be .../B/. However, in rclone, it should be .../B/folder.