Setting up rclone for the Cell Browser
rclone is 'rsync for cloud services' and allows you to manage your files on various cloud services. For the Cell Browser, it's most useful for downloading files from cloud storage providers via the command line.
Installation
You should install rclone using conda in a new environment:
conda create rclone python=3.9 conda activate rclone conda install -c conda-forge mamba mamba install -c conda-forge rclone
Setup
Once you have rclone installed, you will have to go through some steps to get it to work with Box, Google Drive, Dropbox, and any other online services submitters might use. Would recommend just setting up new profiles (or ‘remotes’ as rclone calls them) as needed.
Here are links on setting up rclone with two of the most common cloud storage provides Cell Browser submitters use:
In both cases, you’ll essentially keep the default settings for everything except for this step:
Use auto config? * Say Y if not sure * Say N if you are working on a remote or headless machine
At this step, say ‘No’ as hgwdev is a remote machine. You will be given a URL to go to on your personal computer. To finish this step of the setup, you will need rclone installed on your personal computer. If you’re on a Mac with OSX, it’s easiest to do so with homebrew, which after you install that, you can do: brew install rclone. They also provide executables for Windows: https://rclone.org/downloads/.
Downloading a file
To download a file using rclone, it has to be in your Drive/Box/whatever. You can’t just provide it with a link to a public file. (Though who knows, maybe there is a way to do that!)
The command to download a file from Drive/Box/whatever is:
rclone copy /home/source remote:path/to/file.name
Where ‘remote’ is the name you gave that particular remote/profile during setup (e.g. work_gdrive) and ‘path/to/file.name’ is the path to the file you are downloading (e.g. cb_files/file.h5ad). The path will be relative to the top-level directory of that storage provider.
