Globus File Transfer Services
For large data transfers between NREL’s high-performance computing (HPC) systems and another data center, or even a laptop off-site, we recommend using Globus.
What Is Globus?
Globus provides services for research data management, including file transfer. It enables you to quickly, securely and reliably move your data to and from locations you have access to.
Globus transfers files using GridFTP. GridFTP is a high-performance data transfer protocol which is optimized for high-bandwidth wide-area networks. It provides more reliable high performance file transfer and synchronization than scp or rsync. It automatically tunes parameters to maximize bandwidth while providing automatic fault recovery and notification of completion or problems.
Get a Globus Account
To get a Globus account, sign up on the Globus account website.
To get access to the NREL endpoint (nrel#globus), please contact us.
If you would like to move files to or from Peregrine, please request that a directory be created for you that is accessible to Globus. This directory will be be /scratch/globus/<username>. If you need to move files to or from storage areas not associated with Peregrine, please contact us to discuss options.
You will activate the NREL endpoint (nrel#globus) using your NREL HPC username and password (not the globus.org username/password).
Globus can be used either through the "Globus Online" service or via a CLI (command line interface).
Globus Online is a hosted service that allows you to use a browser to transfer files between trusted sites called "endpoints". To use it, the Globus software must be installed on the systems at both ends of the data transfer. The NREL endpoint is nrel#globus.
- Click Login on the Globus web site. On the login page select "Globus ID" as the login method and click continue. Use the Globus credentials you used to register your Globus.org account.
- Go to the Transfer Files page, the link is located under the Manage Data tab at the top of the page.
- Select nrel#globus as the endpoint on one right side. In the box asking for authentication, enter your Peregrine (NREL HPC) username and password. Do not use your globus.org username and password when authenticating with the nrel#globus endpoint.
- Select another Globus endpoint, such as a personal endpoint or an endpoint at another institution that you have access to. To use your personal endpoint, first start the Globus Connect Personal application. Then enter "USERNAME#ENDPOINT" on the left side or use the drop down menu to find it. Click "go".
- To transfer files
- select the files you want to transfer someplace else from the system from the dialog box on the left.
- select the destination location (a folder or directory) from the dialog box on right right.
- click the large blue button at the top of the screen to begin to transfer the files
When your transfer is complete, you will be notified by email.
Configuring your Globus.org account to allow ssh CLI access
To use the CLI you must have a Globus account with ssh access enabled. To enable your account for ssh access you must add your ssh public key to your Globus account by visiting the Manage Identities page and clicking "manage SSH and X.509 keys" and then "Add a New Key". If you do not have an ssh key, follow the directions here to create one.
Globus.org CLI examples
ssh <globus username>@cli.globusonline.org <command> <options> <params>
ssh <globus username>@cli.globusonline.org help
A one-liner can be used to integrate globus.org CLI commands into shell scripts
ssh <globus username>@cli.globusonline.org scp nrel#globus:/globusro/file1.txt myuser#laptop:/tmp/myfile.txt
The globus.org CLI can be used interactively
$ ssh <globus username>@cli.globusonline.org
Welcome to globusonline.org, dwhitesi. Type 'help' for help.
$ scp nrel#globus:/globusro/file1.txt myuser#laptop:/tmp/myfile.txt
You can find more information on the Globus CLI from the official Globus CLI documentation.
Globus Personal Endpoints
You can set up a "Globus Connect Personal EndPoint", which turns your personal computer into an endpoint, by downloading and installing the Globus Connect Personal application on your system. We use a personal endpoint to demonstrate how to transfer files to and from Peregrine.
Set Up a Personal EndPoint
- Login to the Globus website. From the Manage Data drop down menu, select Transfer Files. Then click Get Globus Connect Personal.
- Pick a name for your personal endpoint and select Generate Startup Key. Follow the instructions on the web page to save your key.
- Download and install the Globus Connect Personal software on your personal system.
- copy the startup key from the Globus web page to this application.
Once Globus Connect Personal is installed on your system, set up the permissions for reading or writing files to your local system. If you are using a Mac, click on the "g" icon on the upper right portion of your screen to access the Globus Connect Personal application. Select Preferences. To allow Globus to copy files to your local system, make sure that the directory (folder) they will go in is Writable.