Globus File Transfer Services
For large data transfers between NREL’s high-performance computing (HPC) systems and HPC Mass Storage, data centers, or even a laptop off-site, we recommend using Globus.
What Is Globus?
Globus provides services for research data management, including file transfer. It enables you to quickly, securely and reliably move your data to and from locations you have access to.
Globus transfers files using GridFTP. GridFTP is a high-performance data transfer protocol, which is optimized for high-bandwidth wide-area networks. It provides more reliable high-performance file transfer and synchronization than scp or rsync. It automatically tunes parameters to maximize bandwidth while providing automatic fault recovery and notification of completion or problems.
Get a Globus Account
To get a Globus account, sign up on the Globus account website.
Once you have a GlobusID, you will have access to the NREL endpoints via your NREL HPC user account credentials. Please submit a help ticket to HPC-Help@nrel.gov or contact us if you have any questions or issues.
If you need to move files to or from storage areas not associated with NREL HPC, please contact us to discuss options.
You will activate an NREL endpoint (nrel#globus) using your NREL HPC username and password (not the globus.org username/password).
These are the current NREL Globus Endpoints
- nrel#globus - This endpoint will give you access to any files you have on Peregrine or Eagle:/scratch and /projects.
- nrel#globus-s3 - This endpoint allows you to copy files to/from AWS S3 buckets.
- nrel#globus-mss - This endpoint allows you to copy files to/from NREL’s Mass Storage System (MSS).
- nrel#eglobus1; nrel#eglobus2; nrel#eglobus3. These endpoints allow you to transfer files to/from Eagle’s /scratch, /projects, and your Eagle /home directory
Globus Personal Endpoints
You can set up a "Globus Connect Personal EndPoint", which turns your personal computer into an endpoint, by downloading and installing the Globus Connect Personal application on your system. You may use 'Globus Connect Personal' to transfer between your system and any NREL Globus Endpoint (e.g. nrel#globus)
Set Up a Personal EndPoint
- Login to the Globus website. From the left globus menu, select Endpoints > Create new endpoint (top right).
- Pick a name for your personal endpoint and select Generate Startup Key. Follow the instructions on the web page to save your key.
- Download and install the Globus Connect Personal software on your personal system.
- Copy the startup key from the Globus web page to this application.
Once Globus Connect Personal is installed on your system, set up the permissions for reading or writing files to your local system. If you are using a Mac, click on the "g" icon on the upper right portion of your screen to access the Globus Connect Personal application. Select Preferences. To allow Globus to copy files to your local system, make sure that the directory (folder) they will go in is Writable.
Globus can be used either through the "Globus Online" service or via a CLI (command line interface).
Globus Online is a hosted service that allows you to use a browser to transfer files between trusted sites called "endpoints." To use it, the Globus software must be installed on the systems at both ends of the data transfer. The following is an example for moving project data from Peregrine (endpoint nrel#globus) to HPC Mass Storage (endpoint nrel#globus-mss.)
- Click Login on the Globus website. If redirected to the Web App page, select "Use Globus ID to sign in" as the login method and click continue. Use the Globus credentials you used to register your Globus.org account.
- Use File Manager (left menu) and search Collection for: nrel.
- Select nrel#globus as the endpoint. In the box asking for authentication, enter your NREL HPC username and password. Do not use your globus.org username and password when authenticating with the nrel#globus endpoint.
- Select Transfer or Sync to... and in the new panel, search for another Globus endpoint (nrel) and select nrel#globus-mss. (To use your personal endpoint, first start the Globus Connect Personal application. Then enter the endpoint name or your username in the Collection search bar and select your endpoint.) After the first use, you should see your endpoints in the Recent tab. You may also setup an endpoint/directory as a bookmark.
- To transfer files
- select the files you want to transfer from the nrel#globus endpoint. (Highlight as many dirs/files as you need.)
- select the destination location (a folder or directory) from the dialog box on right right.
- click the large blue Start button at the bottom of the screen to begin to transfer the files (be sure to verify the direction of transfer using the Start button from the source.)
When your transfer is complete, you will be notified by email.
For a detailed Globus walkthrough, please see Moving Data from Peregrine to Eagle using Globus.
Using the Globus CLI interface
Globus CLI is installed on both Peregrine and Eagle. You may want to utilize this method to automate transfers to Mass Storage.
These instructions assume that you already have a GlobusID and are using bash. See globus <command> --help for further usage information
- To login, use globus login
You will be given an authentication link. Open this link in your web browser and authenticate. You will be given a one time authentication code to paste in your terminal.
Once you authenticate successfully, the globus CLI utility implicity uses what is known as a "refresh token." If you issue any globus commands with this utility you will remain authenticated for 6 months after the last globus command, meaning you will not need to visit a URL like above in browser for up to 6 months of inactivity.
- Run globus whoami to determine the account used.
- You will need to activate endpoints to transfer between:
$ globus endpoint search nrel --filter-owner-id firstname.lastname@example.org
ID | Owner | Display Name
------------------------------------ | -------------------| ----------------
e55b4d7c-6d04-11e5-ba46-22000b92c6ec | email@example.com | nrel#globus
4aa57cfa-1039-11e9-9833-0262a1f2f698 | firstname.lastname@example.org | nrel#globus-mss
You will need to authenticate using your HPC User account (not your GlobusID to access endpoint data.)
$ globus endpoint activate --myproxy $nrel_globus
Myproxy username: hpc_user
Myproxy password: ************
Endpoint activated successfully using a credential fetched from a MyProxy server.
$ globus endpoint activate --myproxy $nrel_mss
... (enter credentials)
$ globus transfer -r --preserve-mtime --sync-level=[exists|size|mtime|checksum] $nrel_globus:/projects/handle/dir $nrel_mss:/nfs/mss/projects/handle/dir
$ globus task list
$ globus task show <task ID>
Please reference the official Globus CLI documentation.