Skip to main content

Data Storage and Transfer

Learn about storing and transferring data on NREL High-Performance Computing (HPC) systems.

Photo of computer server wiring and lights, blurred to show movement.

Globus File Transfer Services

Use for large data transfers from NREL’s HPC systems to other systems.

Mass Storage System — Gyrfalcon

Gyrfalcon is NREL’s primary system for storing long-term user data.

WinSCP for Windows File Transfers

Use to transfer files from a local computer to a remote computer.

Robinhood for File Management

Use this tool to manage your data files on Peregrine.

Best Practices

rsync is the recommended tool for transferring data between NREL systems. It allows you to easily restart transfers if they fail, and also provides more consistency when dealing with symbolic links, hard links, and sparse files than either scp or cp. It is recommended you do not use compression for transfers within NREL systems. An example command is:

rsync -aP /scratch/username/dataset1/ /mss/users/username/dataset1/

rsync or scp will be your best option for small transfers (<100GB) outside of the NREL network. If your rsync/scp transfers are taking hours to complete then you should consider using Globus.

If you're transferring many files then you should use rsync:

$ rsync -azP /mss/users/username/dataset1/

If you're transferring an individual file then use scp:

$ scp /home/username/example.tar.gz user@desthost:/home/username/

Mass Storage has quotas that limit the number of individual files you can store. If you are copying thousands of files then it is best to tar these files up prior to copying to Mass Storage.

From Peregrine you can use wget or curl to transfer datasets smaller than 100GB. You should avoid using wget or curl to transfer datasets larger than 100GB since web traffic through the proxy is not optimized for large data transfers. If your only option for downloading large datasets (>100GB) involves wget or curl, please contact us to request assistance.

Globus is optimized for file transfers between data centers and anything outside of the NREL network. It will be several times faster than any other tools you will have available. Documentation about Globus is available on the Globus Services page

WinSCP is the recommended tool for transferring files to and from NREL Linux systems using Windows. Learn about WinSCP. If your Windows system or computer is outside of the NREL network, then Globus will provide better performance than WinSCP. See the Globus Services page.