Data Storage and Transfer
Learn about storing and transferring data on NREL High-Performance Computing (HPC) systems.
Use for large data transfers from NREL’s HPC systems to other systems.
Gyrfalcon is NREL’s primary system for storing long-term user data.
Use to transfer files from a local computer to a remote computer.
For information, see Eagle System Configuration.
For information, see Peregrine System Configuration.
For information, see Data Retention Policy.
rsync is the recommended tool for transferring data between NREL systems. It allows you to easily restart transfers if they fail, and also provides more consistency when dealing with symbolic links, hard links, and sparse files than either scp or cp. It is recommended you do not use compression for transfers within NREL systems. An example command is:
rsync -aP /scratch/username/dataset1/ /mss/users/username/dataset1/
rsync or scp will be your best option for small transfers (<100GB) outside of the NREL network. If your rsync/scp transfers are taking hours to complete then you should consider using Globus.
If you're transferring many files then you should use rsync:
$ rsync -azP /mss/users/username/dataset1/
If you're transferring an individual file then use scp:
$ scp /home/username/example.tar.gz user@desthost:/home/username/
Mass Storage has quotas that limit the number of individual files you can store. If you are copying thousands of files then it is best to tar these files up prior to copying to Mass Storage.
From Peregrine you can use wget or curl to transfer datasets smaller than 100GB. You should avoid using wget or curl to transfer datasets larger than 100GB since web traffic through the proxy is not optimized for large data transfers. If your only option for downloading large datasets (>100GB) involves wget or curl, please contact us to request assistance.
Globus is optimized for file transfers between data centers and anything outside of the NREL network. It will be several times faster than any other tools you will have available. Documentation about Globus is available on the Globus Services page.