Mass Storage System
NREL’s Amazon Web Services (AWS) Mass Storage System (MSS) is an additional data archival resource available to active projects and users on the Eagle high-performance computing (HPC) system.
The AWS MSS keeps and protects important data, primarily as an addition to Eagle’s high-performance Lustre filesystem (/projects, /shared-projects, and /datasets).
NREL implemented the AWS MSS to take advantage of S3 Deep Glacier archiving, replacing the previous on-premises MSS, Gyrfalcon, which reached end-of-life at the end of 2020.
How To Copy/Move Data from Eagle
AWS charges per inode. Therefore, to keep costs down it is recommended users create a compressed tarball of any files and/or directories desired to be archived to AWS MSS.
The recommended command is:
$ tar czvf /destination/descriptor-YYYMMDD.tgz <source-files-directories>
Example, from Eagle’s /projects/csc000/data-to-be-copied from an Eagle Login node:
$ cd /projects/csc000
$ tar czvf /shared-projects/MSS/projects/csc000/data-to-be-copied-20211215.tgz data-to-be-copied
Data placed in /shared-projects/MSS/projects/<project_handle> and /shared-projects/MSS/home/<username> is synced to AWS MSS and then purged from /shared-projects/MSS.
How To Restore Data
- Restore requests of AWS MSS data will require a request to the HPC Help Desk and may require 48 hours or more to be able to stage from Deep Archive to recover.
- Users can request a list of the archived files they have on AWS MSS by sending a request to the HPC Help Desk.
- Let the HPC Help Desk know specifically what file(s) you would like to recover, and where the recovered files should be placed.
Follow the AWS MSS policies.
Contact the HPC Help Desk if you have any questions or issues.