Mass Storage System Policies

When using NREL's Amazon Web Services (AWS) Mass Storage System (MSS) to archive high-performance computing (HPC) data, please follow these policies.

Quotas

  • Both the amount of data and the number of files will be managed with quotas.
  • Users with accounts on any NREL HPC system will be provided with storage space on the AWS MSS system for personal storage. Every system user receives a default 1 terabyte (TB), 1 million file quota automatically when their account is created.
  • Projects that receive an allocation on HPC systems will be provided with MSS project quota determined as part of the allocation process.  Allocations receive a default 2 TB, 2 million file quota when the allocation is created.

Data Retention and Restoration

  • AWS MSS data will be retained for 15 months from the date written. This differs from the previous retention policy of minimum 15 months with deletion if needed.
  • Restore requests of MSS data that are archived in AWS MSS will initially require a request to the HPC Help Desk and may require 48 hours to be able to recover.

Also, see the Data Retention Policy

File Size and Number Limitations

  • The largest file size AWS S3 can manage is limited to 5 TB.
  • A limit of 1,025 files and/or directories (inodes) at a time has been placed on /shared-projects/MSS/home/<username> and /shared-projects/MSS/projects/<project_handle> to help minimize the cost of data transfers.

Share