Mass Storage System Policies
When using NREL's Amazon Web Services (AWS) Mass Storage System (MSS) to archive high-performance computing (HPC) data, please follow these policies.
- Both the amount of data and the number of files will be managed with quotas.
- Users with accounts on any NREL HPC system will be provided with storage space on the AWS MSS system for personal storage. Every system user receives a default 1 terabyte (TB), 1 million file quota automatically when their account is created.
- Projects that receive an allocation on HPC systems will be provided with MSS project quota determined as part of the allocation process. Allocations receive a default 2 TB, 2 million file quota when the allocation is created.
Data Retention and Restoration
- AWS MSS data will be retained for 15 months from the date written. This differs from the previous retention policy of minimum 15 months with deletion if needed.
- Restore requests of MSS data that are archived in AWS MSS will initially require a request to the HPC Help Desk and may require 48 hours to be able to recover.
Also, see the Data Retention Policy.
File Size and Number Limitations
- The largest file size AWS S3 can manage is limited to 5 TB.
- A limit of 1,025 files and/or directories (inodes) at a time has been placed on /shared-projects/MSS/home/<username> and /shared-projects/MSS/projects/<project_handle> to help minimize the cost of data transfers.