System Resource Allocation Units
NREL uses an allocation unit (AU) to allocate and charge time used on its high-performance computing (HPC) systems.
NREL allocates time on compute nodes and space on its file systems and MSS (archive) system. A Kestrel CPU node hour is equal to 10 AUs if a job is set at normal priority, while a Kestrel GPU node hour is equal to 100 AUs. For comparison with other HPC systems, each Kestrel CPU node has a theoretical peak performance of 8,320 GigaFLOPS and has 104 cores. A Kestrel GPU node has a theoretical peak performance of 245.42 TeraFLOPS.
Computing Allocation Unit Charges
AU charges are calculated for each job run upon job completion. The cost of the job is computed using this formula:
Walltime in hours * Number of Nodes * QoS Factor * Charge Factor
Quality of Service Factor
The quality of service (QoS) factor reflects the priority given to a job. The default QoS factor for all jobs is 1, meaning the job is run at normal priority. A user can change the job to run at high priority by adding “--qos=high” when the job is submitted. Setting the job to high-priority will give the job a priority boost, so it runs sooner than other jobs in the queue. High-priority jobs have a QoS factor of 2. This means they will be charged at twice the normal rate.
The conversion from node hours to AUs is the charge factor. For Eagle, the charge factor is 3, consistent with one AU being one-third of a node hour. For Kestrel CPU nodes, the charge factor is 10 and for Kestrel GPUs the charge factor is 100. Keep in mind that when running jobs on Kestrel GPUs, users have the ability to request the number of GPUs needed for the job, which can be one, two or four GPUs available per node. The charge factor per GPU is the corresponding fraction ¼, ½ or 1.
Users should take care to make efficient use of the node hours they request. Users whose codes are not parallelized are encouraged to run arrays of jobs on one node, in order to efficiently make use of the AUs they will be charged.
Estimating Allocation Units for Allocation Requests
When possible, running jobs on NREL HPC systems prior to making an annual allocation request is encouraged, both to make sure the code is ready to go, and to run test jobs. You can request a pilot allocation at any time.
The basic formula for estimating AUs is:
Per-Job walltime in hours * Number of Nodes * Charge Factor * Number of Runs anticipated
Reasoning Behind Allocation Units
Giving out allocations in AUs, instead of node hours or core hours, lets NREL keep the definition of an allocation consistent as we move from HPC system to HPC system. This is a best-practice that NREL borrowed from other HPC facilities in the U.S. Department of Energy system.
AUs are indexed to a node-hour on Peregrine—the first system deployed at the HPC User Facility within the Energy Systems Integration Facility. A Peregrine node (for purposes of the AU standard) was a 24-core Intel Xeon (Haswell) node, which had a theoretical peak performance of 883.2 GigaFLOPS.