Kestrel System Configuration

Learn about the Kestrel system configuration, expected to be available during fiscal year 2023.

Architecture Description 

The Kestrel system is a high-performance computing (HPC) system for compute intensive and parallel jobs. User facing login and data analysis and visualization (DAV) nodes will utilize Red Hat Enterprise Linux. Compute nodes will utilize Rocky Linux. The system utilizes a high-speed, 200 gigabit (Gb)/s Hewlett Packard Enterprise (HPE) Slingshot interconnect. A brief description of the nodes, interconnect, and file systems is provided below.

Compute Node Hardware Details

Kestrel will have 2436 compute nodes available for HPC jobs. Below are the node types on Kestrel:

Number of Nodes Processors Memory Accelerators Local Storage
2304 Dual socket Intel Xeon Sapphire Rapids (52-core) processors 256 GB DDR5 N/A 256 nodes with 1.92 TB NVMe M.2
132 Dual socket AMD Genoa  
(96-core) processors 
384 GB 4 NVIDIA H100 SXM GPUs,  
80 GB Memory
2 x 1.6 TB NVMe
10 Dual socket Intel Xeon Sapphire Rapids (52-core) processors 2 TB DDR5 N/A 8 x 1.6 TB NVMe
10 Dual socket Intel Xeon Sapphire Rapids (52-core) processors 256 GB DDR5 2 NVIDIA A40 GPUs 2 x 3.84 TB NVMe

Interconnect 

All compute nodes will be connected to the high-performance HPE Slingshot 11 fabric in a dragonfly topology. In the case of the accelerated nodes, each node will have two 200 Gb/s Network Interface Cards (NICs) to the interconnect. 

Parallel File System 

The Parallel File System (PFS) ProjectFS and ScratchFS on Kestrel is a ClusterStor Lustre file system intended for high-performance I/O. ScratchFS uses a Lustre file system in a hybrid flash-disk configuration providing a total of 27 petabytes (PB) of capacity with 354 gigabytes (GB)/s of IOR bandwidth. ProjectFS will provide 68 PB of capacity with 200 GB/s of IOR bandwidth. 

Home File System

The Home File System (HFS) on Kestrel is part of the ClusterStor used for PFS, providing highly reliable storage for user home directories and NREL-specific software. HFS will provide 1.2 PB of capacity. Snapshots of files on the HFS will be available up to 30 days after change/deletion.