Peregrine System Configuration
Learn about the Peregrine system configuration.
The Peregrine system is a high-performance computing (HPC) system with different types of servers (nodes) configured to run compute intensive and parallel computing jobs. All of the nodes run the Linux operating system, either Red Hat Linux or the derivative CentOS distribution. The nodes and storage are connected by a high-speed InfiniBand network. Compute nodes are diskless with an NFS file system for /home and a high-speed parallel Lustre file system for /scratch. The home directories are mounted on all nodes, along with a file system dedicated to shared projects. A brief description of the configuration and features of the nodes, interconnect, and file systems is provided below.
Peregrine had 2592 compute nodes at its peak, but the system is nearing its end-of-life and portions of the system have been, or are in the process of being, decommissioned. Node types included:
- 144 nodes have dual 8-core Intel Sandy Bridge Xeon chips. 88 of these nodes have 16 cores and 32 GB of memory. 56 of these nodes have 16 cores and 256 GB of memory.
- 288 nodes have dual 8-core Intel Sandy Bridge Xeon chips plus two Intel Xeon Phi coprocessors. These nodes have 32 GB of memory.
- 1,008 nodes have dual 12-core Ivy Bridge Xeon processors. 288 of these have 64 GB of memory and the remainder have 32 GB of memory.
- 1,152 nodes have dual 12-core Haswell Xeon processors with 64 GB of memory.
All nodes are connected to the high-speed Infiniband network and and a management ethernet. The /home, /scratch, /projects and /nopt file systems are mounted on all compute nodes.
The Data Analysis & Visualization (DAV) nodes are each equipped with dual 8-core Intel E5-2670 processors running at 2.3 GHz, 384 GB of memory, and an NVIDIA Quadro 6000 GPU. These nodes supports OpenCL and CUDA programming models. These nodes support hardware-accelerated remote visualization of data on the parallel file system.
Users may connect to dav.hpc.nrel.gov. This will connect to one of the 3 DAV nodes. Users also have the option of connecting directly to an individual DAV node using one of the following:
There are four login nodes on the system, HP Proliant DL380 G8 servers with Intel E5-2670 processors, 64 GB memory, and local disk drives for the operating system. The /home, /scratch, /projects, /mss and /nopt file systems are mounted on all login nodes.
Users may connect to peregrine.hpc.nrel.gov. This will connect to one of the four login nodes. Users also have the option of connecting directly to an individual login node using one of the following names:
There are seven nodes referred to as service nodes that perform system administration functions, support file system access for the compute nodes, and manage the file system images that get deployed to the diskless nodes.
The system is a collection of tightly connected nodes we call "Scalable Units." The high-speed InfiniBand network operates at 4X FDR speeds. There are 16 Mellanox spine switches and 8 scalable unit (SU) leaf switches configured that are fully connected (non-blocking between servers within an SU). Bisection bandwidth is 1.9 Terabytes/sec while latency is minimized to three or fewer switches hops within an SU, and no more than 7 hops across the entire system. The connection to the parallel file system is capable of 40GB/s, and the interconnect is designed for future expansion.
The /home file system on Peregrine is a robust NFS file system that is intended to hold small files. These include shell startup files, scripts, source code, executables and data files. The capacity of /home is 10 TB.
/scratch is a parallel Lustre file system intended for high-performance I/O. Use /scratch for running jobs and any other intensive I/O activity.
The capacity of the /scratch file system is 1.5 PB. This capacity is provided by 108 Object Storage Targets (OSTs) that are attached to 24 Object Storage Servers (OSSs). The default stripe count is 1 and the default stripe size is 1 MB.
/projects is a parallel Lustre file system intended for high performance I/O associated with files that are shared by members of a project.
The capacity of the /projects file system is 768 TB. This capacity is provided by 54 OSTs that are attached to the 24 Lustre OSSs. The default strip count is 1 and the default stripe size is 1 MB.
The /nopt file system is a robust NFS file system where NREL-specific software, module files, licenses and licensed software is kept. The capacity of the /nopt file system is 2 TB.