Skip to main content

Peregrine System

Peregrine is NREL's flagship high-performance computing (HPC) system. Learn whether it has the capabilities to meet your research modeling, simulation, and visualization needs.

Photo of computer server stacks with the name "Peregrine" on the side.

System Architecture

Peregrine is a high-performance computing (HPC) system which is configured to run compute-intensive and parallel computing jobs. It is a cluster comprised of over 2500 nodes (servers) that run the Linux operating system (Red Had Linux or the derivative CentOS distribution), with a peak performance of 2.26 PetaFLOPS.

The nodes are connected to each other and to storage by a high-speed FDR InfiniBand network.

The system has NFS file systems for home directories (/home) and application software (/nopt) as well as high-speed parallel Lustre file systems for both temporary (/scratch) and longer-term (/projects) storage.  These file systems are mounted on all nodes.

Peregrine has three classes of nodes that users access:

Peregrine has four login nodes, each of which has Intel E5-2670 Xeon processors and 64 GB of memory. In addition to mounting the /home, /nopt, /projects and /scratch file systems, the /mss file system is mounted on all login nodes.

Peregrine has 2592 dual-socket compute nodes, each with 16 or 24 processors (cores). Together, these nodes have 6,912 Intel Xeon E5-2670 "SandyBridge" processor cores, 24,192 Intel Xeon E5-2695v2 "IvyBridge" processor cores, and 27,648 Intel Xeon E5-2670v3 "Haswell" cores for a total of 58,752 Intel Xeon processor cores. These processors provide a total of about 1.68 PetaFLOPS. Peregrine also has 576 Intel Phi many-core co-processors with an aggregate performance of about 582 TeraFLOPS. In total Peregrine is capable of 2.26 PetaFLOPS.

Number of Nodes in Peregrine

Processors

# cores/chip

# cores/node

Memory/node

Peak (DP) performance per node

88

Intel Xeon E5-2670 "Sandy Bridge"

8

16

32 GB

332.8 GF

56

Intel Xeon E5-2670 "Sandy Bridge"

8

16

256 GB

332.8 GF

288

Sandy Bridge + 2 Xeon Phi co-processors

8

16

32 GB

2.3 TF

288

Intel Xeon E5-2695v2 (Ivy Bridge)

12

24

64 GB

260.8 GF

720

Intel Xeon-2695v2 "Ivy Bridge:

12

24

32 GB

460.8 GF

1152

Intel Xeon E5-2670v3 "Haswell"

12

24

64 GB

883.2 GF

Access to compute nodes is managed by a job scheduler, which makes nodes available either for either "batch" or interactive use.

Peregrine has two nodes that each have an NVIDIA Quadro 6000 GPU card and 384 GB of memory. These nodes support the OpenCL and CUDA programming models and support hardware-accelerated remote visualization of data that resides on the parallel file systems (/scratch and /projects) using VirtualGL/Turbo VNC.

Transition to CentOS7

Learn about the Peregrine transition from CentOS6 to CentOS7.

System Configuration

Learn more about the Peregrine's nodes, processors, interconnect and file systems.

Running Jobs

Learn how to run jobs on Peregrine.

Software

Learn about the software and tools available to use on Peregrine.

 

LEARN HOW TO GET STARTED

If you're a new Peregrine user, follow our guidance for how to use this HPC system at NREL. We have instructions for those with prior HPC experience and for those with no or limited experience.

Learn How