Skip to main content

Peregrine System User Basics

If you are a new user of Peregrine, follow these instructions to get started.

Before you can start using Peregrine, you need an HPC user account.

Users with Prior High-Performance Computing Experience

If you have prior experience with HPC, start with these basic instructions.

There are four login nodes for the Peregrine system:


Access to Peregrine is available using the Secure Shell (SSH) protocol 2. Unix/Linux systems typically have SSH built-in. For Windows systems, an SSH client such as PuTTY needs to be installed.

To access Peregrine from outside NREL, first create a VPN to NREL, then SSH to Peregrine. 

Users may ssh to or to one of the login nodes.

Example commands to access Peregrine from a Linux or Mac OS X system at NREL are:

% ssh -l username peregrine-login[1-4]
% ssh -Y

Idle login sessions will be automatically logged out.

Learn more about connecting to a system.

Create a file called hello.F90 containing the following code:

program hello
write(6,*)'hello, world'
end program hello

Compile it using the Intel Fortran compiler:

% ifort -o hello hello.F90

Peregrine uses Moab for job scheduling and workload management and Torque for resource management. Jobs can be submitted to Torque by using the qsub command. 

Create a script file containing the commands to run your program. The file may contain options in PBS/Toque job submission language. These options, which are preceeded with "#PBS", are used to specify resource limits such as wall clock time, number of nodes, etc. You can also provide these limits as command line arguments, which have precedence over script options.

Sample Job Script:

#! /bin/bash
#PBS -l walltime=3600    # WALLTIME LIMIT
#PBS -N serialjob        # Name of job
./path_to_executable/hello   # run FORTRAN hello program

Submit this job with the following command:

% qsub <batch_file> -A <project-handle> 

HPC systems at NREL set an environment variable (NREL_CLUSTER) to help you identify what cluster your scripts are running on.  The below example shows how you can determine what cluster the script is running on and submit jobs differently to accomodate each clusters differences.

if [[ ${NREL_CLUSTER} = "peregrine" ]]; then
qsub <batch_file> -A <project-handle>
elif [[ ${NREL_CLUSTER} = "eagle" ]]; then
sbatch <batch_file>

For more information and sample batch scripts, see running jobs on Peregrine.

Users with Limited High-Performance Computing Experience

If you have no or limited experience with HPC, then follow these instructions.

 Also see Ten Steps to Linux Survival.

Access to Peregrine is available only from within the NREL firewall, by the Secure Shell (SSH) protocol 2.

To access Peregrine from outside NREL, first create a VPN to NREL.

A big difference between your laptop or personal workstation and Peregrine is that Peregrine is used with a command line interface via a terminal application. If you are using a Mac, your computer already has a terminal application and SSH.

If you are connecting using a Windows systems, a terminal package such as PuTTY, which supports SSH, needs to be installed. You may download PuTTY from its website. Configuring PuTTY to connect to Peregrine is only necessary the first time Peregrine is accessed. You may then log into Peregrine using your user ID and password. For instructions on configuring PuTTY to connect to Peregrine, see system connection

To access Peregrine from a Mac OS X system at NREL, start the terminal application and enter ssh -Y, where username is your NREL HPC user id. Remember to include return at the end of this command.


Once you have entered your password, you are logged in to Peregrine and all subsequent commands you type in your terminal application, whether on Windows or Mac, are commands on Peregrine.

In the examples on this and many other web pages, we represent the prompt by the symbol $. On Peregrine, by default the prompt will include the hostname of the node you have logged into, such as [username@login4 ~]$ . When typing the commands shown in this and other pages on this web site, you do not type the $ symbol.

In addition, all commands you type need to end by typing return or enter, which tells the system that you are done typing.

In places where you should insert information, we usually represent that information by enclosing it in brackets < >. For example:

$ ssh -Y <username>

When the ssh command completes you will be logged in to one of the login nodes and you will see a prompt that looks like

[icarpent@login4 ~]$

If you are new to Unix/Linux systems and command line interfaces, study the Quick guide to get started with the Linux command line. Good tutorials are available from the National Institute for Computational Sciences, TACC, and many other HPC centers.

The Python programming and runtime environment is provided on Peregrine primarily through the Anaconda Python distribution. This enables simple package maintenance, and permits the user to create and manage multiple Python 2 and 3 environments to suit their needs. As of Anaconda 5, the "root" environment activates an Anaconda-provided GCC environment that permits user installation of many Linux packages as well as Python modules, which allows (in many cases) seamless integration of toolchain, libraries, and applications and gives the user great freedom to customize their computing environment.

For more information about using Python see using Python on the Peregrine system.

Besides the login nodes, Peregrine has a large number of other nodes that are used to run programs. Please do not run programs directly on the login nodes since. Each of the other nodes can only be used by one user at a time. Because Peregrine is shared by many users, it uses software to control which user is using each node. To ask Peregrine to run your program, you give the instructions for how to run it to an application called Torque which keeps track of which nodes are already being used and which may be available to run your job. The set of instructions is put in a file called a "job script".

The command used to submit the job script to be run on the system is qsub.

Besides the name and location of the program, the system needs some additional information. This information can either be provided as options to the qsub command or you can save typing by including it in the script itself (which can be re-used many times). One of the critical pieces of information that is needed is the project handle for your allocation. This is included with the -A option, for example, the command

$ qsub -A <project-handle> <job_script>

provides both the project handle and the name of the job script.

To specify this information inside the script, the options are placed on lines that start with #PBS. To specify the project handle, one of the lines near the beginning of your script will contain #PBS -A project-handle. There are other options to specify resource limits such as the maximum time the job will take, the number of nodes needed, the type of nodes the job should run on, etc. Options that are provided to the qsub command take precedence over the same options that are inside the script.

To run the program you created, we will need to create a job script file. To do this, start your text editor again and give it the name of the file the script will be in. For example, type

$ nano

To run the sample code, type the following lines into the file using nano.

#! /bin/bash
#PBS -l walltime=3600    # WALLTIME LIMIT
#PBS -N serialjob        # Name of job


Save your text (^o) and exit the editor (^x). This created the file called

What does this cryptic text mean?

  • The first line of the file tells the system that the commands that follow are "bash" commands.
  • The next line tells torque that your job will have a maximum runtime of 3600 seconds. If your program takes longer than 3600 seconds, the system will kill your job when you reach this limit.
  • The third line gives a name (serialjob) to your job.
  • The forth line tells the system to change to the directory (folder) where your files are located.
  • The fifth line tells the system to run the program in the file
  • The text after the second # symbol on lines 2 and 3 are comments to remind you what information the line contains. Comments are ignored when processing the file.

To submit the job, type

$ qsub -A <CSC001>

but replace <CSC001> with the project handle of an allocation you have access to. The number that appears on your screen is the "job id". This number is handy for checking on the status of the job or reporting a problem. Did the job run and complete? Is it sitting in a queue waiting for nodes to become available? To find out what is going on, you can use the "checkjob" command. Type

$ checkjob <JobID>

An example of output from the checkjob command is shown below:

[icarpent@login2 ~/symm]$ checkjob 1104184
job 1104184

AName: mic_test_mpi_2phis.csh
State: Running
Creds:  user:icarpent  group:icarpent-upg  account:CSC000  class:phi  qos:default
WallTime:   00:00:00 of 00:40:00
SubmitTime: Tue Mar 31 09:44:51
  (Time Queued  Total: 00:00:24  Eligible: 00:00:24)

StartTime: Tue Mar 31 09:45:15
TemplateSets:  DEFAULT
NodeMatchPolicy: EXACTNODE
Total Requested Tasks: 1

Req[0]  TaskCount: 1  Partition: torque
Opsys: ---  Arch: ---  Features: phi
NodeCount:  1

Allocated Nodes:

SystemID:   Moab
SystemJID:  1104184
Notification Events: JobFail

StartCount:     1
Attr:           BACKFILL,checkpoint
StartPriority:  10
Reservation '1104184' (00:00:00 -> 00:40:00  Duration: 00:40:00)

We can see that this job is running on a node named "n1575".

To see the output of your programs, look for the output file from the job. The file name will start with the name assigned in the script (serialjob in this case) followed by .o and the JobID. Type ls to see what files exist in your directory (folder).

[jstickel2@login4 ~]$ ls
hello             serialjob.e1097699  serialjob.o1097699
hello2             serialjob.e1097700  serialjob.o1097700
hello.F90              serialjob.e1098464  serialjob.o1098464

To see what the output file contains, you can use the cat or more command:

cat serialjob.oXXXXX
[jstickel2@login4 ~]$ cat serialjob.o1097700
Hello, world from Python!

For more information and sample batch scripts, see running jobs on Peregrine.