Skip to main content

Using Python on the Peregrine System

For NREL high-performance computing, learn how to use Python on the Peregrine system.

Anaconda Python is our actively supported distribution on Peregrine.

To use Anaconda, run module load conda/5.1. The base package is built on Python 2.7, but you can create custom environments with other versions including Python 3+. In all cases, the numerically intensive routines in numpy and scipy are MKL-enabled for speed.

Custom Environments

Custom environments are available for users to install libraries. Create environments in a directory with write privileges using the -p PATH option. Do not use the --clone=root option when creating custom environments, it was not designed for use in a shared computing environment. The following example creates a new Python 2 environment with numpy, scipy and pandas:

[user@HOSTNAME~]$ module purge; module load conda/5.1
[user@HOSTNAME ~]$ conda create -n exampleenv python=2 numpy scipy pandas
Fetching package metadata .......
Solving package specifications: ..........
<SNIP>Extracting packages ... [ COMPLETE ]|###################################| 100%
Linking packages ... [ COMPLETE ]|###################################| 100%#
#
To activate this environment, use:
# $ source activate exampleenv
#
# To deactivate this environment, use:
# $ source deactivate
#[user@HOSTNAME ~]$ source activate exampleenv
(exampleenv) [user@login1 ~]$ type python
python is /home/user/.conda-envs/exampleenv/bin/python

Using Environment.yml Files

Anaconda supports creating environments from a yml-formatted file. This allows the same environment to be used wherever the dependent code is run. For example an environment.yml file can be created on the developer's laptop and used on the laptop and the HPC to create the environment that will allow the code to run. The conda module has its own environment.yml that is used to create the root environment. This file can be downloaded and modified to produce the environment needed for custom code.

If you require something not available in the default environment and are unable to create a custom environment, contact us to determine if the default environment can be adapted to your needs.

Python Interactive Shell /Jupyter

NOTE: Since the CentOS 7 transition, it has become necessary to unset a Jupyter-related variable by hand in order to get the workflow below to succeed. We are working to resolve this, but until we can, please add the following instruction to any job scripts using Jupyter, or interactive workflows before launching Jupyter:

[user@HOSTNAME ~]$ unset XDG_RUNTIME_DIR

 

Jupyter is supported in the default conda environment. However the HPC firewall prohibits direct connections to the server due to security concerns. A ssh pipe is required to access the notebook server.

  1. Request a compute node as an interactive shell and start the notebook server:
[user@HOSTNAME ~]$ qsub -I -q debug -A MY_ALLOCATION -l nodes=1:ppn=16,walltime=1:00:00
[user@NODENAME ~]$ jupyter notebook --no-browser --ip=0.0.0.0
[I 10:00:14.868 NotebookApp] Serving notebooks from local directory: /home/user
[I 10:00:14.868 NotebookApp] 0 active kernels
[I 10:00:14.869 NotebookApp] The Jupyter Notebook is running at: http://0.0.0.0:8888/?token=somerandomstring
[I 10:00:14.869 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 10:00:14.870 NotebookApp]

   Copy/paste this URL into your browser when you connect for the first time,    to login with a token:
        http://0.0.0.0:8888/?token=somerandomstring
  1. On your local machine start a ssh tunnel:
laptop:~ user$ ssh -L 8888:NODENAME:8888 HOSTNAME.hpc.nrel.gov
[user@HOSTNAME ~]$

* Replace the HOSTNAME with the login node and NODENAME with computenode

   e.g., HOSTNAME=login1 and NODENAME=n0290

        

  1. On your local machine start a web browser and go to http://localhost:8888/?token=somerandomstring, using the token as shown in the notebook startup

See quick reference sheet for iPython.

Python Debugging

Knowing how to use the Python pdb / pdb debugging module can speed development.

See quick reference sheet for pdb.