ARM Parallel Profiling and Debugging Tools on the Eagle System

Start ARM Forge tools with reverse connect option on a compute node then, get a remote desktop session using FastX and login to a DAV node.
# Set your tmp directory to point to a place in scratch
$ export TMPDIR=/scratch/$USER/<temp directory>
$ module load arm/19.1.4
$ ddt  
You will get a prompt for a reverse connect request from your application running on the remote session. In the run configuration, please select the MPI Implementation you would like to use. For example, if you are using "srun" to launch your MPI application, select "SLURM (MPMD)" and then hit "run".

Although it is not necessary, we strongly recommend that you setup the remote client to use this tool to overcome latencies that arise when x-forwarding the display.

Download the remote client from ARM's website (v 19.1.4); download the client for your platform. (Mac/Windows/Linux)

1. Open the ARM Forge app
2. Select the configure option in the "Remote Launch" dropdown menu and set the hostname to "$USER@eagle.hpc.nrel.gov".
3. In the Remote Installation Directory filed, set the path to the ARM installation on Eagle (/nopt/nrel/apps/arm/19.1.4) 
4. Hit "Test Remote Launch" to test the configuration. 

Before you being please make sure that your application is compiled with the debug flag (-g), to enable profiling and debugging:

# Module to load
$ module load arm/19.1.4
$ which map
/nopt/nrel/apps/arm/19.1.4/bin/map
$ which ddt
/nopt/nrel/apps/arm/19.1.4/bin/ddt

ARM MAP is a parallel profiler that can be used to profile serial and parallel applications. It is easy to use and can be used from the GUI or the command line or through a remote client:

# To profile an executable
$ map --profile <name of the executable>
# this will create a *.map file.
# The "--profile" option lets you collect profiling data from the
# command line.
# You can then view the data from the desktop client. To view it
# directly through x11 forwarding just type 'map',
# it will open a GUI.
$ map 
# to profile an mpi executable
$ map --profile -np 4 <executable>
# to run map interactively
# NOTE: you should enable x-forwarding when connecting to
# Eagle.
$ map
# This will open the GUI

Using DDT Interactively

1. Start the remote client and connect to Eagle. You will
   get a prompt to accept the remote connection from the DDT instance after step 2.
2. Get an interactive compute node on Eagle and start DDT with reverse connect:

$ module load arm/19.1.4
$ ddt --connect <executable>
# follow the usual steps for debugging from the remote desktop client.

Offline Debugging with DDT

It is possible to create custom session files and use it with DDT from command line. Although the various flags for memory debugging, trace points, watch points, etc., can be specified through command line, you can set these in your code in the session file. Save the session file.

1. Open DDT in GUI mode, set the required debug points in the session and save the sessionfile (mysession.session) 
2. You can then submit a  job with DDT in offline mode pointing to the session file created in the previous step. At the end of the run, you can view the generated debugging report in html or text mode

$module load arm/19.1.4
$ddt --offline --session=<YOUR SESSION FILE.session> <other flags> -o <NAME of OUTPUT FILE.txt.html>   ./executable 

Module to use:

# set your tmpdir, and don't forget to clean it after your job
# completes. 
$ export TMPDIR=/scratch/$USER/tmp
$ module load arm/19.1.4
$ which perf-report
/nopt/nrel/apps/arm-pr/19.1.4/bin/perf-report

All you need to do is prefix your execution command with 'perf-report'
$ perf-report <executable>
$ perf-report mpirun -n 4 <executable>
# This will generate a .html file and a .txt file that you can
# view in a browser or text editor.