Kestrel Programming Environments Overview
Definitions
Toolchain: a combination of a compiler and an mpi library. Sometimes associated scientific libraries (scalapack, blas, etc.) or bundles of scientific libraries (MKL, libsci, etc.) are considered part of the toolchain.
Environment: a set of modules, including a toolchain. A "build environment" refers to the set of modules (including compiler and MPI library) used to compile a code. A "run-time environment" is the set of modules used to execute a code. The two typically, but not always, match.
Environments
There are three types of module-based Toolchains available on Kestrel:
- "PrgEnv-" Environments, shipped with Kestrel
- NREL-built Environments
- NREL-built Environments with
cray-mpich-abi
The "PrgEnv-" environments are new on Kestrel. PrgEnv stands for "programming environment," and Kestrel ships with several of these. There are advantages to using a PrgEnv environment, as these environments are tailored for some of the Cray-specific features of Kestrel. For example, Cray MPICH utilizes Kestrel's Cray Slingshot network more effectively than OpenMPI or Intel MPI, so it runs noticeably faster than the other two for jobs that require two or more nodes. All PrgEnv- environments utilize Cray MPICH by default.
The NREL-built environments function similarly to those on Eagle, and it is up to the user to load all necessary modules to build and run their applications.
NREL-built environments can make use of Cray MPICH via the cray-mpich-abi. As long as program is compiled with an MPICH-based MPI (e.g., Intel MPI but not Open MPI), the cray-mpich-abi can be loaded at runtime, which causes the program to use Cray MPICH for dynamically built binaries.
A note on OpenMPI
Currently, OpenMPI does not run performantly or stably on Kestrel. You should do your best to avoid using OpenMPI. Please reach out to hpc-help@nrel.gov if you need help working around OpenMPI.
Summary of available compiler environments
- (Cray) denotes that the module belongs to the default Cray module set.
- (NREL) denotes that the module belongs to the NREL-built module set. If a compiler module is denoted (NREL), then the corresponding MPI module is also (NREL).
GNU
| PrgEnv |
Compiler Module |
MPI module |
Language |
Wrapper |
Compiler |
MPI |
| gnu |
gcc (Cray) |
cray-mpich |
Fortran |
ftn |
gfortran |
Cray MPICH |
| gnu |
gcc (Cray) |
cray-mpich |
C |
cc |
gcc |
Cray MPICH |
| gnu |
gcc (Cray) |
cray-mpich |
C++ |
CC |
g++ |
Cray MPICH |
| n/a |
gcc (NREL) |
openmpi/4.1.5-gcc |
Fortran |
mpifort |
gfortran |
Open MPI |
| n/a |
gcc (NREL) |
openmpi/4.1.5-gcc |
C |
mpicc |
gcc |
Open MPI |
| n/a |
gcc (NREL) |
openmpi/4.1.5-gcc |
C++ |
mpic++ |
g++ |
Open MPI |
Cray
| PrgEnv |
Compiler Module |
MPI module |
Language |
Wrapper |
Compiler |
MPI |
| cray |
cce (Cray) |
cray-mpich |
Fortran |
ftn |
crayftn |
Cray MPICH |
| cray |
cce (Cray) |
cray-mpich |
C |
cc |
craycc |
Cray MPICH |
| cray |
cce (Cray) |
cray-mpich |
C++ |
CC |
crayCC |
Cray MPICH |
Intel
| PrgEnv |
Compiler Module |
MPI Module |
Language |
Wrapper |
Compiler |
MPI |
| intel |
intel (Cray) |
cray-mpich |
Fortran |
ftn |
ifort |
Cray MPICH |
| intel |
intel (Cray) |
cray-mpich |
C |
cc |
icc |
Cray MPICH |
| intel |
intel (Cray) |
cray-mpich |
C++ |
CC |
icpc |
Cray MPICH |
| n/a |
intel-oneapi (NREL) |
intel-oneapi-mpi |
Fortran |
mpiifort |
ifort |
intel MPI |
| n/a |
intel-oneapi (NREL) |
intel-oneapi-mpi |
C |
mpiicc |
icc |
intel MPI |
| n/a |
intel-oneapi (NREL) |
intel-oneapi-mpi |
C++ |
mpiicpc |
icpc |
intel MPI |
Note:
The Cray MPICH used for each different PrgEnv- is pointing to a different instance of MPICH, E.g. for PrgEnv-intel the MPICH used is located under /opt/cray/pe/mpich/8.1.21/ofi/intel/19.0 and for PrgEnv-cray the MPICH used is located under /opt/cray/pe/mpich/8.1.20/ofi/crayclang/10.0.
PrgEnv- Programming Environments
Module Known Issues
As of July 30th, 2024, there are some modules that do not work correctly. The following points describe the issues and workarounds. We are working on permanent fixes for these issues.
* If using PrgEnv-intel, cray-libsci/23.12 is loaded by default. Load cray-libsci/22.12 instead.
* If using PrgEnv-gnu with gcc version 10, load cray-libsci/22.12 instead of the default cray-libsci version.
* On the GPU nodes, if using PrgEnv-nvhpc, you need to load nvhpc/23.9. nvhpc/24.1 is loaded by default and will not work.
* The module restore command could cause an "Unable to find cray-mpich" libraries error when used with PrgEnv-intel. If this happens, remove module restore from the list of commands.
Introduction
These environments come packaged with:
- A compiler, which corresponds to the name of the environment. E.g.,
PrgEnv-intel uses intel compilers
- Cray MPICH
- Cray LibSci, which can be used in place of MKL
- Additional communication and network libraries
Upon logging into the machine, the PrgEnv-gnu is loaded by default on both the CPU and GPU login nodes. If we module list, we can see the modules associated with PrgEnv-gnu.
We can swap between programming environments using the module swap command. For example, if PrgEnv-gnu is loaded but we want to use PrgEnv-cray instead, we can module swap PrgEnv-gnu PrgEnv-cray.
What is a PrgEnv module doing?
PrgEnv modules can seem a bit mysterious. We can check out the inner workings of a PrgEnv module with the module show command. For example, for PrgEnv-gnu we can:
module show PrgEnv-gnu
Which outputs:
/opt/cray/pe/modulefiles/PrgEnv-gnu/8.3.3:
conflict PrgEnv-amd
conflict PrgEnv-aocc
conflict PrgEnv-cray
conflict PrgEnv-gnu
conflict PrgEnv-intel
conflict PrgEnv-nvidia
setenv PE_ENV GNU
setenv gcc_already_loaded 1
module swap gcc/12.1.0
module switch cray-libsci cray-libsci/22.10.1.2
module switch cray-mpich cray-mpich/8.1.20
module load craype
module load cray-dsmml
module load craype-network-ofi
module load cray-mpich
module load cray-libsci
setenv CRAY_PRGENVGNU loaded
This tells us that PrgEnv-gnu conflicts with all other PrgEnvs. The modulefile sets some environment variables (the setenv lines), and loads the modules associated with the programming environment.
We can use the module whatis command to give us a brief summary of a module. For example, the command:
module whatis craype
outputs:
craype : Setup for Cray PE driver set and targeting modules
We mentioned previously that the different PrgEnvs use different locations for Cray-MPICH. We can see this by using module show cray-mpich in each different PrgEnv, and examining (for example) the CRAY_LD_LIBRARY_PATH environment variable.
Compiling inside a PrgEnv: ftn, cc, and CC
ftn, cc, and CC are the Cray compiler wrappers for Fortran, C, and C++, respectively, which are part of the craype module. When a particular PrgEnv- programming environment is loaded, these wrappers will make use of the corresponding compiler. For example, if we load PrgEnv-gnu with:
module swap PrgEnv-cray PrgEnv-gnu
we would expect ftn to wrap around gfortran, the GNU fortran compiler. We can test this with:
ftn --version
Which outputs:
GNU Fortran (GCC) 12.1.0 20220506 (HPE)
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
As expected. We can also which ftn:
/opt/cray/pe/craype/2.7.17/bin/ftn
cc, CC and ftn is always the same /opt/cray/pe/craype/2.7.17/bin/ftn and does NOT depend on the loaded PrgEnv.
Note2: cc, CC and ftn are also wrappers around their mpi couterparts. For mpi codes, the wrappers call the necessary mpi compilers depending on which PrgEnv is loaded.
Note3: When changing between PrgEnvs, it is better to use module swap [current prgenv] [new prgenv] instead of module purge; module load [new prgenv] due to the way the environments set some environment variables.
ftn is part of the craype module. If we module unload craype and then type which ftn we find:
/usr/bin/which: no ftn in (/opt/cray/pe/mpich/8.1.20/ofi/gnu/9.1/bin:/opt/cray/pe/mpich/8.1.20/bin:/opt/cray/libfabric/1.15.2.0/bin:/opt/cray/pe/gcc/12.1.0/bin:/home/ohull/.local/bin:/home/ohull/bin:/opt/clmgr/sbin:/opt/clmgr/bin:/opt/sgi/sbin:/opt/sgi/bin:/nopt/xalt/xalt/bin:/nopt/nrel/utils/bin:/nopt/slurm/current/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/c3/bin:/sbin:/bin)
I.e., it can't find the path to ftn, because craype is not loaded into the environment.
What happens if we module swap PrgEnv-gnu PrgEnv-cray, so that we're now using PrgEnv-cray, and then check ftn?
[ohull@eyas1 ~]$ ftn --version
Cray Fortran : Version 14.0.4
ftn is now using Cray Fortran under the hood.
Note: you can still directly access the underlying compiler. For example, if we're using PrgEnv-gnu (so our compilers are the GCC compilers), we can use ftn, or we can use gfortran or mpifort directly. It is considered best practice to use the Cray wrappers (ftn, cc, CC) on a Cray machine like Kestrel.
In fact, the use of mpifort can be quite confusing. Inside the PrgEnv-gnu environment, we might assume that mpifort is a wrapper around OpenMPI. This is not correct, as mpifort wraps around Cray MPICH inside PrgEnv-gnu. If we module unload PrgEnv-gnu and then module load openmpi, then mpifort will wrap around OpenMPI. Using the Cray wrappers (ftn, cc, CC) helps avoid this confusion.
Submitting a job within a PrgEnv
Submitting a Slurm job using a PrgEnv environment is no different than how you would normally submit a job. In your slurm script, below the #SBATCH directives, include:
module swap PrgEnv-cray [new PrgEnv]
We swap from PrgEnv-cray because this is the default PrgEnv that is loaded when logging onto Kestrel.
[new PrgEnv] can be PrgEnv-gnu or PrgEnv-intel.
Depending on the software you're trying to run, you may need to load additional modules like cray-hdf5 or cray-fftw.
NREL-built environments
The NREL build modules are similar to Eagle, where the module are separate and no dependecy is created between modules.
The modules are grouped by type compilers_mpis utilities_libraries and applications, and a module can be loaded using module load $module_name.
The modules are optimized for Kestrel architecture and will be updated/upgraded every 6/12months or upon request. If there is a module you need but is not available, email hpc-help@nrel.gov
NREL-built environments with cray-mpich-abi
For binaries dyanamically built with an MPICH-based MPI such as intel-mpi, the user can choose to use cray-mpich-abi at runtime to leverage its optimization for Kestrel. To check if your executable was dynamically built with intel MPI, you can ldd [your program name] | grep mpi.
the module cray-mpich-abi will cause the program to run with Cray MPICH at runtime instead of Intel MPI. In your slurm submit script, you must include the two lines:
module load craype
module load cray-mpich-abi
in order for the Cray MPICH abi to work properly.
Note: If your code depends on libmpicxx, the Cray MPICH ABI is unlikely to work. You can check this by ldd [your program name] | grep mpicxx.