EasyBuild
Use EasyBuild installing software
EasyBuild is a software build and installation framework that allows you to manage scientific software on high performance computing systems in an efficient way. The official website of EasyBuild can be found in EasyBuild website. Below we give instructions on how to make use of EasyBuild in Spider.
List of supported software
A full list of supported software packages is available in Supported software.
Installing EasyBuild for the project team
Install EasyBuild as a software manager. Note that you need to have write access to directory /project/<project-name>/Software
to be able to follow the instructions below. Please change the <project-name>
in the commands to your project name.
After logging into Spider worker node, run the following commands:
python3.9 -m pip install wheel
python3.9 -m pip install --prefix /project/<project-name>/Software easybuild
Wheel is a necessary package for installation if it is not already installed. Because of the compatibility with Python, it is recommended to use newer version of Python, such as Python3.9. Next update the $PATH
environment variable to make sure the eb
command is available:
export PATH=/project/<project-name>/Software/bin:$PATH
To confirm EasyBuild is successfully installed and check the version, run:
which eb
You should see something like
/project/<project-name>/Software/bin/eb
Next specify the name or the full path to the python command that should be used by the eb
command via the $EB_PYTHON
environment variable:
export EB_PYTHON=python3.9
Later if you use EasyBuild installing a software, EasyBuild will generate a module which you can load. To update the module search path environment variable $MODULEPATH
, run command:
module use /project/<project-name>/Software/easybuild/modules/all
If you may use EasyBuild to install Python packages, you also need to update the Python search path environment variable $PYTHONPATH
to instruct Python where it can find the EasyBuild Python packages:
export PYTHONPATH=/project/<project-name>/Software/lib/python3.9/site-packages:$PYTHONPATH
IMPORTANT: keep in mind that you will have to make these environment changes again if you start a new shell session in Spider. To avoid this, you can update one of the shell startup scripts in your home directory. For example you can edit the .bashrc
file found in a users’ home directory:
cd /home/<user-name>
nano /home/<user-name>/.bashrc
Next, add the environment changes to the .bashrc
file and save.
export PATH=/project/<project-name>/Software/bin:$PATH
export EB_PYTHON=python3.9
export PYTHONPATH=/project/<project-name>/Software/lib/python3.9/site-packages:$PYTHONPATH
module use /project/<project-name>/Software/easybuild/modules/all
Now you are good to explore the EasyBuild world!
Configure EasyBuild
After EasyBuild is installed, here we give instruction on how to configure EasyBuild in Spider.
To get an overview of the current EasyBuild configuration, run command:
eb --show-config
It then shows a couple of selected important configuration settings with default values: build path, install path, path to easyconfigs repository, the robot search path, source path.
Configure EasyBuild via the environment variable $EASYBUILD_PREFIX
which changes all inportant configuration settings. To significantly speed up the builds, you can also change the build path to /tmp
. Note that the build directories are emptied and removed by EasyBuild when the installation is completed (by default).
export EASYBUILD_PREFIX='/project/<project-name>/Software/easybuild'
export EASYBUILD_BUILDPATH='/tmp'
For a full description of what you can configure in EasyBuild, please check Demo on configuing EasyBuild.
Install software
Here the most important range of topics are covered if you are new to EasyBuild. eb
is EasyBuild’s main command line tool, to interact with the EasyBuild framework and hereby the most common command line options are being documented.To instruct EasyBuild which software packages it should build/install and which build parameters it should use, one or more easyconfig files must be specified.
Searching for available easyconfig files can be done using the --search
(long output) and -S
(short output) command line options:
eb --search matplotlib
You will see all the easyconfig files available in the robot search path related to matplotlib
, and searching is done case-insensitive.
Next you can get an overview of planned installations by -D/--dry-run
:
eb matplotlib-3.3.3-foss-2020b.eb -D
Note how the different status symbols denote distinct handling states by EasyBuild:
[ ] The build is not available, EasyBuild will deliver it
[x] The build is available, EasyBuild will skip building this module
[F] The build is available, however EasyBuild has been asked to force a rebuild via –force and will do so
[R] The build is available, and the application will be rebuilt as request via –rebuild
You can also obtain a list of missing installations only using --missing-modules/-M
:
eb matplotlib-3.3.3-foss-2020b.eb -M
To install the software using the easyconfig files and its all software dependencies, use --robot/-r
:
eb matplotlib-3.3.3-foss-2020b.eb --robot
The dependency resolution mechanism of EasyBuild will construct a full dependency graph for the software package(s) being installed. Each of the retained dependencies will then be built and installed, in the required order as indicated by the dependency graph.
If necessary, use --force/-f
to force the reinstallation of a given easyconfig/module:
eb matplotlib-3.3.3-foss-2020b.eb --robot --force
Note
Please only run this command in a worker node. Note that for the first time installation may take longer than you expect, because toolchain dependencies such as GCCcore need to be installed. Once one software/package is installed, it can be used later in resolving the dependency of other software installations.
To check if the installation is successful and use the software, run commands:
module avail
module load matplotlib/3.3.3-foss-2020b
In case you are not familiar with using modules, here is a simple cheatsheet of most common module commands used in combination with Easybuild:
module avail - list the modules that are currently available to load
module load foss/2022a - load the module foss/2022a
module list - list currently loaded modules
module show foss/2022a - see contents of the module foss/2022a (shows the module functions instead of executing them)
module unload foss/2022a - unload the module foss/2022a
module purge - unload all currently loaded modules
Use software modules
As a user of the project to make use of the software installed by EasyBuild, you need read access to /project/<project-name>/Software
.
First update $MODULEPATH so you can find the modules:
module use /project/<project-name>/Software/easybuild/modules/all
IMPORTANT: keep in mind that you will have to run the module use
command again if you start a new shell session in Spider. To avoid this, you can update one of the shell startup scripts in your home directory. For example you can edit the .bashrc
file found in a users’ home directory:
cd /home/$USER
nano /home/$USER/.bashrc
Next, add the command below to the .bashrc
file and save.
module use /project/<project-name>/Software/easybuild/modules/all
Now you can view available software modules and load them:
module avail
module load matplotlib/3.3.3-foss-2020b
Below is how you can use the modules in a job script:
#!/bin/bash
#SBATCH -n 1
#SBATCH -t 10:00
#SBATCH -c 1
module use /project/<project-name>/Software/easybuild/modules/all
module load matplotlib/3.3.3-foss-2020b
module load Python/3.8.6-GCCcore-10.2.0
echo "I am using the matplotlib module installed by EasyBuild"
echo "I am running on " $HOSTNAME
python /home/$USER/draw_a_plot.py
The draw_a_plot.py can be, for example:
import numpy as np
import matplotlib.pyplot as plt
x = np.arange(0, 5, 0.1)
y = np.sin(x)
plt.plot(x, y)
plt.savefig("output.jpg")
To view the result jpg in your terminal, run
display output.jpg
See also
Still need help? Contact our helpdesk