Containerization

Containerization on Deucalion

Containerization is a lightweight form of virtualization that allows software applications to run in isolated environments. Unlike traditional virtual machines (VMs), containers share the host system's operating system kernel but keep the application and its dependencies separate. This approach reduces overhead, improves performance, and ensures consistency across different computing environments. Containers are especially useful in complex software deployments, as they guarantee that an application will run the same way, regardless of the underlying infrastructure—whether on a developer's laptop, a data center, or a high-performance computing (HPC) cluster.

Popular container platforms include Docker, Kubernetes, and Singularity, each with its specific use cases and optimizations for different environments. In Deucalion we provide Singularity in all three partitions (ARM, x86 and GPU).

Singularity: Containerization for HPC

Singularity is a container platform designed specifically for high-performance computing (HPC) environments. It was created to address the unique security, performance, and flexibility needs of scientific computing. Singularity focuses on enabling users to package their applications and dependencies into portable containers that can be executed seamlessly on HPC infrastructure.

Key Features of Singularity:

User-friendly for HPC: Singularity runs without requiring elevated privileges, ensuring compatibility with HPC systems where users typically do not have root access. This makes it safer for multi-user environments.
Compatibility: It can run containers built with Docker, providing flexibility for developers who create their containers using Docker locally but deploy them on HPC clusters.
Portability and reproducibility: Singularity ensures reproducibility, allowing researchers to package their applications, libraries, and environments into a single file (SIF—Singularity Image Format) that can be easily shared and executed across systems. For compatible definitions file click here.
Native MPI support: Singularity is optimized for parallel computing, allowing seamless integration with Message Passing Interface (MPI) applications, which are essential for HPC workloads.
Security: Singularity does not require root access to execute containers. This reduces the security risks commonly associated with containerization in shared environments.

Singularity provides a powerful, secure, and flexible solution for running containerized applications in HPC environments, making it the go-to platform for many scientists and researchers.

Definitions file for MPI Support

We offer a definitions file from which you can build your own singularity versions in your laptop. Due to architecture differences, you will need to create the singularity container using a machine with the same architecture of the compute nodes where you will run it. We offer the --fakeroot option for Singularity builds, so in general you can start an interactive session. If this does not work, please contact support and we will be happy to help you build your container.

First, take the docker container from Rocky Linux (don't forget to do it in a compute node with the intended architecture):

singularity pull rocky_8_5.sif docker://rockylinux:8.5

Then, do singularity build rocky_8_5.sif openmpi_<arch>.def, where the definitions file depends on the partition you are trying to run in. For the ARM partitions, use the openmpi_arm.def. For the other partitions, use openmpi_x86.def. You will find that one step requires the file with the Infiniband drivers (name starting with MLNX_OFED). To have it on Deucalion please download the file from the NVidia website and transfer it using scp.

x86 PartitionsARM Partitions

openmpi_x86.def

BootStrap: yum
OSVersion: 8.5
MirrorURL: https://dl.rockylinux.org/vault/rocky/%{OSVERSION}/BaseOS/x86_64/os/
Include: dnf

%environment
    export OMPI_DIR=/usr/local
    export SINGULARITY_OMPI_DIR=$OMPI_DIR
    export SINGULARITYENV_APPEND_PATH=$OMPI_DIR/bin
    export SINGULARITYENV_APPEND_LD_LIBRARY_PATH=$OMPI_DIR/lib
    export OMPI_MCA_mtl_ofi_verbose=0
%files
    /path/to/MLNX_OFED_LINUX-24.07-0.6.1.0-rhel8.5-x86_64.tgz /opt
%post
    #restorecon -Rv /var/lib/rpm
    rpmdb --rebuilddb
    rpm --import https://download.rockylinux.org/vault/rocky/8.5/PowerTools/x86_64/os/RPM-GPG-KEY-rockyofficial
    ## Prerequisites
    dnf update -y
    dnf install -y dnf-plugins-core
    dnf config-manager -y --set-enabled devel
    dnf groupinstall -y 'Development Tools'
    dnf install -y wget git bash hostname gcc gcc-gfortran gcc-c++ make file autoconf automake libtool zlib-devel python3
    dnf install -y libmnl lsof numactl-libs ethtool tcl tk

    ## Packages required for OpenMPI and PMIx
    dnf install -y libnl3 libnl3-devel
    dnf install -y libevent libevent-devel
    dnf repolist all
    dnf install -y munge munge-devel
    dnf install -y pciutils #perl-sigtrap pciutils
    # Mellanox OFED matching MeluXina
    mkdir -p /tmp/mofed
    cp /opt/MLNX_OFED_LINUX-24.07-0.6.1.0-rhel8.5-x86_64.tgz /tmp/mofed
    cd /tmp/mofed
    tar xf MLNX_OFED_LINUX-24.07-0.6.1.0-rhel8.5-x86_64.tgz
    cd MLNX_OFED_LINUX-24.07-0.6.1.0-rhel8.5-x86_64
    ./mlnxofedinstall --basic --user-space-only --without-fw-update --distro rhel8.5 --force

    # HWLOC for PMIx and OpenMPI
    mkdir -p /tmp/hwloc
    cd /tmp/hwloc
    wget https://download.open-mpi.org/release/hwloc/v2.9/hwloc-2.9.3.tar.bz2
    tar xf hwloc-2.9.3.tar.bz2
    cd hwloc-2.9.3/
    ./configure --prefix=/usr/local
    make -j
    make install

    # PMIx
    mkdir -p /tmp/pmix
    cd /tmp/pmix
    wget -c https://github.com/openpmix/openpmix/releases/download/v4.2.9/pmix-4.2.9.tar.gz
    tar xf pmix-4.2.9.tar.gz
    cd pmix-4.2.9
    ./configure --prefix=/usr/local --with-munge=/usr --with-libevent=/usr --with-zlib=/usr --enable-pmix-binaries --with-hwloc=/usr/local && \
    make -j
    make install

   # libfabric
   mkdir -p /tmp/libfabric
   cd /tmp/libfabric
   wget -c https://github.com/ofiwg/libfabric/releases/download/v1.15.1/libfabric-1.15.1.tar.bz2
   tar xf libfabric-1.15.1.tar.bz2
   cd libfabric-1.15.1
   ./configure --prefix=/usr/local && \
   make -j
   make install

    ## OpenMPI installation
    echo "Installing Open MPI"
    export OMPI_DIR=/usr/local
    export OMPI_VERSION=4.1.5
    export OMPI_URL="https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-$OMPI_VERSION.tar.bz2"
    mkdir -p /tmp/ompi
    cd /tmp/ompi
    wget -c -O openmpi-$OMPI_VERSION.tar.bz2 $OMPI_URL && tar -xjf openmpi-$OMPI_VERSION.tar.bz2

    # Compile and install
    cd /tmp/ompi/openmpi-$OMPI_VERSION
   ./configure --prefix=$OMPI_DIR --with-pmix=/usr/local --with-libevent=/usr --with-ompi-pmix-rte --with-orte=no --disable-oshmem --enable-mpirun-prefix-by-default --enable-shared --with-ofi=/usr/local --without-verbs --with-hwloc
   make -j
   make install

   # Set env variables so we can compile our applications
   export PATH=$OMPI_DIR/bin:$PATH
   export LD_LIBRARY_PATH=$OMPI_DIR/lib:$LD_LIBRARY_PATH
   export MANPATH=$OMPI_DIR/share/man:$MANPATH

   ## Example MPI applications installation - OSU microbenchmarks
   cd /root
   wget -c https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.2.tar.gz
   tar xf osu-micro-benchmarks-7.2.tar.gz
   cd osu-micro-benchmarks-7.2/
   echo "Configuring and building OSU Micro-Benchmarks..."
   ./configure --prefix=/usr/local/osu CC=$(which mpicc) CXX=$(which mpicxx) CFLAGS=-I$(pwd)/util
   make -j
   make install


%runscript
  echo "Container will run: /usr/local/osu/libexec/osu-micro-benchmarks/mpi/$*"
  exec /usr/local/osu/libexec/osu-micro-benchmarks/mpi/$*

A simple jobscript to test the container would be:

#!/bin/bash -l
#SBATCH -J MPIContainerTest
#SBATCH -A <your account>
#SBATCH -p normal-x86
#SBATCH --nodes 2
#SBATCH --ntasks-per-node=16
#SBATCH -t 0:5:0

module load OpenMPI/4.1.5-GCC-12.3.0

mpirun singularity run rocky_8_5.sif pt2pt/osu_mbw_mr

This jobscript will test the available bandwidth (measured by osu_mbw_mr, part of the OSU Benchmarks).

openmpi.def

BootStrap: yum
OSVersion: 8.5
MirrorURL: https://dl.rockylinux.org/vault/rocky/%{OSVERSION}/BaseOS/aarch64/os/
Include: dnf

%environment
    export OMPI_DIR=/usr/local
    export SINGULARITY_OMPI_DIR=$OMPI_DIR
    export SINGULARITYENV_APPEND_PATH=$OMPI_DIR/bin
    export SINGULARITYENV_APPEND_LD_LIBRARY_PATH=$OMPI_DIR/lib
    export OMPI_MCA_mtl_ofi_verbose=0
%files
    /path/to/MLNX_OFED_LINUX-24.07-0.6.1.0-rhel8.5-aarch64.tgz /opt
%post
    #restorecon -Rv /var/lib/rpm
    rpmdb --rebuilddb
    rpm --import https://download.rockylinux.org/vault/rocky/8.5/PowerTools/aarch64/os/RPM-GPG-KEY-rockyofficial
    ## Prerequisites
    dnf update -y
    dnf install -y dnf-plugins-core
    dnf config-manager -y --set-enabled devel
    dnf groupinstall -y 'Development Tools'
    dnf install -y wget git bash hostname gcc gcc-gfortran gcc-c++ make file autoconf automake libtool zlib-devel python3
    dnf install -y libmnl lsof numactl-libs ethtool tcl tk

    ## Packages required for OpenMPI and PMIx
    dnf install -y libnl3 libnl3-devel
    dnf install -y libevent libevent-devel
    dnf repolist all
    dnf install -y munge munge-devel
    dnf install -y pciutils #perl-sigtrap pciutils
    # Mellanox OFED matching MeluXina
    mkdir -p /tmp/mofed
    cp /opt/MLNX_OFED_LINUX-24.07-0.6.1.0-rhel8.5-aarch64.tgz /tmp/mofed
    cd /tmp/mofed
    tar xf MLNX_OFED_LINUX-24.07-0.6.1.0-rhel8.5-aarch64.tgz
    cd MLNX_OFED_LINUX-24.07-0.6.1.0-rhel8.5-aarch64
    ./mlnxofedinstall --basic --user-space-only --without-fw-update --distro rhel8.5 --force

    # HWLOC for PMIx and OpenMPI
    mkdir -p /tmp/hwloc
    cd /tmp/hwloc
    wget https://download.open-mpi.org/release/hwloc/v2.9/hwloc-2.9.3.tar.bz2
    tar xf hwloc-2.9.3.tar.bz2
    cd hwloc-2.9.3/
    ./configure --prefix=/usr/local
    make -j
    make install

    # PMIx
    mkdir -p /tmp/pmix
    cd /tmp/pmix
    wget -c https://github.com/openpmix/openpmix/releases/download/v4.2.9/pmix-4.2.9.tar.gz
    tar xf pmix-4.2.9.tar.gz
    cd pmix-4.2.9
    ./configure --prefix=/usr/local --with-munge=/usr --with-libevent=/usr --with-zlib=/usr --enable-pmix-binaries --with-hwloc=/usr/local && \
    make -j
    make install

   # libfabric
   mkdir -p /tmp/libfabric
   cd /tmp/libfabric
   wget -c https://github.com/ofiwg/libfabric/releases/download/v1.15.1/libfabric-1.15.1.tar.bz2
   tar xf libfabric-1.15.1.tar.bz2
   cd libfabric-1.15.1
   ./configure --prefix=/usr/local && \
   make -j
   make install

    ## OpenMPI installation
    echo "Installing Open MPI"
    export OMPI_DIR=/usr/local
    export OMPI_VERSION=4.1.5
    export OMPI_URL="https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-$OMPI_VERSION.tar.bz2"
    mkdir -p /tmp/ompi
    cd /tmp/ompi
    wget -c -O openmpi-$OMPI_VERSION.tar.bz2 $OMPI_URL && tar -xjf openmpi-$OMPI_VERSION.tar.bz2

    # Compile and install
    cd /tmp/ompi/openmpi-$OMPI_VERSION
   ./configure --prefix=$OMPI_DIR --with-pmix=/usr/local --with-libevent=/usr --with-ompi-pmix-rte --with-orte=no --disable-oshmem --enable-mpirun-prefix-by-default --enable-shared --with-ofi=/usr/local --without-verbs --with-hwloc
   make -j
   make install

   # Set env variables so we can compile our applications
   export PATH=$OMPI_DIR/bin:$PATH
   export LD_LIBRARY_PATH=$OMPI_DIR/lib:$LD_LIBRARY_PATH
   export MANPATH=$OMPI_DIR/share/man:$MANPATH

   ## Example MPI applications installation - OSU microbenchmarks
   cd /root
   wget -c https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.2.tar.gz
   tar xf osu-micro-benchmarks-7.2.tar.gz
   cd osu-micro-benchmarks-7.2/
   echo "Configuring and building OSU Micro-Benchmarks..."
   ./configure --prefix=/usr/local/osu CC=$(which mpicc) CXX=$(which mpicxx) CFLAGS=-I$(pwd)/util
   make -j
   make install


%runscript
  echo "Container will run: /usr/local/osu/libexec/osu-micro-benchmarks/mpi/$*"
  exec /usr/local/osu/libexec/osu-micro-benchmarks/mpi/$*

A simple jobscript to test the container would be:

#!/bin/bash -l
#SBATCH -J MPIContainerTest
#SBATCH -A <your account>
#SBATCH -p normal-arm
#SBATCH --nodes 2
#SBATCH --ntasks-per-node=16
#SBATCH -t 0:5:0

module load OpenMPI/4.1.5-GCC-12.3.0

mpirun singularity run rocky_8_5.sif pt2pt/osu_mbw_mr

This jobscript will test the available bandwidth (measured by osu_mbw_mr, part of the OSU Benchmarks).