Deucalion
Architecture
Compute nodes
Deucalion has a heterogeneous architecture and is the first in Europe to incorporate arm and x86 microprocessors in the same cluster. The following table will give you insight into Deucalion compute nodes components.
| Partition | Compute Nodes | CPU | GPU | Memory | Disk | Network |
|---|---|---|---|---|---|---|
| arm | 1632 | Fujitsu A64FX 48-core 2.0 GHz | No GPU | 32GB | WD CL SN720 512GB PCIe | NVIDIA ConnectX-6 100 Gb/s |
| x86 | 500 | 2x AMD EPYC 7742 64-core 2.25 GHz | No GPU | 256GB DDR4 | 480GB SSD | NVIDIA ConnectX-6 200 Gb/s |
| a100-40 | 17 | 2x AMD EPYC 7742 64-core 2.25 GHz | 4x NVIDIA Ampere A100 40 GB | 512 GB DDR4 | Micron 5300 480GB SATA | 2x NVIDIA ConnectX-6 200 Gb/s |
| a100-80 | 16 | 2x AMD EPYC 7742 64-core 2.25 GHz | 4x NVIDIA Ampere A100 80 GB | 512 GB DDR4 | Micron 5300 480GB SATA | 2x NVIDIA ConnectX-6 200 Gb/s |
Software
| Component | Description | Version |
|---|---|---|
| Operating System | Rocky Linux | 8 |
| Software environment/package system | Modules | 8.5 |
| Containers | Singularity | 3.11 |
| Resource manager and scheduling | Slurm | 23.11.4 |
Storage
| Purpose | Overview | File System | Mount points |
|---|---|---|---|
| High Speed Storage | Metadata and Hot Pools NVMe – 430TB HDD Datapools – 10PB MDS Servers – 8 Nodes OSS I/O Servers – 32 Nodes Aggregated Performance - 340GB/s in reads, 260GB/s in writes |
Lustre Parallel File System | /share /projects |
| NAS Storage | Total SSD net Capacity – 50TB Data Modules – 2 for redundancy Connection type – 8x10 GbE |
NFSv4 | /home /apps |
Network
| Segment | Switch model | Units | Ports | Network Operting System |
|---|---|---|---|---|
| Top of Rack | NVIDIA Mellanox AS4610 | 64 | 48x 1G Base-T 4x 10G SFP+ |
Cumulus Linux |
| Core | NVIDIA Mellanox SN2410 | 2 | 48x 25G SFP28 8x 100G QSFP |
Cumulus Linux |
| Core | NVIDIA Mellanox SN2010 | 4 | 18x 25G SFP28 4x 100G QSFP |
Cumulus Linux |
ARM Infiniband
Infiniband Fat-Tree with 1:6 blocking factor using 50 switches NVIDIA Mellanox QM8790 with NVIDIA MLNX-OS
| Segment | Units | IB links per switch |
|---|---|---|
| Leaf | 35 | 24 downlinks 15 uplinks |
| Spine | 15 | 35 downlinks 5 uplinks |
x86 Infiniband
Infiniand Fat-Tree with 1:1 non-blocking factor using 27 switches NVIDIA Mellanox QM8790 with NVIDIA MLNX-OS
| Segment | Units | IB links per switch |
|---|---|---|
| Leaf | 17 | 20 downlinks 20 uplinks |
| Spine | 10 | 34 doanlinks 6 uplinks |