Hardware overview

Computing Hardware

Julia 2 consists of:

  • 20x Triple GPU nodes with dual 32 core CPUs
  • 20x Triple GPU nodes with dual 64 core CPUs
  • 3x Octa GPU nodes with L40s cards (MegWare)
  • 2x Octa GPU nodes with L40s cards (Supermicro)
  • 2x Octa GPU nodes with H100 cards
  • 18x Dual AMD Epyc 9654 96-Core machines
  • Some additional octa GPU nodes with L40s cards for special groups
  • Some additional octa GPU nodes with L40 cards

Triple GPU Nodes dual 32 cores

  • DELL PowerEdge R7525 2U servers
  • 2x AMD Epyc 7543 (3rd Gen)
    • 32 Cores / 64 Threads, Base: 2.80 GHz up to 3.7 GHz, 240W
    • 16x 32 GB RAM DDR4 3200 MT/s, Dual Rank (512 GB tot)
  • 3x NVIDIA L40, 300 W
    • 48 GB RAM ECC
    • 18.176 Cuda Cores, 142 RT-Cores (3rd Gen), 568 Tensor Cores (4th Gen)
    • No MiG
    • No NVLINK
  • 2x 480 GB SATA (rootfs)
  • 2x 1.92 TB NVME (cache/tmpfs)
  • 2x 10GB/s Ethernet

Triple GPU Nodes dual 64 cores

  • DELL PowerEdge R7525 2U servers
  • 2x AMD Epyc 7763 (3rd Gen)
    • 64 Cores / 128 Threads, Base: 2.54 GHz up to 3.5 GHz, 280W
    • 16x 32 GB RAM DDR4 3200 MT/s, Dual Rank (512 GB tot)
  • 3x NVIDIA L40, 300 W
    • 48 GB RAM ECC
    • 18.176 Cuda Cores, 142 RT-Cores (3rd Gen), 568 Tensor Cores (4th Gen)
    • No MiG
    • No NVLINK
  • 2x 240 GB SATA (rootfs)
  • 2x 7.68 TB NVME (cache/tmpfs)
  • 2x 10GB/s Ethernet

Octa GPU node with L40s (Megware)

  • Megware Saxonid C128 2U servers
  • 2x AMD Epyc 7713 (3rd Gen)
  • 64 Cores / 128 Threads, Base: 2.00 GHz up to 3.67 GHz, 225W
  • 16x 64 GB RAM DDR4 3200 MT/s, Dual Rank (1024 GB tot)
  • 8x NVIDIA L40s
    • 48 GB RAM ECC
    • 18.176 Cuda Cores, 142 RT-Cores (3rd Gen), 568 Tensor Cores (4th Gen)
    • No MiG
    • No NVLINK
  • 2x 480 GB SATA (rootfs)
  • 2x 1.92 TB NVME (cache/tmpfs)
  • 2x 10GB/s Ethernet

Octa GPU node with L40s (Supermicro)

  • Supermicro AS -4125GS-TNRT 4U servers
  • 2x AMD Epyc 9654 (4th Gen)
  • 96 Cores / 192 Threads, Base: 2.40 GHz up to 3.70 GHz, 360W
  • 32x 64 GB RAM DDR5 4800 MT/s, Dual Rank (1024 GB tot)
  • 8x NVIDIA L40s
    • 48 GB RAM ECC
    • 18.176 Cuda Cores, 142 RT-Cores (3rd Gen), 568 Tensor Cores (4th Gen)
    • No MiG
    • No NVLINK
  • 2x 480 GB SATA (rootfs)
  • 2x 7.68 TB NVME (cache/tmpfs)
  • 2x 10GB/s Ethernet

Octa GPU node with H100

  • DELL PowerEdge XE9680
  • 2x Intel Xeon Platinum 8480+ (4th Gen)
    • 56 Cores / 112 Threads, Base: 2.00 GHz up to 3.8 GHz, 350W
    • 32x 64 GB RAM DDR4 4800 MT/s, Dual Rank (2048 GB tot)
  • 8x NVIDIA H100, SXM, 700W
    • 80 GB RAM ECC
    • up to 7 MiG @ 10 GB
    • NVLINK
  • 1x 480 GB NVME (rootfs)
  • 2x 7.68 TB NVME (cache/tmpfs)
  • 2x 10GB/s Ethernet

Dual AMD Epyc 9654 96-Core machines

  • DELL PowerEdge R7625 2U servers
  • 2x AMD Epyc 9654 (4th Gen)
  • 96 Cores / 192 Threads, Base: 2.40 GHz up to 3.70 GHz, 360W
  • 32x 64 GB RAM DDR5 5600 MT/s, Dual Rank (1024 GB tot)
  • 1x 960 GB SSD (rootfs)
  • 2x 960 GB SSD (cache/tmpfs)
  • 2x 10GB/s Ethernet

Storage

  • 7x NVME nodes for /home
    • 538 TB /home gross
  • 21 HDD nodes for /data
    • 3.360 TB /data gross