GPGPU computation servers

• ctgpgpu4:
  • PowerEdge R730
  • 2 x Intel Xeon E52623v4
  • 128 GB RAM (4 DDR4 DIMM 2400MHz)
  • 2 x Nvidia GP102GL 24GB [Tesla P40]
  • AlmaLinux 9.1
    • Cuda 12.0
    • Mandatory use of Slurm queue manager.

• HPC cluster servers: HPC cluster
• CESGA servers: Access procedure info

* ctgpgpu5:

• PowerEdge R730
• 2 x Intel Xeon E52623v4
• 128 GB RAM (4 DDR4 DIMM 2400MHz)
• 2 x Nvidia GP102GL 24GB [Tesla P40]
• Ubuntu 18.04
  • Slurm as a mandatory use queue manager.
  • Modules for library version management .
  • CUDA 11.0
  • OpenCV 2.4 and 3.4
  • Atlas 3.10.3
  • MAGMA
  • TensorFlow
  • Caffee

• ctgpgpu6:
  • Server SIE LADON 4214
  • 2 processors Intel Xeon Silver 4214
  • 192 GB RAM memory(12 DDR4 DIMM 2933MHz)
  • Nvidia Quadro P6000 24GB (2018)
  • Nvidia Quadro RTX8000 48GB (2019)
  • Operating system Centos 7.7
    • Nvidia Driver 418.87.00 for CUDA 10.1
    • Docker 19.03
    • Nvidia-docker
• ctgpgpu9:
  • Dell PowerEdge R750
  • 2 x Intel Xeon Gold 6326
  • 128 GB RAM
  • 2 x NVIDIA Ampere A100 80 GB
  • AlmaLinux 8.6
    • NVIDIA 515.48.07 driver and CUDA 11.7
• ctgpgpu10:
  • PowerEdge R750
  • 2 x Intel Xeon Gold 5317
  • 128 GB RAM
  • NVIDIA Ampere A100 80 GB
  • Sistema operativo AlmaLinux 8.7
    • Driver NVIDIA 525.60.13 and CUDA 12.0
• ctgpgpu11:
  • Server Gybabyte G482-Z54
  • 2 x AMD EPYC 7413 @2,65 GHz 24c
  • 256 GB RAM
  • 4 x NVIDIA Ampere A100 de 80 GB
  • AlmaLinux 9.1
    • Driver NVIDIA 520.61.05 and CUDA 11.8
• ctgpgpu12:
  • Servidor Dell PowerEdge R760
  • 2 x Intel Xeon Silver 4410Y
  • 384 GB RAM
  • 2 x NVIDIA Hopper H100 de 80 GB
  • Sistema operativo AlmaLinux 9.2
    • Driver NVIDIA 535.104.12 and CUDA 12.2

Not all servers are available to use freely. Access must be requested filling the requests and problem reporting form. Users without access permission will receive an incorrect password error message.

Use SSH. Hostnames and ip addresses are:

• ctgpgpu4.inv.usc.es - 172.16.242.201:22
• ctgpgpu5.inv.usc.es - 172.16.242.202:22
• ctgpgpu6.inv.usc.es - 172.16.242.205:22
• ctgpgpu9.inv.usc.es - 172.16.242.94:22
• ctgpgpu10.inv.usc.es - 172.16.242.95:22
• ctgpgpu11.inv.usc.es - 172.16.242.96:22
• ctgpgpu12.inv.usc.es - 172.16.242.97:22

Connection in only possible from inside the CITIUS network. To connect from other places or from the RAI network it is necessary to use the VPN or the SSH gateway.

The servers switch themselves off after an hour of being idle. To switch them on again use the remote power service.

Servers won't switch themselves off if there is an open SSH or Screen session.

On servers where there is a queue management software installed its use is mandatory to send jobs and avoid conflicts between different processes because two jobs shouldn't be executed at the same time.

To send a job to the queue command srun is used:

srun cuda_program arguments_of_cuda_program

The srun process waits until the job is executed before returning control to the user. If you don't want to wait a console session manager like screen can be used. This way you can leave the the job in the queue and disconnect the session without losing the output of the job witch can be recovered any other moment.

Alternatively nohup can be used and then the job sent to the background with &. This way the output is written in the file nohup.out:

nohup srun cuda_program cuda_program_arguments &

To check the queue status command squeue is used. The command shows an output similar to this one:

JOBID PARTITION     NAME     USER  ST       TIME  NODES NODELIST(REASON)
9  servidore ca_water pablo.qu    PD       0:00      1 (Resources)
10 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
11 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
12 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
13 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
14 servidore ca_water pablo.qu    PD       0:00      1 (Priority)
 8 servidore ca_water pablo.qu     R       0:11      1 ctgpgpu2

An interactive view can be obtained, refreshed every second, with the smap command:

smap -i 1