Resource changes for all Apocrita and Andrena GPU jobs

To make better use of the resources available on GPU nodes, the Apocrita and Andrena GPUs now support 12 cores per GPU. Please update your job scripts from 8 cores and 11G per GPU to 12 cores and 7.5G per GPU - this will maintain approximately the same total RAM per job, while increasing the core count.

This change has been made because the majority of GPU nodes attached to the HPC systems have more than 32 cores, which lead to several cores being unused.

Additionally, the newer GPU nodes contain more than 350G of RAM (the previous maximum memory available inside a single GPU node), so it no longer made sense to offer 88G per GPU (8 cores and 11G RAM per core) on all GPU nodes.

The ability to submit 8-core GPU jobs will be removed in due course.

What has changed?

• All GPU jobs (Apocrita and Andrena) should now request 12 cores per GPU
• All Apocrita GPU jobs should now request 7.5G per core (using 11G per core will restrict which GPU nodes will accept your request)
• The Science and Engineering GPU nodes now accept up to 20G per core (12 core jobs)
• All Andrena GPU jobs should now request 7.5G per core

8 cores per GPU deprecation warning

8 cores per GPU is deprecated and will be removed shortly. Please request 12 cores per GPU for all GPU jobs. See below for further details.

Below shows the expected outcome for various submission parameters when requesting 1 GPU.

Apocrita

This table applies to the following nodes:

• nxg1 and nxg2
• rdg1-13
• sbg1-5
• xdg1 and xdg2

** Number of cores **	** Memory per core **	** Outcome **
8 cores	7.5G	job accepted with 8 -> 12 warning
8 cores	11G	job accepted with 8 -> 12 warning
8 cores	>11G without -l gpuhighmem	job rejected
8 cores	20G with -l gpuhighmem (S&E only)	job accepted with 8 -> 12 warning
12 cores	7.5G	job accepted
12 cores	11G	job accepted with memory warning
12 cores	>11G without -l gpuhighmem	job rejected
12 cores	20G with -l gpuhighmem (S&E only)	job accepted

Users with access to the Science and Engineering RDG and XDG nodes may optionally request 20G per core with the following submission parameters:

#$ -pe smp 12
#$ -l h_vmem=20G
#$ -l gpuhighmem

The above parameters will not work for any other node type.

Andrena

This table applies to the following nodes:

• sbg7-sbg23 (including Apocrita node sbg19)

** Number of cores **	** Memory per core **	** Outcome **
8 cores	7.5G	job accepted with 8 -> 12 warning
8 cores	11G	job rejected
8 cores	>11G without -l gpuhighmem	job rejected
8 cores	20G with -l gpuhighmem (S&E only)	job rejected
12 cores	7.5G	job accepted
12 cores	11G	job rejected
12 cores	>11G without -l gpuhighmem	job rejected
12 cores	20G with -l gpuhighmem (S&E only)	job rejected

Getting help

If you require further assistance, please take a read through our Using GPUs documentation, or contact us.