Skip to content

Getting started with Kubernetes

Creating your first pod template within a job yaml file

To access the GPUs on the service, it is recommended to start with one of the prebuilt container images provided by Nvidia, these images are intended to perform different tasks using Nvidia GPUs.

The list of Nvidia images is available on their website.

The following example uses their CUDA sample code simulating nbody interactions.

apiVersion: batch/v1
kind: Job
metadata:
 generateName: jobtest-
 labels:
  kueue.x-k8s.io/queue-name:  <tre-project-namespace>-user-queue
spec:
 completions: 1
 backoffLimit: 1
 ttlSecondsAfterFinished: 1800
 template:
  metadata:
   name: job-test
  spec:
   containers:
   - name: cudasample
     image: tre-ghcr-proxy.nsh.loc:5000/<github_user>/cuda-sample:nbody-cuda11.7.1
     args: ["-benchmark", "-numbodies=512000", "-fp64", "-fullscreen"]
     resources:
      requests:
       cpu: 2
       memory: '1Gi'
      limits:
       cpu: 2
       memory: '4Gi'
       nvidia.com/gpu: 1
   restartPolicy: Never

The pod resources are defined under the resources tags using the requests and limits tags.

Resources defined under the requests tags are the reserved resources required for the pod to be scheduled.

If a pod is assigned to a node with unused resources then it may spontaneously burst up to use resources beyond those requested, and up to the limits. This may allow the task within the pod to run faster, but it will also throttle back down when further pods are scheduled to the node.

The SHS GPU Cluster requires that all pods have requests and limits tags for CPU and memory defined in order to be accepted.

GPU resources requests are optional and only an entry under the limits tag is needed to specify the use of a GPU, nvidia.com/gpu: 1. Without this no GPU will be available to the pod.

The label kueue.x-k8s.io/queue-name specifies the queue you are submitting your job to. This is part of the Kueue system in operation on the Cluster, to allow for improved resource management for users.

Submitting your first job

  1. Open an editor of your choice and create the file test_NBody.yml
  2. Copy the above job yaml in to the file, filling in <tre-project-namespace>-user-queue, e.g. nsh-2024-0000-ns-user-queue:
  3. Save the file and exit the editor
  4. Run kubectl -n <tre-project-namespace> create -f test_NBody.yml

  5. This will output something like the following:

    job.batch/jobtest-b92qg created

    The five character code appended to the job name, i.e. b92qg above, is randomly generated and will differ from your run.

  6. Run kubectl -n <tre-project-namespace> get jobs

  7. This will output something like the following:

    NAME            COMPLETIONS   DURATION   AGE
jobtest-b92qg   1/1           48s        29m

    There may be more than one entry, as this displays all the jobs in the current namespace, listing their name, number of completions against required completions, duration and age.

  8. Inspect your job further using the command kubectl -n <tre-project-namespace> describe job jobtest-b92qg, updating the job name with your five character code.

  9. This will output something like the following:

    Name:             jobtest-b92qg
Namespace:        t4
Selector:         controller-uid=d3233fee-794e-466f-9655-1fe32d1f06d3
Labels:           kueue.x-k8s.io/queue-name=t4-user-queue
Annotations:      batch.kubernetes.io/job-tracking:
Parallelism:      1
Completions:      3
Completion Mode:  NonIndexed
Start Time:       Wed, 14 Feb 2024 14:07:44 +0000
Completed At:     Wed, 14 Feb 2024 14:08:32 +0000
Duration:         48s
Pods Statuses:    0 Active (0 Ready) / 3 Succeeded / 0 Failed
Pod Template:
    Labels:  controller-uid=d3233fee-794e-466f-9655-1fe32d1f06d3
            job-name=jobtest-b92qg
    Containers:
        cudasample:
            Image:      tre-ghcr-proxy.nsh.loc:5000/<github_user>/cuda-sample:nbody-cuda11.7.1
            Port:       <none>
            Host Port:  <none>
            Args:
                -benchmark
                -numbodies=512000
                -fp64
                -fullscreen
            Limits:
                cpu:             2
                memory:          4Gi
                nvidia.com/gpu:  1
            Requests:
                cpu:        2
                memory:     1Gi
            Environment:  <none>
            Mounts:       <none>
    Volumes:        <none>
Events:
Type    Reason            Age    From                        Message
----    ------            ----   ----                        -------
Normal  Suspended         8m1s   job-controller              Job suspended
Normal  CreatedWorkload   8m1s   batch/job-kueue-controller  Created Workload: t4/job-jobtest-b92qg-3b890
Normal  Started           8m1s   batch/job-kueue-controller  Admitted by clusterQueue project-cq
Normal  SuccessfulCreate  8m     job-controller              Created pod: jobtest-b92qg-lh64s
Normal  Resumed           8m     job-controller              Job resumed
Normal  SuccessfulCreate  7m44s  job-controller              Created pod: jobtest-b92qg-xhvdm
Normal  SuccessfulCreate  7m28s  job-controller              Created pod: jobtest-b92qg-lvmrf
Normal  Completed         7m12s  job-controller              Job completed

  10. Run kubectl -n <tre-project-namespace> get pods

  11. This will output something like the following:

    NAME                  READY   STATUS      RESTARTS   AGE
jobtest-b92qg-lh64s   0/1     Completed   0          11m

    Again, there may be more than one entry as this displays all the jobs in the current namespace. Also, each pod within a job is given another unique five-character code appended to the job name.

  12. View the logs of a pod from the job you ran kubectl -n <tre-project-namespace> logs jobtest-b92qg-lh64s - again update with you run's pod and job five-letter code.

  13. This will output something like the following:

    Run "nbody -benchmark [-numbodies=<numBodies>]" to measure performance.
    -fullscreen       (run n-body simulation in fullscreen mode)
    -fp64             (use double precision floating point values for simulation)
    -hostmem          (stores simulation data in host memory)
    -benchmark        (run benchmark to measure performance)
    -numbodies=<N>    (number of bodies (>= 1) to run in simulation)
    -device=<d>       (where d=0,1,2.... for the CUDA device to use)
    -numdevices=<i>   (where i=(number of CUDA devices > 0) to use for simulation)
    -compare          (compares simulation results running once on the default GPU and once on the CPU)
    -cpu              (run n-body simulation on the CPU)
    -tipsy=<file.bin> (load a tipsy model file for simulation)

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

> Fullscreen mode
> Simulation data stored in video memory
> Double precision floating point simulation
> 1 Devices used for simulation
GPU Device 0: "Ampere" with compute capability 8.0

> Compute 8.0 CUDA device: [NVIDIA A100-SXM4-40GB]
number of bodies = 512000
512000 bodies, total time for 10 iterations: 10570.778 ms
= 247.989 billion interactions per second
= 7439.679 double-precision GFLOP/s at 30 flops per interaction

  14. Delete your job with kubectl -n <tre-project-namespace> delete job jobtest-b92qg - this will delete the associated pods as well.

Specifying GPU requirements

At present, the SHS GPU Cluster only provides one type of GPU:[NVIDIA A100-SXM4-40GB].

If you create multiple jobs with the same definition file and compare their log files you may notice the CUDA device may differ from Compute 8.0 CUDA device: [NVIDIA A100-SXM4-40GB].

The GPU Operator on K8s is allocating the pod to the first node with a GPU free that matches the other resource specifications irrespective of the type of GPU present on the node.

For future reference (if additional GPU-types become available), you can make the GPU resource request more specific by adding a node selector to your pod template, as per the following example:

  • nvidia.com/gpu.product: 'NVIDIA-A100-SXM4-40GB'

Example yaml file with GPU type specified

The nodeSelector: key at the bottom of the pod template states the pod should be ran on a node with a NVIDIA-A100-SXM4-40GB GPU.

Exact GPU product names only

K8s will fail to assign the pod if you misspell the GPU type.

Be especially careful with memory allocation, as attempting to load GPUs with more data than its memory can handle can have unexpected consequences.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: jobtest-
  labels:
    kueue.x-k8s.io/queue-name:  <tre-project-namespace>-user-queue
spec:
  completions: 1
  backoffLimit: 1
  ttlSecondsAfterFinished: 1800
  template:
    metadata:
      name: job-test
    spec:
      containers:
      - name: cudasample
        image: tre-ghcr-proxy.nsh.loc:5000/<github_user>/cuda-sample:nbody-cuda11.7.1
        args: ["-benchmark", "-numbodies=512000", "-fp64", "-fullscreen"]
        resources:
          requests:
            cpu: 2
            memory: '1Gi'
          limits:
            cpu: 2
            memory: '4Gi'
            nvidia.com/gpu: 1
      restartPolicy: Never
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-A100-SXM4-40GB

Running multiple pods with K8s jobs

Wrapping a pod within a job provides additional functionality on top of accessing the queuing system.

Firstly, the restartPolicy within a job enables the self-healing mechanism within K8s so that if a node dies with the job's pod on it, then the job will find a new node to automatically restart the pod.

Jobs also allow users to define multiple pods that can run in parallel or series, and will continue to spawn pods until a specific number of pods successfully terminate.

See below for an example K8s job that requires three pods to successfully complete the example CUDA code before the job itself ends.

apiVersion: batch/v1
kind: Job
metadata:
 generateName: jobtest-
 labels:
    kueue.x-k8s.io/queue-name:  <tre-project-namespace>-user-queue
spec:
 completions: 1
 backoffLimit: 1
 ttlSecondsAfterFinished: 1800
 parallelism: 1
 template:
  metadata:
   name: job-test
  spec:
   containers:
   - name: cudasample
     image: tre-ghcr-proxy.nsh.loc:5000/<github_user>/cuda-sample:nbody-cuda11.7.1
     args: ["-benchmark", "-numbodies=512000", "-fp64", "-fullscreen"]
     resources:
      requests:
       cpu: 2
       memory: '1Gi'
      limits:
       cpu: 2
       memory: '4Gi'
       nvidia.com/gpu: 1
   restartPolicy: Never

Change the default kubectl namespace in the project kubeconfig file

Passing the -n <tre-project-namespace> flag every time you want to interact with the cluster can be cumbersome.

You can alter the kubeconfig on your VM to send commands to your project namespace by default. Pleaase follow the below instructions to achieve that:

  1. Open the command line on your SHS VM with access to the SHS GPU Cluster.

  2. Open the kubeconfig file.

    vi ~/.kube/config

  3. Add the namespace line with your project's kubernetes namespace to the "shs-gpu-cluster" context entry in your copy of the config file.

    *** MORE CONFIG ***

contexts:
- name: "shs-gpu-cluster"
  context:
    user: "shs-gpu-cluster"
    namespace: "<tre-project-namespace>" # INSERT LINE
    cluster: "shs-gpu-cluster"

*** MORE CONFIG ***

  4. Check kubectl connects to the Cluster. If this does not work contact your dedicated helpdesk.

    kubectl get pods