GPU Sharing in Kubernetes: Step-by-Step Guide

When a workload requests a GPU in Kubernetes, by default, it allocates an entire GPU, even if the workload uses only a small portion of the GPU.

GPU sharing solves this by allowing workloads to share the same GPU.

In this guide, we will configure and test the three NVIDIA GPU sharing strategies in Kubernetes, such as Time-Slicing, MPS, and MIG.

We will use the NVIDIA GPU Operator and its custom Helm values to configure each strategy.

Also, we used the NVIDIA H100 GPU because we are configuring and testing NVIDIA MIG as well, which needs a MIG supported GPU type, and the H100 supports MIG.

If you are using a different GPU type. make sure it supports MIG.

Lets get started.

Setup Prerequisites

Below are the prerequisites for this setup.

Kubernetes Cluster with an NVIDIA GPU node
kubectl
Helm

💡

You can try this on any Kubernetes cluster which has atleast one NVIDIA GPU node avaibale

Clone Repository

We have pushed every Helm values file and test file for each sharing strategy into our GitHub repository.

Fork and clone the repository locally. Then move into the mlops-for-devops/platform-tools/gpu-sharing directory.

git clone https://github.com/techiescamp/mlops-for-devops.git

cd mlops-for-devops/phase-3-gpu-infra/gpu-sharing

There you can find the following directory structure.

gpu-sharing
  ├── mig
  │   ├── mix
  │   │   ├── mig-heavy.yaml
  │   │   ├── mig-light.yaml
  │   │   └── values.yaml
  │   └── single
  │       ├── mig-single.yaml
  │       └── values.yaml
  ├── mps
  │   ├── mps.yaml
  │   └── values.yaml
  └── slicing
      ├── slicing.yaml
      └── values.yaml

Lets look into each strategy in detail with hands-on example.

Lets start the hands-on section.

Here, we will configure and test each sharing strategy one by one.

Add NVIDIA GPU Operator Helm Chart

Since we are going to use the NVIDIA GPU Operator Helm Chart, add its repository first if it is not already added.

Run the following command to add it.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
    && helm repo update

We will be using custom Helm values of the NVIDIA GPU Operator to configure and use each sharing strategy one by one and deploy an example workload on it.

Configure Time-Slicing Strategy

Lets start with slicing.

In this section, we will use a custom Helm values file to enable time-slicing GPU sharing and deploy a test deployment to test it.

We will use the GPU Operator Helm chart to do the configurations.

Assuming you are in the mlops-for-devops/phase-3-gpu-infra/gpu-sharing directory, move into the slicing directory using the following command.

cd slicing

There you can find the values file values.yaml and a deployment file slicing.yaml which we are going to use in this section.

In the values file, it is set to replicas: 4, so the GPU will be advertised as 4 GPUs to Kubernetes, and it can schedule 4 workloads to the GPU at the same time.

Now, run the following Helm install command inside the slicing folder.

helm install --wait gpu-operator \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    -f values.yaml \
    --version=v26.7.1

Once the installation is completed, run the following command with your GPU node's name.

kubectl describe no <node-name> | grep -A10 "Allocatable:"

You will get a similar output showing that there are 4 GPUs.

Allocatable:
  cpu:                23840m
  ephemeral-storage:  684666794899
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             235610848Ki
  nvidia.com/gpu:     4
  pods:               110

Also, there will be labels added to the GPU node based on the sharing strategy we use.

Run the following command to check if the node is labeled with time-slicing label.

kubectl get no --show-labels | grep time-slicing

You can see the following label in the output.

nvidia.com/gpu.sharing-strategy=time-slicing

Testing Time Slicing Strategy

For testing the time slicing strategy, we will use the manifest file slicing.yaml inside the slicing folder.

The manifest will create a deployment with 4 replicas that will use the same GPU.

Run the following command to apply it.

kubectl apply -f slicing.yaml

Once applied, run the following command to check if the pods are up and running.

$ kubectl get po

NAME                                READY   STATUS    RESTARTS   AGE

timeslicing-demo-6f974bdf84-hlpsp   1/1     Running   0          8s
timeslicing-demo-6f974bdf84-px457   1/1     Running   0          8s
timeslicing-demo-6f974bdf84-wsmwz   1/1     Running   0          8s
timeslicing-demo-6f974bdf84-zg9mj   1/1     Running   0          8s

Always keep in mind that, while using the Time Slicing Strategy, all workloads share the same GPU without any isolation.

To check it, you use the following command.

for pod in $(kubectl get pods -l app=timeslicing-demo -o name); do
  echo "--- $pod ---"
  kubectl exec "$pod" -- \
    nvidia-smi --query-gpu=uuid --format=csv,noheader
done

You can see that all the pods use the same GPU.

--- pod/timeslicing-demo-6f974bdf84-4qccg ---
GPU-808c2abc-6e32-babd-e395-27e1ce85be2e
--- pod/timeslicing-demo-6f974bdf84-lkwwl ---
GPU-808c2abc-6e32-babd-e395-27e1ce85be2e
--- pod/timeslicing-demo-6f974bdf84-x7ksx ---
GPU-808c2abc-6e32-babd-e395-27e1ce85be2e
--- pod/timeslicing-demo-6f974bdf84-x9jkd ---
GPU-808c2abc-6e32-babd-e395-27e1ce85be2e

Before moving to the next strategy, remove the deployment we used for testing time-slicing using the following command.

kubectl delete deployment timeslicing-demo

The pods deployed on GPU might take more time to terminate than the pods deployed on CPU.

Because GPU workloads need time to clean up VRAM, CUDA, and workload memory from the GPU.

Configuring MPS Strategy

The second strategy we are going to configure is Multi-Process Service(MPS).

For that, move into the mps folder using the following command.

cd ../mps

There you can find two files, Helm values file values.yaml and a deployment file mps.yaml.

Since we will be using the same GPU Operator Helm chart, we will be using the Helm upgrade command to change the strategy to MPS.

helm upgrade --install --wait gpu-operator \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    -f values.yaml \
    --version=v26.7.1

Once applied, wait for the pods to restart.

For MPS, a new controller pod will be created to manage the MPS strategy.

$ kubectl get po -n gpu-operator | grep mps

nvidia-device-plugin-mps-control-daemon-j58wn  2/2     Running     0          2m32s

Same as the previous strategy, you will see the following labels on the GPU nodes.

nvidia.com/gpu.sharing-strategy=mps
nvidia.com/mps.capable=true

Testing MPS Strategy

For testing MPS, apply the manifest mps.yaml inside the MPS folder.

We will use the same deployment with 4 replicas as we did for the time-slicing strategy.

Run the following command to apply the manifest.

kubectl apply -f mps.yaml

You can see the following pods running in the default namespace.

$ kubectl get po

NAME                        READY   STATUS    RESTARTS   AGE

mps-demo-6f674bdf4-hlssp   1/1     Running   0          6s
mps-demo-6f674bdf4-pq457   1/1     Running   0          6s
mps-demo-6f674bdf4-wstwz   1/1     Running   0          6s
mps-demo-6f674bdf4-zu9mj   1/1     Running   0          6s

Since MPS is for equal resource allocation for each replica, we will use the following command to check how much of the GPU each pod is using.

⚠️

Update on of your pods name in the following command

kubectl exec <pod-name> -- \
  sh -c '
    echo get_default_active_thread_percentage |
    nvidia-cuda-mps-control
  '

You will get output as:

25.0

This means a single pod is limited by MPS to use 25% of the total GPU resource. Because we configured 4 replicas in the MPS configuration, it allocates the GPU resource in 4 equal parts.

Even if you reduce the deployment's replica count from 4 to 3, the percentage will be the same because we changed the deployment replica count, not the GPU replica count.

If you reduce the GPU replicas from 4 to 3, the usage will be approximately 33% for each replica.

Now, remove the deployment before moving to MIG.

kubectl delete deployment mps-demo

Configuring MIG Strategy

Now, lets configure the MIG strategy.

Unlike the previous two strategies, MIG slices the GPU at the hardware level instead of just sharing.

MIG has two strategy types, single and mixed.

MIG Single Strategy

First, move into the mig/single folder.

cd ../mig/single

There you can find a values file values.yaml and the deployment file mig-single.yaml.

Then, change to MIG Single strategy using the following Helm upgrade command.

💡

The GPU instance type we are using is H100 so 1g.10gb profile will support for it, if your GPU type is different, check the supported profiles and update your supported profile in the values file instead of 1g.10gb.

helm upgrade --install --wait gpu-operator \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    -f values.yaml \
    --version=v26.7.1

Once the GPU operator pods are restarted, label the node with the profile name.

⚠️

Update the node name and change the profile name if you are using a different profile

kubectl label node <node-name> \
    nvidia.com/mig.config=all-1g.10gb --overwrite

This will restart operator pods, wait until it is done.

💡

We are labeling the nodes because MIG relies on Kubernetes node labels to choose the profile, we need to activate.

Then, check the GPU allocation.

kubectl describe no <node-name> | grep -A10 "Allocatable:"

You will get the following output.

Allocatable:
  cpu:                      19850m
  ephemeral-storage:        684666794899
  hugepages-1Gi:            0
  hugepages-2Mi:            0
  memory:                   235611936Ki
  nvidia.com/mig-1g.10gb:   7
  pods:                     110

Testing MIG Single Strategy

Lets test the MIG Single strategy by creating a deployment using the mig-single.yaml manifest.

This will create a deployment with 4 replicas that use the MIG profile we configured before. Run the following command to apply it.

kubectl apply -f mig-single.yaml

Once applied, you can see the pods running in the default namespace.

$ kubectl get po

NAME                               READY   STATUS    RESTARTS   AGE
mig-single-demo-678f69c779-c6g42   1/1     Running   0          2s
mig-single-demo-678f69c779-drhhh   1/1     Running   0          2s
mig-single-demo-678f69c779-hw5s8   1/1     Running   0          2s
mig-single-demo-678f69c779-khm2p   1/1     Running   0          2s

And check each pod.

for pod in $(kubectl get pods -l app=mig-single-demo -o name); do
  echo "--- $pod ---"
  kubectl exec "$pod" -- nvidia-smi -L
done

Each pod will have a different MIG UUID as shown below.

💡

MIG UUID is an unique identifier assigned for each MIG slices.

--- pod/mig-single-demo-57658ff6b7-5bkmv ---
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-808c2abc-6e32-babd-e395-27e1ce85be2e)
  MIG 1g.10gb     Device  0: (UUID: MIG-b0abc112-d00d-5ef2-bcc5-78d1c28ac621)
--- pod/mig-single-demo-57658ff6b7-82zsd ---
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-808c2abc-6e32-babd-e395-27e1ce85be2e)
  MIG 1g.10gb     Device  0: (UUID: MIG-dbc2c893-6f17-5f5b-90f1-177d8b93a07d)
--- pod/mig-single-demo-57658ff6b7-g99kz ---
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-808c2abc-6e32-babd-e395-27e1ce85be2e)
  MIG 1g.10gb     Device  0: (UUID: MIG-c00d8185-f253-59fd-ac09-491a65b20ac9)
--- pod/mig-single-demo-57658ff6b7-hjsnk ---
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-808c2abc-6e32-babd-e395-27e1ce85be2e)
  MIG 1g.10gb     Device  0: (UUID: MIG-1d5247a0-b4c5-58b2-9ca5-8e887d6a2441)

This means each pod is using isolated MIG slices.

Now, delete the deployment before moving to the next section.

kubectl delete deployment mig-single-demo

MIG Mixed Strategy

This is the second type of MIG strategy.

With it, we will use different profile types at the same time.

Move into the mig/mix folder using the following command.

cd ../mix

There, you can see a Helm values file values.yaml and two deployment files mig-light.yaml and mig-heavy.yaml.

First, you need to reset the node label before upgrading. Because the operator needs to clear the existing configuration before changing the strategy.

kubectl label node <node-name> \
    nvidia.com/mig.config=all-disabled --overwrite

Now, run the Helm upgrade command to change the strategy to MIG Mix.

⚠️

Change the profile name in the values file, if you are using a different profile

helm upgrade --install --wait gpu-operator \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    -f values.yaml \
    --version=v26.7.1

Wait for the operator pods to restart and start running.

Then, add the label with the profile name.

kubectl label node <node-name> \
    nvidia.com/mig.config=large-plus-small --overwrite

Again, wait for pods to restart and describe the node.

kubectl describe no pool-bj2660mwe-3of18z | grep -A10 "Allocatable:"

You can see both profiles are used at the same time.

Allocatable:
  cpu:                     19850m
  ephemeral-storage:       684666794899
  hugepages-1Gi:           0
  hugepages-2Mi:           0
  memory:                  235611936Ki
  nvidia.com/gpu:          0
  nvidia.com/mig-1g.10gb:  2
  nvidia.com/mig-3g.40gb:  1
  pods:                    110

Testing MIG Mixed Strategy

To test the MIG Mixed strategy, we will be using two deployments to use two different profiles.

⚠️

Change the profile names in the deployment files if you are using a different profiles

First, apply mig-heavy.yaml manifest, which creates a deployment that uses nvidia.com/mig-3g.40gb.

kubectl apply -f mig-heavy.yaml

Then, apply mig-light.yaml manifest, which creates another deployment that uses nvidia.com/mig-1g.10gb.

kubectl apply -f mig-light.yaml

Once both are applied, check the pods are running.

$ kubectl get po

NAME                         READY   STATUS    RESTARTS   AGE
mig-heavy-7f46f99c6b-gbvtq   1/1     Running   0          2m2s
mig-light-6c95b9648b-497t9   1/1     Running   0          114s
mig-light-6c95b9648b-bfvzd   1/1     Running   0          114s

If the pods start running, run the following command, which shows the MIG UUID and the MIG profile the pods are using.

for pod in $(kubectl get pods \
  -l 'app in (mig-heavy,mig-light)' \
  -o name); do
  echo "--- $pod ---"
  kubectl exec "$pod" -- nvidia-smi -L
done

You can see each deployment's pods are using different profiles.

--- pod/mig-heavy-7f46f99c6b-gbvtq ---
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-808c2abc-6e32-babd-e395-27e1ce85be2e)
  MIG 3g.40gb     Device  0: (UUID: MIG-2bb86c61-14b3-54a4-b66f-7ce790800659)
--- pod/mig-light-6c95b9648b-497t9 ---
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-808c2abc-6e32-babd-e395-27e1ce85be2e)
  MIG 1g.10gb     Device  0: (UUID: MIG-1d5247a0-b4c5-58b2-9ca5-8e887d6a2441)
--- pod/mig-light-6c95b9648b-bfvzd ---
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-808c2abc-6e32-babd-e395-27e1ce85be2e)
  MIG 1g.10gb     Device  0: (UUID: MIG-cba658f3-fb61-52a8-8079-0ed3f1470615)

Clean Up

First, delete the two deployments created for the mig-heavy and mig-light example.

kubectl delete deployment mig-heavy

kubectl delete deployment mig-light

Then, clean up the operator pods using the following command.

helm uninstall gpu-operator -n gpu-operator

Conclusion

In this guide, we have configured GPU sharing strategies like Time-Slicing, MPS, and MIG.

And we have tested each strategy with an example deployment to see how they work.

We cannot use a single sharing strategy for all types of workloads. Choose based on what your workloads need. For example, if your workload needs hardware-level isolation, use MIG.