> ## Content Index > Fetch the complete content index at: https://devopscube.com/llms.txt > Use this file to discover other available public pages before exploring further. # Deploy ML Model on Kubernetes with KServe (Step-by-Step Guide) - URL: https://devopscube.com/deploy-ml-model-kubernetes-kserve/ - Published: 2026-05-22T16:49:00.000Z - Updated: 2026-07-23T11:10:57.000Z - Author: Bibin Wilson - Tags: #Migrated-1741795015845, #wp, #wp-post, #Import 2025-03-12 15:57, #blog, Kubernetes, AI/ML, #syntax-highlight, MLOPS In this guide, you will learn how to deploy a machine learning model on a Kubernetes cluster using **KServe** model serving. Here is what we will cover in this guide. - What is Kserve? - Deploying KServe on Kubernetes - Deploying a sample scikit-learn ML model using KServe - Test the deployed model using its inferencing endpoint. 🎯 ****Learning Focus**: The goal of this guide is to understand how to deploy ML models with KServe, not how to build ML models. We have given a simple model so you can concentrate on the Kubernetes and KServe deployment concepts. Anyone can try this guide without AI/Ml background. ## What is KServe? When we train a machine learning model, the next step is to **serve** it so others can use it for predictions. Serving means loading the trained model, running it inside an inference server, and exposing an endpoint for apps or users to send requests and get results. 💡 Inference in machine learning means using a trained model to make predictions on new and unseen data. You cannot deploy an ML model on Kubernetes the same way you deploy regular workloads. Models need an inference server that handles prediction requests. KServe makes this process simple. [KServe](https://kserve.github.io/website/?ref=devopscube.com) is an open-source ML model serving tool for [Kubernetes](https://devopscube.com/kubernetes-tutorials-beginners/), which helps you serve your ML models on a Kubernetes cluster with minimal effort. You can deploy Kserve in two modes. 1. **Knative (default mode):** This mode reqires Knative components and this is particularly good for advanced setups. 2. **RawDeployment mode:** If you plan to deploy your models in a simple setup RawDeployment is the best option. In this guide, we will focus on the Kserver installation using the RawDeployment mode and show you how to serve a simple ML model on Kubernetes. ## Kserve Model Serving Workflow To get started with Kserve, we are going to use the following Kserve workflow to serve a model. ![](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/09/kserve-3.png) Here is how in works. 1. The KServe controller running in the Kubernetes cluster continuously looks for **`InferenceService`** resources that are created. 2. When a user creates the **`InferenceService`** resource, KServe detects it and creates the following required objects. - A Deployment with a **Pod** to run the model server. - A **Service** to expose the Pod as an endpoint. - **HPA** toscale up/down based on load. 3. The Pod then pulls a SKlearn model server **image** from the **container registry**. This image contains the libraries to serving the model. 4. The pod then uses the PVC URI to access the model.pkl file. 5. As explained in second point, KServe automatically exposes a **Kubernetes Service** endpoint. This becomes the URL where clients can send **API requests** for predictions. 6. Finally, you or an app can send data to the **Model Endpoint** for inferencing. In Kserver, you can store your model in following ways: 1. It can be stored in an object storage like [AWS S3](https://devopscube.com/configure-loki-s3/) or Azure Blob Storage. 2. Store it as a [container image](https://devopscube.com/build-docker-image-kubernetes-pod/). 3. Store it in your clusters [Persistent Volume](https://devopscube.com/provsion-persistent-volume-on-eks/). For other storage options, refer the [offical documentation](https://kserve.github.io/website/docs/model-serving/storage/overview?ref=devopscube.com). 💡 In this guide, we are going to store the ****model in a PVC** and use it for the deployment. ## ML Model Details & Context For this setup, We will use a sample model that is a [**scikit-learn**](https://scikit-learn.org/stable/?ref=devopscube.com) **text classification** pipeline that categorizes words into three types. - **Animals** (label: 0) - **Birds** (label: 1) - **Plants** (label: 2) You give a word to the model and it tells you whether that word represents an animal, bird, or plant. This is a common type of machine learning problem called **supervised learning classification**. 💡 The pre-trained `model.pkl` file is already included in the GitHub repository, so you don't need to create or train any model yourself. This model is provided ****purely for learning purposes** to demonstrate KServe deployment concepts. ## Setup Prerequisites Following are the pre-requisites to follow this setup. 1. [Kubernetes Cluster](https://devopscube.com/setup-kubernetes-cluster-kubeadm/) 2. [Kubectl](https://devopscube.com/kubectl-set-context/) 3. [Docker](https://devopscube.com/what-is-docker/) 4. [Helm](https://devopscube.com/install-configure-helm-kubernetes/) Lets get started. ## Install KServe in Kubernetes Follow the steps below to install KServe on your Cluster. ### Step 1: Install Cert Manager Let's start with installing Cert Manager, which is essential for creating and managing [TLS certificates](https://devopscube.com/configure-ingress-tls-kubernetes/) for KServe. You refer to the official site for the [latest version](https://cert-manager.io/docs/releases/?ref=devopscube.com). Run the following command to install [Cert Manager](https://devopscube.com/nginx-ingress-with-cert-manager/). All the components gets deployed in the **`cert-manager`** namespace. ```bash kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.0/cert-manager.yaml ``` Ensure all the three cert-manager components are in running state. ```bash kubectl get po -n cert-manager ``` ### Step 2: Install KServe CRDs Once the Cert Manager is installed, install KServe. First, start by installing the required CRDs of KServe using helm. To install the CRDs, run the following command. It also creates the kserve namespace where the **`kserve`** controller will be deployed. ```bash helm install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd --version v0.18.0 -n kserve --create-namespace ``` Now, verify the kserver CRD's ```bash kubectl get crds | grep kserve ``` ### Step 3: Deploy KServe Controller Now run the following command to install the KServe controller. ```bash helm install kserve oci://ghcr.io/kserve/charts/kserve-resources --version v0.18.0 \ --set kserve.controller.deploymentMode=Standard \ -n kserve ``` In the above command, you can see a flag that specifies the deployment mode to `Standard`. Verify if the kserve controller is in a running state. The controller pod runs **`kube-rbac-proxy`** and the controller containers. ```bash $ kubectl get po -n kserve NAME READY STATUS RESTARTS AGE kserve-controller-manager-59d84566d-grswq 2/2 Running 0 103s ``` ### Step 4: Install Runtimes KServe has build in runtimes that give the details to serving pod to serve a model. Use the following command to deploy them. ```bash helm install kserve-runtime-configs oci://ghcr.io/kserve/charts/kserve-runtime-configs \ --set kserve.servingruntime.enabled=true \ --version v0.18.0 ``` Then use the following command to check if they are created. ```bash $ kubectl get clusterservingruntimes NAME DISABLED MODELTYPE CONTAINERS AGE kserve-huggingfaceserver false huggingface kserve-container 6s kserve-huggingfaceserver-multinode false huggingface kserve-container 6s kserve-lgbserver false lightgbm kserve-container 6s kserve-mlserver false sklearn kserve-container 6s kserve-paddleserver false paddle kserve-container 6s kserve-pmmlserver false pmml kserve-container 6s kserve-predictiveserver false sklearn kserve-container 6s kserve-sklearnserver false sklearn kserve-container 6s kserve-tensorflow-serving false tensorflow kserve-container 6s kserve-torchserve false pytorch kserve-container 6s kserve-tritonserver false tensorrt kserve-container 6s kserve-xgbserver false xgboost kserve-container 6s ``` Apart from built-in runtimes, you can also create custom runtimes as per your need. Now that the setup is done, let's move on to the model deployment. ## KServe Sample Project Repository All the files and model we are going to use in this guide are from our GitHub repository. Run the following command to clone the repository. ```bash git clone https://github.com/devopscube/predictor-model.git ``` You can see the following directory structure. ```bash predictor-model ├── Dockerfile ├── README.md ├── inference.yaml ├── job.yaml └── model └── model.pkl ``` 1. **Dockerfile** \- For dockerizing the model. 2. **inference.yaml** \- Manifest file to create a [kubernetes resource](https://devopscube.com/kubernetes-resoruces/) that hosts the model on Kubernetes using KServe. 3. **job.yaml** \- Manifest that create a PVC and a job that copies the model into the PVC. CD into the `predictor-model` directory and follow the steps below. ```bash cd predictor-model ``` ## Deploy a Sample ML Model with KServe Follow the steps given below to deploy the model. ### Step 1: Dockerize the Model (Optional) ⚠️ Use this section if you are going to create your own container image. If you dont want to build your own image, use our `devopscube/predictor-model:1.0` image to follow the tutorial. The Dockerfile used to Dockerize the model is given below. ```Dockerfile FROM alpine:latest WORKDIR /app COPY model/ ./model/ ``` Here is the [Dockerfile](https://devopscube.com/create-dockerfile-using-docker-init/) explanation. 1. The Dockerfile uses the `alpine:latest` as the base image. 2. It sets the `/app` directory as the work directory and copies the model directory and the model file inside it to the `/app` directory. Now, run the following command to dockerize the model. ⚠️ Update your Docker registry name in below command or use our image. ```bash docker build -t devopscube/predictor-model:1.0 . ``` Once it's built, run the following command to push the image to the registry. ```bash docker push devopscube/predictor-model:1.0 ``` 💡 In real-world production setups, models are usually stored in ****cloud storage**. This is because ML models can be very large (hundreds of MBs or even GBs), and storing them in object storage allows for better scalability and retrieval. Another Kubernete native ML feature called [imageVolumes](https://devopscube.com/oci-image-volume-kubernetes-pods/) let you package models as contianer image and mount as volumes in Pods. ### Step 2: Store the Model in PVC To copy the model into a PV, we are going to create a PVC and a [job](https://devopscube.com/create-kubernetes-jobs-cron-jobs/) that copies the model to the PVC Here is the **`job.yaml`** with PVC manifest. ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: predictor-model-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi --- apiVersion: batch/v1 kind: Job metadata: name: predictor-model-copy-job spec: ttlSecondsAfterFinished: 10 backoffLimit: 1 template: spec: restartPolicy: OnFailure containers: - name: model-writer image: devopscube/predictor-model:1.0 command: [ "/bin/sh", "-c" ] args: - | echo ">>> Copying model to PVC..."; cp -r /app/model/* /mnt/models/; echo ">>> Verifying contents in PVC..."; ls -lh /mnt/models; echo ">>> Verification complete. Job finished."; volumeMounts: - name: model-storage mountPath: /mnt/models volumes: - name: model-storage persistentVolumeClaim: claimName: predictor-model-pvc ``` This file creates a PVC, a Job that copies the model into a PV, and the pod gets deleted after 10 seconds once the job is completed. Run the following to apply the manifest ```bash kubeclt apply -f job.yaml ``` If you check the logs of the pod you will get the following output. ```bash $ kubectl logs job/predictor-model-copy-job >>> Copying model to PVC... >>> Verifying contents in PVC... total 4K -rw-r--r-- 1 root root 1.7K Sep 16 06:57 model.pkl >>> Verification complete. Job finished. ``` ⚠️ If the job pod is stuck in Pending state, check PVC and storage class and ensure the PVC is created and in bound state. ### Step 3: Deploy InferenceService Resource To deploy the model, we are going to apply the **`inference.yaml`** file to create KServe's inference resource on the cluster. ```yaml apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata: name: model spec: predictor: sklearn: storageUri: pvc://predictor-model-pvc resources: requests: cpu: 500m memory: 1Gi ``` In the above manifest, - The `spec.predictor` section defines how the model will be served - Since the model we are using is based on scikit-learn, we are using **sklearn** framework. - `storageUri: pvc://predictor-model-pvc` means the model files are stored on a Persistent Volume Claim (PVC) named "predictor-model-pvc". 💡 Since we are using the `sklearn` predictor block, KServe knows that we want to use the ****built-in SKLearn model server image**. KServe will pull the default **`kserve/sklearnserver:`** and run the model in it. Run the following command to apply the inference service manifest. ```bash kubectl apply -f inference.yaml ``` Then run the following command to check if the related objects are created by the inferenceService object. ```bash kubectl get po,svc,hpa,inferenceservice ``` You will get the following output. ![verifing if the objects are created](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/09/image-12.png) ⚠️ If the model-predictor pod is stuck in Pending state check the node CPU and memory resources. Ensure atleast 1 GB memory is present in the nodes. The internal endpoint will be: ```bash http://model-predictor.default.svc.cluster.local/v1/models/model:predict ``` In the above endpoint: - **model-predictor.default.svc.cluster.local** \- Internal DNS of the service attached to the [pod](https://devopscube.com/kubernetes-pod/). - **v1/models/** \- API version - **model** \- Name of the inference service - **predict** \- It's a standard endpoint for predictor models. ## Test the KServe Inference Endpoint To test the model, we are going to port forward the deployment service and send a request to it using curl. Run the following command to port-forward the service. ```bash kubectl port-forward service/model-predictor 8000:80 ``` Then, run the following command to send the request for prediction. ```bash curl -X POST \ -H "Content-Type: application/json" \ -d '{ "instances": [ "sparrow", "elephant", "sunflower" ] }' \ "http://localhost:8000/v1/models/model:predict" ``` You will get the following output. ![checking output from the curl request](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/09/image-10.png) In the prediction model, `0` is for animal, `1` is for bird, and `2` is for the plant. And the predictor output is correct based on the input. Thats it! You have deployed a model using Kserver and made a inference request! ## Kserve FAQ's Following are some of the frequently asked questions about Kserve. ### What is the difference between KServe RawDeployment and Knative mode? RawDeployment is lightweight and easier deployment option for beginners. While, Knative mode supports advanced use cases with autoscaling and networking functionalities. ### How can I test a deployed KServe model? The easiest way is to port-forward the inference service and send test requests with `curl` or a program to the inference endpoint. ### Can I deploy models other than scikit-learn with KServe? Yes. KServe supports TensorFlow, PyTorch, XGBoost, ONNX, and more. ### What should I do if my KServe pod is stuck in Pending state? Check PVC storage, node CPU and memory, and controller logs. Ensure at least 1 GB memory is available while deploying simple models. ### Is KServe used by companies in production? Yes. Companies like Bloomberg, IBM, Red Hat, Gojek, and Cisco use KServe in their production environments. ## Conclusion You have just taken your first step into MLOps. The practice of deploying and maintaining machine learning system. In summary, you have learned how to serve ML models using KServe and how to check if the model is served successfully by sending a curl request to it. Give it a try and if you have any doubts or face any issues, drop a comment below. We will help you out. Also, if you want to know more about AI/ML features of Kubernetes, refer the [Kubernetes AI/ML features](https://devopscube.com/kubernetes-ai-ml-features/) blog.