In this blog we will look at what is Kube State Metrics and its setup on Kubernetes.
What is Kube State Metrics?
Kube State metrics is s service which talks to Kubernetes API server to get all the details about all the API objects like deployments, pods, daemonsets etc. Basically it provides kubernetes API object metrics which you cannot get directly from native Kubernetes monitoring components.
Kube state metrics service exposes all the metrics on /metrics URI. Prometheus can scrape all the metrics exposed by kube state metrics. Here are the few key objects you can monitor with kube state metrics.
- Monitor node status, node capacity (CPU and memory)
- Monitor replica-set compliance (desired/available/unavailable/updated status of replicas per deployment)
- Monitor pod status (waiting, running, ready, etc)
- Monitor the resource requests and limits.
- Monitor Job & Cronjob Status
You can check out the detailed supported metrics from the documentation here.
You Might Like: How To Setup Prometheus on Kubernetes
Kube State Metrics Setup
kube state metrics is available as a public docker image. You will have to deploy the following Kubernetes objects for kube state metrics to work.
- A Service Account
- Cluster Role – For kube state metrics to access all the Kubernetes API objects.
- Cluster Role Binding – Binds the service account with the cluster role.
- Kube State Metrics Deployment
- Service – To expose the metrics
All the above kube state metrics objects will be deployed in the kube-system namespace
Lets deploy the components. All the deployment objects are available in Github. You can also find same deployment object in the official repo as well.
Step 1: Clone the Github repo
git clone https://github.com/devopscube/kube-state-metrics-configs.gitStep 2: Create all the objects by pointing to the cloned directory.
kubectl apply -f kube-state-metrics-configs/Step 3: Check the deployment status using the following command.
kubectl get deployments kube-state-metrics -n kube-systemkube State Metrics Prometheus Config
All the kube static metrics can be obtained from the kube state service endpoint on /metrics URI.
This configuration can be added as part of prometheus job configuration. You need to add the following job configuration to your prometheus config for prometheus to scrape all the kube state metrics.
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
6 comments
Hi Bipin
Thanks for nice explanation. This is very helpful. I have one doubt:
As last step you mentioned to add job configuration to your prometheus config. Are you talking about this file: https://raw.githubusercontent.com/bibinwilson/kubernetes-prometheus/master/config-map.yaml and i should add this in scrape_configs section.
You are correct Balkrishan.
Hi Bibin, thank you for writing this awesome blog.
I had worked on this setting up kube state metric but faced endpoint is down in Prometheus target and showing error Context Deadline Exceeded.
Hi Shivam, Did you check the Kube state metrics deployment? is it running without any errors?
Hey Bibin, After long search atleast got some good documentation abt kube-state-metrics. thank you for writing it. Do you know how to add this kube-state metrics entry with helm for installing it as automation ??
I have added entry like below (last line). but its not wring. Do i missing anything ? or is this not correct way ?
else
helm upgrade –install –tiller-namespace tiller-full –force prometheus stable/prometheus –wait –namespace prometheus –kube-context=${lower(var.environment)}.${data.terraform_remote_state.base_kops_config.base_domain}
–set rbac.create=true
–set server.ingress.enabled=true
–set server.ingress.hosts=”{prometheus.${lower(var.environment)}.${data.terraform_remote_state.base_kops_config.base_domain},prometheus.ingress.${lower(var.environment)}.${data.terraform_remote_state.base_kops_config.base_domain}}”
–set server.image.tag=”v2.11.1″
–set kubeStateMetrics.image.tag=”v1.7.0″