How to Setup Prometheus Monitoring On Kubernetes Cluster

Prometheus is an open source monitoring framework. Explaining Prometheus is out of scope of this article. In this article, I will guide you to setup Prometheus on a Kubernetes cluster and collect node, pods and services metrics automatically using Kubernetes service discovery configurations. If you want to know more about Prometheus, You can watch all Prometheus related videos from here.

Prometheus Monitoring on Kubernetes

I assume that you have a kubernetes cluster ready with kubectl setup on your workstation. If you don’t have a kubernetes setup, you can setup a cluster on google cloud by following this article.

Latest Prometheus is available as a docker image in its official docker hub account. We will use that image for the setup.

Let’s get started with the setup.

All the configuration files I mentioned in this guide is hosted on Github. You can clone the repo using the following command. Thanks to James for contributing to this repo. Please don’t hesitate to contribute to the repo for adding features.

Connect to the Cluster

Connect to your Kubernetes cluster and set up the proxy for accessing the Kubernetes dashboard.

Create a Namespace

First, we will create a Kubernetes namespace for all our monitoring components. Execute the following command to create a new namespace called monitoring.

You need to assign cluster reader permission to this namespace so that prometheus can fetch the metrics from kubernetes API’s.

1. Create a file named cluster-role.yaml and copy the content of this file –> ClusterRole Config

2. Create the role using the following command.

Create a Config Map

We should create a config map with all the prometheus scrape config which will be mounted to the Prometheus container in /etc/prometheus as prometheus.yaml file. This config map contains all the configuration to dynamically discover pods and services running in the kubernetes cluster.

1. Create a file called config-map.yaml and copy the contents of this file –> Prometheus Config File

2. Execute the following command to create the config map in kubernetes.

Create a Prometheus Deployment

1. Create a file named prometheus-deployment.yaml and copy the following contents onto the file. In this configuration, we are mounting the Prometheus config map as a file inside /etc/prometheus. It uses the official Prometheus image from docker hub.

READ  How to Setup a Replicated GlusterFS Cluster on AWS EC2

4. Create a deployment on monitoring namespace using the above file.

5. You can check the created deployment using the following command.

You can also get details from the kubernetes dashboard like shown below.

prometheus on kubernetes

Connecting To Prometheus

You can connect to the deployed Prometheus in two ways.

  1. Using Kubectl port forwarding
  2. Exposing the Prometheus deployment as a service with NodePort or a Load Balancer.

We will look at both the options.

Using Kubectl port forwarding

Using kubectl port forwarding, you can access the pod from your workstation using a selected port on your localhost.

1. First, get the Prometheus pod name.

The output will look like the following.

2. Execute the following command with your pod name to access Prometheus from localhost port 8080.

READ  How to Setup Slaves on Jenkins 2.0 Using Password and ssh Keys

Note: Replace prometheus-monitoring-3331088907-hm5n1 with your pod name.

3. Now, if you access http://localhost:8080 on your browser, you will get the Prometheus home page.

Exposing Prometheus as a Service

To access the Prometheus dashboard over a IP or a DNS name, you need to expose it as kubernetes service.

1. Create a file named prometheus-service.yaml and copy the following contents. We will expose Prometheus on all kubernetes node IP’s on port 30000.

Note: If you are on AWS or Google Cloud, You can use Loadbalancer type, which will create a load balancer and points it to the service.

2. Create the service using the following command.

3. Once created, you can access the Prometheus dashboard using any Kubernetes node IP on port 30000. If you are on the cloud, make sure you have the right firewall rules for accessing the apps.

4. Now if you go to status –> Targets, you will see all the Kubernetes endpoints connected to Prometheus automatically using service discovery as shown below. So you will get all kubernetes container and node metrics in Prometheus.

prometheus kubernetes target configuration

5. You can head over the homepage and select the metrics you need from the drop-down and get the graph for the time range you mention. An example graph for container memory utilization is shown below.

prometheus kubernetes metrics

In the next article, I will be covering pod monitoring using custom application metrics and set up alert rules and alert manager for Prometheus. Subscribe to the email list to get the new article updates.


ONLINE COURSE: The Complete Kubernetes Course

Learn how you can run, deploy, manage and maintain containerized Docker applications on Kubernetes

  • Learn to launch kubernetes cluster
  • Get started with Containerization of apps
  • Deploy applications on kubernetes cluster
  • Run stateful and stateless applications on containers

 

Free DevOps Resources

Get DevOps news, tutorials and resources in your inbox. A perfect way If you want to get started with devops. Like you, we dont like spam.

24 comments

  1. Alok T Reply

    Thanks a Ton !! I am new to Kubernetes and while Exposing Prometheus As A Service i am not getting external IP for it. My Graphana dashboard cant consume localhost. Can you please guide me how to Exposing Prometheus As A Service with external IP.

    Thanks
    Alok

  2. Greg K. Reply

    Thanks for the tutorial.
    Looks like the arguments need to be changed from
    “-config.file=/etc/prometheus/prometheus.yml”
    and
    “-storage.local.path=/prometheus/”

    to

    “–config.file=/etc/prometheus/prometheus.yml”
    “–storage.tsdb.path=/prometheus/”

    Thank again!

  3. Atoz Reply

    Nice article. There is a Syntax change for command line arguments in the recent Prometheus build, it should two minus ( — ) symbols before the argument not one.

  4. Aotz Prasad Reply

    Nice article. There is a Syntax change for command line arguments in the recent Prometheus build, it should two minus ( — ) symbols before the argument not one.

  5. suresh Reply

    Nice Article, I’m new to this tools and setup. Now got little bit idea before entering into spike.

  6. Vlad Reply

    Great article. I am using this for a GKE cluster, but when I got to targets I have nothing. Could you please advise?

  7. Debo Reply

    Thanks! Great Tutorial. Can you please provide me link for the next tutorial in this series.

  8. Joshua Reply

    Appreciate the article, it really helped me get it up and running. I do have a question though. Where did you get the contents for the config-map and the Prometheus deployment files. Is this something Prometheus provides? I have no other pods running in my monitoring namespace and can find no way to get Prometheus to see the pods in other namespaces. I went ahead and changed the namespace parameters in the files to match namespaces I had but I was just curious. Thanks

  9. MJ Reply

    When I run ./kubectl get pods –namespace=monitoring I also get the following:

    NAME READY STATUS RESTARTS AGE
    prometheus-deployment-5cfdf8f756-mpctk 1/1 Running 0 1d

    When this article tells me I should be getting

    prometheus-monitoring-xxxxxxxxx-xxxx

    Could you please advise on this?
    Then when I run this command – kubectl port-forward prometheus-deployment-5cfdf8f756-mpctk 8080:9090 I get the following

    Error from server (NotFound): pods “prometheus-deployment-5cfdf8f756-mpctk” not found

    Could someone please help?
    Many thanks in advance

    • MP Reply

      You need to specificate the namespace.

      Try
      kubectl port-forward prometheus-deployment-5cfdf8f756-mpctk 8080:9090 -n monitoring
      (if the namespace is called “monitoring”)

  10. MJ Reply

    Hi does anyone know when the next article is? I need to set up Alert manager and alert rules to route to a web hook receiver. If anyone has attempted this with the config-map.yaml given above could they let me know please?

    Thanks in advance!

  11. Mark Reply

    Hi – I’m getting the following error

    From the k8s log

    parsing YAML file /etc/prometheus/prometheus.yml: yaml: line 58: mapping values are not allowed in this context”

    prometheus-deployment-79c7cf44fc-p2jqt 0/1 CrashLoopBackOff

    Could you help me with this ?

    Thanks in advance

  12. xyz Reply

    Hi ,
    Thanks to your artical was able to set prometheus. can you post the next article soon. for alert configuration. waiting…!!!

  13. Sachin Reply

    Thanks, great post.

    It worked like a charm for me.

    Regards,
    ~ Sachin. K.

  14. Hari Krishnan Reply

    Hello Sir, I am currently exploring the Prometheus to monitor k8s cluster. I would like to know how to Exposing Prometheus As A Service with external IP, you please guide me..
    kubernetes-service-endpoints is showing down when I try to access from external IP. I believe we need to modify in configmap.yaml file, but not sure what need to make change. Need your help on that.

Leave a Reply

Your email address will not be published. Required fields are marked *