How To Setup Grafana On Kubernetes

byBibin Wilson
January 29, 2022

Grafana is an open-source lightweight dashboard tool. It can be integrated with many data sources like Prometheus, AWS cloud watch, Stackdriver, etc. Running Grafana on Kubernetes

In our previous posts, we have looked at the following.

This tutorial explains how to run Grafana on Kubernetes cluster. Using Grafana you can simplify Kubernetes monitoring dashboards from Prometheus metrics.

Grafana Kubernetes Manifests

All the Kubernetes manifests (YAML files) used in this tutorial are hosted on Github as well. You can clone it and use it for the setup.

git clone https://github.com/bibinwilson/kubernetes-grafana.git

Deploy Grafana On Kubernetes

Let’s look at the Grafana setup in detail.

Step 1: Create a file named grafana-datasource-config.yaml

vi grafana-datasource-config.yaml

Copy the following contents.

Note: The following data source configuration is for Prometheus. If you have more data sources, you can add more data sources with different YAMLs under the data section.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
               "access":"proxy",
                "editable": true,
                "name": "prometheus",
                "orgId": 1,
                "type": "prometheus",
                "url": "http://prometheus-service.monitoring.svc:8080",
                "version": 1
            }
        ]
    }

Step 2: Create the configmap using the following command.

kubectl create -f grafana-datasource-config.yaml

Step 3: Create a file named deployment.yaml

vi deployment.yaml

Copy the following contents on the file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests: 
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources

Note: This Grafana deployment does not use a persistent volume. If you restart the pod all changes will be gone. Use a persistent volume if you are deploying Grafana for your project requirements. It will persist all the configs and data that Grafana uses.

Step 4: Create the deployment

kubectl create -f deployment.yaml

Step 5: Create a service file named service.yaml

vi service.yaml

Copy the following contents. This will expose Grafana on NodePort 32000. You can also expose it using ingress or a Loadbalancer based on your requirement.

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '3000'
spec:
  selector: 
    app: grafana
  type: NodePort  
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 32000

Step 6: Create the service.

kubectl create -f service.yaml

Now you should be able to access the Grafana dashboard using any node IP on port 32000. Make sure the port is allowed in the firewall to be accessed from your workstation.

http://<your-node-ip>:32000

You can also use port forwarding using the following command.

kubectl port-forward -n monitoring <grafana-pod-name> 3000 &

For example,

[email protected]:~$ kubectl get po -n monitoring
NAME                       READY   STATUS    RESTARTS   AGE
grafana-64c89f57f7-kjqrb   1/1     Running   0          10m
[email protected]:~$ kubectl port-forward -n monitoring grafana-64c89f57f7-kjqrb 3000 &

You will be able to access Grafana a from http://localhost:3000

Use the following default username and password to log in. Once you log in with default credentials, it will prompt you to change the default password.

User: admin
Pass: admin

Create Kubernetes Dashboards on Grafana

Creating a Kubernetes dashboard from the Grafana template is pretty easy. There are many prebuilt Grafana templates available for Kubernetes. You can easily have prebuilt dashboards for ingress controllers, volumes, API servers, Prometheus metrics, and much more.

To know more, see Grafana templates for Kubernetes monitoring

Follow the steps given below to set up a Grafana dashboard to monitor kubernetes deployments.

Step 1: Get the template ID from grafana public template. as shown below.

Step 2: Head over to the Grafana dashbaord and select the import option.

Step 3: Enter the dashboard ID you got in step 1

Step 4: Grafana will automatically fetch the template from the Grafana website. You can change the values as shown in the image below and click import.

Note: If you are behind the corporate firewall and cannot download the template using id, you can download the template JSON and paste the JSON in the text box to import it.

You should see the dashboard immediately.

When Grafana is used with Prometheus, it uses PromQL to query metrics from Prometheus. You can use the same PromQL Prometheus queries to build custom dashboards on Grafana.

Conclusion

Grafana is a very powerful tool when it comes to Kubernetes monitoring dashboards.

It is used by many organizations to monitor their Kubernetes workloads. With the wide range of pre-built templates, you can get started with the templates pretty quickly. What more can you ask for right?

Let me know how you are using Grafana in your organization.

Also, let me know if you want to add more information to this article.

Bibin Wilson

An author, blogger, and DevOps practitioner. In his spare time, he loves to try out the latest open source technologies. He works as an Associate Technical Architect. Also, the opinions expressed here are solely his own and do not express the views or opinions of his previous or current employer.

42 comments

Brice FOTSING says:

June 10, 2022 at 9:15 am

merci , merci , merci encore pour ce bel article tout est propre comme l’eau de roche.
j’ai tout déployer de bout en bout sans problème.
merci encore et surtout bon courage
Cordialement

Reply
1. Bibin Wilson says:
  
  June 14, 2022 at 4:14 am
  
  Hi Brice,
  
  Je suis content que l’article ait aidé.. 🙂
  
  Reply
Liwies says:

April 5, 2022 at 9:16 am

This is a great tutorial, very easy to read and implement, Thank you.

Reply
1. Bibin Wilson says:
  
  April 8, 2022 at 9:46 am
  
  Really glad it helped Liwies 🙂
  
  Reply
Sheriffo says:

March 14, 2022 at 11:38 am

Nice tutorial. I managed to set up a ks8 cluster using kind and deploy Prometheus and Grafana following your guide. However, it seems some of the PromQL queries for the Prometheus dashboard are not working. E.g. sum (container_memory_working_set_bytes{pod_name=~”^$Deployment$Statefulset$Daemonset.*$”, kubernetes_io_hostname=~”^$Node$”, pod_name!=””}) returns an NA.

Reply
1. Bibin Wilson says:
  
  March 17, 2022 at 2:48 am
  
  Thanks Sheriffo. Regarding the merits, I believe these metrics come from the advisor. Can you check if cadvisor is running as expected?
  
  Reply
Gabriel says:

February 21, 2022 at 2:31 pm

Hi!

Awesome tutorial, but i got a CreatingContainer on the Grafana pod, right after executing the service part:
`kubectl create -f service.yaml`

I am using Kind and my local machine has 8gb of RAM and it is a Core i5, running Fedora 35.

Does anybody know what could have happened?

Again, thanks for the tutorial, i am learning a lot from your posts!

Reply
1. Bibin Wilson says:
  
  February 21, 2022 at 4:20 pm
  
  Hi Gabriel,
  Glad the content helps Gabriel
  
  When you describe the Grafan pod, what exact error do you see?
  
  Reply
  1. Gabriel says:
    
    February 22, 2022 at 6:30 pm
    
    I am totally noob on this matter, but when loading up my workstation this morning, i got the grafana container up and running.
    
    I don´t remember what i had done to get it work…
    
    The error I have now it is a Bad gateway on the Grafana dashboard.
    Both grafana and PRometheus containers are up and running.
    
    The only error i got is this:
    [[email protected] ~]$ kubectl port-forward -n monitoring grafana-64c89f57f7-b9ld8 3000 &
    [1] 989504
    
    [[email protected] ~]$ Unable to listen on port 3000: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:3000: bind: address already in use unable to create listener: Error listen tcp6 [::1]:3000: bind: address already in use]
    error: unable to listen on any of the requested ports: [{3000 3000}]
    
    Thanks for the help!
    
    Reply
    1. Bibin Wilson says:
      
      February 23, 2022 at 4:34 am
      
      Hi Gabriel,
      
      Looks like you have some other application running on port 3000 in your local system.
      
      Either you can stop that service or change port 3000 to some other port in the Kubectl port-forward command.
      
      Reply
tony says:

July 13, 2021 at 12:14 pm

Very well written tutorial. I followed it but when I went to add Prometheus as a data source http:127.0.0.1:9090/ I get a bad gateway. Not sure why I am getting a network related issue. I see the loadbalancer status by typing kubectl get all
service/grafana LoadBalancer 10.49.98.169 3000:30407/TCP 3d21h and not sure if assigning a public ip to the LB will fix this. Would liek your input thank you!

Reply
1. devopscube says:
  
  July 13, 2021 at 12:57 pm
  
  Hi Tony,
  
  1. Instead of http:127.0.0.1:9090, you should give the Prometheus service endpoint as given in the article. ie, http://prometheus-service.monitoring.svc:8080
  2. If it is a test setup, either you can use the NodePort option or Kubectl port forward to access from the local workstation.
  
  Reply
Raghav says:

May 31, 2021 at 11:13 am

Is it possible to show old data on grafana from prometheus?
By old data I mean data of terminated nodes which was scraped by Prometheus earlier.
Also, as I can see Grafana shows data of last 15 days only, even when PV and PVC has been configured, Is there a way to fix this as well?

Reply
1. devopscube says:
  
  May 31, 2021 at 2:33 pm
  
  HI Raghava,
  
  In Prometheus, the default local storage retention is 15 days. You can modify it with the --storage.tsdb.retention.time parameter to increase the retention days. Please refer https://prometheus.io/docs/prometheus/latest/storage/#local-storage for more details
  
  Reply
Gaurav says:

April 22, 2021 at 5:08 pm

Hi,
[email protected]:~$ k port-forward -n monitoring grafana-548fdc7598-2mczp 3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

I am not able to port-forward grafana UI to nodeIP:NodePort. It does not display the Login page.

Can access the Prometheus dashboard fine. Any clue what could be wrong?

Reply
1. Bibin Wilson says:
  
  April 23, 2021 at 5:23 am
  
  Hi Gaurav,
  
  It should work.. When you run the port-forward command, make sure you run it in the background by adding &
  
  For example,
  
  kubectl port-forward -n monitoring grafana-64c89f57f7-kjqrb 3000 &
  Also, if you have deployed the NodePort services, you can access Grafana on HTTP://:32000
  Let us know if the issue is rectified.
  
  Reply
Dmitry says:

March 28, 2021 at 7:13 am

Good day!
[email protected]:~/Yaml$ kubectl get pods –namespace=monitoring
NAME READY STATUS RESTARTS AGE
grafana-548fdc7598-225vq 0/1 Pending 0 77s
prometheus-deployment-6b58f86445-rs9dw 1/1 Running 0 13h

Grafana does not work(( What is a problem?

Reply
1. devopscube says:
  
  March 28, 2021 at 12:23 pm
  
  Hi Dmitry,
  
  Did you check the Grafana pod logs? Your output shows the pod is in a pending state. Also, check the worker node has enough capacity to run all the pods?
  
  Reply
  1. Dmitry says:
    
    March 28, 2021 at 2:08 pm
    
    kubectl logs grafana
    Error from server (NotFound): pods “grafana” not found
    My nodes is free. I made their on cluster GCP (default three nodes).
    
    Reply
    1. devopscube says:
      
      March 29, 2021 at 12:17 pm
      
      Hi Dmitry,
      
      JFYI, first you need to get the pod name by listing the pods. Then use the pod name to get the specific logs.
      
      For example, In your case
      
      kubectl logs grafana-548fdc7598-225vq
      
      Reply
      1. Dmitry says:
        
        March 29, 2021 at 7:33 pm
        
        Thank you very much! I happy that all working!
      2. devopscube says:
        
        March 30, 2021 at 2:28 am
        
        You are Welcome, Dmitry. Glad it worked!
  2. Dmitry says:
    
    March 29, 2021 at 11:40 am
    
    You are right! I made 4 nods for Cloud Google and it working!
    
    Reply
    1. devopscube says:
      
      March 29, 2021 at 12:16 pm
      
      Great! Let us know if you face any other error!
      
      Reply
Mehmet Ali Aydin says:

March 20, 2021 at 5:22 pm

Excellent guide, Bibin, thanks! Installation is straight forward, just struggled a bit with custom alerts. How can I create a custom alert?

Reply
1. Bibin Wilson says:
  
  March 21, 2021 at 2:17 am
  
  Welcome Mehmet…For custom alerts, you need to write Alert rules PROMQL queries. Same queries that you use in Grafana but with conditions…Refer https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
  
  Reply

Hi,
The above setup works for me. But i am trying to keep dashboards in one specific folder in my cluster and then i want to read it from grafana pod.

This is config map i have created:-

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
  namespace: monitoring
data:
  dashboards.yaml: |-
    {
      "apiVersion": 1,
      "providers": [
        {
          "name": 'default',
          "orgId": 1,
          "folder": '',
          "type": "file",
          "disableDeletion": false,
          "updateIntervalSeconds": 10,
          "options": {
            "path": "/downloads/templated_deployments/monitoring/06-dashboards"
          }
        }
      ]
    }

And in deployment file I am adding following config:

volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-storage
            - mountPath: /etc/grafana/provisioning/datasources
              name: grafana-datasources
              readOnly: false
            - mountPath: /etc/grafana/provisioning/dashboards
              name: grafana-dashboards

and

volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
            defaultMode: 420
            name: grafana-datasources
        - name: grafana-dashboards
          configMap:
            name: grafana-dashboards

But i am getting following error:

t=2020-07-06T11:03:11+0000 lvl=eror msg="failed to search for dashboards" logger=provisioning.dashboard type=file name=default error="stat /downloads/templated_deployments/monitoring/06-dashboards: no such file or directory"

Pallavi says:

January 14, 2021 at 12:14 pm

Did you find the solution to this. Facing same issue

Reply

Michael says:

May 8, 2020 at 12:05 pm

Could you possibly correct the following typo? I looked at the configs several times and asked what I was doing wrong, but I hadn’t read enough.

You wrote “Now you should be able to access the Grafana dashboard using any node IP on port 3200.”

It should be “Now you should be able to access the Grafana dashboard using any node IP on port 32000.”. A zero is missing.

Reply
1. devopscube says:
  
  May 8, 2020 at 12:12 pm
  
  Thanks, Michael for letting us know. We have updated the content with the correct port.
  
  Reply
Sathish Neelam says:

April 23, 2020 at 7:54 am

Hi, I’m not getting the error “Templating init failed Bad gateway”

Reply
1. cheng says:
  
  May 10, 2020 at 4:38 am
  
  Press F12 to see if you get lots of errors, if so, your datasource url in prometheus.yaml is wrong. Change “http://prometheus-service.monitoring.svc:8080” to “http://prometheus-service:8080”, it works for me, hope to help you 🙂
  
  Reply
  1. devopscube says:
    
    May 10, 2020 at 5:06 am
    
    Correct Cheng. http://prometheus-service.monitoring.svc:8080 is added assuming that Prometheus is running in the monitoring namespace.
    
    Reply
Newbie11222 says:

April 21, 2020 at 8:06 am

HI, how do you get the node ip to get the grafana dashboard? Thanks!!

Reply
Newbie11222 says:

April 21, 2020 at 8:06 am

Hi,
how do i get the node ip to view the dashboard? Thanks!!

Reply
1. Devopscube says:
  
  April 21, 2020 at 8:17 am
  
  you can use any one of the nodes IP in the cluster.
  
  Reply
Janith Hasantha Panditharathne says:

April 21, 2020 at 8:04 am
Hi, I want to install Grafana with MySQL database rather than default SQLite? Do you know how to do this? I tried a lot but no success.

Update: Yeah! First, we have to create centralized MySQL DB and then I passed env variables like this.
```
env:
- name: GF_DATABASE_TYPE
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_TYPE
- name: GF_DATABASE_HOST
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_HOST
- name: GF_DATABASE_NAME
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_NAME
- name: GF_DATABASE_USER
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_USER
- name: GF_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_PASSWORD
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_SECURITY_ADMIN_PASSWORD
```
Reply
1. Devopscube says:
  
  April 21, 2020 at 8:04 am
  
  Never done it before! I will consider testing it out!
  
  Please let us know if you are able to setup it up before we do!
  
  Reply
Thomas says:

April 21, 2020 at 7:37 am

Helpful post! Thanks

Statefulset grafana will be helpful as well

Reply
Agnip Dasgupta says:

April 21, 2020 at 8:17 am

Hi , can you please provide what would be a proper datasource-config file for mysql on google cloud sql ?

Reply
vishal says:

April 21, 2020 at 8:17 am

Could you please provide article to setup ELK on Kubernetes same like this document.

Reply
Praveen Sam says:

April 21, 2020 at 8:17 am

Thanks for the nice article , does the volume is persistence?

Reply

How To Setup Grafana On Kubernetes

Grafana Kubernetes Manifests

Deploy Grafana On Kubernetes

Create Kubernetes Dashboards on Grafana

Conclusion

Tags:

Bibin Wilson

42 comments

Leave a Reply Cancel reply

How To Create Kubernetes Jobs and Cronjobs – Getting Started Guide

Kubernetes monitoring With Sensu: Setting up Container Sidecar Agent

How to Create kubernetes Role for Service Account

Vault Agent Injector Tutorial: Inject Secrets to Pods Using Vault Agent

How to Setup Jenkins Build Agents on Kubernetes Pods

How To Deploy MongoDB on Kubernetes – Beginners Guide

How To Setup Grafana On Kubernetes

Grafana Kubernetes Manifests

Deploy Grafana On Kubernetes

Create Kubernetes Dashboards on Grafana

Conclusion

Tags:

Bibin Wilson

42 comments

Leave a Reply Cancel reply

You May Also Like