HHOW TO

How To Setup Grafana On Kubernetes

Grafana on Kubernetes
4
Shares

Grafana is an open-source lightweight dashboard tool. It can be integrated with many data sources like Prometheus, AWS cloud watch, Stackdriver, etc. Running Grafana on Kubernetes

When Grafana is used with Prometheus, it caused PromQL to query metrics from Prometheus

In our previous posts, we have looked at the following.

  1. Setup Prometheus on Kubernetes
  2. Setup Kube State Metrics
  3. Setup alert manager on Kubernetes

This tutorial explains how to run Grafana on Kubernetes cluster. Using Grafana you can simplify Kubernetes monitoring dashboards from Prometheus metrics.

Grafana Kubernetes Manifests

All the Kubernetes manifests (YAML files) used in this tutorial are hosted on Github as well. You can clone it and use it for the setup.

git clone https://github.com/bibinwilson/kubernetes-grafana.git

Deploy Grafana On Kubernetes

Let’s look at the Grafana setup in detail.

Step 1: Create file named grafana-datasource-config.yaml 

vi grafana-datasource-config.yaml

Copy the following contents.

Note: The following data source configuration is for Prometheus. If you have more data sources, you can add more data sources with different YAMLs under the data section.

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
               "access":"proxy",
                "editable": true,
                "name": "prometheus",
                "orgId": 1,
                "type": "prometheus",
                "url": "http://prometheus-service.monitoring.svc:8080",
                "version": 1
            }
        ]
    }

Step 2: Create the configmap using the following command.

kubectl create -f grafana-datasource-config.yaml

Step 3: Create a file named deployment.yaml

vi deployment.yaml

Copy the following contents on the file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests: 
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources

Note: This Grafana deployment does not use a persistent volume. If you restart the pod all changes will be gone. Use a persistent volume if you are deploying Grafana for your project requirements. It will persist all the configs and data that Grafana uses.

Step 4: Create the deployment 

kubectl create -f deployment.yaml

Step 5: Create a service file named service.yaml 

vi service.yaml

Copy the following contents. This will expose Grafana on NodePort 32000. You can also expose it using ingress or a Loadbalancer based on your requirement.

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '3000'
spec:
  selector: 
    app: grafana
  type: NodePort  
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 32000

Step 6: Create the service.

kubectl create -f service.yaml

Now you should be able to access the Grafana dashboard using any node IP on port 32000. Use the following default username and password to log in. Once you log in with default credentials, it will prompt you to change the default password.

User: admin
Pass: admin
Grafana dashboard on Kubernetes

Setup Kubernetes Dashboards on Grafana

There are many prebuilt Grafana templates available for Kubernetes. To know more, see Grafana Kubernetes Dashboard templates

Setting up a dashboard from a template is pretty easy. Follow the steps given below to set up a Grafana dashboard to monitor kubernetes deployments.

Step 1: Get the template ID from grafana public template. as shown below.

image

Step 2: Head over to grafana and select the import option.

image 1

Step 3: Enter the dashboard ID you got it step 1

image 2

Step 4: Grafana will automatically fetch the template from Grafana website. You can change the values as shown in the image below and click import.

image 3

You should see the dashboard immediately.

image 4

Conclusion

Grafana is a very powerful tool when it comes to monitoring dashboards.

It is used by many organizations to monitor their workloads.

Let us know how you are using Grafana in your organization. Also, let us know if you want to add more information to this article.

4 Shares:
Share 4
Tweet 0
Share 0
Share 0

An author, blogger and DevOps practitioner. In spare time, he loves to try out the latest open source technologies. He works as an Associate Technical Architect

26 comments

  1. Good day!
    [email protected]:~/Yaml$ kubectl get pods –namespace=monitoring
    NAME READY STATUS RESTARTS AGE
    grafana-548fdc7598-225vq 0/1 Pending 0 77s
    prometheus-deployment-6b58f86445-rs9dw 1/1 Running 0 13h

    Grafana does not work(( What is a problem?

    Reply

    1. Hi Dmitry,

      Did you check the Grafana pod logs? Your output shows the pod is in a pending state. Also, check the worker node has enough capacity to run all the pods?

      Reply

      1. kubectl logs grafana
        Error from server (NotFound): pods “grafana” not found
        My nodes is free. I made their on cluster GCP (default three nodes).

        Reply

        1. Hi Dmitry,

          JFYI, first you need to get the pod name by listing the pods. Then use the pod name to get the specific logs.

          For example, In your case

          kubectl logs grafana-548fdc7598-225vq
          Reply

  3. Hi,
    The above setup works for me. But i am trying to keep dashboards in one specific folder in my cluster and then i want to read it from grafana pod.

    This is config map i have created:-

    apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
  namespace: monitoring
data:
  dashboards.yaml: |-
    {
      "apiVersion": 1,
      "providers": [
        {
          "name": 'default',
          "orgId": 1,
          "folder": '',
          "type": "file",
          "disableDeletion": false,
          "updateIntervalSeconds": 10,
          "options": {
            "path": "/downloads/templated_deployments/monitoring/06-dashboards"
          }
        }
      ]
    }

    And in deployment file I am adding following config:

    volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-storage
            - mountPath: /etc/grafana/provisioning/datasources
              name: grafana-datasources
              readOnly: false
            - mountPath: /etc/grafana/provisioning/dashboards
              name: grafana-dashboards

    and 

    volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
            defaultMode: 420
            name: grafana-datasources
        - name: grafana-dashboards
          configMap:
            name: grafana-dashboards

    But i am getting following error: 

    t=2020-07-06T11:03:11+0000 lvl=eror msg="failed to search for dashboards" logger=provisioning.dashboard type=file name=default error="stat /downloads/templated_deployments/monitoring/06-dashboards: no such file or directory"
    Reply

  4. Could you possibly correct the following typo? I looked at the configs several times and asked what I was doing wrong, but I hadn’t read enough.

    You wrote “Now you should be able to access the Grafana dashboard using any node IP on port 3200.”

    It should be “Now you should be able to access the Grafana dashboard using any node IP on port 32000.”. A zero is missing.

    Reply

    1. Press F12 to see if you get lots of errors, if so, your datasource url in prometheus.yaml is wrong. Change “http://prometheus-service.monitoring.svc:8080” to “http://prometheus-service:8080”, it works for me, hope to help you 🙂

      Reply

  8. Hi, I want to install Grafana with MySQL database rather than default SQLite? Do you know how to do this? I tried a lot but no success.

    Update: Yeah! First, we have to create centralized MySQL DB and then I passed env variables like this.

    env:
- name: GF_DATABASE_TYPE
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_TYPE
- name: GF_DATABASE_HOST
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_HOST
- name: GF_DATABASE_NAME
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_NAME
- name: GF_DATABASE_USER
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_USER
- name: GF_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_DATABASE_PASSWORD
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-secret
key: GF_SECURITY_ADMIN_PASSWORD
    Reply

  10. Hi , can you please provide what would be a proper datasource-config file for mysql on google cloud sql ?

    Reply
Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like