Grafana is an open-source lightweight dashboard tool. It can be integrated with many data sources like Prometheus, AWS cloud watch, Stackdriver, etc. Running Grafana on Kubernetes
When Grafana is used with Prometheus, it caused PromQL to query metrics from Prometheus
In our previous posts, we have looked at the following.
This tutorial explains how to run Grafana on Kubernetes cluster. Using Grafana you can simplify Kubernetes monitoring dashboards from Prometheus metrics.
Grafana Kubernetes Manifests
All the Kubernetes manifests (YAML files) used in this tutorial are hosted on Github as well. You can clone it and use it for the setup.
git clone https://github.com/bibinwilson/kubernetes-grafana.gitDeploy Grafana On Kubernetes
Let’s look at the Grafana setup in detail.
Step 1: Create file named grafana-datasource-config.yaml
vi grafana-datasource-config.yamlCopy the following contents.
Note: The following data source configuration is for Prometheus. If you have more data sources, you can add more data sources with different YAMLs under the data section.
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: monitoring
data:
prometheus.yaml: |-
{
"apiVersion": 1,
"datasources": [
{
"access":"proxy",
"editable": true,
"name": "prometheus",
"orgId": 1,
"type": "prometheus",
"url": "http://prometheus-service.monitoring.svc:8080",
"version": 1
}
]
}Step 2: Create the configmap using the following command.
kubectl create -f grafana-datasource-config.yamlStep 3: Create a file named deployment.yaml
vi deployment.yamlCopy the following contents on the file.
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
name: grafana
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:latest
ports:
- name: grafana
containerPort: 3000
resources:
limits:
memory: "1Gi"
cpu: "1000m"
requests:
memory: 500M
cpu: "500m"
volumeMounts:
- mountPath: /var/lib/grafana
name: grafana-storage
- mountPath: /etc/grafana/provisioning/datasources
name: grafana-datasources
readOnly: false
volumes:
- name: grafana-storage
emptyDir: {}
- name: grafana-datasources
configMap:
defaultMode: 420
name: grafana-datasourcesNote: This Grafana deployment does not use a persistent volume. If you restart the pod all changes will be gone. Use a persistent volume if you are deploying Grafana for your project requirements. It will persist all the configs and data that Grafana uses.
Step 4: Create the deployment
kubectl create -f deployment.yamlStep 5: Create a service file named service.yaml
vi service.yamlCopy the following contents. This will expose Grafana on NodePort 32000. You can also expose it using ingress or a Loadbalancer based on your requirement.
apiVersion: v1
kind: Service
metadata:
name: grafana
namespace: monitoring
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '3000'
spec:
selector:
app: grafana
type: NodePort
ports:
- port: 3000
targetPort: 3000
nodePort: 32000Step 6: Create the service.
kubectl create -f service.yamlNow you should be able to access the Grafana dashboard using any node IP on port 32000. Make sure the port is allowed in the firewall to be accessed from your workstation.
http://<your-node-ip>:32000You can also use port forwarding using the following command.
kubectl port-forward -n monitoring <grafana-pod-name> 3000 &For example,
[email protected]:~$ kubectl get po -n monitoring
NAME READY STATUS RESTARTS AGE
grafana-64c89f57f7-kjqrb 1/1 Running 0 10m
[email protected]:~$ kubectl port-forward -n monitoring grafana-64c89f57f7-kjqrb 3000 &You will be able to access Grafana a from http://localhost:3000
Use the following default username and password to log in. Once you log in with default credentials, it will prompt you to change the default password.
User: admin
Pass: admin
Setup Kubernetes Dashboards on Grafana
There are many prebuilt Grafana templates available for Kubernetes. To know more, see Grafana Kubernetes Dashboard templates
Setting up a dashboard from a template is pretty easy. Follow the steps given below to set up a Grafana dashboard to monitor kubernetes deployments.
Step 1: Get the template ID from grafana public template. as shown below.
Step 2: Head over to grafana and select the import option.
Step 3: Enter the dashboard ID you got it step 1
Step 4: Grafana will automatically fetch the template from Grafana website. You can change the values as shown in the image below and click import.
You should see the dashboard immediately.
Conclusion
Grafana is a very powerful tool when it comes to monitoring dashboards.
It is used by many organizations to monitor their workloads.
Let us know how you are using Grafana in your organization. Also, let us know if you want to add more information to this article.
30 comments
Is it possible to show old data on grafana from prometheus?
By old data I mean data of terminated nodes which was scraped by Prometheus earlier.
Also, as I can see Grafana shows data of last 15 days only, even when PV and PVC has been configured, Is there a way to fix this as well?
HI Raghava,
In Prometheus, the default local storage retention is 15 days. You can modify it with the
--storage.tsdb.retention.timeparameter to increase the retention days. Please refer https://prometheus.io/docs/prometheus/latest/storage/#local-storage for more detailsHi,
[email protected]:~$ k port-forward -n monitoring grafana-548fdc7598-2mczp 3000
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000
I am not able to port-forward grafana UI to nodeIP:NodePort. It does not display the Login page.
Can access the Prometheus dashboard fine. Any clue what could be wrong?
Hi Gaurav,
It should work.. When you run the port-forward command, make sure you run it in the background by adding
&For example,
kubectl port-forward -n monitoring grafana-64c89f57f7-kjqrb 3000 &
Also, if you have deployed the NodePort services, you can access Grafana on HTTP://:32000
Let us know if the issue is rectified.
Good day!
[email protected]:~/Yaml$ kubectl get pods –namespace=monitoring
NAME READY STATUS RESTARTS AGE
grafana-548fdc7598-225vq 0/1 Pending 0 77s
prometheus-deployment-6b58f86445-rs9dw 1/1 Running 0 13h
Grafana does not work(( What is a problem?
Hi Dmitry,
Did you check the Grafana pod logs? Your output shows the pod is in a pending state. Also, check the worker node has enough capacity to run all the pods?
kubectl logs grafana
Error from server (NotFound): pods “grafana” not found
My nodes is free. I made their on cluster GCP (default three nodes).
Hi Dmitry,
JFYI, first you need to get the pod name by listing the pods. Then use the pod name to get the specific logs.
For example, In your case
Thank you very much! I happy that all working!
You are Welcome, Dmitry. Glad it worked!
You are right! I made 4 nods for Cloud Google and it working!
Great! Let us know if you face any other error!
Excellent guide, Bibin, thanks! Installation is straight forward, just struggled a bit with custom alerts. How can I create a custom alert?
Welcome Mehmet…For custom alerts, you need to write Alert rules PROMQL queries. Same queries that you use in Grafana but with conditions…Refer https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
Hi,
The above setup works for me. But i am trying to keep dashboards in one specific folder in my cluster and then i want to read it from grafana pod.
This is config map i have created:-
apiVersion: v1 kind: ConfigMap metadata: name: grafana-dashboards namespace: monitoring data: dashboards.yaml: |- { "apiVersion": 1, "providers": [ { "name": 'default', "orgId": 1, "folder": '', "type": "file", "disableDeletion": false, "updateIntervalSeconds": 10, "options": { "path": "/downloads/templated_deployments/monitoring/06-dashboards" } } ] }And in deployment file I am adding following config:
volumeMounts: - mountPath: /var/lib/grafana name: grafana-storage - mountPath: /etc/grafana/provisioning/datasources name: grafana-datasources readOnly: false - mountPath: /etc/grafana/provisioning/dashboards name: grafana-dashboardsand
volumes: - name: grafana-storage emptyDir: {} - name: grafana-datasources configMap: defaultMode: 420 name: grafana-datasources - name: grafana-dashboards configMap: name: grafana-dashboardsBut i am getting following error:
Did you find the solution to this. Facing same issue
Could you possibly correct the following typo? I looked at the configs several times and asked what I was doing wrong, but I hadn’t read enough.
You wrote “Now you should be able to access the Grafana dashboard using any node IP on port 3200.”
It should be “Now you should be able to access the Grafana dashboard using any node IP on port 32000.”. A zero is missing.
Thanks, Michael for letting us know. We have updated the content with the correct port.
Hi, I’m not getting the error “Templating init failed Bad gateway”
Press F12 to see if you get lots of errors, if so, your datasource url in prometheus.yaml is wrong. Change “http://prometheus-service.monitoring.svc:8080” to “http://prometheus-service:8080”, it works for me, hope to help you 🙂
Correct Cheng. http://prometheus-service.monitoring.svc:8080 is added assuming that Prometheus is running in the monitoring namespace.
HI, how do you get the node ip to get the grafana dashboard? Thanks!!
Hi,
how do i get the node ip to view the dashboard? Thanks!!
you can use any one of the nodes IP in the cluster.
Hi, I want to install Grafana with MySQL database rather than default SQLite? Do you know how to do this? I tried a lot but no success.
Update: Yeah! First, we have to create centralized MySQL DB and then I passed env variables like this.
Never done it before! I will consider testing it out!
Please let us know if you are able to setup it up before we do!
Helpful post! Thanks
Statefulset grafana will be helpful as well
Hi , can you please provide what would be a proper datasource-config file for mysql on google cloud sql ?
Could you please provide article to setup ELK on Kubernetes same like this document.
Thanks for the nice article , does the volume is persistence?