How to Setup Grafana Loki on Kubernetes & Query Logs

In this blog, we will look into a step-by-step guide to setup Grafana Loki on Kubernetes using Helm. We will also look at how to query the aggregated logs using promtail.

Grafana Loki is an open-source log aggregation system that supports large-scale log management. It is a widely adopted tool for organizations observability stack.

If you want to learn in detail about Grafana Loki’s architecture, read the detailed Grafana Loki Architecture blog.

Grafana Loki Workflow

A workflow overview of Grafana Loki is shown in the below diagram

Let me explain the diagram

1. As you can see in the diagram Loki gets logs from Promtail, which is the log-collecting agent of Loki.
2. Promtail collects logs and gives them to Loki, Loki compresses the logs and saves them as chunks.
3. To view the logs the Grafana dashboard is used, Grafana connects with Loki through port 3100 as default.
4. The user can access the Grafana dashboard on the browser and query the logs using the Grafana dashboard.

Setup Prerequisites

The following are the prerequisites for this setup.

1. Kubernetes Cluster
2. Kubectl
3. Helm installed in your system

Steps to Setup Grafana Loki on Kubernetes

If you are ready with the above prerequisites follow the below steps to set up Grafana Loki Logging on Kubernetes using Helm.

Step 1: Add Grafana repo

Add Grafana repo to your system to download the helm chart to set up Grafana Loki Logging. Run the following command to add the repo

helm repo add grafana https://grafana.github.io/helm-charts

Once the repo is added, run the following command to update the repo to make sure the repo is up-to-date

helm repo update

Now, list every repo with the word Loki using the command

helm search repo loki

You will get the list of repos with the word Loki as shown below

From these repos, we are going to use grafana/loki-stack which has the Helm chart to deploy Promtail, and Grafana, and configure Loki to Grafana.

Step 2: Customize Helm Chart Configuration Values

Before deploying we need to update some configurations in the helm chart.

Use the below command to save the default values of the helm chart in a YAML file.

helm show values grafana/loki-stack > loki.yaml

Now, in the loki.yaml, make changes to the Grafana configuration block because by default it is set to false and would not install Grafana.

As shown in the above image enable Grafana and set the tag to the latest, this will deploy Grafana with the latest version. You can also change other values as per your requirements.

If you want to store the log in a Persistent Volume instead of the file system configure a PVC to Loki.

Since Grafana Loki is deployed as a StatefulSet the logs will be saved on the same persistent volume even if the Grafana Loki pod is deleted and deployed again.

Following are the images used in this Helm chart.

1. grafana/loki
2. docker.io/grafana/promtail

Step 3: Deploy Loki

Once the values are changed deploy the Loki Helm chart with the YAML file using the following command

helm upgrade --install --values loki.yaml loki grafana/loki-stack -n grafana-loki --create-namespace

This creates a namespace grafana-loki and deploys every component for the Grafana Loki Logging on Kubernetes.

Once everything has deployed, you will get the following output

Step 4: Login to Grafana UI

First, verify everything has deployed and running properly using the command

kubectl get pod -n grafana-loki

You get the following output

I am using 2 worker nodes that’s why Promtail is running in two nodes because Promtail deploys as a DaemonSet.

Now, every component has been deployed properly on the Kubernetes cluster, port-forward Grafana so that we can access the Grafana dashboard on the browser because Loki uses the Grafana dashboard to display logs.

Use the following command to port-forward Grafana on port 9090.

kubectl port-forward pod/loki-grafana-7dd5f9d5c7-4d8jm -n grafana-loki 9090:3000

Make sure to replace loki-grafana-7dd5f9d5c7-4d8jm with your Grafana pod name, which you can get using the command kubectl get pod -n grafana-loki as shown below

You can also expose the Grafana service as NodePort and access it using the node IP and node port assigned to your service.

Step 5: Log in to Grafana

Access your Grafana on the browser by searching as localhost:9090, you will get the following window

Login to Grafana using the username and password. Your default username will be admin and to get the password run the following command

kubectl get secret --namespace grafana-loki loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

This command will show your password for Grafana, use the password and login to Grafana.

Once you log in you can see the home screen of Grafana, press the three lines at the top left corner you can see the menu then go to Connections > Data sources as shown in the below image.

In Data sources you can see Loki has been configured as shown below

Step 6: Query Logs

Now, check if you are getting logs.

Go to Explore by pressing the Explore button.

To query logs select a Label and Value, Loki will collect every log in your Kubernetes cluster and label it according to container, pod, namespace, and other objects of Kubernetes.

After selecting a Label and Value, press the blue button at the top right corner to query logs. You can also set a time to query logs.

For example, if you set the time to 5s it queries logs every 5s.

After pressing the blue button, scroll down to view the logs from the Label and Value you select.

Conclusion

I believe this helped you to to setup of Grafana Loki with simple six steps. We also learned how to query logs using Loki from the Grafana dashboard.

If you want to deploy a standalone grafana instance, you can use our guide on setting up Grafana on Kubernetes.