How to Setup Grafana Loki on Kubernetes & Query Logs

setup grafana loki

In this blog, we will look into a step-by-step guide to setup Grafana Loki on Kubernetes using Helm. We will also look at how to query the aggregated logs using promtail.

Grafana Loki is an open-source log aggregation system that supports large-scale log management. It is a widely adopted tool for organizations observability stack.

If you want to learn in detail about Grafana Loki’s architecture, read the detailed Grafana Loki Architecture blog.

Grafana Loki Workflow

A workflow overview of Grafana Loki is shown in the below diagram

Grafana Loki Workflow

Let me explain the diagram

  1. As you can see in the diagram Loki gets logs from Promtail, which is the log-collecting agent of Loki.
  2. Promtail collects logs and gives them to Loki, Loki compresses the logs and saves them as chunks.
  3. To view the logs the Grafana dashboard is used, Grafana connects with Loki through port 3100 as default.
  4. The user can access the Grafana dashboard on the browser and query the logs using the Grafana dashboard.

Setup Prerequisites

The following are the prerequisites for this setup.

  1. Kubernetes Cluster
  2. Kubectl
  3. Helm installed in your system

Steps to Setup Grafana Loki on Kubernetes

If you are ready with the above prerequisites follow the below steps to set up Grafana Loki Logging on Kubernetes using Helm.

Step 1: Add Grafana repo

Add Grafana repo to your system to download the helm chart to set up Grafana Loki Logging. Run the following command to add the repo

helm repo add grafana https://grafana.github.io/helm-charts

Once the repo is added, run the following command to update the repo to make sure the repo is up-to-date

helm repo update

Now, list every repo with the word Loki using the command

helm search repo loki

You will get the list of repos with the word Loki as shown below

list repos with word loki

From these repos, we are going to use grafana/loki-stack which has the Helm chart to deploy Promtail, and Grafana, and configure Loki to Grafana.

Step 2: Customize Helm Chart Configuration Values

Before deploying we need to update some configurations in the helm chart.

Use the below command to save the default values of the helm chart in a YAML file.

helm show values grafana/loki-stack > loki.yaml

Now, in the loki.yaml, make changes to the Grafana configuration block because by default it is set to false and would not install Grafana.

Altering the values of loki helm chart

As shown in the above image enable Grafana and set the tag to the latest, this will deploy Grafana with the latest version. You can also change other values as per your requirements.

If you want to store the log in a Persistent Volume instead of the file system configure a PVC to Loki.

Since Grafana Loki is deployed as a StatefulSet the logs will be saved on the same persistent volume even if the Grafana Loki pod is deleted and deployed again.

Following are the images used in this Helm chart.

  1. grafana/loki
  2. docker.io/grafana/promtail

Step 3: Deploy Loki

Once the values are changed deploy the Loki Helm chart with the YAML file using the following command

helm upgrade --install --values loki.yaml loki grafana/loki-stack -n grafana-loki --create-namespace

This creates a namespace grafana-loki and deploys every component for the Grafana Loki Logging on Kubernetes.

Once everything has deployed, you will get the following output

Grafana Loki installation command

Step 4: Login to Grafana UI

First, verify everything has deployed and running properly using the command

kubectl get pod -n grafana-loki

You get the following output

Checking the pod status of Grafana Loki

I am using 2 worker nodes that’s why Promtail is running in two nodes because Promtail deploys as a DaemonSet.

Now, every component has been deployed properly on the Kubernetes cluster, port-forward Grafana so that we can access the Grafana dashboard on the browser because Loki uses the Grafana dashboard to display logs.

Use the following command to port-forward Grafana on port 9090.

kubectl port-forward pod/loki-grafana-7dd5f9d5c7-4d8jm -n grafana-loki 9090:3000

Make sure to replace loki-grafana-7dd5f9d5c7-4d8jm with your Grafana pod name, which you can get using the command kubectl get pod -n grafana-loki as shown below

port forwarding grafana loki to access on web

You can also expose the Grafana service as NodePort and access it using the node IP and node port assigned to your service.

Step 5: Log in to Grafana

Access your Grafana on the browser by searching as localhost:9090, you will get the following window

Grafana UI for Loki

Login to Grafana using the username and password. Your default username will be admin and to get the password run the following command

kubectl get secret --namespace grafana-loki loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

This command will show your password for Grafana, use the password and login to Grafana.

Once you log in you can see the home screen of Grafana, press the three lines at the top left corner you can see the menu then go to Connections > Data sources as shown in the below image.

Grafana UI

In Data sources you can see Loki has been configured as shown below

Add adding loki as data source on grafana

Step 6: Query Logs

Now, check if you are getting logs.

Go to Explore by pressing the Explore button.

To query logs select a Label and Value, Loki will collect every log in your Kubernetes cluster and label it according to container, pod, namespace, and other objects of Kubernetes.

Filtering the logs using loki

After selecting a Label and Value, press the blue button at the top right corner to query logs. You can also set a time to query logs.

For example, if you set the time to 5s it queries logs every 5s.

Filtering the logs using loki

After pressing the blue button, scroll down to view the logs from the Label and Value you select.

Filtering the logs using loki

Conclusion

I believe this helped you to to setup of Grafana Loki with simple six steps. We also learned how to query logs using Loki from the Grafana dashboard.

If you want to deploy a standalone grafana instance, you can use our guide on setting up Grafana on Kubernetes.

1 comment
  1. Hello! Can u help me? I have the problem with plugin, Can’t fetch the labels, i checked the plugin: unable to connect with loki. please check the server logs for more details

Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like