How to Setup Prometheus Operator on Kubernetes

In this blog, we are going to set up and configure the Prometheus Operator.

The Prometheus is not a native object of the Kubernetes, but we can make it a Kubernetes native object using the Prometheus operators.

What is Prometheus Operator

Prometheus Operator is a Kubernetes Custom Controller that can communicate with the Kubernetes API Server and allow Prometheus to do the operations.

We will install Prometheus on our Kubernetes cluster but the main moto of installing the Prometheus Operator is to reduce the complexity of the configuration.

In normal Prometheus deployment, configMap is an important object where we inform the targets and rules information to the Prometheus.

Instead of that, the Prometheus Operator has dedicated objects for that. For example, prometheusRuler for rules, podMonitor object for monitoring the Pods, scrapeConfig object for static targets and more.

In a normal Prometheus stack, if you want to make any modifications, that will distrube the present configuration but you do that on the Prometheus Operator whenever you want.

If you set up the Prometheus operators, you won’t feel that the Prometheus is an external object also it will simplify the overall integration and configuration process.

If you want to install Prometheus on your Kubernetes cluster without the Operator, here is the blog.

Prometheus Operator Workflow

Prometheus Operator always watches its resources such as Custom Resources and deployments.

We inform the Operator via Custom Resource what outcome we are expecting.

For example, I am telling on a Custom Resource that I have a service and the endpoints should be monitored.

The operator will get this information and the service labels that I provide, apart from that no information we provide.

But the Operator has all the logic in it, which means that the operator will identify the Service with the labels that we provide and collect the endpoint information then make a structure and place it on the Prometheus instance.

What are the Prometheus Supported Custom Resource Definitions

Prometheus

Prometheus Custom Resource Definition (CRD) is used to create the Prometheus Custom Resource or we can simply say the Prometheus deployment on the Kubernetes cluster, also offers the replication, and storage-related configuration.

Alertmanager

Alertmanager CRD is used to create an Alertmanager deployment on the cluster.

ThanosRuler

ThanosRuler CRD is used to manage and validate the Prometheus Recording Rules and the Alerting Rules.

ServiceMonitor

ServiceMonitor CRD helps to dynamically add the Kubernetes services on the Prometheus Custom Resources.

PodMonitor

PodMonitor CRD is used to dynamically add the Pods on the Prometheus Custom Resource.

Probe

Probe CRD is used to add the static targets on the Prometheus Custom Resource.

PrometheusRule

Add alerting rules and recording rules on the Prometheus.

AlertmanagerConfig

Alertmanager-related configuration such as inhibition, routing, silencing, etc.

Prometheus Operator Setup – Helm

The Helm chart will deploy all the Prometheus Operator components such as Custom Resource Definitions as well as the Deployments.

Prerequisites

• Helm
• kubectl

Create a Namespace for the Prometheus Operator.

kubectl create namespace prometheus-operator

Add the Prometheus community repository in Helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Update the Helm repository

helm repo update

Now, we can install the Prometheus Operator using the Helm

helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus-operator

It will take a few minutes to complete the setup and once it is completed we can check the Pods.

kubectl get pods -n prometheus-operator

In this, Grafana, Prometheus Operator and Kube State Metrics are deployed as Deployment Objects.

kubectl get deployments -n prometheus-operator

Grafana: Visualize the metrics

Prometheus Operator: It is a controller to manage and configure the Prometheus stack

Kube state metrics: Collect the Kubernetes API Server metrics

Alertmanager and Prometheus are deployed as StatefulSets.

kubectl get statefulsets -n prometheus-operator

Alertmanager: Sending alerts via email or notifications

Prometheus: Collect metrics

Prometheus and Alertmanager are deployed as StatefulSet objects because Prometheus has the Time Series Database, that stores the metrics and both these Objects have configurations so it does need persistent volumes.

Because of the persistent volume, the data will persist even if the Pod is removed or rescheduled.

The Node Exporter is deployed as a Daemonset because this object ensures that the Node Exporter is available on each node to collect the metrics.

kubectl get daemonsets -n prometheus-operator

Now, let’s view their intended Services

kubectl get svc -n prometheus-operator

When deploying the Prometheus Operator, it will create some Custom Resources to monitor the default Kubernetes resources such as Nodes, API Server, kube-proxy, kubelet, etc.

To see the web UI of Prometheus, Alertmanager, or Grafana, we need to perform the port forwarding.

For that, we need the service name prometheus-kube-prometheus-prometheus and the related port number 9090.

To view the Dashboard of the Prometheus, use the following command.

kubectl port-forward -n prometheus-operator svc/prometheus-kube-prometheus-prometheus 9090:9090

These Custom Resources are created by using the ServiceMonitor Custom Resource Definition.

kubectl get servicemonitors -n prometheus-operator

To see the dashboard of the Alertmanager, use the following command.

kubectl port-forward --namespace prometheus-operator svc/kube-stack-prometheus-kube-alertmanager 9093:9093

Let’s try to access the dashboard from our local system.

Most of the configurations are already defined and if you want modifications you can do it, also in this setup, by default, you can’t access the dashboards of the services such as Prometheus, Alertmanager, Grafana, etc.

So if you want to access over the internet you need to create a new Service Object for them with type NodePort or Loadbalancer.

To view the list of all the Custom Resource Definitions related to the Prometheus Operator.

kubectl get crds -n prometheus-operator

You can create as many Custom Resources with these Custom Resource Definitions.

How to Install Prometheus Operator – YAML Manifest

In this method, you can customize the Prometheus Operator installation. For example, if you only want Prometheus but don’t need Alertmanager or Grafana, you can easily do that with this setup.

Download the bundle.yaml file from the Prometheus Operator GitHub repository to define the Prometheus Custom Resource Definitions and the Prometheus Operator.

wget https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.70.0/bundle.yaml

To deploy the Custom Resource Definitions (CRDs) to the cluster, use the following command.

kubectl create -f bundle.yaml

These resources will be deployed in the default namespace.

kubectl get deployments -n default

To view and filter the Prometheus Operator Custom Resource Definitions.

kubectl get crds | grep "monitoring.coreos.com"

Test the Prometheus Operator with an Application

To test the workflow of the Prometheus Operator, we need a target application.

For testing purposes, we are using an application, and you have to make sure that the application you want to monitor should be instrumented by Prometheus Client Libraries.

For testing purposes, I am going to deploy this application with 3 replicas and will create a service for this deployment.

Then, I will use the ServiceMonitor Custom Resource to collect the metrics from the endpoints of the Service, basically, the 3 replica Pods of the deployment.

First, we need to create a Deployment manifest for the application instrumented-app-deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: fabxc/instrumented_app
        ports:
        - name: web
          containerPort: 8080

This application will expose the metrics at Port 8080 and I am providing 3 replicas for this application.

To deploy this application, use the following command.

kubectl apply -f instrumented-app-deploy.yaml

Check the Deployment and Pods to ensure the deployment is properly done.

Now, I am creating a service file for this application instrumented-app-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: web
    port: 8080

To deploy this application service, use the following command.

kubectl apply -f instrumented-app-svc.yaml

We need to ensure this application service is running properly.

kubectl get svc

Create Service Monitor Custom Resource

ServiceMonitor Custom Resource is used to collect the metrics from the service endpoints.

Labels play an important role in these configurations, so provide your labels simple and meaningful.

I am going to use the labels of the Service as the Selector of the ServiceMonitor object.

Now, we have to create the ServiceMonitor Custom Resource file service-monitor-cr.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: web

Here, you can see, under the spec section, I have provided the label of the application service app: example-app. This is how ServiceMonitor identifies the service object and its endpoints.

To apply this configuration, use the following command.

kubectl apply -f service-monitor-cr.yaml

To list the ServiceMonitor object.

kubectl get servicesmonitor

Create a Service Account

If you are performing RBAC authorization on your cluster, then first you need to set up the ClusterRole and ClusterRoleBinding with a ServiceAccount.

Create a service account file service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-operator-svc-ac

To apply this configuration, use the following command.

kubectl apply -f service-account.yaml

To list the service accounts in the current namespace.

kubectl get serviceaccount

Create ClusterRole

Create a ClusterRole file clusterrole.yaml and provide the necessary permission for the cluster-scoped resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

To apply this configuration, use the following command.

kubectl apply -f clusterrole.yaml

To list the cluster roles in the current namespace.

kubectl get clusterrole prometheus

Create ClusterRoleBinding

Create a ClusterRoleBinding file clusterrolebinding.yaml and integrate the information about the ClusterRole and the ServiceAccount.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus-operator-svc-ac
  namespace: default

To apply this configuration, use the following command.

kubectl apply -f clusterrolebinding.yaml

To view the details about the ClusterRoleBinding.

kubectl describe clusterrolebinding prometheus

If the configuration is properly done, you can able to see the ClusterRole, ServiceAccount, and Namespace names in the details of the ClusterRoleBinding.

Create a Prometheus Custom Resource

Now we need to create the Prometheus Custom Resources and also add the information about the ServiceAccount and the ServiceMonitor.

Create a configuration file to deploy the Prometheus Custom Resource Prometheus-cr.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus-operator-svc-ac
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false

In this configuration, I am providing the value for the spec.serviceMonitorSelector.matchLabels is the label of the ServiceMonitor Object that we created earlier.

This is how the Prometheus grabs the target details with the help of the ServiceMonitor Custom Resource

To apply this configuration, use the following command.

kubectl apply -f prometheus-cr.yaml

To list the Prometheus object.

kubectl get prometheus

I want to access Prometheus over the internet, so I am creating a service with nodePort. Create a service file prometheus-cr-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  type: NodePort
  ports:
  - name: web
    nodePort: 30900
    port: 9090
    protocol: TCP
    targetPort: web
  selector:
    prometheus: prometheus

To deploy this service, use the following command.

kubectl apply -f prometheus-cr-svc.yaml

List the services to ensure that the service is running and note down the nodePort number.

Open any browser and paste any one of the instance’s public IPs and the port number, and then you will get this same output.

Here, we can see three Pods have been monitored because I have given 3 replicas for the application. ServiceMonitor got the endpoint information of the three of them as well and Prometheus got it from the ServiceMonitor.

If the Pods don’t have service then we can use the PodMonitor Custom Resource to monitor the pods.

Create AlertmanagerConfig Custom Resource

We need Alertmanager to send the notifications to the receiver, based on the Prometheus rules.

But before that, we need to create the AlertmanagerConfig Custom Resource with the routing and receiver information.

Create a manifest for the AlertmanagerConfig Object alertmanager-config-cr.yaml

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: config-example
  labels:
    alertmanagerConfig: example
spec:
  route:
    groupBy: ['job']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: 'webhook'
  receivers:
  - name: 'webhook'
    webhookConfigs:
    - url: 'http://example.com/'

To deploy this configuration, use the following command.

kubectl apply -f alertmanager-config-cr.yaml

To list the AlertmanagerConfig object.

kubectl get alertmanager

AlertmanagerConfig contains the configuration information of the Alertmanager.

Create Alertmanager Custom Resource

Create a manifest for the Alertmanager Object alertmanager-cr.yaml and add the following contents.

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: example
spec:
  replicas: 3
  alertmanagerConfigSelector:
    matchLabels:
      alertmanagerConfig: example

Here, you can see that spec.alertmanagerConfigSelector pointing to the AlertmanagerConfig Custom Resource labels.

To apply this configuration, use the following command.

kubectl apply -f alertmanager-cr.yaml

To view the Alertmanager Custom Resource.

kubectl get alertmanager

I am creating a service for the Alertmanager Custom Resource to access it over the internet.

Create a service file alertmanager-cr-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-example
spec:
  type: NodePort
  ports:
  - name: web
    nodePort: 30903
    port: 9093
    protocol: TCP
    targetPort: web
  selector:
    alertmanager: example

To apply this configuration, use the following command.

kubectl apply -f alertmanager-cr-svc.yaml

Now, we can able to access the Alertmanager dashboard over the internet, but before that ensure the Pod is running properly.

To access the dashboard, we need the nodePort number 30903 and the Public IP of any one of the instances.

Initially when we deployed the Prometheus Custom Resource, didn’t mention anything about the Alertmanager so now, we need to make some changes in the existing Prometheus configuration file prometheus-cr.yaml.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus-operator-svc-ac

  replicas: 2

  alerting:
    alertmanagers:
    - namespace: default
      name: alertmanager-example
      port: web

  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false

Now, we have to redeploy this manifest to update the changes, and this modification will not affect the existing setup.

kubectl apply -f prometheus-cr.yaml

If we check the Prometheus dashboard now, we can able to see the Alertmanager information in the dashboard.

Advantages Of Prometheus Operator

Here are the few advantages of Prometheus Operator.

1. If you are fed up with the manual integration of the Prometheus in the Kubernetes, you can use the operators because the setup process is really easy.
2. In the manual configuration, you might face some inconsistencies if you are working in multiple clusters but if you are using operators then the Custom Resource Definition (CRD) will ensure consistency across the cluster.
3. If you are required to handle the scaling process in the Prometheus stack, manual configuration modification is difficult, but the operators will simplify the process and automate the job.
4. Integrating other components such as Alertmanager and Grafana is also easy when compared to manual integration and you will not be confused about the configurations because the Prometheus manifests are the same as the Kubernetes native objects manifest.

Conclusion

I have tried this setup to know the configuration of the Prometheus Custom Resources also I have done a test setup to explain the concept, but you can do more in the configuration segment.

Also try the other Custom Resources such as PodMonitor, Probe, ThanosRuler, etc.

Know about their use cases and implement them if your requirements need them and you can create your custom resources if you want.