> ## Content Index
> Fetch the complete content index at: https://devopscube.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# How To Set Up Blackbox Exporter On VM - Ultimate Guide
- URL: https://devopscube.com/set-up-blackbox-exporter-on-vm/
- Published: 2024-04-04T15:28:56.000Z
- Updated: 2025-03-15T12:50:36.000Z
- Author: Bibin Wilson
- Tags: DEVOPS, #arunlal, MONITORING, #Migrated-1741795015845, #wp, #wp-post, #Import 2025-03-12 15:57, #blog

In this blog, let's explore how to set up Black Box Exporter on VM(virtual machine). By the end, you'll have a clear understanding of the process.

The working nature of the [Black Box exporter](https://devopscube.com/blackbox-exporter-on-kubernetes/) is a little different than other exporters, if you require to monitor the endpoint of a server then the Blackbox exporter is useful.

This exporter will probe the metrics based on monitoring the health of the server and whether the endpoint is available or not, using protocols such as HTTP, HTTPS, ICMP, TCP, gRPC, and DNS.

## **Blackbox Exporter Workflow**

![set up Black box Exporter: This is the Workflow of Blackbox Exporter ](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/blackbox.png)

In this setup, we use two separate servers for [Prometheus](https://devopscube.com/prometheus-architecture/) and Blackbox Exporter, but if you want, you can install both utilities on the same server.

After installing the Blackbox Exporter, we have to configure it for probing the metrics from the target, the configuration is a YAML formatted file, which contains modules, and modules are responsible for choosing the endpoint type.

Blackbox itself doesn't collect the information from the target, so for that we need [Prometheus.](https://devopscube.com/prometheus-alert-manager/)

We have to modify the configuration in Prometheus about the scraping interval, and target information. Also, we have to link the Blackbox Exporter to the Prometheus.

Based on the scrape interval, Prometheus tries to scrape the metrics from the Blackbox Exporter, once that response reaches the Blackbox, Blackbox will be triggered to collect the information from the target.

Blackbox Exporter collects the information and stores it in **/probes** the directory, from there. Prometheus pulls them and stores them in the **Time Series Data Base (TSDB).**

## **Setting Up Prometheus Blackbox Exporter on Virtual Machine(VM)**

I am using two servers in [AWS](https://devopscube.com/terraform-aws-vpc/) for this tutorial.

**Server 1**: The instance type is **t2.medium** and the utilities in this server are **Prometheus** and **Grafana**`.`

**Server 2**: This instance type is **t2.micro** and the utilities are **Prometheus Blackbox Exporter**.

I assume you have the server 1 setup already, so I am skipping that installation part, and let's begin to install Blackbox Exporter on the server.

Download Blackbox Exporter binaries from the [official repository](https://github.com/prometheus/blackbox%5Fexporter/releases?ref=devopscube.com). I am using the latest version **v0.24.0** for this setup.

```
https://github.com/prometheus/blackbox_exporter/releases/download/v0.24.0/blackbox_exporter-0.24.0.linux-amd64.tar.gz
```

Unzip the files.

```
tar -xvf blackbox_exporter-0.24.0.linux-amd64.tar.gz
```

This will contain the Blackbox executable file and the configuration file.

Open the directory

```
cd blackbox_exporter-0.24.0.linux-amd64
```

Create a user for Blackbox with no shell access.

```
sudo useradd -rs /bin/false blackbox
```

Create a directory in the **/etc** directory to store the configuration file.

```
sudo mkdir -p /etc/blackbox
```

Move the configuration `blackbox.yml` file to the directory

```
sudo mv blackbox.yml /etc/blackbox
```

Move the executable file **blackbox\_exporter** to the **/usr/local/bin** directory

```
sudo mv blackbox_exporter /usr/local/bin
```

Change the ownership to the executable file and configuration file to the **Blackbox** user.

```
sudo chown blackbox:blackbox /usr/local/bin/blackbox_exporter
sudo chown -R blackbox:blackbox /etc/blackbox/*
```

Create a service file for the Blackbox Exporter **blackbox.service**

```
sudo cat <<EOT > /lib/systemd/system/blackbox.service
[Unit]
Description=Blackbox Exporter Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=blackbox
Group=blackbox
ExecStart=/usr/local/bin/blackbox_exporter \
  --config.file=/etc/blackbox/blackbox.yml \
  --web.listen-address=":9115"

Restart=always

[Install]
WantedBy=multi-user.target
EOT

```

Enable and start the **blackbox.service**

```
sudo systemctl enable blackbox.service
sudo systemctl start blackbox.service
```

To check the status of the service, use the following command.

```
sudo systemctl status blackbox.service
```

The port number of the Blackbox Exporter is **9115**, by default, the **external probe metrics** will be saved in the `/probe` directory and the **internal metrics** will be stored in **/metrics** the directory.

If you want to check it from inside the instance, use the following method.

```
curl http://localhost:9115/metrics
```

If you want to access it over the internet, in the browser, use your instance public IP with the port number **<Public IP>:9115**. Then you will get this output.

![prometheus blackbox exporter dashboard
](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-152-6.png)

Based on modules, the probe data will be stored here.

## **Prometheus Blackbox Exporter Probe Module**

We have to create **modules** in Blackbox, then we will get the **probe metrics**. Fundamentally, we have to know which type of **endpoint** we are going to monitor because the module should be related to the endpoint.

A simple example module for an HTTP endpoint is

```
http_endpoint:
  prober: http
  timeout: 10s
  http:
    valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
    valid_status_codes: [200, 204]
    no_follow_redirects: false
    preferred_ip_protocol: "ip4"
```

This module will be configured in the Blackbox Exporter configuration file, which is **/etc/blackbox/blackbox.yml.**

The collected probe metrics by this particular module are this

![probe metrics
](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-2-30.png)

We can further filter and visualize these metrics using [Prometheus](https://devopscube.com/setup-prometheus-using-docker/) and Grafana utilities.

## **Prometheus Blackbox Exporter Endpoint**

The Blackbox Exporter probes the metrics using responses from the endpoint, protocol defining the endpoint types and the supporting protocols,

1. HTTP
2. HTTPS
3. ICMP
4. TCP
5. DNS
6. gRPC

We should choose one of the protocols to create a module, for example, if I am taking the **DNS** protocol to create a module, I could get the probe metrics like DNS query failure counts, query duration, query count, response size, etc

## **Prometheus Blackbox Exporter Real-World Examples**

### Create a module using `HTTP(S)` protocol

HTTP(S) module can track the SSL/TLS certificate validation and expiration, so I am creating this module to know that the target web server has a valid SSL/TLS certificate.

Add this configuration to your Blackbox Exporter configuration file **/etc/blackbox/blackbox.yml**.

```
modules:
  https_endpoint:
    prober: http
    timeout: 15s
    http:
      method: GET
      valid_http_versions:
        - HTTP/1.1
        - HTTP/2.0
      fail_if_not_ssl: true
      no_follow_redirects: false
      ip_protocol_fallback: false
      preferred_ip_protocol: ip4

```

Here, you can see that the prober is **HTTP**, which indicates this module is meant to be an HTTP endpoint, also I am passing another argument **fail\_if\_not\_ssl: true** so that if the web server doesn't have an SSL/TLS certificate, the status would fail.

Restart and check the status of the Blackbox Exporter.

```
sudo systemctl restart blackbox.service
sudo systemctl status blackbox.service
```

### Update the Prometheus configuration file

Open the [Prometheus](https://devopscube.com/setup-prometheus-monitoring-on-kubernetes/) configuration file **/etc/prometheus/prometheus.yml**, and append the Blackbox configuration with your existing Prometheus configuration.

In the global section, I have given the **scrape\_interval** time as **5s**, which means [Prometheus](https://devopscube.com/node-exporter-kubernetes/) will pull the metrics from the exporters every 5 seconds. you can modify with your required value.

Under the **scrape\_configs** section, we have to create a new job for Blackbox Exporter and have to give the `metrics_path`, we already know that the probe metrics will be stored in the**/probe** directory.

I am giving the module value **https\_endpoint, which** is my module name, and modifying the replacement value with your Blackbox Exporter servers private (if same network) or public IP with port number.

```
global:
  scrape_interval: 5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: "blackbox"
    metrics_path: /probe
    params:
      module: [https_endpoint]
    static_configs:
      - targets:
        - 35.80.157.25
        - devopscube.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.31.19.73:9115
```

I have given **two targets** in this configuration file, which are **`35.80.157.25`** and **devopscube.com**. The first one has a Nginx web server and doesn't have an SSL/TLS certificate, but the second one has a valid certificate.

![ssl certificate check](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-167-4.png)

Let's see if our setup identifies that, before that, we have to restart the Blackbox service and ensure the configurations are properly done.

```
sudo systemctl restart blackbox.service
sudo systemctl status blackbox.service
```

Check the **Blackbox Exporter** directly. For that open any browser, and paste your Blackbox Exporter public IP and the port number.

![prometheus black box exporter output](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-168-3.png)

Here it shows that the result of the **devopscube.com** is successful and the **35.80.157.24** is failed.

If you click the logs, you can view all the probe metrics, which is related to the target.

![prometheus blackbox exporter metrics dashboard](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/carbon-1-1-1.png)

Now I am picking one of the metrics from that to see the results in Prometheus.

To see the presence of the SSL/TLS certificate, we can use **probe\_http\_ssl** metrics.

```
probe_http_ssl
```

**Prometheus output**

![prometheus dashboard query
](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-169-3.png)

**Grafana output**

![grafana dashboard visualization
](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-170-2.png)

## Create a module using `TCP` protocol

The main use case of the `TCP` the based module is used to identify whether a user can reach the target over TCP or not also we can check some SSL-related metrics.

Add this to the Blackbox configuration file **/etc/blackbox/blackbox.yml**

```
modules:
  tcp_module:
    prober: tcp
    timeout: 10s
    tcp:
      preferred_ip_protocol: "ip4"
      tls: true
      tls_config:
        insecure_skip_verify: false

```

`tcp_module` is the module name, you can modify it if you want, and the `prober` value should be **tcp**.

Don't forget to restart and view the status of the black-box service **blackbox.service**, this will ensure the configurations are properly done.

Update the Prometheus configuration file **/etc/prometheus/prometheus.yml.** Add another job or replace it with the existing job if you want.

```
scrape_configs:
  - job_name: "blackbox-tcp"
    metrics_path: /probe
    params:
      module: [tcp_module]
    static_configs:
      - targets:
        - 35.80.157.25:443
        - devopscube.com:443
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.31.19.73:9115
```

After updating the configuration file, restart and check the status of the Prometheus service **prometheus.service**

> Providing target with the port number is necessary for the TCP module.

The Blackbox Exporter output is

![prometheus blackbox exporter output](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-3-34.png)

Here, the first server doesn't have the Transport Layer Security, so it fails and we can view the probe metrics of the second server.

This **tcp\_module** module collects the probe metrics.

![blackbox exporter metrics](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-4-30.png)

I want to know the SSL/TLS certificate expiry date of the server, for that, I am using **probe\_ssl\_last\_chain\_expiry\_timestamp\_seconds** metrics.

```
probe_ssl_last_chain_expiry_timestamp_seconds
```

**Prometheus output**

![prometheus query](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-5-40.png)

Let's calculate this value **1706436139** to find the actual date and time of the expiration.

![timestamp conversion](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-6-26.png)

We can cross-verify this by searching the website on Google and viewing the certificate details.

![ssl certificate](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-8-28.png)

### Create a Module Using ICMP the Protocol

Example for the ICMP module in Blackbox configuration file **`/etc/blackbox/blackbox.yml`**.

```
modules: 
  ping:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"
```

With this module, we can able to know whether the particular web server or network is reachable or not.

Update the Prometheus configuration file with this module **/etc/prometheus/prometheus.yml**

```
scrape_configs:
  - job_name: "blackbox-icmp"
    metrics_path: /probe
    params:
      module: [ping]
    static_configs:
      - targets:
        - 35.80.157.25
        - devopscube.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.31.19.73:9115
```

Before starting testing, I am blocking the port for ICMP protocol in my EC2 instance so that the result should fail.

![ec2 instance security group inbound rule](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-172-3.png)

In the Blackbox Exporter, you can see the failure result

![blackbox exporter dashboard](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-177-2.png)

I am enabling the **ICMP** protocol in the target EC2 instance.

![aws security group inbound rules](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-178-2.png)

Now the result is **Success**, this ensures that the target server is reachable.

![blackbox exporter dashboard
](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-179-2.png)

If you click the **Logs**, you can able to see what are the probe metrics this module collects.

![blackbox exporter metrics](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-180-3.png)

Let's take one of the metrics from this and make a query in Prometheus to see the results, also we can visualize them with Grafana.

I am choosing the **probe\_icmp\_duration\_seconds** metric.

**Prometheus output**

![prometheus operator query](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-181-2.png)

**Grafana output**

![grafana dashboard](https://storage.ghost.io/c/5f/2f/5f2f4d20-2abf-4534-8d40-7aa233aedd43/content/images/2025/03/image-182-3.png)

## **Prometheus Blackbox Exporter Use Cases**

1. We can know whether a particular website or server is currently reachable or not and also analyze how much time it takes to reach it.
2. Create alerts based on the server's health, for example, if the server is not reachable or it is taking too much time to respond, we will get alerts.
3. SSL/TLS certificate is one of the essential security components of our servers, we can track whether the certificate is present in the server or is valid.
4. We all know, that the DNS resolution should be very fast, only then can we access a website real quick, if it is taking too much time, that will kill the user experience, this exporter can check the DNS-related responses.

## **Conclusion**

This setup gives you an overall idea about, what is **Blackbox** **Exporter** and how this utility works with Prometheus.

Here, I have created modules, only for testing purposes, but you can create modules with a lot of customization because the requirements will differ from person to person.

I am attaching the [official documentation](https://github.com/prometheus/blackbox%5Fexporter/blob/master/CONFIGURATION.md?ref=devopscube.com) with this for your further customization.