How To Monitor Linux Servers Using Prometheus Node Exporter

Node exporter is the best way to collect all the Linux server related metrics and statistics for monitoring.

Monitor Linux Servers Using Prometheus

In this guide, you will learn how to setup Prometheus node exporter on a Linux server to export all node level metrics to the Prometheus server.

Before You Begin

  1. Prometheus Node Exporter needs Prometheus server to be up and running. If you would like to setup Prometheus, please see the Prometheus setup guide for Linux.
  2. Port 9100 opened in server firewall as Prometheus reads metrics on this port.

Setup Node Exporter Binary

Step 1: Download the latest node exporter package. You should check the Prometheus downloads section for the latest version and update this command to get that package.

cd /tmp
curl -LO https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

Step 2: Unpack the tarball

tar -xvf node_exporter-0.18.1.linux-amd64.tar.gz

Step 3: Move the node export binary to /usr/local/bin

sudo mv node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin/

Create a Custom Node Exporter Service

Step 1: Create a node_exporter user to run the node exporter service.

sudo useradd -rs /bin/false node_exporter

Step 2: Create a node_exporter service file under systemd.

sudo vi /etc/systemd/system/node_exporter.service

Step 3: Add the following service file content to the service file and save it.

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Step 4: Reload the system daemon and star the node exporter service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter

Step 5: check the node exporter status to make sure it is running in the active state.

sudo systemctl status node_exporter

Step 6: Enable the node exporter service to the system startup.

sudo systemctl enable node_exporter

Now, node exporter would be exporting metrics on port 9100. 

You can see all the server metrics by visiting your server URL on /metrics as shown below.

http://<server-IP>:9100/metrics

Configure the Server as Target on Prometheus Server

Now that we have the node exporter up and running on the server, we have to add this server a target on the Prometheus server configuration.

Note: This configuration should be done on the Prometheus server.

Step 1: Login to the Prometheus server and open the prometheus.yml file.

sudo vi /etc/prometheus/prometheus.yml

Step 2: Under the scrape config section add the node exporter target as shown below. Change 10.142.0.3 with your server IP where you have setup node exporter. Job name can be your server hostname or IP for identification purposes.

- job_name: 'node_exporter_metrics'
  scrape_interval: 5s
  static_configs:
    - targets: ['10.142.0.3:9100']

Step 3: Restart the prometheus service for the configuration changes to take place.

sudo systemctl restart prometheus

Now, if you check the target in prometheus web UI (http://<prometheus-IP>:9090/targets) , you will be able to see the status as shown below.

prometheus node exporter status

Also, you can use the Prometheus expression browser to query for node related metrics. Following are the few key node metrics you can use to find its statistics.

node_memory_MemFree_bytes
node_cpu_seconds_total
node_filesystem_avail_bytes
rate(node_cpu_seconds_total{mode="system"}[1m]) 
rate(node_network_receive_bytes_total[1m])
prometheus node exporter metric queries

Follow “Integrate And Visualize Prometheus Metrics In Grafana” blog to visualize the node exporter metrics in Grafana.

10 comments
  1. Hi everyone, just so you know, at least in Debian 10, when editing the prometheus.yml you need to have the job name as well as the rest of the copy and paste to line up with the above job name that is already there. If you don’t you will get an error when doing sudo systemctl status prometheus

  2. Hi Guys,
    I have installed prometheus and Node Exporter and Grafana , it is working properly in Cent OS 7.4. I have few client server centos 6.4 64 bit. try to Node Exporter but service is not starting. Please help me.

  3. hi i want create snmp exporter as sevice with toturial but i have this erorr can you help me

    snmp-exporter.service – snmp Exporter
    Loaded: loaded (/etc/systemd/system/snmp-exporter.service; disabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Mon 2019-12-30 14:06:15 +0330; 4s ago
    Process: 12884 ExecStart=/usr/local/bin/snmp_exporter/snmp-exporter (code=exited, status=1/FAILURE)
    Main PID: 12884 (code=exited, status=1/FAILURE)

  4. I followed your steps. when check the status of promtheus service it shows failed .

    # systemctl status prometheus
    ● prometheus.service – Prometheus
    Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
    Active: failed (Result: exit-code) since Fri 2019-06-14 16:49:54 +0530; 8s ago
    Process: 8125 ExecStart=/usr/local/bin/prometheus –config.file /etc/prometheus/prometheus.yml –storage.tsdb.path /var/lib/prometheus/ –web.console.templates=/etc/prometheus/consoles –web.console.libraries=/etc/prometheus/console_libraries (code=exited, status=1/FAILURE)
    Main PID: 8125 (code=exited, status=1/FAILURE)

    1. I had some similar – looks like selinux doesn’t help either. It was because I was in the home directory then moved node_exporter to /usr/local/bin/.
      to fix:
      cd /usr/local/bin/
      sudo restorecon -rv node_exporter

      mine then worked.

  5. I have done everything you explained in these two tutorial but ı am getting state “down” for node_exporter_metrics. How can I fix this?

    1. Please check the firewall rules..make sure the right Prometheus ports are open for communication…Meanwhile, if you fixed it, please let us know the fix so that we add it to the error list.

Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like