How to Run Prometheus on a Raspberry Pi

Prometheus is a cool network monitoring software to make sure the servers and services that you set up continue to run on your network. Monitoring can be a part of your home Raspberry Pi network!

To get started, go to the Prometheus website https://prometheus.io/download/ and use the pulldowns to find the server software. I’m going to install it on an RPi4, so I choose “armv7”. If you’re planning on different hardware, adjust accordingly. Right-click on the blue “prometheus-2.23.0.linux-armv7.tar.gz” link and copy it.

Now, ssh to your server. Next, create a non-root user to run the server.

useradd -m -s /bin/bash prometheus

sudo to the new user and wget software from the link you just copied.

root@prometheus:~# useradd -m -s /bin/bash prometheus
root@prometheus:~# su - prometheus
prometheus@prometheus:~ $ wget https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-armv7.tar.gz
 --2020-12-29 15:13:26--  https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-armv7.tar.gz
 HTTP request sent, awaiting response… 200 OK
 Length: 59186922 (56M) [application/octet-stream]
 Saving to: ‘prometheus-2.23.0.linux-armv7.tar.gz’
 prometheus-2.23.0.linux-armv7.tar.gz           100%[=================================================================================================>]  56.44M  16.6MB/s    in 4.2s
 2020-12-29 15:13:31 (13.5 MB/s) - ‘prometheus-2.23.0.linux-armv7.tar.gz’ saved [59186922/59186922]

Untar the distribution and move into prometheus’s home:

tar xvfz prometheus-2.23.0.linux-armv7.tar.gz 
mv prometheus-2.23.0.linux-armv7/* /home/prometheus/

After that, create a new ‘data’ directory that will be used as a ‘tsdb’ storage.

mkdir -p ~/prometheus/data

cd to the ~ directory and just run the binary:

prometheus@prometheus:/home/prometheus# ./prometheus
 level=info ts=2020-12-29T20:19:05.640Z caller=main.go:322 msg="No time or size retention was set so using the default time retention" duration=15d
 level=info ts=2020-12-29T20:19:05.640Z caller=main.go:360 msg="Starting Prometheus" version="(version=2.23.0, branch=HEAD, revision=26d89b4b0776fe4cd5a3656dfa520f119a375273)"
 level=warn ts=2020-12-29T20:19:05.640Z caller=main.go:362 msg="This Prometheus binary has not been compiled for a 64-bit architecture. Due to virtual memory constraints of 32-bit systems, it is highly recommended to switch to a 64-bit binary of Prometheus." GOARCH=arm
 level=info ts=2020-12-29T20:19:05.640Z caller=main.go:365 build_context="(go=go1.15.5, user=root@37609b3a0a21, date=20201126-11:53:03)"
 level=info ts=2020-12-29T20:19:05.640Z caller=main.go:366 host_details="(Linux 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l prometheus (none))"
 level=info ts=2020-12-29T20:19:05.640Z caller=main.go:367 fd_limits="(soft=1024, hard=1048576)"
 level=info ts=2020-12-29T20:19:05.640Z caller=main.go:368 vm_limits="(soft=unlimited, hard=unlimited)"
 level=info ts=2020-12-29T20:19:05.645Z caller=main.go:722 msg="Starting TSDB …"
 level=info ts=2020-12-29T20:19:05.645Z caller=web.go:528 component=web msg="Start listening for connections" address=0.0.0.0:9090
 level=info ts=2020-12-29T20:19:05.662Z caller=head.go:645 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
 level=info ts=2020-12-29T20:19:05.662Z caller=head.go:659 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=8.5µs
 level=info ts=2020-12-29T20:19:05.662Z caller=head.go:665 component=tsdb msg="Replaying WAL, this may take a while"
 level=info ts=2020-12-29T20:19:05.663Z caller=head.go:717 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
 level=info ts=2020-12-29T20:19:05.663Z caller=head.go:722 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=118.628µs wal_replay_duration=691.786µs total_replay_duration=896.117µs
 level=info ts=2020-12-29T20:19:05.673Z caller=main.go:742 fs_type=EXT4_SUPER_MAGIC
 level=info ts=2020-12-29T20:19:05.673Z caller=main.go:745 msg="TSDB started"
 level=info ts=2020-12-29T20:19:05.673Z caller=main.go:871 msg="Loading configuration file" filename=prometheus.yml
 level=info ts=2020-12-29T20:19:05.699Z caller=main.go:902 msg="Completed loading of configuration file" filename=prometheus.yml totalDuration=25.969176ms remote_storage=4.334µs web_handler=1.37µs query_engine=2.371µs scrape=24.757249ms scrape_sd=217.7µs notify=124.258µs notify_sd=76.184µs rules=4.074µs
 level=info ts=2020-12-29T20:19:05.699Z caller=main.go:694 msg="Server is ready to receive web requests."

So far, so good. It runs. Keep in mind that we’re just running this in the shell. We’ll need to create a unit file to make sure that this is running as a service.

Prometheus Unit File

Create a new file in /lib/systemd/system/prometheus.service. The content should be as follows:

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=prometheus
Restart=on-failure
ExecStart=/home/prometheus/prometheus \
  --config.file=/home/prometheus/prometheus.yml \
  --storage.tsdb.path=/home/prometheus/data

[Install]
WantedBy=multi-user.target

Next, start and enable your new service:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus
sudo systemctl enable prometheus

The default configuration in “prometheus.yml” just monitors itself. Great. Let’s add some more hosts (or targets) to the config and then we can look at installing the agent on those hosts.

Adding Prometheus Targets

The base config is pretty simple. Let’s grab the end of the file that contains the targets, rip out the comments, and copy it for each new target:

scrape_configs:
 - job_name: 'prometheus'
 static_configs:
 - targets: ['localhost:9090']

This is the end of the yml file on the server. Copy, paste, and edit. You can see where the hostnames change in each paragraph in the example:

scrape_configs:
 - job_name: 'prometheus'
 static_configs:
 - targets: ['localhost:9090']

 - job_name: 'salt'
 static_configs:
 - targets: ['salt:9090']

 - job_name: 'mons'
 static_configs:
 - targets: ['mons:9090']

And I’ve effectively added two more hosts, salt and mons! Now, we need to get the agent installed on each of those hosts.

Installing the Prometheus Agent on New Targets

Head back to the Prometheus website, https://prometheus.io/download/ and scroll down a little further to node-exporter. This is the basic agent that scrapes and reports on machine info. Make sure you’ve got the pulldown set for your target hardware type.

Same routine, right click>copy the link and then wget the software on the desired node. You can just untar it and run it, but it probably makes sense to put together a salt script to install the proper version for the hardware on which it’s getting installed and then create a control script to run it. (Watch for upcoming posts!) Saltstack is optional, but we are going to need a unit script to keep the agent running. Here’s a basic unit file that you can customize to your liking. Copy this text into /lib/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter 

[Service] 
User=node_exporter 
Group=node_exporter 
ExecStart=/opt/node_exporter $OPTIONS 

[Install] 
WantedBy=multi-user.target

Probably best to create a user and group to run the exporter.

useradd --no-create-home --shell /bin/false node_exporter

Then enable the new service file:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl status node_exporter
sudo systemctl enable node_exporter

Done! You’ve got the server running and the exporter running on two targets as a service!

Next, we’ll look at creating a saltstack script to install more exporters on all the computers!

Author: John

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.