screenly / balena-prometheus-exporter

Monitoring and Alerting - screenly/balena-prometheus-exporter

To set up monitoring and alerting for the balena-prometheus-exporter project, you can use Prometheus and the Prometheus Alertmanager. Here’s how to do it:

  1. Configure Prometheus

Prometheus configuration is in YAML format. The default configuration file is prometheus.yml. Here is a sample configuration for scraping the metrics from the balena-prometheus-exporter:

global:
scrape_interval:     15s
evaluation_interval: 15s

scrape_configs:
- job_name: 'balena-prometheus-exporter'
static_configs:
- targets: ['exporter_address:9323']

Replace exporter_address with the address of your balena-prometheus-exporter instance.

  1. Set up Alerting Rules

You can create alerting rules in Prometheus to generate alerts based on the metrics. For example, you can create an alert for high CPU usage:

alert: HighCPUUsage
expr: node_cpu_seconds_total{mode="idle"} < 50
for: 5m
annotations:
summary: High CPU usage on node
description: The CPU usage is higher than 50% for 5 minutes

This alert will trigger if the idle CPU time is below 50% for 5 minutes.

  1. Configure Alertmanager

The Alertmanager is responsible for managing and notifying alerts. Here is a sample configuration for Alertmanager:

global:
resolve_timeout: 5m

route:
receiver: 'team-X-mails'
routes:
- match:
alertname: DeadMansSwitch
continue: false

receivers:
- name: 'team-X-mails'
email_configs:
- to: '[email protected]'

This configuration will send email notifications to [email protected] for all alerts, except for the DeadMansSwitch alert.

  1. Synthetic Monitoring

You can use Grafana’s Synthetic Monitoring to monitor the availability of your balena-prometheus-exporter instance. Here is an example of a Ping check configuration:

  • Check type: PING
  • Job name: balena-prometheus-exporter-ping
  • Target: exporter_address
  • Probe locations: All

You can create similar checks for HTTP/HTTPS, TCP, and DNS.

  1. Optional Environment Variable

You can use the ALERTMANAGER_URL environment variable to specify the URL of the Alertmanager. The exporter will send the alerts to the Alertmanager if this variable is set.

For more information, see: