Recording rules in Prometheus are used to pre-compute frequently needed or computationally expensive expressions and save their result as a new set of time series. This is useful if you want to run alerts on aggregated data or if you have dashboards that query computationally expensive expressions repeatedly. Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.
Prometheus supports two types of rules: recording rules and alerting rules. Recording rules are just normal queries that are run periodically (default 1m) and whose result is stored in another metric. Alerting rules, on the other hand, are used to define alerts and are evaluated at regular intervals.
To include rules in Prometheus, you need to create a file containing the necessary rule statements and have Prometheus load the file via the rule_files field in the Prometheus configuration. Rule files use YAML and can be reloaded at runtime by sending SIGHUP to the Prometheus process.
Here is an example of a recording rule:
groups:
- name: example
rules:
- record: job:http_inprogress_requests:sum
expr: sum(http_inprogress_requests) by (job)
In this example, the record key is used to specify the name of the new time series that will be created. The expr key is used to specify the expression that will be evaluated periodically. In this case, the expression is sum(http_inprogress_requests) by (job), which will calculate the sum of the http_inprogress_requests metric for each job.
It is important to note that recording rules can have a significant impact on performance, especially if they are computationally expensive. Therefore, it is recommended to use them judiciously and to carefully evaluate their impact on performance.
For more information on recording rules, please refer to the Prometheus documentation.
Sources:
- Prometheus documentation on recording rules: https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/
- Grafana documentation on recording rules: https://grafana.com/docs/grafana/latest/alerting/fundamentals/alert-rules/recording-rules/
- Prometheus blog post on one year of open source development: https://prometheus.io/blog/2016/01/26/one-year-of-open-prometheus-development/
- Prometheus command line tool documentation: https://prometheus.io/docs/prometheus/latest/command-line/promtool/
- KubeVela documentation on metrics collection: https://kubevela.io/docs/platform-engineers/operations/o11y/metrics
- Prometheus blog post on subqueries: https://prometheus.io/blog/2018/06/21/subqueries-evaluation-ordering-and-timing/
- Grafana Enterprise Metrics documentation on downloads: https://grafana.com/docs/enterprise-metrics/latest/downloads/
