Summary of Open Issue: Promtool Support for Test and Coverage of Rules
Issue Number: #11848
Created by: michael-doubez
Date Created: 2023-01-13
Current Status: Ongoing discussion; focus on coverage aspect
Proposal Overview
The issue proposes extending promtool to support better testing and coverage reporting for Prometheus rules, particularly in GitOps workflows. Currently, promtool check config/rules outputs results only to standard output and communicates failures through return codes, which is sufficient for deployment gating but does not ensure rule coverage in tests.
Key Points
- Current Limitation: Validating Prometheus rules relies heavily on code reviews, making it easy to overlook rules lacking tests.
- Desired Features:
- An output format that can be easily parsed by scripts — ideally minimal xunit/junit format for compatibility with CI tools like Jenkins.
- Comprehensive reporting to indicate:
- Individual test successes and failures.
- Coverage of rules — identifying untested rules, potentially as warnings or errors.
Example Output Format
A suggested XML format is provided for reporting test results, including suites, cases, and failure messages.
Discussion and Progress
- As of October 2024, there was an indication that other issues (specifically #14506) may address some parts of this request. However, coverage reporting remains unhandled, keeping this issue open.
- A new contributor expressed interest in working on this issue (reinainblood). The community has encouraged investigation into related issue #12253 to determine if it is a duplicate or a distinctly separate concern.
Labels
- good first issue
- help wanted
- kind/enhancement
Next Steps
New contributor to explore issue #12253 and determine its relationship to this issue. Continued discussion and contributions are encouraged to enhance the testing and coverage capabilities of promtool.
