thanos-io / thanos

Rule: remote-write.config should not be the trigger to enable stateless ruler

Summary of Open Issue: remote-write.config Should Not Trigger Stateless Ruler

Issue Title: Rule: remote-write.config should not be the trigger to enable stateless ruler

Reported By: ahurtaud (contributor)

Date: June 23, 2023

Problem: The current implementation automatically enables the stateless mode for the Thanos ruler when the --remote-write.config flag is set. As a result, no series are stored in the ruler’s TSDB, preventing a stateful ruler from remotely writing a subset of generated recording rule metrics to external systems. This functionality is available in Prometheus and is a limitation when using the stateful ruler in Thanos.

Proposed Solution: The contributor suggests implementing a dedicated flag to enable stateless mode independently of the remote-write.config, allowing for the execution of recording rules in stateful mode while also supporting the remote writing of specific series to an external system. This change would enable Ruler users to manage how metric processing occurs without being forced into a stateless architecture.

Alternatives Considered: While migrating rulers to a stateless architecture could be an option, the contributor prefers to maintain stateful mode for their use case.

Community Feedback:

  • yeya24 supports the need for a dedicated flag.
  • manulpatel expresses interest in working on the issue and seeks guidance.
  • moki1202 proposes a potential implementation strategy by defining a stateless mode flag in rule.go and modifying the RegisterPathOrContent function. However, they are unsure about changes needed for the handling of time series storage in stateful mode.
  • There is ongoing discussion regarding the complexity of allowing both remote write and TSDB storage, indicating that the implementation might require more than just a new flag.
  • aditya7302 expresses a desire to take on the issue if it remains unresolved.

Labels: component: rule, good first issue, help wanted

This summary serves as an overview of the ongoing discussions and proposed developments regarding the handling of the stateless and stateful modes of the Thanos ruler in relation to the remote-write.config flag. Advanced developers are invited to contribute to resolving the proposed changes.