thanos-io / thanos

Improve compatibility testing against Prometheus and TSDB.

Summary of Open Issue: Compatibility Testing Enhancements for Thanos

Issue Overview

The goal is to enhance compatibility testing between Thanos and various versions of Prometheus and its Time Series Database (TSDB). The need arises from ensuring that Thanos supports multiple Prometheus versions, particularly focusing on TSDB format and HTTP API compatibility.

Acceptance Criteria

  1. Determine compatible versions of Prometheus for each Pull Request (PR) in Thanos.
  2. Support should encompass:
  • TSDB format: Ensure backward compatibility with TSDB versions used by different Prometheus releases.
  • HTTP API: Guarantee support for API changes, noting that while major functionalities remain stable, new features may be introduced across minor versions.

Current Issues

  • Testing Methodology: The current approach runs all tests against a set of hardcoded supported Prometheus versions (e.g., 2.2.1, 2.3.2, etc.) using a Makefile loop. This has led to issues with:
  • Golang Test Caching: Environment variable changes may not be detected, leading to stale tests being executed.
  • Signal Handling: Erroneous passing of tests that should fail.
  • Dependencies: Upgrading TSDB dependencies can hinder compatibility testing, as newer TSDB formats are not forward-compatible with older Prometheus versions.

Proposed Improvements

  1. Versioned TSDB Dependencies: Introduce a TSDB dependency versioned to match Prometheus 2.0 to maintain understanding across all Prometheus versions during testing.
  2. Refine Testing Approach:
  • Replace external loops with an internal for-loop managed within the test code to improve caching and signal handling issues.
  • Consider mechanisms to selectively skip tests based on version requirements, enhancing clarity and control.
  1. Optimization of Test Execution: Potentially clean the Go test cache (go clean -testcache) before each test loop to prevent caching-related problems.

Additional Considerations

  • It would be beneficial to add integration tests that validate the compatibility of TSDB blocks generated by Thanos when used in Prometheus, ensuring seamless utilization of stored blocks.

Stakeholders

  • Participants: Members like @bwplotka have initiated discussions and proposed solutions, with ongoing engagement from others.
  • Interest for Implementation: Recent inquiries from users indicate a desire to pick up the issue and contribute.

Labels

  • The issue is tagged as medium difficulty and is marked as a feature request/improvement, with indications for being a good first issue for newcomers.

Status

The issue remains open with some prior discussions but lacks recent active contributions. There is renewed interest in addressing the problem, indicating potential for upcoming work on the issue.