thanos-io / thanos

Downsampling - thanos-io/thanos

Downsampling in Thanos

Thanos is an open-source CNCF Sandbox project that builds upon Prometheus components to create a global-scale highly available monitoring system. It seamlessly extends Prometheus in a few simple steps and is already used in production by dozens of companies that aim for high multi-cloud scale for metrics while keeping low maintenance cost.

Downsampling is a process of rewriting series’ to reduce overall resolution of the samples without losing accuracy over longer time ranges. Thanos uses downsampling to provide an opportunity to get fast results for range queries of big time intervals like months or years.

Thanos Compactor is responsible for downsampling of data. There is a time delay before downsampling at a given resolution is possible. This is necessary because downsampled chunks will have fewer samples in them, and as chunks are fixed size, data spanning more time will be required to fill them.

  • Creating 5m downsampling for blocks older than 40 hours (2d)
  • Creating 1h downsampling for blocks older than 10 days (2w)

(Source: docs/components/compact.md)

Possible Options

Thanos Compactor provides the following options for downsampling:

  1. Raw data: This option allows querying only raw data, without any downsampling.
thanos compact --data-dir /tmp/thanos-compact --objstore.config-file=bucket.yml --downsample.resolution=0s
  1. 5m downsampling: This option allows querying data downsampled at a 5-minute resolution.
thanos compact --data-dir /tmp/thanos-compact --objstore.config-file=bucket.yml --downsample.resolution=5m
  1. 1h downsampling: This option allows querying data downsampled at a 1-hour resolution.
thanos compact --data-dir /tmp/thanos-compact --objstore.config-file=bucket.yml --downsample.resolution=1h

(Source: docs/components/compact.md)

Example

Here’s an example of how to use Thanos Compactor for downsampling:

  1. Create a configuration file for the object store (e.g., bucket.yml).
type: GCS
config:
bucket: example-bucket
  1. Run Thanos Compactor with the desired downsampling resolution.
thanos compact --data-dir /tmp/thanos-compact --objstore.config-file=bucket.yml --downsample.resolution=5m

This will create downsampled data at a 5-minute resolution for blocks older than 40 hours.

(Source: docs/components/compact.md)