Techniques and best practices for optimizing Thanos’ performance
Caching
Thanos uses a technique called index-header caching to reduce the number of disk reads and improve query performance. The Store Gateway does not fully load the entire index-header content into memory but only loads 1/32 of postings offsets. At query time, postings offsets are looked up both from memory and from the mmap-ed index-header files. This optimization technique allows for significant reduction in Store Gateway memory utilization while increasing CPU and disk read OPS.
To control the ratio of postings offsets loaded into memory, Thanos provides a hidden CLI flag --store.index-header-posting-offsets-in-mem-sampling. Larger values will lower memory usage and increase CPU and disk read OPS, while lower values will increase memory and lower CPU and disk read OPS. A value of 1 will keep all in memory.
Downsampling
Thanos performs downsampling at query time to reduce the amount of data that needs to be transferred between local Queriers and the central Querier. This is achieved by rewriting queries using a logical optimizer into a form that is suitable for distributed execution.
The proposed algorithm for distributing PromQL queries is as follows:
- Start AST traversal from the bottom up.
- If both the current node and its parent can be distributed, move up to the parent.
- If the current node can be distributed and its parent cannot, rewrite the current node into its distributed form.
- If the current node cannot be distributed, stop traversal.
With this algorithm, Thanos tries to distribute as much of the PromQL query as possible. Even queries without aggregations, like rate(http_requests_total[2m]), will be rewritten into:
coalesce(
rate(http_requests_total[2m]),
rate(http_requests_total[2m])
)
Since PromQL queries are limited in the number of steps they can evaluate, this algorithm achieves downsampling at query time by only sending a small number of samples from local Queriers to the central Querier.
Thanos documentation - Performance Optimization
