Summary of Open Issue: Concurrent Replication in tools bucket replicate
Issue Title: Concurrent Replication for tools bucket replicate
Date Created: May 27, 2021
Original Creator: GiedriusS
Problem Statement:
The current implementation of the tools bucket replicate function is slow because it uploads objects sequentially, one at a time.
Proposed Solution:
Enhance tools bucket replicate to support concurrent uploads of multiple objects, significantly improving replication speed.
Rationale: The convenience of the existing tool is highlighted, as it allows seamless replication between different object storage providers. Given this unique utility, alternative solutions (e.g., using other replication tools) are seen as less viable.
Implementation Notes:
- The implementation is perceived to be straightforward.
- Contributors have engaged with the issue; questions have been asked about the codebase and where to facilitate the concurrent upload changes, particularly in
thanos/pkg/replicate/scheme.go.
Recent Activities:
- Multiple contributors expressed their interest in working on the issue, including Hangzhi, Subhasish-Behera, and Kartik-Garg, the latter of whom has submitted a pull request.
- There was an initial period where the issue saw little activity, leading the bot to prompt for updates.
Current Status: As of March 1, 2023, a pull request has been raised in response to this issue. Potential contributors are encouraged to explore existing PRs for implementation insights, as there currently is no dedicated design documentation for the involved components.
Labeling:
- Component: bucket tools
- Difficulty: easy
- Tags: feature request/improvement, good first issue
This issue represents an ongoing effort to enhance Thanos’ utility by leveraging concurrency in object replication processes.
