thanos-io / thanos

s3/aliyunOSS: Support multipart upload without pre known object size.

Summary of Open Issue: Support Multipart Upload Without Pre-Known Object Size in s3/aliyunOSS

Issue Overview

  • Title: Support multipart upload without pre-known object size in the s3/aliyunOSS provider.
  • Created by: bwplotka on December 13, 2018.
  • Current Status: Open and seeking contributors.
  • Discussion Reference: #617.

Context and Motivation The challenge arises in integrating multipart uploads for scenarios where the size of the object to be uploaded is not known beforehand. This is essential, especially when utilizing providers such as MinIO and Aliyun OSS within the Thanos architecture, which involves storage and retrieval of time-series data.

Acceptance Criteria

  • The MinIO S3 provider must intelligently determine when to activate multipart uploads using the Upload(ctx context.Context, name string, r io.Reader) interface.
  • For Aliyun, implementation may involve either devising a custom solution or extending the existing objstore interface.

Key Contributions and Ideas

  • Early discussions highlight the urgency for this feature due to use cases like observatorium/thanos-replicate#22, which necessitate compatibility with large file uploads.
  • Insights were shared by contributors on utilizing resumable uploads for Aliyun without enforcing size constraints, potentially simplifying integration.

Recent Status Updates

  • There have been numerous comments and inquiries about the status of the issue over the years, notably by yeya24 and xiaozongyang, indicating continued relevance and highlighting production use cases where failures occur due to unsupported io.Reader types.
  • The issue has experienced recurrent stale bot interventions, prompting periodic requests for updates.

Current Activity and Call to Action

  • As of July 2024, a new contributor bdharsan04 expressed interest in tackling this issue, emphasizing its open status and potential for engagement.

Labels for the Issue

  • Difficulty: Medium
  • Type: Feature Request/Improvement
  • Good First Issue
  • Help Wanted

This issue remains a critical enhancement for support and functionality within the Thanos project, particularly for developers looking to improve integration with cloud storage solutions. Advanced developers intrigued by storage optimizations and S3 compatibility are encouraged to contribute.