The SFTP uploader used io.Copy(dstFile, srcFile). Since the source is an
*os.File it implements io.WriterTo, so io.Copy took the os.File.WriteTo
branch and issued one synchronous SFTP WRITE per packet, waiting for the
server reply every time. Throughput was therefore bounded by packetSize/RTT
instead of the link capacity: on a ~68ms RTT link a 440MB backup reached
only ~0.25 MB/s, while rclone / OpenSSH sftp reach 16-19 MB/s on the very
same link and target.
pkg/sftp already provides a pipelined write path (File.ReadFrom, enabled by
UseConcurrentWrites and bounded by MaxConcurrentRequestsPerFile, default 64).
Use it for uploads, and truncate the remote file to the number of bytes
actually written when the concurrent write fails, to avoid leaving holes.
Measured against the same 68ms RTT target: a 96MiB upload went from 6m40s
(~0.25 MB/s) to ~6s (~16 MB/s); a 950MB three-site backup job from ~2h40m to
~2m04s.
Co-authored-by: 4kercc <13767585+4kercc@users.noreply.github.com>
* chore: update dependencies and migrate S3 client to MinIO SDK
- Removed AWS SDK dependency and replaced S3 client implementation with MinIO SDK.
- Updated various dependencies in go.mod and go.sum, including version upgrades for `klauspost/compress`, `minio/minio-go`, and `rs/xid`.
- Adjusted S3 client methods to utilize MinIO's API for bucket listing, object existence checks, uploads, downloads, and deletions.
* refactor: enhance S3 client with context management and TLS support
- Introduced context management with timeouts for S3 client operations to improve reliability.
- Added TLS configuration to support secure connections based on the endpoint scheme.
- Updated S3 client methods to utilize the new context and transport settings for enhanced performance and security.
---------
Co-authored-by: ssongliu <sloooop1x@gmail.com>
4kerccand4kercc
ssongliu
KOMATAandssongliu
ssongliu
cui
蘭
igophper
zhengkunwang223
ssongliu