The SFTP uploader used io.Copy(dstFile, srcFile). Since the source is an
*os.File it implements io.WriterTo, so io.Copy took the os.File.WriteTo
branch and issued one synchronous SFTP WRITE per packet, waiting for the
server reply every time. Throughput was therefore bounded by packetSize/RTT
instead of the link capacity: on a ~68ms RTT link a 440MB backup reached
only ~0.25 MB/s, while rclone / OpenSSH sftp reach 16-19 MB/s on the very
same link and target.
pkg/sftp already provides a pipelined write path (File.ReadFrom, enabled by
UseConcurrentWrites and bounded by MaxConcurrentRequestsPerFile, default 64).
Use it for uploads, and truncate the remote file to the number of bytes
actually written when the concurrent write fails, to avoid leaving holes.
Measured against the same 68ms RTT target: a 96MiB upload went from 6m40s
(~0.25 MB/s) to ~6s (~16 MB/s); a 950MB three-site backup job from ~2h40m to
~2m04s.
Co-authored-by: 4kercc <13767585+4kercc@users.noreply.github.com>
4kerccand4kercc
ssongliu
ssongliu
ssongliu