starred/1Panel
1
0
Fork 0
mirror of https://github.com/1Panel-dev/1Panel.git synced 2026-10-09 16:00:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: speed up SFTP backup upload with concurrent writes (#13976)

Browse Source 
The SFTP uploader used io.Copy(dstFile, srcFile). Since the source is an
*os.File it implements io.WriterTo, so io.Copy took the os.File.WriteTo
branch and issued one synchronous SFTP WRITE per packet, waiting for the
server reply every time. Throughput was therefore bounded by packetSize/RTT
instead of the link capacity: on a ~68ms RTT link a 440MB backup reached
only ~0.25 MB/s, while rclone / OpenSSH sftp reach 16-19 MB/s on the very
same link and target.

pkg/sftp already provides a pipelined write path (File.ReadFrom, enabled by
UseConcurrentWrites and bounded by MaxConcurrentRequestsPerFile, default 64).
Use it for uploads, and truncate the remote file to the number of bytes
actually written when the concurrent write fails, to avoid leaving holes.

Measured against the same 68ms RTT target: a 96MiB upload went from 6m40s
(~0.25 MB/s) to ~6s (~16 MB/s); a 950MB three-site backup job from ~2h40m to
~2m04s.

Co-authored-by: 4kercc <13767585+4kercc@users.noreply.github.com>
This commit is contained in:
4kercc
2026-10-08 09:45:51 +08:00
committed by GitHub
co-authored by 4kercc
parent d21288788e
commit 3571fb8d86
1 changed files with 10 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent/utils/cloud_storage/client/sftp.go
+10 -2
View File
@@ -97,7 +97,7 @@ func (s sftpClient) Upload(ctx context.Context, src, target string) (bool, error
}()
defer close(done)
client, err := sftp.NewClient(sshClient)
client, err := sftp.NewClient(sshClient, sftp.UseConcurrentWrites(true))
if err != nil {
return false, err
}
@@ -127,7 +127,15 @@ func (s sftpClient) Upload(ctx context.Context, src, target string) (bool, error
}
defer dstFile.Close()
if _, err := io.Copy(dstFile, srcFile); err != nil {
// ReadFrom uses the pipelined write path of pkg/sftp. A plain io.Copy would
// pick os.File.WriteTo (the source is an *os.File), which issues one
// synchronous SFTP WRITE per packet and waits for the reply every time, so
// the throughput is bounded by packetSize/RTT.
written, err := dstFile.ReadFrom(srcFile)
if err != nil {
// A failed concurrent write may leave holes in the remote file, so cut
// it back to the number of bytes actually written.
_ = dstFile.Truncate(written)
if ctxErr := ctx.Err(); ctxErr != nil {
return false, ctxErr
}