Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[sftp] Improve write & memory performance when saving files (#1194)
* Move seekability test to a separate helper function Having a third copy of this code would feel silly. * Stop calling content.open() from SFTPStorage._save The open method isn't available in all file-like objects one might want to save using the storage framework, so instead of relying on it calling seek(0) for us do it ourselves (conditionally). * Require at least version 1.10.0 of paramiko This version introduced the SFTPClient.putfo() method. It was released in early 2013, so this shouldn't be a great hardship. * Use SFTPClient.putfo() in SFTPStorage._save() Instead of using SFTPClient.open() to get a file-like object, reading the entire content into memory and then calling .write() on that. The measured performance difference is stunning, and the uploaded file data no longer needs to fit into memory.
- Loading branch information
1 parent
b04de35
commit 04676b5
Showing
6 changed files
with
22 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -46,7 +46,7 @@ google = | |
libcloud = | ||
apache-libcloud | ||
sftp = | ||
paramiko | ||
paramiko >= 1.10.0 | ||
|
||
[flake8] | ||
exclude = | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters