[TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files #3125

zoewangg · 2022-03-24T23:17:36Z

Motivation and Context

#3108
#2300

Modifications

Expose an option to overwrite an existing file in FileAsyncResponseTransformer.
Note that it's not added to FileResponseTransformer right now because it's not trivial to do so. I will create a backlog item for it.

Testing

Added unit tests and integ tests

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

I have read the CONTRIBUTING document
Local run of mvn install succeeds
My code follows the code style of this project
My change requires a change to the Javadoc documentation
I have updated the Javadoc documentation accordingly
I have added tests to cover my changes
All new and existing tests passed
I have added a changelog entry. Adding a new entry must be accomplished by running the scripts/new-change script and following the instructions. Commit the new file created by the script in .changes/next-release with your changes.
My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

I confirm that this pull request can be released under the Apache 2 license

core/sdk-core/src/main/java/software/amazon/awssdk/core/FileTransformerConfiguration.java

core/sdk-core/src/main/java/software/amazon/awssdk/core/async/AsyncResponseTransformer.java

...e/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformer.java

.changes/next-release/feature-AWSSDKforJavav2-1ee4d61.json

core/sdk-core/src/main/java/software/amazon/awssdk/core/FileTransformerConfiguration.java

dagnir · 2022-03-25T18:22:24Z

...e/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformer.java

+    private long determineFilePositionToWrite(Path path) {
+        if (fileModificationOption == CREATE_OR_APPEND_EXISTING) {
+            try {
+                return Files.size(path);
+            } catch (NoSuchFileException e) {
+                // Ignore
+            } catch (IOException exception) {
+                throw SdkClientException.create("Cannot determine the current file size " + path, exception);
+            }


This seems like an expensive thing to do all the time (e.g. if the file is always being created for the first time). Is there any way we can defer this until we need it or use a solution that doesn't rely on catching NoSuchFileException?

File position has to be determined in the ctor because otherwise retry would not work for appending in the case where certain bytes have been written. We could callFiles.notExists, but it also relies on catching NoSuchFileException behind the scene.

Hmm couldn't we do it at the time we open the file?

No not really, we open the file in onStream method, so the position we get at this point may not be correct if the previous attempt had written a few bytes already before it failed.

…ansformer

…e option by default

zoewangg · 2022-03-25T23:25:11Z

Changed the target branch to zoewang/tm-download-pause-resume to make sure it works with TM download pause/resume before we merge it to mainline.

~~I'm still updating the PR~~

sonarcloud · 2022-03-29T02:47:01Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
11 Code Smells

88.4% Coverage
0.0% Duplication

millems · 2022-03-29T19:51:22Z

core/sdk-core/src/main/java/software/amazon/awssdk/core/FileTransformerConfiguration.java

+     */
+    public static FileTransformerConfiguration defaultCreateOrAppend() {
+        return builder().fileWriteOption(FileWriteOption.CREATE_OR_APPEND_EXISTING)
+                        .failureBehavior(FailureBehavior.LEAVE)


Any specific reason for this default?

Yeah, retry would not work as expected if we delete the file when there is an exception.

But it also might not work as expected if we started appending, failed and then on a retry reappended the same data.

The initial position is recorded in the ctor, so we will always append on the same offset on retries (overwrite whatever bytes that have been written in the previous attempt)

dagnir · 2022-03-29T22:23:14Z

...e/src/main/java/software/amazon/awssdk/core/internal/async/FileAsyncResponseTransformer.java

+        if (configuration.fileWriteOption() == CREATE_OR_APPEND_EXISTING) {
+            try {
+                return Files.size(path);
+            } catch (NoSuchFileException e) {
+                // Ignore
+            } catch (IOException exception) {
+                throw SdkClientException.create("Cannot determine the current file size " + path, exception);
+            }
+        }
+        return  0L;


Can we add a backlog item to revisit this? It makes me nervous to rely on catching an exception on a possibly hot code path.

We could call Files#exists first but it uses the same try-catch though... I'm not sure how we can optimize it, but yeah, I can add a TODO

* [TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files (#3125) * Expose an option to overwrite an existing file in FileAsyncResponseTransformer * Add changelog entries and make TM use CREATE_OR_REPLACE_EXISTING write option by default * Address feedback * Update and address feedback * [TM DownloadFile Pause and Resume] Part 2: Implement pause for downloadFile operation (#3094) * Part 1: Implement pause for downloadFile operation * Address feedback * Refactor the logic * Address feedback * Fix merging issue * [TM DownloadFile Pause and Resume] Part 3: Implement resumeDownloadFile (#3157) * Implement resumeDownloadFile * Move test code around * Address feedback * Fix flaky test * fix flaky integ test * add changelog entry * Troubleshooting flaky test * Add file length check when checking if file has modified or not * Address feedback

zoewangg requested a review from a team as a code owner March 24, 2022 23:17

millems reviewed Mar 24, 2022

View reviewed changes

dagnir reviewed Mar 25, 2022

View reviewed changes

zoewangg changed the base branch from master to zoewang/tm-download-pause-resume March 25, 2022 19:01

zoewangg added 3 commits March 25, 2022 12:08

Expose an option to overwrite an existing file in FileAsyncResponseTr…

cc6cb52

…ansformer

Add changelog entries and make TM use CREATE_OR_REPLACE_EXISTING writ…

97cc2a2

…e option by default

Address feedback

1576120

zoewangg force-pushed the zoewang/fileOverwrite branch from 6c8a211 to 1576120 Compare March 25, 2022 22:52

Update and address feedback

307f465

zoewangg changed the title ~~Add configuration to enable overwriting existing files~~ [TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files Mar 29, 2022

millems approved these changes Mar 29, 2022

View reviewed changes

dagnir approved these changes Mar 29, 2022

View reviewed changes

zoewangg merged commit eade04e into zoewang/tm-download-pause-resume Mar 29, 2022

zoewangg deleted the zoewang/fileOverwrite branch March 29, 2022 23:49

zoewangg mentioned this pull request Apr 25, 2022

Implement pause/resume for downloadFile in S3 transfer manager #3169

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files #3125

[TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files #3125

zoewangg commented Mar 24, 2022 •

edited

dagnir Mar 25, 2022

zoewangg Mar 25, 2022 •

edited

dagnir Mar 25, 2022

zoewangg Mar 25, 2022

zoewangg commented Mar 25, 2022 •

edited

sonarcloud bot commented Mar 29, 2022

millems Mar 29, 2022

zoewangg Mar 29, 2022

millems Mar 29, 2022

zoewangg Mar 29, 2022 •

edited

dagnir Mar 29, 2022

zoewangg Mar 29, 2022 •

edited

[TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files #3125

[TM DownloadFile Pause and Resume] Part 1: Add configuration to enable overwriting existing files #3125

Conversation

zoewangg commented Mar 24, 2022 • edited

Motivation and Context

Modifications

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

dagnir Mar 25, 2022

Choose a reason for hiding this comment

zoewangg Mar 25, 2022 • edited

Choose a reason for hiding this comment

dagnir Mar 25, 2022

Choose a reason for hiding this comment

zoewangg Mar 25, 2022

Choose a reason for hiding this comment

zoewangg commented Mar 25, 2022 • edited

sonarcloud bot commented Mar 29, 2022

millems Mar 29, 2022

Choose a reason for hiding this comment

zoewangg Mar 29, 2022

Choose a reason for hiding this comment

millems Mar 29, 2022

Choose a reason for hiding this comment

zoewangg Mar 29, 2022 • edited

Choose a reason for hiding this comment

dagnir Mar 29, 2022

Choose a reason for hiding this comment

zoewangg Mar 29, 2022 • edited

Choose a reason for hiding this comment

zoewangg commented Mar 24, 2022 •

edited

zoewangg Mar 25, 2022 •

edited

zoewangg commented Mar 25, 2022 •

edited

zoewangg Mar 29, 2022 •

edited

zoewangg Mar 29, 2022 •

edited