Amplify Storage put fails to upload after an hour with session token expiry

[vinu-ablabs]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React

Amplify APIs

Storage

Amplify Version

v5

Amplify Categories

storage

Backend

None

Environment information

# Put output below this line
 System:
    OS: macOS 14.1.2
    CPU: (8) arm64 Apple M1 Pro
    Memory: 867.25 MB / 16.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 20.11.1 - /usr/local/bin/node
    Yarn: 1.22.10 - /usr/local/bin/yarn
    npm: 10.2.4 - /usr/local/bin/npm
  Browsers:
    Chrome: 124.0.6367.119
    Safari: 17.1.2
  npmPackages:
    @abrightlab/client-rbac: ^1.2.1 => 1.2.1 
    @aws-amplify/auth: ^5.4.0 => 5.6.9 
    @babel/plugin-proposal-private-property-in-object: ^7.21.11 => 7.21.11 (7.21.0-placeholder-for-preset-env.2)
    @emotion/react: ^11.11.3 => 11.11.3 
    @emotion/styled: ^11.11.0 => 11.11.0 
    @material-ui/core: ^4.12.4 => 4.12.4 
    @material-ui/lab: ^4.0.0-alpha.61 => 4.0.0-alpha.61 
    @mui/icons-material: ^5.15.3 => 5.15.7 
    @mui/material: ^5.15.3 => 5.15.7 
    @mui/x-date-pickers: ^6.18.6 => 6.19.2 
    @reduxjs/toolkit: ^2.2.2 => 2.2.3 
    @reduxjs/toolkit-query:  1.0.0 
    @reduxjs/toolkit-query-react:  1.0.0 
    @reduxjs/toolkit-react:  1.0.0 
    @svgr/webpack: ^8.1.0 => 8.1.0 (5.5.0)
    @testing-library/jest-dom: ^6.1.4 => 6.4.1 
    @testing-library/react: ^14.0.0 => 14.2.0 
    @testing-library/user-event: ^14.5.1 => 14.5.2 
    @types/google-map-react: ^2.1.9 => 2.1.10 
    @types/jest: ^29.5.6 => 29.5.11 
    @types/lodash: ^4.14.200 => 4.14.202 
    @types/node: ^20.2.5 => 20.11.14 
    @types/react: ^18.2.8 => 18.2.49 
    @types/react-datepicker: ^4.19.1 => 4.19.5 
    @types/react-dom: ^18.2.4 => 18.2.18 
    @types/uuid: ^9.0.6 => 9.0.8 
    @typescript-eslint/eslint-plugin: ^6.8.0 => 6.20.0 (5.62.0)
    @typescript-eslint/parser: ^6.8.0 => 6.20.0 (5.62.0)
    aws-amplify: ^5.2.5 => 5.3.15 
    dayjs: ^1.11.10 => 1.11.10 
    eslint: ^8.52.0 => 8.56.0 
    eslint-config-prettier: ^9.0.0 => 9.1.0 
    eslint-import-resolver-typescript: ^3.6.1 => 3.6.1 
    eslint-plugin-header: ^3.1.1 => 3.1.1 
    eslint-plugin-import: ^2.28.1 => 2.29.1 
    eslint-plugin-jest: ^27.4.3 => 27.6.3 (25.7.0)
    eslint-plugin-prettier: ^5.0.1 => 5.1.3 
    eslint-plugin-promise: ^6.1.1 => 6.1.1 
    eslint-plugin-sonarjs: ^0.21.0 => 0.21.0 
    eslint-plugin-unicorn: ^48.0.1 => 48.0.1 
    example:  0.0.0 
    google-map-react: ^2.2.1 => 2.2.1 
    i18next: ^23.7.8 => 23.8.2 
    lodash: ^4.17.21 => 4.17.21 
    material-react-table: ^2.3.1 => 2.10.0 
    moment: ^2.29.4 => 2.30.1 
    prettier: ^3.0.3 => 3.2.4 
    react: ^18.2.0 => 18.2.0 
    react-datepicker: ^4.15.0 => 4.25.0 
    react-dom: ^18.2.0 => 18.2.0 
    react-geocode: ^1.0.0-alpha.1 => 1.0.0-alpha.1 
    react-google-places-autocomplete: ^4.0.1 => 4.0.1 
    react-hook-form: ^7.45.0-next.1 => 7.49.3 
    react-i18next: ^13.5.0 => 13.5.0 
    react-material-ui-carousel: ^3.4.2 => 3.4.2 
    react-otp-input: ^3.0.2 => 3.1.1 
    react-places-autocomplete: ^7.3.0 => 7.3.0 
    react-redux: ^9.1.0 => 9.1.2 
    react-router-dom: ^6.12.1 => 6.21.3 
    react-scripts: ^5.0.1 => 5.0.1 
    recharts: ^2.7.1 => 2.11.0 
    typescript: ^5.2.2 => 5.3.3 
    uuid: ^9.0.0 => 9.0.1 (3.4.0, 8.3.2)
    web-vitals: ^3.3.2 => 3.5.2 
  npmGlobalPackages:
    @aws-amplify/cli: 10.4.1
    @nestjs/cli: 9.0.0
    aws-appsync-codegen: 0.17.5
    aws-appsync: 4.1.9
    aws-cdk: 2.135.0
    aws-cli: 0.0.2
    eas-cli: 7.3.0
    expo-cli: 6.0.8
    n: 9.2.0
    ngrok: 5.0.0-beta.2
    npm-check-updates: 16.10.12
    ts-node: 10.9.1
    webpack: 5.83.1
    wscat: 5.2.0

Describe the bug

We use amplify with Cognito and have some user uploaded files pushed to S3. Both both reactJs webapp and react native mobile app do similar functions.

If the user leaves the screen unattended for more than an hour, further invocation of Strorage.Put keeps failing, saying the session token has expired. It resumes working only after refreshing the page. We checked the amplify-user in IAM console and it is configured to have access keys

It looks like Amplify signs the S3 URL and does not refresh the token even after the session expires. Ideally, since it is a key based signature, the token is an optional parameter. All further attempts give the same error and the session error. Initially we thought it to be a cognito token expiry but turns out that is not relevant here

Expected behavior

Ideally, the token should refresh if it fails and a new signed url should be created. If not we need to do this at our backend servers and push the new signed url to front end. Trying to avoid this.

Reproduction steps

Use amplify Storage in any reactJs app
leave the app idle for an hour
Try uploading a file

Code Snippet

// Put your code below this line.
let body;
  let storedDocument;
  let modifiedConfig = config;

  try {
    if (fileExtension && fileExtensionExtraction(fileExtension) !== '') {
      modifiedConfig = { ...config, contentType: fileExtensionExtraction(fileExtension) };
    }
    body = await requestBlob(url);
    storedDocument = await Storage.put(s3BucketPath, body, modifiedConfig);
    return { file: storedDocument, error: {} };
  } catch (error: any) {
    throw new Error(error);
  }

Log output

Request URL:
https://XXXX-qa.s3.us-east-1.amazonaws.com/public/XXXX/a00aa016-2636-4737-b2a4-XXXXX/bid-attachments/7e622eb5-9851-4e12-be93-XXXX/XXXX/10373b9f-e4d3-4fb8-a681-XXXX.png
Request Method:
PUT
Status Code:
400 Bad Request
Remote Address:
XXXX:443
Referrer Policy:
strict-origin-when-cross-origin
Access

aws-exports.js

/* eslint-disable */
// WARNING: DO NOT EDIT. This file is automatically generated by AWS Amplify. It will be overwritten.

const config = {
  aws_project_region: 'us-east-1',
  aws_cognito_region: 'us-east-1',
  aws_cognito_identity_pool_id: 'us-east-1:XXXXXX',
  aws_user_pools_id: 'us-east-1_XXXXX',
  aws_user_pools_web_client_id: 'XXXXXXXX',
  oauth: {
    domain: 'abrightlab.auth.us-east-1.amazoncognito.com',
    scope: ['aws.cognito.signin.user.admin', 'email', 'openid', 'phone', 'profile'],
    redirectSignIn:
      'XXXXX',
    redirectSignOut:
      'XXXX',
    responseType: 'code',
  },
  federationTarget: 'COGNITO_USER_POOLS',
  aws_cognito_username_attributes: ['EMAIL'],
  aws_cognito_social_providers: ['FACEBOOK', 'GOOGLE', 'APPLE'],
  aws_cognito_signup_attributes: ['NAME'],
  aws_cognito_mfa_configuration: 'OPTIONAL',
  aws_cognito_mfa_types: ['SMS', 'TOTP'],
  aws_cognito_password_protection_settings: {
    passwordPolicyMinLength: 6,
    passwordPolicyCharacters: [],
  },
  aws_cognito_verification_mechanisms: ['EMAIL', 'PHONE_NUMBER'],
  aws_appsync_region: 'us-east-1',
  aws_appsync_authenticationType: 'AMAZON_COGNITO_USER_POOLS',
  aws_appsync_graphqlEndpoint: 'https://XXXXXX.appsync-api.us-east-1.amazonaws.com/graphql',
  aws_appsync_apiKey: 'da2-XXXXXX',
  aws_user_files_s3_bucket: 'arkstoragedev132731-qa',
  aws_user_files_s3_bucket_region: 'us-east-1',
  aws_cloud_logic_custom: [
    {
      name: 'getIdsApi',
      endpoint: 'https://XXXXXX/analytics/customerDashboard/submitQuery',
      region: 'us-east-1',
    },
    {
      name: 'getFacilities',
      endpoint: `https://XXXXXX/analytics/customerDashboard/getQueryResults`,
      region: 'us-east-1',
    },
    {
      name: 'taskSummary',
      endpoint: ` https://XXXXXX/analytics/dashboard/taskSummary`,
      region: 'us-east-1',
    },
    {
      name: 'paginatedTaskSummary',
      endpoint: `https://XXXXXX/analytics/dashboard/taskSummary/paginated`,
      region: 'us-east-1',
    },
    {
      name: 'noShowFacilities',
      endpoint: `https://XXXXXX/analytics/dashboard/noShowFacilities`,
      region: 'us-east-1',
    },
    {
      name: 'oneTimeJobs',
      endpoint: `https://XXXXXX/analytics/dashboard/oneTimeJobs`,
      region: 'us-east-1',
    },
  ],
};

export default config;

Manual configuration

const [localRedirectSignIn] = awsConfig.oauth.redirectSignIn.split(',');
const [localRedirectSignOut] = awsConfig.oauth.redirectSignOut.split(',');

Amplify.configure({
  ...awsConfig,
  oauth: {
    ...awsConfig.oauth,
    redirectSignIn: isLocalhost ? localRedirectSignIn : '',
    redirectSignOut: isLocalhost ? localRedirectSignOut : '',
  },
});

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

[israx]

hello @vinu-ablabs . Sorry for any inconvenience with the library. Is it possible for you to upgrade to v6, and test this behavior ? You can decrease the expiration time of your access token to 5 minutes for faster testing.

[vinu-ablabs]

The issue is not with cognito access token. Upgrading would need us to migrate some of the code. This was a production. What is expiring is the session token from sts Thanks and Regards, Vineeth Vijayan +91 9902921500 *Head of Engineering*

…

On Wed, 8 May 2024 at 21:58, israx ***@***.***> wrote: hello @vinu-ablabs <https://github.com/vinu-ablabs> . Sorry for any inconvenience with the library. Is it possible for you to upgrade to v6, and test this behavior ? You can decrease the expiration time of your access token to 5 minutes for faster testing. — Reply to this email directly, view it on GitHub <#13347 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2DSXOCF45CLD3A4QDVWS6TZBJHBFAVCNFSM6AAAAABHMKTABCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBQHE2TMOBTGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[israx]

Thank you for the context. I'll bring this to the team, try to reproduce it and be back with next steps. Thanks for the patience!

[cwomack]

Related to #13307

[israx]

hello @vinu-ablabs . Are you seeing this error after authenticating via a social provider, username and password, guess access or all of them ?

[israx]

hey @vinu-ablabs . Is this issue also happening for both light and heavy size files ?

[vinu-ablabs]

@israx Yes. Even any API calls going through Amplify prior to the upload calls do not revive the session

[vinu-ablabs]

hello @vinu-ablabs . Are you seeing this error after authenticating via a social provider, username and password, guess access or all of them ?

@israx Yes it happens to login via email/password. I believe it could happen to all of them

[israx]

Hello @vinu-ablabs . I'm unable to reliable reproduce this issue. I see the library is able to refresh tokens and credentials on the following scenarios:

1. leave the screen unattended until credentials expire -> call Storage.put
2. upload a heavy file while credentials expire.

In both cases the library is constantly refreshing AWS credentials and authentication tokens.

Can you upgrade to the latest version of Amplify v5 (5.3.18) and check if the issue still persist ?

[vinu-ablabs]

HI @israx Tried with 5.3.18 but still no luck

Getting the below error still

Response

Request URL:
https://xxxx-qa.s3.us-east-1.amazonaws.com/public/vendor/a00aa016-2636-4737-b2a4-xxxx/bid-attachments/19fff378-1f09-4674-a0fc-xxxx/version0/d211fbe1-772b-4a42-8a3b-xxxx.png
Request Method:
PUT
Status Code:
400 Bad Request
Remote Address:
52.216.50.106:443
Referrer Policy:
strict-origin-when-cross-origin

Request Headers

PUT /public/vendor/xxxxx-2636-4737-b2a4-xxxx/bid-attachments/19fff378-1f09-4674-a0fc-xxxx/version0/d211fbe1-772b-4a42-8a3b-xxxxx.png HTTP/1.1
Accept: /
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Connection: keep-alive
Content-Length: 965205
Host: arkstoragedev132731-qa.s3.us-east-1.amazonaws.com
Origin: http://localhost:3000
Referer: http://localhost:3000/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
authorization: AWS4-HMAC-SHA256 Credential=ASIA2RSZYKMI6PAZUAMT/20240514/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-user-agent, Signature=7ae8133104becf95eb63eb0f1261fbe2e6f3974ddc46df87ae2b38c362babd34
content-type: image/png
sec-ch-ua: "Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20240514T040227Z
x-amz-security-token: IQoJb3JpZ2luX2VjEA8aCXVzLWVhc3QtMSJIMEYCIQDE6e5XBARuPy9JuSWxP6pJi/Ye7kgJgyhA52rTpNCC6gIhAPSI+HnjbS5WYtzNV634NPL+09gfuYN1TrFGdZRIpOcpKsQECHgQARoMNzI0OTYzOTcxODU3IgwwrJ37r7QpZsXsh6IqoQQ0Y8fWA+joA4dOL+dVHgSllowObt6qIryNMRD76qdeqrHeaAm7tw/E0c5OTTPuWaIu7u6mLwpHaCN77KK0Wra2SBmircrGA8p0GWtX2kB6IIlqoB9qpVgu6td6fjvcOwDYEZ7RV3YvYz/22O5CNWvPuhk4qkLQzMVII0eALkN2jJacSkT8nlJnSirkzxYsZiZM9VOM7CvLO48bC9zMGgbNAq2JG9yb6kH1FqrCUxfOnTda4YHvJgTbTOJrbDLTG62Hd5eNZ4TkSryKmlr85jJCTgg3Sm8gSuntqKs9jjw1v5AYJKwjeldzoJoSF8bM3V8rGnF7d/hAkQX/9Ik5yJijOv5EsCacPGcI08utpA7ZbuxcZET/sl/04XAEXqGJezxF0s+4J5SZ0a0XXCy701+pZVFgjk/nxL3JKAZ2SyerPVBXzXON6Scm2ROxyASotWtXH3Z/4VD//NoNrz9Q25OgTh+cme+ebE7gOKX+agKojceLVIH8AqKvHTY4MmHLxzqpKOzPFfiSeq5SxqDvGC7X8p7Uyy1CiIDi5QGePjHTzm8aUV4BPBS5lmWFRIgxv/kWwbpBaf9ot8HdaaHuie1YHEn36E9n4qatadDpEZy0cVecozCcGQyPDvbF3r07gYtAS22AGyimL+kqzcXuI4d+jzkty/dFyuXTNofT7iHNn9OYdDZGb+nYxVtWkgXl4Xo3cVXuk8RHMbJvHpC81MAbbzCqzYiyBjqEAkD2Hr2JleOlrkp0+3yts3c9uTcjiLy0E99BlfOqB5OEB4guuP2EuO2AwUFIx6ryTww37nKIZXwYUjfD74K4KhSzRyQf06t3ccy4TsQYpR9ofCrDMt86LTDKXlNIaak4vQPrLFmM50v514U5lVzzohUKchDv82jMJNk6luv3j1/HiJ8cMezjjbVNknSfoE4KVmiUZkjq3WqVfOeWu2v2/Mg8i7/wf8Q2kqjU4VDL8ZD5/cAxJJ7H0GagRshz2wY8nV39YJ00kCJY7pR6ie7h2ioADFrl1nrl3l2D6BpMFuoO9EzMAxweRqmdQ9w4sEor91dG+Oo8MGi8fV4S05dPftZglv0L
x-amz-user-agent: aws-amplify/5.3.18 storage/1 framework/1

Response headers

HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3000
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
x-amz-request-id: 2WNZP2QWZTBJK8SD
x-amz-id-2: C4DDPVSLx3Trgmb0kEkJg41N6O+NnKfwXlWFAkjCiksNg8tOhjskAB3POzziRpXFH3ibxPEzOeA=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Tue, 14 May 2024 04:02:35 GMT
Server: AmazonS3
Connection: close
access-control-allow-headers: *
access-control-expose-headers: *
access-control-allow-credentials: true
access-control-allow-methods: *

[vinu-ablabs]

@israx another issue was looped in on my issue. You think its relavant?

#13307

[israx]

That issue is related to creds being expired while uploading a heavy file, and we currently have a PR to address it. We highly recommend to upgrade to v6 as we are constantly bringing updates.

[vinu-ablabs]

@israx I have a feeling that the same thing is happening here as well. When you do not use the S3 APIs for an hour, the credentials expire, and the signed URL won't work after that. I can see the credentials (Cognito) are refreshing, but I don't think that is being used here. Upgrading to v6 at this juncture would require us to do a through round of testing

As a workaround, I am moving away from the storage plugin and getting a presigned url from the backend to upload the file. I will let you know how that goes

[israx]

Thanks @vinu-ablabs for all the context. We will dig deep into the root cause in v5 and follow up with next steps. Thanks for the patience

[israx]

hello @vinu-ablabs . In the mean time can you try the following setting in your config and see if that ends up working for you?

 {
    AWSS3: {
      bucket: ******,
      region: *******,
      resumable: true
    }
 }

[vinu-ablabs]

@israx That didnt work. Still getting the error

[israx]

hey @vinu-ablabs . Following up on the issue. You are mentioning that after 1 hour even the preSigned URL won't work. This is expected behavior as credentials have a default expiration of 50 minutes. A potential work around is to refetch the preSigned URL on an interval or just manually increasing the expiration date.

What I'm still not able to repro is the failing calls to the put API after credentials have expired. I see that the library will refresh them after expiration

[vinu-ablabs]

@israx When i said presignedUrl, I didnt mean the url you fetched via preSIgnedUrl method. From what I understood/assume is assumed, the sdk signs the put url with a credentials which expires like you mentioned. Once it fails its not really refreshing the url/credentials and further attempts to upload fails. It works once you refresh the whole screen and the library reinitializes.

The repo we have is tied to the company. Will see if we can reproduce the issue on a barebone repo and share it with you. Might take some time to do that

[israx]

hey @vinu-ablabs . The way am testing and not able to reproduce this issue is by doing the following:

1. configure storage
2. setup a react app
3. login with username/password
4. call Storage.put
5. wait until credentials expire
6. call Storage.put. Here I see the library is refreshing credentials and able to upload a file successfully.

Something that might be happening is that your refresh_token might be expired when calling Storage.put . In that case the library will clear tokens. You can check the expiration date of your refresh_token by going to your user pool and select your app client id.

If that is not an issue, then something that can help is to provide a sample app or more specific steps to reliable reproduce the issue.

[israx]

btw this issue is v6 specific and it happens only when creds and tokens are expired while uploading a file. If you do the same on v5 the library will refresh tokens and creds on demand while the file is being uploaded.