Retrieving oAuth token fails every time on first call

[yarivlifchuk]

Description
Kyma version 1.11.1

When calling the following curl to retrieve token there is a failure response and only in the 2nd call it succeed with new token.

curl -X POST -u {ClientId}:{ClientSecret} -d "grant_type=client_credentials&scope=application:read" https://oauth2.{domain}/token -H "Accept: application/json" -H "Accept-Language: en_US"

1st failed response
error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authenticatio n method)","status_code":401}

2nd response
300{"access_token":"hbhwWJDunwBFlEAWJjHiqK3CvrICJ8dt7g-VMGI7TmI.yHg41WlnOOPzzm64cmscope":"application:read","token_type":"bearer"}

Expected result

Get success response on first shot.
300{"access_token":"hbhwWJDunwBFlEAWJjHiqK3CvrICJ8dt7g-VMGI7TmI.yHg41WlnOOPzzm64cmscope":"application:read","token_type":"bearer"}

Actual result
a failure response in the 1st attempt and only in the 2nd call it succeed with new token.

1st failed response
error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authenticatio n method)","status_code":401}

2nd response
300{"access_token":"hbhwWJDunwBFlEAWJjHiqK3CvrICJ8dt7g-VMGI7TmI.yHg41WlnOOPzzm64cmscope":"application:read","token_type":"bearer"}

Steps to reproduce
Call the following URL (replace clientId,clientSecret and Kyma domain)

curl -X POST -u {ClientId}:{ClientSecret} -d "grant_type=client_credentials&scope=application:read" https://oauth2.{domain}/token -H "Accept: application/json" -H "Accept-Language: en_US"

Troubleshooting
Calling it several times until succeed response

[piotrmsc]

@jakkab please provide more details here what happened, why we have updated version etc and close it ;)

[jakkab]

Please refer to ory/hydra#1599. It looks like switching the db driver from lib/pq to jackc/pgx in Hydra solves the problem. Moreover, we might want to set max_conn_lifetime=10s in the DSN should the issue persist (see https://community.ory.sh/t/hydra-broken-pipe/1691).

For now, Hydra has been bumped to v1.4.6 (#9016). This version features updated drivers.
We should also synchronise our ORY chart with ory/k8s repo as described here: #9020

[colunira]

The issue persists, we need to investigate further.

[jakkab]

@yarivlifchuk #9317 should stabilize the connection. Feel free to reopen in case the issue persists.

[yarivlifchuk]

Thanks

[tgorgol]

This issue started appearing again, some example logs from this week:
LOG 1
LOG 2
LOG 3
LOG 4

[Tomasz-Smelcerz-SAP]

Happened again: kyma-integration-gardener-aws

[Tomasz-Smelcerz-SAP]

See also: #10965

[kyma-stale-bot]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[kyma-stale-bot]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[pbochynski]

The test is replaced by fast-integration tests.