couldn't start a new transaction: could not create new transaction write: broken pipe

[mstrYoda]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• This issue affects my Ory Cloud project.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Describe the bug

We are running 10 pods of keto instances on kubernetes.

When we send create request to Keto after a while Keto prints the following logs and our write requests fail. In the meantime when I execute keto check/expand commands inside that pod it works.

error=couldn't start a new transaction: could not create new transaction: write failed: write tcp ...:55274->...:5432: write: broken pipe

Reproducing the bug

helm install ory/keto

Use acl WriteService

Relevant log output

error=couldn't start a new transaction: could not create new transaction: write failed: write tcp 10.233.68.125:55274->10.84.90.178:5432: write: broken pipe grpc.code=Unknown grpc.method=TransactRelationTuples grpc.service=ory.keto.acl.v1alpha1.WriteService

Relevant configuration

No response

Version

0.8.0-alpha.0

On which operating system are you observing this issue?

No response

In which environment are you deploying?

Kubernetes with Helm

Additional Context

No response

[zepatrik]

I think this is related to your database closing connections. You can configure various timeouts and other connection options according to https://www.ory.sh/docs/ecosystem/deployment#postgresql

[mstrYoda]

I think this is related to your database closing connections. You can configure various timeouts and other connection options according to https://www.ory.sh/docs/ecosystem/deployment#postgresql

Hi @zepatrik, thank you for your response.

Even though db closes the one session, should not that keto create new session/connection?

[zepatrik]

Even though db closes the one session, should not that keto create new session/connection?

No, we don't do that. You should instead set the connection parameters in a way that the connection works as you need it. Or is there any other recommendation @aeneasr?

[aeneasr]

Exactly, this is heavily dependent on your environment and database!

[sagkhr23]

time=2023-03-21T20:49:20Z level=error msg=failed to look up direct access in db audience=application error=map[message:write failed: write tcp {IP}:49656->{IP}:5432: write: broken pipe] method=checkDirect service_name=Ory Keto service_version=master
time=2023-03-21T20:49:20Z level=error msg=An error occurred while handling a request audience=application error=map[message:unable to fetch records: write failed: write tcp 10.23.25.105:46334->172.20.213.42:5432: write: broken pipe]

I am also facing the same issue with ory sometimes when I am calling the check api I am getting the same error, is it related to my max connection timout as it is getting timed out or do I have to retry with db if I face any error once and add the retry logic to the code.
I am using Postgres with default configurations, haven't added any new parameters in the configuration.
Can you please suggest some configuration change as what might be causing this and what needs to be done?