master in aws not able to connect to rds🐛[bug]

[humbleearth]

Describe the bug

INFO[2023-09-24T06:39:21Z] Determined master 0.26.0-dev0 (built with go1.21.0)
INFO[2023-09-24T06:39:21Z] connecting to database determined-ai-stack-database-8.cluster-clmqsrouqcuk.us-gov-west-1.rds.amazonaws.com:5432
WARN[2023-09-24T06:39:25Z] failed to connect to postgres, trying again in 4s error="failed to connect to host=determined-ai-stack-database-gmauvb4ghjrp.cluster-8.us-gov-west-1.rds.amazonaws.com user=postgres database=determined: hostname resolving error (lookup determined-ai-stack-database-8.cluster-clmqsrouqcuk.us-gov-west-1.rds.amazonaws.com on 127.0.0.11:53: no such host)"
WARN[2023-09-24T06:39:29Z] failed to connect to postgres, trying again in 4s error="failed to connect to host=determined-ai-stack-database-8.cluster-clmqsrouqcuk.us-gov-west-1.rds.amazonaws.com user=postgres database=determined: hostname resolving error (lookup determined-ai-stack-database-8.cluster-clmqsrouqcuk.us-gov-west-1.rds.amazonaws.com on 127.0.0.11:53: no such host)"
WARN[2023-09-24T06:39:33Z] failed to connect to postgres, trying again in 4s error="failed to connect to host=determined-ai-stack-database-8.cluster-clmqsrouqcuk.us-gov-west-1.rds.amazonaws.com user=postgres database=determined: hostname resolving error (lookup determined-ai-stack-database-8.cluster-clmqsrouqcuk.us-gov-west-1.rds.amazonaws.com on 127.0.0.11:53: no such host)"
WARN[2023-09-24T06:39:37Z] failed to connect to postgres, trying again in 4s error="failed to connect to host=determined-ai-stack-database-8.cluster-clmqsrouqcuk.us-gov-west-1.rds.amazonaws.com user=postgres database=determined: hostname resolving error (lookup determined-ai-stack-database-8.cluster-clmqsrouqcuk.us-gov-west-1.rds.amazonaws.com on 127.0.0.11:53: no such host)"

Reproduction Steps

1. deploy determined
2. check docker logs for agent
3. not able to lookup error

Expected Behavior

should be able to connect to database

Screenshot

na

Environment

• Device or hardware: [aws]
• OS: [linux]
• Browser [chrome]
• Version [25]

Additional Context

No response

[ioga]

in another issue of yours, master is on govcloud and has successfully connected to RDS. what changed?

[humbleearth]

It's still happening. I am manually doing nslookup and other means to connect and restart the docker container many times until it connects back.

[ioga]

first of all, I am sorry you're seeing all these issues. we do not have any testing for govcloud today, and we primarily rely on the community for support.

I don't understand what's different about DNS configuration in govcloud which would make it break intermittently like this.