Harbor core error connexion reset by peer

[igorrenquin]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:

I'm experiencing an issue with my self deployed harbor's instance host on Azure AKS.
I'm unable to connect to Harbor (API or UI)

• OIDC connection button disappear from Harbor login page
• can't push or pull images on the registry

When it happened I notice in the logs the error reset by peer.
It looks like the harbor-core service failed to connect on external PostgreSQL instance.

here under the logs I can collect:

[/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /service/ request in transaction failed: write tcp [Harbor_pod_IP]:49134-\u003e[PostgreSQL_IP]:5432: write: connection reset by peer"}]}
--
stdout F 2021-08-10T01:47:53Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /chartrepo/ request in transaction failed: write tcp [Harbor_pod_IP]:60688-\u003e[PostgreSQL_IP]:5432: write: connection reset by peer"}]}
stdout F 2021-08-10T01:47:28Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /v2/ request in transaction failed: write tcp [Harbor_pod_IP]:49136-\u003e[PostgreSQL_IP]:5432: write: connection reset by peer"}]}
stdout F 2021-08-10T01:47:21Z [WARNING] [/common/utils/oidc/helper.go:72]: Failed to refresh configuration, error: failed to load OIDC setting: write tcp [Harbor_pod_IP]:56508->[PostgreSQL_IP]:5432: write: connection reset by peer
stdout F 2021-08-10T01:47:12Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /c/ request in transaction failed: write tcp [Harbor_pod_IP]:60696-\u003e[PostgreSQL_IP]:5432: write: connection reset by peer"}]}
stdout F 2021-08-10T01:46:57Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"UNKNOWN","message":"unknown: deal with /service/token request in transaction failed: write tcp [Harbor_pod_IP]:60702-\u003e[PostgreSQL_IP]:5432: write: connection reset by peer"}]}

PostgreSQL server resources used:

• I count 60 opened connexion when the instance can handle 145 users connexion. 100 connexion configured in chart.yaml
• CPU load is under 5%
• RAM load is under 50%
• IO load is under 10%

The quick fix is to restart Harbor is to restart harbor-core pod

Steps to reproduce the problem:

So far I did not find a way to reproduce the issue. It can happen any time. 5min, 1d or 10d after restart.

I can't match the root causes to:

• any component performance issue (postgresql, BLOB, POD restarts,etc..)
• any workload (pull/push on harbor instance)

Versions:

• harbor version: v2.2.2-56d7937f
• docker engine version: ContainerD v1.4.4+azure
• Kerbernetes version: AKS 1.19.7
• Azure PostrgreSQL version: 9.6 (General Purpose, 2 vCore(s))
• Azure storage : container (BLOB) Standard

Additional context:

chart.yaml

    [prod]
    httpport = 8080
  PORT: "8080"
  DATABASE_TYPE: "postgresql"
  POSTGRESQL_HOST: "[server url]"
  POSTGRESQL_PORT: "[sqlserverport]"
  POSTGRESQL_USERNAME: "[serverlogin]"
  POSTGRESQL_DATABASE: "registry"
  POSTGRESQL_SSLMODE: "require"
  POSTGRESQL_MAX_IDLE_CONNS: "100"
  POSTGRESQL_MAX_OPEN_CONNS: "1000"

[heww]

@igorrenquin could you try to upgrade to harbor 2.3.2? See #14856 for more info

[igorrenquin]

@igorrenquin could you try to upgrade to harbor 2.3.2? See #14856 for more info

It looks like harbor 2.3.2 solve the issue. I can see in my dashboard harbor release SQL connexion.

[kavyavvv]

while testing with this load for harbor 2.10.0
getting a lot many connection reset by peer messages
VUS: getEnvInt('HARBOR_VUS', '20'),
      ProjectsCount: 1,
      RepositoriesCountPerProject: 60,
      ArtifactsCountPerRepository: 1,
      ArtifactTagsCountPerArtifact: 1,
      UsersCount: 1,
      ProjectMembersCountPerProject: 1,
      AuditLogsCount: 5000,
      BlobSize: '5 MiB',
      BlobsCountPerArtifact: 20

many connection reset by peer is coming failed to push image due to connection reset by peer