CPU burn / slow >= v0.14.0 #3881
Comments
|
Looked into the issue and it seems to be caused by a long-running background check which was using a lot of CPU. We have started working on a fix for this issue: #3884 |
|
@guyguy333 The latest Boundary 0.14.2 release has the the fix to address this issue. Let me know if that helps address the issue. |
|
Thanks a lot @elimt, I can confirm it solved CPU burning issue. However, I still have huge slowness in production mode with latency-ms ~900ms for most requests.
|
|
|
Hey @guyguy333 -- are you still encountering this issue? |
|
Hi @AdamBouhmad, yes I still have the issue. App is slow and request latency is about 900ms, resulting is really slow UI. To answer @elimt and provide more details, I run container on a K8S cluster (hashicorp/boundary:0.14.3). There is no CPU or memory limit for this container. My config: disable_mlock = true
log_format = "json"
controller {
name = "kubernetes-controller"
description = "Boundary Controller"
public_cluster_addr = "boundary-cluster.example.com:443"
database {
url = "env://BOUNDARY_POSTGRES_URL"
max_open_connections = 10
max_idle_connections = 10
}
}
# Ingress TCP Route
worker {
name = "kubernetes-worker"
description = "Boundary Worker"
address = "localhost"
initial_upstreams = ["boundary:9201"]
public_addr = "boundary-worker.example.com:443"
}
# Ingress
listener "tcp" {
address = "0.0.0.0"
purpose = "api"
tls_disable = true
cors_enabled = true
cors_allowed_origins = ["https://boundary.example.com"]
}
listener "tcp" {
address = "0.0.0.0"
purpose = "cluster"
tls_cert_file = "/certs/tls.crt"
tls_key_file = "/certs/tls.key"
}
listener "tcp" {
address = "0.0.0.0"
purpose = "proxy"
tls_cert_file = "/certs/tls.crt"
tls_key_file = "/certs/tls.key"
}
kms "transit" {
purpose = "root"
address = "https://vault.example.com"
// Key configuration
key_name = "boundary-root"
mount_path = "transit/"
}
kms "transit" {
purpose = "recovery"
address = "https://vault.example.com"
// Key configuration
key_name = "boundary-recovery"
mount_path = "transit/"
}
kms "transit" {
purpose = "worker-auth"
address = "https://vault.example.com"
// Key configuration
key_name = "boundary-worker-auth"
mount_path = "transit/"
}Certs are mounted from a K8S secret in /certs. Host machine uses AMD64 arch and has 4 cores and 32Gb RAM Currently, there is no load, only me testing solution to have something stable before opening it. Thanks for considering the issue :) |
Describe the bug
Running Hashicorp Boundary in a container (at least) burns CPU and each API request from UI is slow (~1s).
To Reproduce
Steps to reproduce the behavior in dev mode:
docker run --net=host -v /var/run/docker.sock:/var/run/docker.sock --rm hashicorp/boundary:latest devSteps to reproduce the behavior in dev mode with external Postgres 15 (another machine):
docker run --rm hashicorp/boundary:latest dev -database-url=XXXXXSteps to reproduce the behavior in production mode with external Postgres 15 (another machine):
However, in production mode, I found that disabling worker, stops CPU burning but API requests are still really slow from UI.
If I run in dev mode on my laptop (Apple M1 PRO) without Docker, I don't have CPU burn issue and everything is fast. If I run on VMs without container, I also have the issue (Linux Ubuntu 22.04 LTS related issue ?)
It has been tested on two different cloud providers for VMs, both running Ubuntu 22.04 LTS
I don't have the CPU burn issue with v0.13.1 but I've the issue with v0.14.0 and v0.14.1. However, I've really slow API requests and so slow UI with v0.13.1.
Expected behavior
Hashicorp boundary is responsive and no longer burn CPU.
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: