Failed to boot firecracker VM, failed to fetchRepository?

[bgoosman]

hocus-local-hocus-agent-1 | 2023-05-07T18:51:16.089Z [INFO] Booting firecracker VM with pid 254 took: 456.09 ms, TOTAL: 505.93 ms
hocus-local-hocus-agent-1 | 2023-05-07T18:51:18.255Z [WARN] firecracker process with pid 254 closed: 0
hocus-local-hocus-agent-1 | 2023-05-07T18:52:11.437Z [WARN] Activity failed {
hocus-local-hocus-agent-1 | error: FetchError: The request failed and the interceptors did not return an alternative response
hocus-local-hocus-agent-1 | at DefaultApi.BaseAPI.fetchApi (/app/node_modules/firecracker-client/dist/runtime.js:100:31)
hocus-local-hocus-agent-1 | at processTicksAndRejections (node:internal/process/task_queues:96:5)
hocus-local-hocus-agent-1 | at DefaultApi.request (/app/node_modules/firecracker-client/dist/runtime.js:136:26)
hocus-local-hocus-agent-1 | ... 5 lines matching cause stack trace ...
hocus-local-hocus-agent-1 | at /app/agent.js:2259:16 {
hocus-local-hocus-agent-1 | cause: TypeError: fetch failed
hocus-local-hocus-agent-1 | at fetch (/app/node_modules/undici/index.js:105:13)
hocus-local-hocus-agent-1 | at processTicksAndRejections (node:internal/process/task_queues:96:5)
hocus-local-hocus-agent-1 | at DefaultApi.BaseAPI.fetchApi (/app/node_modules/firecracker-client/dist/runtime.js:84:28)
hocus-local-hocus-agent-1 | at DefaultApi.request (/app/node_modules/firecracker-client/dist/runtime.js:136:26)
hocus-local-hocus-agent-1 | at DefaultApi.createSyncActionRaw (/app/node_modules/firecracker-client/dist/apis/DefaultApi.js:83:26)
hocus-local-hocus-agent-1 | at DefaultApi.createSyncAction (/app/node_modules/firecracker-client/dist/apis/DefaultApi.js:96:9)
hocus-local-hocus-agent-1 | at FirecrackerService.shutdownVM (/app/agent.js:3817:5)
hocus-local-hocus-agent-1 | at FirecrackerService.withVM (/app/agent.js:3798:9)
hocus-local-hocus-agent-1 | at /app/agent.js:4082:7
hocus-local-hocus-agent-1 | at /app/agent.js:2259:16 {
hocus-local-hocus-agent-1 | cause: [Error]
hocus-local-hocus-agent-1 | }
hocus-local-hocus-agent-1 | },
hocus-local-hocus-agent-1 | durationMs: 55931,
hocus-local-hocus-agent-1 | isLocal: false,
hocus-local-hocus-agent-1 | attempt: 1,
hocus-local-hocus-agent-1 | namespace: 'default',
hocus-local-hocus-agent-1 | taskToken: '...',
hocus-local-hocus-agent-1 | workflowId: '271718a2-c2e1-4e43-8ffc-695c43e5c6d0',
hocus-local-hocus-agent-1 | workflowRunId: '6d4504b2-5b2d-4d8b-89d6-961d0605af71',
hocus-local-hocus-agent-1 | workflowType: 'runBuildfsAndPrebuilds',
hocus-local-hocus-agent-1 | activityId: '2',
hocus-local-hocus-agent-1 | activityType: 'fetchRepository',
hocus-local-hocus-agent-1 | taskQueue: 'main'
hocus-local-hocus-agent-1 | }

[bgoosman]

I'm on bitbucket, and the project page gave me a green checkmark on the git repository, so I'm not sure what to do now. I already added the publish ssh key to my workspace on bitbucket.

[hugodutka]

It looks like the VM that fetchRepository uses was killed before the Hocus agent could shut it down normally. This could have happened because your host was out of memory and the OOM killer did its thing. Are you running it on a system that has less than 8-16GB of free RAM? Hocus is currently very memory hungry and we are working on reducing that.

If memory is the culprit, an easy fix would be creating a swap file on your host. I would suggest 16GB of size. https://linuxize.com/post/create-a-linux-swap-file/

[bgoosman]

Hmm I should have 64 GB RAM total, on an Ubuntu 20.04 Focal OS.

top - 15:38:46 up 20:48,  1 user,  load average: 0.23, 0.55, 0.54
Tasks: 267 total,   1 running, 266 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  64241.8 total,  35695.6 free,    466.1 used,  28080.1 buff/cache
MiB Swap:  32735.0 total,  32735.0 free,      0.0 used.  63065.2 avail Mem

I forgot I got this warning which I didn't know what to do with. (Sorry, not a linux pro 😅)

> HOCUS_HOSTNAME="x" ops/bin/local-up.sh
HOCUS_HOSTNAME="x" ops/bin/local-up.sh
[WARNING] Host kernel *might* be too old. If you encounter issues with nested virtualization please first try running Hocus on at least the 5.10 kernel
Building docker images 👷📦
Building vm-builder done in 1.32 s ✅
Building db-autosetup done in 0.52 s ✅
Building keycloak done in 0.80 s ✅
Building temporal-codec done in 1.34 s ✅
Building ui done in 0.82 s ✅
Building agent done in 0.88 s ✅
Pulling docker images 📥 - ✅ in 1.26 s
Building MicroVMs 👷🖥️ - ✅ in 4.21 s
Seeding the DB 🌱 - ✅ in 0.77 s
Starting the DB 📙 - ✅ in 2.00 s
Starting Keycloak 🔑 - ✅ in 11.01 s
Starting Temporal ☁️  - ✅ in 12.52 s
Starting Hocus 🧙🪄  - ✅ in 3.98 s

You may access Hocus here: http://x:3000/ Creds: dev/dev
Keycloak: http://x:4200/ Creds: admin/admin
Temporal: http://x:8080/

To delete all data ./ops/bin/local-cleanup.sh
To get debug logs: ./ops/bin/local-cmd.sh logs
To stop the deploy: ./ops/bin/local-cmd.sh down

Hmm I ran as root, but some of the processes are showing up as a sub account I created earlier.

top - 15:42:13 up 20:51,  1 user,  load average: 0.22, 0.46, 0.51
Tasks: 345 total,   2 running, 341 sleeping,   0 stopped,   2 zombie
%Cpu(s):  1.7 us,  0.6 sy,  0.0 ni, 97.2 id,  0.5 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :  64241.8 total,  34705.6 free,   1400.9 used,  28135.2 buff/cache
MiB Swap:  32735.0 total,  32735.0 free,      0.0 used.  62099.2 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
3153183 ben       20   0  843896 119528  44260 S   8.0   0.2   0:04.17 temporal-server
   6448 root      20   0 2171064  54520  34280 S   1.7   0.1  15:32.09 containerd
   6569 root      20   0 2743272 139344  61024 S   1.7   0.2  18:06.06 dockerd
3153479 70        20   0  174716  18744  15452 S   1.0   0.0   0:00.13 postgres
3154508 root      20   0  104.7g 374672  59088 S   1.0   0.6   0:12.05 node
3154130 root      20   0 1596484  95224  42048 S   0.7   0.1   0:01.48 node
      1 root      20   0  169012  12596   8512 S   0.3   0.0   4:01.05 systemd
    548 message+  20   0    7404   4520   3868 S   0.3   0.0   0:56.35 dbus-daemon
3138296 root      20   0   19268   9724   8032 S   0.3   0.0   0:00.97 systemd
3147404 root      20   0       0      0      0 I   0.3   0.0   0:00.03 kworker/10:4-events
3152434 root      20   0  720752   9720   6812 S   0.3   0.0   0:00.40 containerd-shim
3152662 ben       20   0 5547528 320828  29992 S   0.3   0.5   0:11.37 java
3153457 70        20   0  174420  23824  20852 S   0.3   0.0   0:00.16 postgres
3153480 70        20   0  173872  15716  12900 S   0.3   0.0   0:00.01 postgres
3154111 root      20   0  720752  10388   7388 S   0.3   0.0   0:00.35 containerd-shim
3170811 70        20   0  175236  19208  15352 S   0.3   0.0   0:00.04 postgres

Here are my prebuild limits. I just raised prebuild to 4096*4.

On the brightside the git integration seems to be working, since a push to dev triggered a prebuild

Raising the prebuild limit did not help 😞

[gorbak25]

@bgoosman

• What filesystem are you running on, how much free storage is available on the system?
• Are you running plain docker, docker in user namespaces or podman ?
• How large is your bitbucket repository?
• I've checked on my setup that Hocus works well with Bitbucket, the issue must lie somewhere in the setup.
• Do you have an hocus.yml file in the repository?