Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CLI] Workaround to set a custom heap snapshot dir #170756

Merged
merged 3 commits into from
Nov 8, 2023

Conversation

dokmic
Copy link
Contributor

@dokmic dokmic commented Nov 7, 2023

Summary

When Kibana is running inside a Docker container, the root filesystem is read-only. If we try to use --heapsnapshot-signal=SIGUSR2 in the NODE_OPTIONS and then send this signal to the Kibana process, the process crashes with an error complaining about the read-only filesystem.

The --diagnostic-dir parameter is supposed to set the destination folder for the heap snapshots, but there is a bug in Node <20 (nodejs/node#39493).

Since Kibana is running under non-root user, there is no way to work this around with mount --bind. And apparently, the Node 20 upgrade is not going to happen shortly (#162696). Due to those facts, the easiest option would be to use the writeHeapSnapshot() call.

The proposed workaround takes place only if both options --heapsnapshot-signal and --diagnostic-dir are present at the same time, and can be safely removed right in #162696.

Checklist

@dokmic dokmic added Feature:Console Dev Tools Console Feature Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Team:Operations Team label for Operations Team release_note:skip Skip the PR/issue when compiling release notes v8.12.0 labels Nov 7, 2023
@dokmic dokmic marked this pull request as ready for review November 7, 2023 17:10
@dokmic dokmic requested a review from a team as a code owner November 7, 2023 17:10
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@jbudz
Copy link
Member

jbudz commented Nov 7, 2023

Since Kibana is running under non-root user, there is no way to work this around with mount --bind. And apparently, the Node 20 upgrade is not going to happen shortly (#162696). Due to those facts, the easiest option would be to use the writeHeapSnapshot() call.

@watson are you planning on picking up the Node 20 PR in the near future? It looks mostly ready?

@watson
Copy link
Contributor

watson commented Nov 8, 2023

@watson are you planning on picking up the Node 20 PR in the near future? It looks mostly ready?

I haven't had time due to serverless work. If someone wants to take it over I'll happily do hand over. Or I can probably find time this month.

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@dokmic dokmic merged commit a173300 into elastic:main Nov 8, 2023
29 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Nov 8, 2023
@dokmic dokmic deleted the feature/heap-snapshot-dir branch November 8, 2023 14:05
@rudolf rudolf removed the Feature:Console Dev Tools Console Feature label Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Team:Operations Team label for Operations Team v8.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants