[PRODUCTION] vault ui just stopped working, no config changes and no new binary... just won't start ui

[pracplayopen]

Describe the bug

Been using vault1.9.1 for about a year.
Node running vault was restarted w/same os image. (alpine linux)
Vault restarts w/same script/config ok, but UI won't access.

To Reproduce
Steps to reproduce the behavior:

1. start vault: server -config config.vault.hcl
2. go to admin to unseal server: :8200/ui
3. get error 'Vault UI is not available in this binary'

Expected behavior
Normally.... w/this same image and vault binary, should be able to unseal and start serving requests.

Environment:
./vault --version
Vault v1.9.1

• Server Operating System/Architecture:
• uname -r -v
  5.10.11-0-rpi Initial Website Import #1-Alpine SMP PREEMPT Wed Jan 27 19:13:11 UTC 2021

Vault server configuration file(s):

storage "raft" {
  path    = "/mnt/sda4/vault/data"
  node_id = "node1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "false"
  tls_cert_file = "/mnt/sda4/vault/keys/cert.pem"
  tls_key_file = "/mnt/sda4/vault/keys/key.pem"
  tls_min_version = "tls12"
}

api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
ui = true

Additional context

a little bit freaked that hashicorp can just turn off the UI from a static binary/image.
not what you would expect from a secure installation.
certificate being used on this server isn't public/valid, but it hasn't expired and it's same cert used for over a year.

didn't find anyone receiving this error on github issues nor hasicorp valut discussion forum.
only place i could find it was on homebrew package site, which is obviously not case here.
again, just using github binary from hashicorp for above.

if we can't expect versions to run unmodified and unconfied for more than X time, might be good to state that and save people some time.

anyways not expecting much but if i'm I'm missing something please lmk.
we had CISSP configure this thing, really hoping don't need to do that again just to keep it running.
best

[hsimon-hashicorp]

Vault 1.9.1 has only been out since December 2021, so I don't think you've been using it for a year. :) What method did you use to create this install? Docker image, helm chart, etc?

[pracplayopen]

thx for follow up.

not trying to hyperbolize... based on the docs I tried to read to figure this out, we first installed it a few days into 2021.

current keys and datastore are a year old:

~/vault/keys $ ls -lh
total 8K     
ZZZZZZ    WW XXXX YYYYY 1.2K Feb 10  2021 cert.pem
ZZZZZZ    WW XXXX YYYYY 1.6K Feb 10  2021 key.pem
~/vault/data$ ls -lhd raft
total 128K   
ZZZZZZ    WW XXXX YYYYY         4.0K Feb 11  2021 raft

to your point, appears we installed 1.9.1 on dec 23.
prior to that we were using vault 1.6.2 (installed feb 21 '21)

~/vault$ ls -lh vault*
ZZZZZZ    WW XXXX YYYYY         161.6M Dec 23 09:29 vault
ZZZZZZ    WW XXXX YYYYY         168.6M Feb 21  2021 vault-1.6.2

whatever version we used on feb 10th or before not sure.
just know how to restart it from docs, fix basic problems and configure via ui.

appears both were downloaded from github downloads using wget.
based on security log, it appears the first version might've been a package from alpine but because it was older they went w/newer binaries mentioned above.

[hsimon-hashicorp]

Which path did you download the 1.9.1 release from? That can make a difference with expected behavior.

[pracplayopen]

not a public cloud service, it's downloaded from a local package server.
that's why it's really confusing it suddenly not serving the ui.

here are signatures of version running:

~/vault$ md5sum vault && sha256sum vault && sha512sum vault
233f2a092807800c61bfa81699e9ad56  vault
ba737c536d0dfa152b8b1d571e3e14b389e1cf819f197dc69a9ee0bd08e5acea  vault
d8988b57e6c74a966fab992f9d444b3569546239ea2b4baaa9ea5cb9af0ba6b6afcf9c52f500eacd130e4a66b6f84384e2cc5c78456ca8b9b38664d39b9c1504  vault

based on notes it came from arch linux repository, which i think downloaded it from hashicorp vault release 1.9.1... however (what seems to be) original link is serving 404.

[hsimon-hashicorp]

Can you please try downloading 1.9.1 from our releases page? https://www.vaultproject.io/downloads
Thanks!

[pracplayopen]

would be happy to but it's running on aarch64, don't see that there.

this is where the original package appeared to be getting 1.9.1 from (i think), but don't see aarch64 there now:

https://releases.hashicorp.com/vault/1.9.1/

i supposed if it's not supported i can look at installing a newer version from a repo, or getting it a cd pipeline.
that could be quite a bit of work though so will have to be evaluated against using other alternatives.

i guess what is most troubling me is that... as mentioned:

1. it's a static binary download
2. the binary hasn't changed
3. config is in version control, it hasn't changed
4. the script starting it is in version control, it hasn't changed
5. the binary still starts and runs ok
6. the cert is private but is not expired, loads in browser
7. yet suddenly vault is telling us UI isn't available?

unless something else missing.... feels like you can disable things even in a static binary, which we could avoid presumably by building source.

idk, seems like there might be a lot more uncertainty in supporting this long-term than can be understood.

anyways, if you have more insight into why this specific error just shows up w/no context... that would be helpful to more than just us i'm certain.

[hsimon-hashicorp]

Not every binary has every support, it can depend on how it's built and who's doing the building. The rest of the error provides more context:
Vault UI is not available in this binary. To get Vault UI do one of the following: Download an official release Run make release to create your own release binaries. Run make dev-ui to create a development binary with the UI.

This has been previously reported to arch linux: https://bugs.archlinux.org/task/58256

Please let me know if that helps. I think you'll want to ask arch to ensure that they compile with UI support.

[ncabatoff]

this is where the original package appeared to be getting 1.9.1 from (i think), but don't see aarch64 there now:

FYI arm64 is the name the Go compiler uses for aarch64 builds. In other words, in this context arm64 == aarch64.

[pracplayopen]

@hsimon-hashicorp thanks i understand that but like i said, we've been using the ui via this binary for months. (the ticket you opened is super old and not relevant, was closed as fixed in 2018).

@ncabatoff nick! that did it, thanks so much.

for anybody else suddenly having this issue, went with: https://releases.hashicorp.com/vault/1.9.1/vault_1.9.1_linux_arm64.zip

everything back to normal, whew!

[ncabatoff]

@pracplayopen sounds like Arch has re-created the bug in their release that @hsimon-hashicorp discovered from 2018, you might want to file an issue with them (and encourage them to use our packages/builds if that's an option for them... I know arch likes to build things themselves so maybe not, but it would avoid these problems...)

[pracplayopen]

@ncabatoff respectfully you might've missed previous comments, but 100% sure the 1.9.1 binary we've used has working ui support. we've been using the UI w/this binary for months. indeed we wouldn't have upgraded if the ui didn't work, because our support guide is 99% based around the ui. the ui is definately there and working in 1.9.1, no doubts.

as it was working until server stopped recently. this happens periodically, we simply restart it and it works great. however this time i'm wondering if somehow our original binary was corrupted. can't verify this entirely because our documentation was somewhat lacking and our install script was a bit incomplete as to original source (have used both alpine and arch packages in past, both build from source but whatever we have used always had UI. also both alpine and arch have moved to 1.9.2 release so it's a bit hard to verify based on checksums).

anyways this source binary discrepency is our our fault and i'm making sure it's corrected.

thx again for your assistance.

[pracplayopen]

final follow-up- vault is quite a nice product. thx to entire hashicorp product/dev/support teams!

general feedback:

• vault ui is useful because while they both have similiar syntax for very specific commands, it's easier to ensure that tokens/keys/passwords stay out of command histories when using the ui

thx again