Provide Information On Remote Logging

[jseiser]

What I'd like:
We currently use Fluent-bit, and Promtail for remote logging. Fluent-bit was configured using this guide: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html but since we switched to Bottle Rocket, we lost the host logs.

Same with Promtail, we ship all the container logs etc to Loki, but again, we are not getting kubelet logs etc.

Just some basic information on how to get this working would be amazing. From Promtail/Fluent-bits perspective, we just really need to know which volumes to mount on the host, and if they are files we tail or systemd logging.

Running Bottle Rocket has left a huge gap in what information we are able to use to troubleshoot, since we are hitting an issue where a node goes notready, and the kubelet dies, and AWS SSM access stops, we are basically only left with deleting the node.

[stmcginnis]

Hey @jseiser - there is a blog post out there that has some details. It would be great to hear if this is missing anything.

[jseiser]

Going to take a bit to parse that, since im not hip to the CRD from's the Fluent Bit operator.

Of hand, it appears we need to mount /var/log/journal which is fine. But It doesnt appear to cover the ipam logs.

[stockholmux]

@stmcginnis and @jseiser should we transfer this to bottlerocket-os/bottlerocket-project-website? Sounds like this is a decent FAQ item.

[jseiser]

@stockholmux

Anything even close to this: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html

Would have been a huge help for us. We use fluent-bit and Loki, and install FB from a helm chart, not as raw manifests like that guide, but it was definitely easy to just see what AWS expects to be getting sent.

I also still have not been able to take a stab at the guide that was posted above yet. Hopefully this week if my QA cluster would cooperate.

[stockholmux]

@jseiser Cool. Let's transfer this issue over to the other repo.

FWIW, The website is always looking for new guides, so if you have insights you want to share about bottlerocket + promtail/fluent bit/loki, i'm sure it would be a valuable contribution.

Also, Please let me know about how the guide works for you. It's a blog post, which is a slice-in-time how-to (i.e. it worked when published but details change), so if it needs work, please let us know. The broad strokes shouldn't change though.

[jseiser]

I wanted to post back, that I am able to make this work using fluent-bit helm chart. The chart already mounts /var/log and the /etc/machine-id

Just need an INPUT

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Systemd_Filter _SYSTEMD_UNIT=docker.service
        Systemd_Filter _SYSTEMD_UNIT=containerd.service
        Read_From_Tail On

A FILTER

    [FILTER]
        Name                modify
        Match               host.*
        Rename              _HOSTNAME                   hostname
        Rename              _SYSTEMD_UNIT               systemd_unit
        Rename              MESSAGE                     message
        Remove_regex        ^((?!hostname|systemd_unit|message).)*$

Then configure an OUTPUT matching your label.