Skip to content

Commit

Permalink
[DOCS-7911] Update log.md (#23066)
Browse files Browse the repository at this point in the history
* Update log.md

Revamp the troubleshooting section for Kubernetes Log Collection documentation to clear up the confusion between DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION and DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION

* Apply suggestions from code review

* fixin tabs

---------

Co-authored-by: cecilia saixue watt <cecilia.watt@datadoghq.com>
  • Loading branch information
mrdoggopat and cswatt committed May 10, 2024
1 parent 504e989 commit c717daf
Showing 1 changed file with 54 additions and 4 deletions.
58 changes: 54 additions & 4 deletions content/en/containers/kubernetes/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -409,13 +409,62 @@ For Agent v6.12+, short lived container logs (stopped or crashed) are automatica

## Troubleshooting

If you are missing tags on Kubernetes logs, this may be because the Agent's internal tagger does not yet have the related container or pod tags when logs are sent. To make the Log Agent wait a few seconds for the tagger to be ready, you can use the environment variable `DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION` to set how many seconds to wait. The default value is 0.
#### Missing tags on new containers or pods

When sending logs to Datadog from newly created containers or pods, the Datadog Agent's internal tagger may not yet have the related container/pod tags. As a result, tags may be missing from these logs.

To remediate this issue, you can use the environment variable `DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION` to configure a duration (in seconds) for the Datadog Agent to wait before it begins to send logs from newly created containers and pods. The default value is `0`.

{{< tabs >}}
{{% tab "Datadog Operator" %}}

```yaml
spec:
override:
nodeAgent:
env:
- name: DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION
value: "5"
```
{{% /tab %}}
{{% tab "Helm" %}}
```yaml
datadog:
env:
- name: DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION
value: "5"
```
{{% /tab %}}
{{< /tabs >}}

#### Missing host-level tags on new hosts or nodes

Host-level tags are those seen in the infrastructure list for a given host, and are sourced from either a cloud provider or the Datadog Agent. Common host-level tags include `kube_cluster_name`, `region`, `instance-type`, and `autoscaling-group`.

When sending logs to Datadog from a newly created host or node, it can take a few minutes for host-level tags to be [inherited][12]. As a result, host-level tags may be missing from these logs.

To remediate this issue, you can use the environment variable `DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION` to configure a duration (in minutes). For this duration, the Datadog Agent to manually attaches the host-level tags that it knows about to each sent log. After this duration, the Agent reverts to relying on tag inheritance at intake.

{{< tabs >}}
{{% tab "Datadog Operator" %}}
```yaml
# The number of seconds the Log Agent waits for the internal tagger to add the related container or pod tags to the logs before the logs are sent.
# For example, in order to set the Log Agent to wait five seconds, use an integer in the value:
tagger_warmup_duration: 5
spec:
override:
nodeAgent:
env:
- name: DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION
value: "10m"
```
{{% /tab %}}
{{% tab "Helm" %}}
```yaml
datadog:
env:
- name: DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION
value: "10m"
```
{{% /tab %}}
{{< /tabs >}}

## Further Reading

Expand All @@ -432,3 +481,4 @@ tagger_warmup_duration: 5
[9]: /containers/guide/kubernetes_daemonset/#log-collection
[10]: /getting_started/containers/autodiscovery
[11]: /logs/log_configuration/attributes_naming_convention/
[12]: /getting_started/tagging/assigning_tags/#integration-inheritance

0 comments on commit c717daf

Please sign in to comment.