The datadog-agent fails to deploy on EKS Fargate

[milldr]

Problem

datadog-agent pods fail to start for no clear reason. After debugging, it appears that the pods deployed to Fargate are failing

NAME                                    READY   STATUS    RESTARTS   AGE
datadog-6rm4l                           5/5     Running   0          7m38s
datadog-7fghz                           5/5     Running   0          7m38s
datadog-cluster-agent-95b486955-fqrpg   1/1     Running   0          7m37s
datadog-ghw6z                           5/5     Running   0          7m38s
datadog-gz55x                           0/5     Pending   0          7m38s
datadog-pzsqr                           0/5     Pending   0          7m38s
datadog-vg2nq                           5/5     Running   0          7m38s

The Datadog Cluster Agent reports

Warning  Unhealthy  3m13s (x2 over 3m43s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
Warning  Unhealthy  3m13s (x2 over 3m43s)  kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500

Reference

• Slack thread in reference (private)

[osterman]

aws/containers-roadmap#971 (comment) - they are not supported, hence this issue - the Daemoneset is still trying to create the pod for it

[osterman]

For ECS + Fargate, run Datadog as a Replica Service
For EKS + Fargate, set DD_EKS_FARGATE = true to run the agent as a Sidecar rather than a DaemonSet.

[Benbentwo]

This env var is used to add when deploying the agent as a sidecar to fargate Deployments specifically. The problem here is that we want to deploy the agent (as a helm chart) to report on the cluster that is not managed on fargate (since this is a fargate and karpenter cluster). Since only karpenter runs on fargate we need a way to prevent the agent from running on fargate specifically.

[milldr]

Solution

Datadog Agent shouldn't be deployed to Fargate! Use affinity to declare which nodes should be excluded. For example, add the following values to the datadog-agent component:

components:
  terraform:
    datadog-agent:
      vars:
...
        values:
          agents:
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: eks.amazonaws.com/compute-type
                      operator: NotIn
                      values:
                      - fargate

But there is more

From Datadog documentation:

If the EKS cluster runs Fargate and non-Fargate workloads, and you want to monitor the non-Fargate workload through Node Agent DaemonSet, add the Cluster Agent/Cluster Checks Runner to this deployment. For more information, see the Cluster Agent Setup.

Unanswered

• Problem is that daemonsets are to all nodes, wonder if theres a way to default exclude?
• This is a one-off solution for Datadog. We need to determine if theres a cluster wide solution

[osterman]

I wonder if unanswered questions should be moved to discussions.

[Benbentwo]

Perhaps a link to another question?

[Benbentwo]

The problem context here is that we have a cluster with Karpenter and Fargate. A singular namespace is dedicated to fargate, specifically karpenter. This allows us to spin up a perfectly sized cluster on ec2 via a serverless methodology. This means our nodes are mostly ec2 instances, with a few for Fargate for the karpenter deployments.

The issue arises in that we need datadog monitoring for our average deployments (standard charts deployed on ec2 ASG managed by karpenter). To monitor this we need datadogs default helm chart for the datadog agent. Thus we must use the Helm Charts methodology for Node Affinity.

Another methodology that could work would be to create a scheduler specifically for fargate, and another default scheduler for non fargate nodes. This could work cluster wide so no customizations are needed.

[Nuru]

From Datadog documentation:

If the EKS cluster runs Fargate and non-Fargate workloads, and you want to monitor the non-Fargate workload through Node Agent DaemonSet, add the Cluster Agent/Cluster Checks Runner to this deployment. For more information, see the Cluster Agent Setup.