Thanos hardlink operation not permitted

[ankushbbbr]

I am deploying thanos v0.30.2 with prometheus operator. But I am seeing below error in thanos-sidecar logs -

err="upload 01HPNCX2PD4QEKY3J8JY3DG4KB: hard link block: hard link file chunks/000001: link /prometheus/01HPNCX2PD4QEKY3J8JY3DG4KB/chunks/000001 /prometheus/thanos/upload/01HPNCX2PD4QEKY3J8JY3DG4KB/chunks/000001: operation not permitted" uploaded=0

This is an issue in creating hard link to prometheus TSDB blocks in thanos/ directory.
Upon debugging, I found that -

• Thanos runs as root user (uid=0)
• Prometheus runs as nobody user (uid=65534)
• I tried manually creating a hardlink as nobody user, it works. Hence the underlying file-system supports hardlinks. But when I login from thanos-sidecar as root user, root user is not able to create hardlink, it gives operation not permitted error. However, root user is able to create softlink and copy the same file.

I also tried adding a fsgroup 1001 in k8s Pod level securityContext, but even then root user is not able to create hardlink on files owned by nobody user.

Any pointers on how can I resolve this issue?

[ankushbbbr]

I was able to resolve this by adding

   securityContext:
      fsGroup: 2000
      runAsGroup: 2000
      runAsNonRoot: true
      runAsUser: 1000

to make sure Prometheus and Thanos run as the same user.