thanos.shipper.json.tmp: read-only file system

[davidy718]

Hi
I'm running prometheus with thanos sidecar and I get an error "thanos.shipper.json.tmp: read-only file system"

- name: thanos-sidecar
  image: quay.io/thanos/thanos:main-2021-11-06-abc5d4c3
  args:
              - "sidecar"
              - "--prometheus.url=http://localhost:9090/"
              - "--tsdb.path=/bucket"
              - "--objstore.config-file=/bucket/bucket.yml"
              - "--log.level=debug"

bucket.yml

   type: S3 
    config: 
      bucket: my-s3-bucket
      endpoint: s3.eu-west-1.amazonaws.com

the host has an AMI role to the S3 bucket
this is from in the sidecar container
drwxrwxrwx 3 root root 99 Nov 8 21:24 bucket

level=error ts=2021-11-08T22:27:15.018260899Z caller=sidecar.go:279 err="WAL dir is not accessible. Is this dir a TSDB directory? If yes it is shared with TSDB?: stat /bucket/wal: no such file or directory"

[yeya24]

That's a configuration issue. Please make sure your sidecar can access Prometheus TSDB wal locally.
Is /bucket the correct path?

[yeya24]

I don't get it. Then why you cannot have 2h upload time? You have live data you mean you have backfilling data?

[davidy718]

what I mean is that if my Prometheus fails then I can lose up to 2h of data

[yeya24]

You need a PV for your Prometheus. And upload time is not controlled by Thanos. It is based on how Prometheus works. It generates block every 2h so the upload time of sidecar is 2h as well. Please read the document of sidecar.

[davidy718]

ok thanks so much for all the help : )

[GiedriusS]

You cannot lose up to 2h of data, that's a lie spread around by certain people. All of the saved data is stored in WAL which later ends up in a block. You can only lose data if you physically delete the persistent volume.

[PratapSingh13]

@davidy718 @yeya24 I'm facing same issue but from starting I'm using - "--tsdb.path=/prometheus" but still issue persist

[davidy718]

Hi @PratapSingh13
The - "--tsdb.path=/prometheus" in Thanos sidecar needs to be the same as "--storage.tsdb.path=/prometheus" in your Prometheus container and they both need to be able to access it
LMK if that helps