Qurey：Can not query from Object storage

[btdan]

I run with the command as follows：

nohup thanos receive
--tsdb.path "/aiops/prometheusData/metricsData"
--grpc-address 0.0.0.0:10907
--http-address 0.0.0.0:10909
--receive.replication-factor 1
--label "receive_replica="0""
--label "receive_cluster="aiopssection""
--receive.local-endpoint 10.0.90.202:10907
--receive.hashrings-file /aiops/hashring.json
--remote-write.address 0.0.0.0:10908
--objstore.config-file "/aiops/bucket.yml" > /thanos_receive.log 2>&1 &

nohup thanos query --http-address 0.0.0.0:19192 --grpc-address 0.0.0.0:19193 --store 10.0.90.202:10907 --store 10.0.90.202:19090 > /thanos_query.log 2>&1 &

nohup thanos store --data-dir /aiops/thanosStoreGateway --objstore.config-file /aiops/bucket.yml --http-address 0.0.0.0:19191 --grpc-address 0.0.0.0:19090 > /thanos_store.log 2>&1 &

But I find that the query result on the website url http://10.0.90.202:19192 only include the data from local disk with thanos receive command. That is to say, the data on object store is not queried.

I read the log file thanos_query.log, and find many record like this:
level=warn ts=2021-07-21T07:12:35.910427898Z caller=proxy.go:450 component=proxy request="min_time:1624258500000 max_time:1626850800000 matchers:<type:RE name:"instance" value:"10\\.0\\.90\\.210:9100" > matchers:<name:"name" value:"node_memory_MemAvailable_bytes" > aggregates:COUNT aggregates:SUM " err="receive series from Addr: 10.0.90.202:19090 LabelSets: {receive_cluster="aiopssection", receive_replica="0", tenant_id="default-tenant"} Mint: 16 21123202783 Maxt: 1626847200000: rpc error: code = Aborted desc = fetch series for block 01F9J4Q0B5WYV9SH6KCP2WN3AX: load chunks: get range reader: Get "http://10.0.90.203/api/v1/obj/aiopssection/01F9J4Q0B5WYV9SH6KCP2WN3AX/chunks/000001/?offset=51277073&size=16806\": dial tcp 10.0.90.203:80: socket: too many open files" msg="returning partial response"

We can see there is an log for error. Can anyone help me to solve this problem? Thanks very much.

[amine250]

dial tcp 10.0.90.203:80: socket: too many open files" you need to increase the max number of files opened at time in your OS.

Try running ulimit -a and check the value of the file descriptors, increase it with ulimit -n 120000 for example.

[btdan]

HI, Thanks very much for your reply.
I run ulimit -a command and get the result as follows:

[root@Thanos ~]# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15588
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 409600
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15588
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

however, the problem still not be solved.

[amine250]

Are you running it in a container?

[btdan]

I am runing it in a virtual machine. It is an Centos system. Thanks very much.