Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

processes collector is broken after upgrade to 1.2.0 #2110

Closed
karlism opened this issue Aug 4, 2021 · 10 comments · Fixed by #2112
Closed

processes collector is broken after upgrade to 1.2.0 #2110

karlism opened this issue Aug 4, 2021 · 10 comments · Fixed by #2112

Comments

@karlism
Copy link

karlism commented Aug 4, 2021

Host operating system: output of uname -a

$ uname -a
Linux hostname 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

Prior to upgrade:

node_exporter, version 1.1.2 (branch: HEAD, revision: b597c1244d7bef49e6f3359c87a56dd7707f6719)
  build user:       root@f07de8ca602a
  build date:       20210305-09:29:10
  go version:       go1.15.8
  platform:         linux/amd64

After an upgrade:

node_exporter, version 1.2.0 (branch: HEAD, revision: 12968948aec1e2b216a2ecefc45cf3a50671aecb)
  build user:       root@6b17174de526
  build date:       20210715-16:35:54
  go version:       go1.16.6
  platform:         linux/amd64

node_exporter command line flags

/usr/bin/node_exporter --no-collector.bonding --no-collector.conntrack --no-collector.fibrechannel --no-collector.infiniband --no-collector.ipvs --no-collector.nfs --no-collector.nfsd --collector.textfile.directory=/var/lib/node_exporter/textfile_collector --collector.ntp --no-collector.pressure --collector.processes --no-collector.rapl --no-collector.zfs --web.listen-address=172.16.xxx.xxx:9100

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Upgrade node exporter from 1.1.2 to 1.2.0

What did you expect to see?

I would expect process collector to work

What did you see instead?

Process exporter is broken

Prior to upgrade:

Aug  4 12:28:58 hostname node_exporter: level=info ts=2021-08-04T12:28:58.405Z caller=node_exporter.go:178 msg="Starting node_exporter" version="(version=1.1.2, branch=HEAD, revision=b597c1244d7bef49e6f3359c87a56dd7707f6719)"
Aug  4 12:28:58 hostname node_exporter: level=info ts=2021-08-04T12:28:58.405Z caller=node_exporter.go:179 msg="Build context" build_context="(go=go1.15.8, user=root@f07de8ca602a, date=20210305-09:29:10)"
Aug  4 12:28:58 hostname node_exporter: level=info ts=2021-08-04T12:28:58.405Z caller=filesystem_common.go:74 collector=filesystem msg="Parsed flag --collector.filesystem.ignored-mount-points" flag=^/(dev|proc|sys|var/lib/docker/.+)($|/)
Aug  4 12:28:58 hostname node_exporter: level=info ts=2021-08-04T12:28:58.405Z caller=filesystem_common.go:76 collector=filesystem msg="Parsed flag --collector.filesystem.ignored-fs-types" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
Aug  4 12:28:58 hostname node_exporter: level=info ts=2021-08-04T12:28:58.405Z caller=node_exporter.go:106 msg="Enabled collectors"
[...]
Aug  4 12:28:58 hostname node_exporter: level=info ts=2021-08-04T12:28:58.406Z caller=node_exporter.go:113 collector=processes
[...]
Aug  4 12:28:58 hostname node_exporter: level=info ts=2021-08-04T12:28:58.406Z caller=node_exporter.go:195 msg="Listening on" address=172.16.xxx.xxx:9100
Aug  4 12:28:58 hostname node_exporter: level=info ts=2021-08-04T12:28:58.406Z caller=tls_config.go:191 msg="TLS is disabled." http2=false

After an upgrade:

Aug  4 12:36:32 hostname node_exporter: level=info ts=2021-08-04T12:36:32.533Z caller=node_exporter.go:182 msg="Starting node_exporter" version="(version=1.2.0, branch=HEAD, revision=12968948aec1e2b216a2ecefc45cf3a50671aecb)"
Aug  4 12:36:32 hostname node_exporter: level=info ts=2021-08-04T12:36:32.534Z caller=node_exporter.go:183 msg="Build context" build_context="(go=go1.16.6, user=root@6b17174de526, date=20210715-16:35:54)"
Aug  4 12:36:32 hostname node_exporter: level=info ts=2021-08-04T12:36:32.534Z caller=filesystem_common.go:110 collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|sys|var/lib/docker/.+)($|/)
Aug  4 12:36:32 hostname node_exporter: level=info ts=2021-08-04T12:36:32.534Z caller=filesystem_common.go:112 collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
Aug  4 12:36:32 hostname node_exporter: level=info ts=2021-08-04T12:36:32.534Z caller=node_exporter.go:108 msg="Enabled collectors"
[...]
Aug  4 12:36:32 hostname node_exporter: level=info ts=2021-08-04T12:36:32.534Z caller=node_exporter.go:115 collector=processes
[...]
msg="Listening on" address=172.16.xxx.xxx:9100
Aug  4 12:36:32 hostname node_exporter: level=info ts=2021-08-04T12:36:32.535Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
Aug  4 12:36:46 hostname node_exporter: level=error ts=2021-08-04T12:36:46.012Z caller=collector.go:169 msg="collector failed" name=nvme duration_seconds=5.8419e-05 err="error obtaining NVMe class info: failed to list NVMe devices at \"/sys/class/nvme\": open /sys/class/nvme: no such file or directory"
Aug  4 12:36:46 hostname node_exporter: level=error ts=2021-08-04T12:36:46.016Z caller=collector.go:169 msg="collector failed" name=processes duration_seconds=0.000351222 err="unable to retrieve number of allocated threads: strconv.ParseInt: parsing \"18446744073709551615\": value out of range"

I can also see that metrics like node_processes_max_processes are missing after an upgrade.

@ygxxii
Copy link

ygxxii commented Aug 5, 2021

I got a same issue too.

Linux hostname 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
node_exporter, version 1.2.0 (branch: HEAD, revision: 12968948aec1e2b216a2ecefc45cf3a50671aecb)
  build user:       root@6b17174de526
  build date:       20210715-16:35:54
  go version:       go1.16.6
  platform:         linux/amd64

I queried node_scrape_collector_success and got:

node_scrape_collector_success{collector="processes", instance="localhost:9100", job="node"}  0

@discordianfish
Copy link
Member

Can you try 1.2.1? I think both issues should be fixed there.

@karlism
Copy link
Author

karlism commented Aug 5, 2021

@discordianfish, unfortunately there are still same errors for process collector even after an upgrade to 1.2.1:

Aug  5 12:18:46 hostname node_exporter: level=error ts=2021-08-05T12:18:46.017Z caller=collector.go:169 msg="collector failed" name=processes duration_seconds=0.001882138 err="unable to retrieve number of allocated threads: strconv.ParseInt: parsing \"18446744073709551615\": value out of range"
Aug  5 12:18:58 hostname node_exporter: level=error ts=2021-08-05T12:18:58.868Z caller=collector.go:169 msg="collector failed" name=processes duration_seconds=0.000401219 err="unable to retrieve number of allocated threads: strconv.ParseInt: parsing \"18446744073709551615\": value out of range"
$ node_exporter --version
node_exporter, version 1.2.1 (branch: HEAD, revision: 4200d7f0e15d57e783ac2b814df12edce33513c6)
  build user:       root@e727e6931c4b
  build date:       20210805-09:14:31
  go version:       go1.16.6
  platform:         linux/amd64

nvme collector errors are gone however.

@l0tzi
Copy link

l0tzi commented Aug 5, 2021

I can confirm the same error as @karlism using 1.2.1

@SuperQ
Copy link
Member

SuperQ commented Aug 5, 2021

Odd, I don't see any relevant changes between these two releases for that collector. The collector itself hasn't changed at all. And where strconv.ParseInt() is used in procfs is where it parses the /proc filenames to get the PID list.

I'll have to do some more debugging. If people who can reproduce the problem, please include your OS info (uname -a).

SuperQ added a commit that referenced this issue Aug 5, 2021
Wrap error returns so that the user can know more about what failed.
Returns from getAllocatedThreads() are errors anyway.

Related to #2110

Signed-off-by: Ben Kochie <superq@gmail.com>
@SuperQ
Copy link
Member

SuperQ commented Aug 5, 2021

If someone can try running with --log.level=debug?

Or even better, run the node_exporter with this patch: #2112

@jeffrey4l
Copy link

Got following error with debug log level


level=debug ts=2021-08-06T03:02:39.128Z caller=node_exporter.go:76 msg="collect query:" filters="unsupported value type"
level=debug ts=2021-08-06T03:02:39.130Z caller=tapestats_linux.go:128 collector=tapestats msg="scsi_tape stats not found, skipping"
level=debug ts=2021-08-06T03:02:39.130Z caller=zfs_linux.go:52 collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs
level=debug ts=2021-08-06T03:02:39.130Z caller=collector.go:167 msg="collector returned no data" name=tapestats duration_seconds=8.7961e-05 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.130Z caller=zfs.go:73 collector=zfs err="ZFS / ZFS statistics are not available"
level=debug ts=2021-08-06T03:02:39.130Z caller=collector.go:167 msg="collector returned no data" name=zfs duration_seconds=0.000186033 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.131Z caller=loadavg.go:54 collector=loadavg msg="return load" index=0 load=0.43
level=debug ts=2021-08-06T03:02:39.131Z caller=loadavg.go:54 collector=loadavg msg="return load" index=1 load=0.62
level=debug ts=2021-08-06T03:02:39.131Z caller=loadavg.go:54 collector=loadavg msg="return load" index=2 load=0.83
level=debug ts=2021-08-06T03:02:39.131Z caller=collector.go:173 msg="collector succeeded" name=loadavg duration_seconds=0.000183161
level=debug ts=2021-08-06T03:02:39.131Z caller=collector.go:173 msg="collector succeeded" name=mdadm duration_seconds=0.000195125
level=debug ts=2021-08-06T03:02:39.131Z caller=collector.go:173 msg="collector succeeded" name=cpufreq duration_seconds=0.000266535
level=debug ts=2021-08-06T03:02:39.131Z caller=diskstats_linux.go:195 collector=diskstats msg="Ignoring device" device=sda1
level=debug ts=2021-08-06T03:02:39.131Z caller=diskstats_linux.go:195 collector=diskstats msg="Ignoring device" device=sdb1
level=debug ts=2021-08-06T03:02:39.131Z caller=diskstats_linux.go:195 collector=diskstats msg="Ignoring device" device=sdb2
level=debug ts=2021-08-06T03:02:39.131Z caller=collector.go:173 msg="collector succeeded" name=arp duration_seconds=8.7942e-05
level=debug ts=2021-08-06T03:02:39.131Z caller=collector.go:173 msg="collector succeeded" name=diskstats duration_seconds=0.00024626
level=debug ts=2021-08-06T03:02:39.131Z caller=collector.go:173 msg="collector succeeded" name=sockstat duration_seconds=0.000548901
level=debug ts=2021-08-06T03:02:39.131Z caller=collector.go:173 msg="collector succeeded" name=edac duration_seconds=0.000227584
level=debug ts=2021-08-06T03:02:39.131Z caller=nvme_linux.go:56 collector=nvme msg="nvme statistics not found, skipping"
level=debug ts=2021-08-06T03:02:39.131Z caller=time.go:60 collector=time msg="Return time" now=1.6282189591315703e+09
level=debug ts=2021-08-06T03:02:39.132Z caller=time.go:62 collector=time msg="Zone offset" offset=28800 time_zone=CST
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=softnet duration_seconds=0.00038791
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=time duration_seconds=0.000475059
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=ipvs duration_seconds=0.000106772
level=debug ts=2021-08-06T03:02:39.132Z caller=pressure_linux.go:91 collector=pressure msg="collecting statistics for resource" resource=cpu
level=debug ts=2021-08-06T03:02:39.132Z caller=pressure_linux.go:95 collector=pressure msg="pressure information is unavailable, you need a Linux kernel >= 4.20 and/or CONFIG_PSI enabled for your kernel"
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:167 msg="collector returned no data" name=pressure duration_seconds=0.000110662 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=bcache duration_seconds=2.5483e-05
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:167 msg="collector returned no data" name=nvme duration_seconds=0.001027206 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=netdev duration_seconds=0.000694099
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=udp_queues duration_seconds=0.000305862
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=rapl duration_seconds=5.5305e-05
level=debug ts=2021-08-06T03:02:39.132Z caller=infiniband_linux.go:119 collector=infiniband msg="infiniband statistics not found, skipping"
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:167 msg="collector returned no data" name=infiniband duration_seconds=9.7144e-05 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=powersupplyclass duration_seconds=5.5391e-05
level=debug ts=2021-08-06T03:02:39.132Z caller=meminfo.go:53 collector=meminfo msg="Set node_mem" memInfo="unsupported value type"
level=debug ts=2021-08-06T03:02:39.132Z caller=collector.go:173 msg="collector succeeded" name=meminfo duration_seconds=0.000259414
level=debug ts=2021-08-06T03:02:39.133Z caller=collector.go:173 msg="collector succeeded" name=xfs duration_seconds=0.000917375
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=uname duration_seconds=0.002460773
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=stat duration_seconds=0.001351162
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=textfile duration_seconds=4.9198e-05
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=filefd duration_seconds=6.7485e-05
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=btrfs duration_seconds=2.9097e-05
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=hwmon duration_seconds=3.6782e-05
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=thermal_zone duration_seconds=0.001423837
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=timex duration_seconds=1.9654e-05
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:173 msg="collector succeeded" name=entropy duration_seconds=0.000106204
level=debug ts=2021-08-06T03:02:39.134Z caller=fibrechannel_linux.go:113 collector=fibrechannel msg="fibrechannel statistics not found, skipping"
level=debug ts=2021-08-06T03:02:39.134Z caller=nfs_linux.go:102 collector=nfs msg="Not collecting NFS metrics" err="open /proc/net/rpc/nfs: no such file or directory"
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:167 msg="collector returned no data" name=fibrechannel duration_seconds=2.155e-05 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:167 msg="collector returned no data" name=nfs duration_seconds=7.1296e-05 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.134Z caller=bonding_linux.go:64 collector=bonding msg="Not collecting bonding, file does not exist" file=/sys/class/net
level=debug ts=2021-08-06T03:02:39.134Z caller=nfsd_linux.go:68 collector=nfsd msg="Not collecting NFSd metrics" err="open /proc/net/rpc/nfsd: no such file or directory"
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:167 msg="collector returned no data" name=bonding duration_seconds=4.9535e-05 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.134Z caller=collector.go:167 msg="collector returned no data" name=nfsd duration_seconds=5.0597e-05 err="collector returned no data"
level=debug ts=2021-08-06T03:02:39.136Z caller=collector.go:173 msg="collector succeeded" name=conntrack duration_seconds=0.001628831
level=debug ts=2021-08-06T03:02:39.136Z caller=cpu_linux.go:242 collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu0
level=debug ts=2021-08-06T03:02:39.136Z caller=collector.go:173 msg="collector succeeded" name=vmstat duration_seconds=0.002642096
level=debug ts=2021-08-06T03:02:39.136Z caller=cpu_linux.go:252 collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu0
level=debug ts=2021-08-06T03:02:39.136Z caller=cpu_linux.go:242 collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu1
level=debug ts=2021-08-06T03:02:39.136Z caller=cpu_linux.go:252 collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu1
level=debug ts=2021-08-06T03:02:39.136Z caller=cpu_linux.go:242 collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu2
level=debug ts=2021-08-06T03:02:39.136Z caller=cpu_linux.go:252 collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu2
level=debug ts=2021-08-06T03:02:39.137Z caller=cpu_linux.go:242 collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu3
level=debug ts=2021-08-06T03:02:39.137Z caller=cpu_linux.go:252 collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu3
level=debug ts=2021-08-06T03:02:39.137Z caller=collector.go:173 msg="collector succeeded" name=schedstat duration_seconds=0.005028974
level=debug ts=2021-08-06T03:02:39.137Z caller=cpu_linux.go:242 collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu4
level=debug ts=2021-08-06T03:02:39.137Z caller=cpu_linux.go:252 collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu4
level=debug ts=2021-08-06T03:02:39.137Z caller=processes_linux.go:121 collector=processes msg="error reading stat for pid" pid="unsupported value type" err="strconv.ParseInt: parsing \"18446744071638183230\": value out of range"
level=error ts=2021-08-06T03:02:39.137Z caller=collector.go:169 msg="collector failed" name=processes duration_seconds=0.00520059 err="unable to retrieve number of allocated threads: strconv.ParseInt: parsing \"18446744071638183230\": value out of range"
level=debug ts=2021-08-06T03:02:39.137Z caller=cpu_linux.go:242 collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu5
level=debug ts=2021-08-06T03:02:39.137Z caller=cpu_linux.go:252 collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu5
level=debug ts=2021-08-06T03:02:39.137Z caller=cpu_linux.go:242 collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu6
level=debug ts=2021-08-06T03:02:39.137Z caller=cpu_linux.go:252 collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu6
level=debug ts=2021-08-06T03:02:39.138Z caller=cpu_linux.go:242 collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu7
level=debug ts=2021-08-06T03:02:39.138Z caller=cpu_linux.go:252 collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu7
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys
level=debug ts=2021-08-06T03:02:39.138Z caller=collector.go:173 msg="collector succeeded" name=cpu duration_seconds=0.00357387
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/proc
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/dev
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/security
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/dev/shm
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/dev/pts
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/systemd
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/pstore
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/net_cls,net_prio
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/memory
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/cpuset
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/pids
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/cpu,cpuacct
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/hugetlb
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/blkio
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/perf_event
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/devices
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup/freezer
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/config
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:58 collector=filesystem msg="Ignoring fs" type=rpc_pipefs
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/dev/mqueue
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/dev/hugepages
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/debug
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/proc/sys/fs/binfmt_misc
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/9ec66440652e2dcaaf1aac617a0b396262ccf7d012063e3a9fa7c121c688353c/merged
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:58 collector=filesystem msg="Ignoring fs" type=proc
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/4a6547ac1b93da248fe0245e6921cb5fefddf15013804d1c55236be703a87779/merged
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/5fce2bc55a5903b16e4a1a62118104a836785d05f855c47fd662fc6ab550b37b/merged
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/bd7bd9f6bf6aebd627b43423f5c04c721ee366c14c1c688f11761a115e774529/merged
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/c5f6cc85893d7d05dcdc061f6ddd9d8931ec2bdc579aedf96f40dab268a21411/merged
level=debug ts=2021-08-06T03:02:39.138Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/5359b5a9bd3d2f0caa88cd83435f22e2ad6a64d159a77bd8e3cbff0bc905e22f/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/c73b14f50533dbf949b6a51d7ae350bb68ee72161ebf6b0062d709ec00f3a682/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/73ea7c65f32e1b0d74861077c7e0660bed60dd175666e6b25cac4cef535a666b/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/25735821c16fd9b5c16ba8d5825d5160387a2a2e907f86e0090ab18dd330d264/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/b70c2f6559eb7f884bbdd6ec0e3b88fdd0a72b08f053a6f2945f3965eea5cbbd/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/7a588b934831d8336dcad509da7140da2823d26cc1879f8f192f6bdbdbc59e4a/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/e4d0a75d7775f1e822fb9a183b29c6afe3b75b0caa140e9d742dcf611c9cd92a/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/409612db63889908f9ffce2f30af3260f85a91ceaa264b57f02f5523b08db112/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/eada8e8799324f4929433e9e53d998e5d2ff3ff4bb3e0722052471cc34bbc8d1/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/81886160abb05f19623425cb59c22976c46b5a4282a86c11e92a63c06fb01018/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/96e0f747696c0f1f092dce331cfed617e0070c54e39544a68132e1a67395e149/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/29796ac0d099c1d400b7c9126012568293f196b3ea538b0fa9d46ac2444b77cf/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/4e84ad23180f63fbbb186cc68be7393c74879d75c89b25270c1b25ad1885b519/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/2f7999ff251dd32404fc3b94e6ef8283589b12f5ab55c3e5f2bcd94e23043d76/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/a977789e0e2ed77fd571e6d85b4b751e4cede4ba05e461dd772c5f5aa86dee5b/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/1316ac9d7783788368b2c2c7b456a31760a3554e32887108540b55179da6b1f9/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/c14a90199dbe816b2f97b6e4c860548685e15a96b4e909f0737c7914e3e12206/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/ae8fcdfab8f14f73cf524259cb7ab68e7d311290ed9cc6a0c2a883f7653c838a/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/ff9b94fa79e73a34c063a79c4c19a0a9cf37863a708c495ed6c02b3989a6f834/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/c06156b0e907beae0d8352f58e64864b78ed137118da603999c263efe6598eab/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/a8db4d401116793a41ffad92ea87dd8e001a7557479ce358d2ab19f73ed58cb9/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=collector.go:173 msg="collector succeeded" name=netstat duration_seconds=0.004458802
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/2882f24a0f395249a198a67962ef3a87a8b7d91f4ba0e2d6a48106d8e4e98a00/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/ed6a9b1cf00a1ec1b683e251e84cae792fe875c822794918b06086d7751a90a5/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/09b6dc21d09080af33aca7170453cdf7849010cdb3c4a22f6ccd20576ccdf287/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/0dcb8bc41c8a8f267862fdd298e47b98c6e03a1dd32b5616fb4f3a86e52326fc/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/40b1957c60ebee90709a0115bf1c007b1feb5eb1eb0c4da1d163cbd104eb2bd0/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/aaa0338934980806aacbb755780754c83c29a1d385ddad70ee5ff2f9a8b561c7/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/d964b57309fa9694e0d8171b1dc11ceb835fef6df0619bf03d030e8fc5f468a8/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/2f52c404aecf8d0abd493bba2fdd5fd496c279c3fcee53b17603b56bd690b0e7/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/a26f290d409402240b3476593d64bc41bf3b4d7fbf365144523bcf49be11e70c/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/8055769e11dc7e8f7c3c1339ee6de57d24faccec9e02c671739c64407683208c/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/d51c7687cef6cd532bf024bb8696304dcd28dd81d4428133f75431cc7842c474/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/234af8448ef7cefe2ba0103f951cb6e3cf4b2f935b8f749cc415aa650b44f387/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/9766a11e59c9b59eca6d12e9a5588565505b8185f32aed771693e4e5779e5abf/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/d2faa91694582160ecdf353bf16a5e286b0ad405bb97063b8c04ae08afd8022e/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/abd8000ab78b9e3e5660d303f4f9f1d7454676509237272b4355d1a81f8a42c2/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/a5872304c9485815f2d97de7b8ff99d9d538c4407bda1ec8ed2c39b8f8666b49/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/573effaaaf47474325b1fd14d393466df78b2bded6607603266b0458522a7d45/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/3ab2199e325a343bfd372f893a6db193c6a2ad41310a5e98bd70ebcef42b7fee/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/f5f354e34e7ff8370deef8784981bf63e4f9dd2ea1701beb36481d9d28b42056/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/f85b3b07092e1d883a2047d3b6f36ae5e70d889bfbbe06cb292c2110f0f0200f/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/d05d36278d10b16b8749338a611e39cfca6ee8930f2dc07a317221d8101db66e/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/4e478c2e511318a854b960cad98a1ff0cdee693d7db795861e535c1c9f2f3341/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/7feb765557ab13d6a4d83596722edcb297a9e52dbddca7ac088cf773b707ad64/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/c68c50f5fca7de5ed1e644552dd00d075842c25b2e0d89bf121d014bdc5b82f5/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/76ed0c9431af1c95f8df3516bbd5d0484dd62372c823d69819a50e0bf9282ea2/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/723cbb04ef302bd24350ef898fcfe483ae5c6b512abcbd01c3f6c79bc1ff4ab5/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/59dca58a91214ca8a9382cfee9d8ae54c2a271063fe20e01c0e30172f6d1fd61/merged
level=debug ts=2021-08-06T03:02:39.139Z caller=filesystem_linux.go:54 collector=filesystem msg="Ignoring mount point" mountpoint=/var/lib/docker/overlay2/1ab7411ccf88bc6f39a888a1b66f0023942f2b8eddc91554866a711413eb7714/merged
level=debug ts=2021-08-06T03:02:39.140Z caller=collector.go:173 msg="collector succeeded" name=filesystem duration_seconds=0.008552683

@SuperQ
Copy link
Member

SuperQ commented Aug 6, 2021

Thanks, that helps. I'm guessing that's related prometheus/procfs#402. That bugfix didn't make it into the release.

SuperQ added a commit that referenced this issue Aug 6, 2021
Update procfs library to include ignored fields ParseInt handling.

Wrap error returns so that the user can know more about what failed.
Returns from getAllocatedThreads() are errors anyway.

Fixes: #2110

Signed-off-by: Ben Kochie <superq@gmail.com>
@SuperQ
Copy link
Member

SuperQ commented Aug 6, 2021

@jeffrey4l Can you give this patch a try? #2112

@jeffrey4l
Copy link

yeah, #2112 works. thanks @SuperQ

SuperQ added a commit that referenced this issue Aug 6, 2021
Update procfs library to include ignored fields ParseInt handling.

Wrap error returns so that the user can know more about what failed.
Returns from getAllocatedThreads() are errors anyway.

Fixes: #2110

Signed-off-by: Ben Kochie <superq@gmail.com>
oblitorum pushed a commit to shatteredsilicon/node_exporter that referenced this issue Apr 9, 2024
Update procfs library to include ignored fields ParseInt handling.

Wrap error returns so that the user can know more about what failed.
Returns from getAllocatedThreads() are errors anyway.

Fixes: prometheus#2110

Signed-off-by: Ben Kochie <superq@gmail.com>
oblitorum pushed a commit to shatteredsilicon/node_exporter that referenced this issue Apr 9, 2024
Update procfs library to include ignored fields ParseInt handling.

Wrap error returns so that the user can know more about what failed.
Returns from getAllocatedThreads() are errors anyway.

Fixes: prometheus#2110

Signed-off-by: Ben Kochie <superq@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants