Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mountstats collector support NFS RDMA protocol stats #2749

Closed
miloaissatu opened this issue Jul 18, 2023 · 2 comments · Fixed by prometheus/procfs#558 · May be fixed by #2825
Closed

mountstats collector support NFS RDMA protocol stats #2749

miloaissatu opened this issue Jul 18, 2023 · 2 comments · Fixed by prometheus/procfs#558 · May be fixed by #2825
Labels
accepted help wanted

Comments

@miloaissatu
Copy link

Host operating system: output of uname -a

Linux 5.14.0-162.23.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 11 19:09:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d69b645cb691dd3e84dc3afc88f5c006962)
  build user:       root@f9c3ed0cfbd3
  build date:       20230527-12:03:54
  go version:       go1.20.4
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

/usr/local/bin/node_exporter --collector.disable-defaults --collector.systemd --collector.cpu --collector.loadavg --collector.filesystem --collector.meminfo --collector.nfs --collector.stat --collector.schedstat --collector.netstat --collector.netdev --collector.mountstats --collector.processes

node_exporter log output

Jul 12 23:49:00 sn3htcw06 node_exporter[2431]: ts=2023-07-12T13:49:00.826Z caller=collector.go:169 level=error msg="collector failed" name=mountstats duration_seconds=0.00067901 err="failed to parse mountstats: invalid NFS protocol \"rdma\" in stats 1.1 statement: [0 0 1093 11 0 442050369 442050342 27 8268101819 1239649 98568716 1057099561 2009384 115759957791482 115758862768800 202114800 57231381 0 4652 0 0 0 0 44440 1157660127 0 78]"
Jul 12 23:49:30 sn3htcw06 node_exporter[2431]: ts=2023-07-12T13:49:30.827Z caller=collector.go:169 level=error msg="collector failed" name=mountstats duration_seconds=0.000677778 err="failed to parse mountstats: invalid NFS protocol \"rdma\" in stats 1.1 statement: [0 0 1093 11 0 442060684 442060657 27 8268393343 1239649 98571877 1057100074 2009460 115759976309794 115758881287112 202117772 57231381 0 4652 0 0 0 0 44440 1157663877 0 78]"
Jul 12 23:50:00 sn3htcw06 node_exporter[2431]: ts=2023-07-12T13:50:00.872Z caller=collector.go:169 level=error msg="collector failed" name=mountstats duration_seconds=0.047233018 err="failed to parse mountstats: invalid NFS protocol \"rdma\" in stats 1.1 statement: [0 0 1093 11 0 442069986 442069959 27 8268656563 1239649 98574744 1057103953 2009504 115760378112416 115759283044678 202120496 57231381 0 4652 0 0 0 0 44440 1157670667 0 78]"
Jul 12 23:50:30 sn3htcw06 node_exporter[2431]: ts=2023-07-12T13:50:30.855Z caller=collector.go:169 level=error msg="collector failed" name=mountstats duration_seconds=0.000692826 err="failed to parse mountstats: invalid NFS protocol \"rdma\" in stats 1.1 statement: [0 0 1093 11 0 442078360 442078333 27 8268896734 1239649 98577095 1057111098 2009555 115761210161171 115760115044281 202120496 57231381 0 4652 0 0 0 0 44440 1157680214 0 78]"
Jul 12 23:51:00 sn3htcw06 node_exporter[2431]: ts=2023-07-12T13:51:00.932Z caller=collector.go:169 level=error msg="collector failed" name=mountstats duration_seconds=0.066178698 err="failed to parse mountstats: invalid NFS protocol \"rdma\" in stats 1.1 statement: [0 0 1093 11 0 442089937 442089910 27 8269242100 1239649 98582064 1057117381 2009611 115761918746806 115760823489874 202120980 57240350 0 4652 0 0 0 0 44440 1157691522 0 78]"
Jul 12 23:51:30 sn3htcw06 node_exporter[2431]: ts=2023-07-12T13:51:30.827Z caller=collector.go:169 level=error msg="collector failed" name=mountstats duration_seconds=0.003460234 err="failed to parse mountstats: invalid NFS protocol \"rdma\" in stats 1.1 statement: [0 0 1093 11 0 442101286 442101259 27 8269562384 1239649 98584262 1057117501 2009661 115761925121512 115760829802743 202123752 57245792 0 4652 0 0 0 0 44440 1157693890 0 78]"

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Enable collector.mountstats, system has NFS mounts using RDMA protocol

What did you expect to see?

Node exporter able to parse mountstats for NFS mounts with RDMA protocol
Output of /proc/self/mountstats for a mount

# stats snippet for 1 mount from /proc/self/mountstats
device <nfsserver>:<nfsmount> mounted on <mountpoint> with fstype nfs statvers=1.1
        opts:   ro,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=120,acregmax=120,acdirmin=120,acdirmax=120,hard,nocto,forcerdirplus,proto=rdma,nconnect=16,port=20049,timeo=600,retrans=2,sec=sys,mountaddr=172.16.40.20,mountvers=3,mountport=0,mountproto=tcp,local_lock=none
        age:    1270876
        caps:   caps=0xf,wtmult=4096,dtsize=131072,bsize=0,namlen=255
        sec:    flavor=1,pseudoflavor=1
        events: 512052 36601115 0 68 1498583 16514 38815015 0 41584 2654459933 0 0 0 0 1527715 0 0 1498575 0 0 0 0 0 0 0 0 0
        bytes:  3104202770327296 0 0 0 2013200952170479 0 491504202537 0
        RPC iostats version: 1.1  p/v: 100003/3 (nfs)
        xprt:   rdma 0 0 5808 62 0 494490723 494490687 36 10032963746 1282789 107150285 1226637531 2673889 135120843409861 135119397156505 266368832 75716996 0 7853 0 0 0 0 119328 1336431717 0 96
        xprt:   rdma 0 0 14094 145 0 492392334 492392307 27 7078693624 2509627 105561370 1280878332 2659446 142218924010291 142217463504063 276368040 94761838 0 7610 0 0 0 0 207977 1389069860 0 103
        xprt:   rdma 0 0 16107 156 0 522755125 522755092 33 9119562599 1147699 109077860 1491898147 2566003 167152062826463 167149287506014 284931680 83011025 0 6229 0 0 0 0 221408 1603518232 0 82
        xprt:   rdma 0 0 7808 82 0 441542046 441542010 36 7226132207 2519174 111096004 955223347 2676765 105741904708009 105740125663595 275613584 80373159 0 8893 0 0 0 0 149479 1068962768 0 76
        xprt:   rdma 0 0 15018 167 0 508091827 508091764 63 19817677255 36702583 108265928 1258185459 2438516 138247436686102 138246196289594 270162080 74962306 0 13328 0 0 0 0 268433 1368837472 0 66
        xprt:   rdma 0 0 14321 149 0 530246310 530246275 35 9723190432 2392024 111099700 1494204555 2589805 166691166581904 166689567426908 289995492 85067377 0 8010 0 0 0 0 214511 1607864447 0 100
        xprt:   rdma 0 0 7863 84 0 459019689 459019642 47 11809253102 1716688 111825219 1032758664 2564226 114416685286438 114414936423706 290494252 73702102 0 6927 0 0 0 0 134453 1147121864 0 79
        xprt:   rdma 0 0 7702 84 3 497598986 497598931 55 11816221496 3924722 106922130 1382063307 2506108 153967067193941 153965665472218 286222584 84094006 0 5875 0 0 0 0 127347 1491469045 0 66
        xprt:   rdma 0 0 18341 202 0 477721151 477721073 78 15204400959 40562626 106645745 1291616653 3091375 144533696686651 144529688231163 278135800 73821525 0 6795 0 0 0 0 251097 1401327563 0 64
        xprt:   rdma 0 0 8228 90 4 453155092 453155063 29 7884786894 1591225 112197590 1026006338 2742688 114591819605673 114590175821191 275541944 85857259 0 7487 0 0 0 0 143044 1140917892 0 76
        xprt:   rdma 0 0 7843 83 0 446480377 446480324 53 12267986428 2958997 111971246 963162784 2693433 107176282309753 107174637802555 290269096 101100410 0 7825 0 0 0 0 141735 1077797328 0 83
        xprt:   rdma 0 0 7582 86 0 423315608 423315567 41 10197484604 2076993 109207538 785978455 2650354 86090211449474 86088475571312 279912524 87676008 0 7491 0 0 0 0 137533 897807641 0 101
        xprt:   rdma 0 0 7767 84 0 482538465 482538424 41 8935200479 1344778 112200583 1192341640 2644896 132860698423762 132858881459050 273354060 75337030 0 5941 0 0 0 0 127842 1307164736 0 97
        xprt:   rdma 0 0 14526 148 2 537745063 537745007 56 20756072620 3970332320 109539564 1363647371 2503250 148793734936250 148791264145401 291888720 90344151 0 7471 0 0 0 0 211057 1475661285 0 82
        xprt:   rdma 0 0 14300 151 0 495357347 495357316 31 8703101643 1451809 112315311 1303804607 2620502 145680743007170 145678880292235 288046696 98018259 0 7241 0 0 0 0 209396 1418712657 0 139
        xprt:   rdma 0 0 7700 82 0 466611083 466611050 33 8540498291 4082864 114740300 1059770596 2523155 117376668239921 117375375683167 260927576 78437075 0 6691 0 0 0 0 130878 1177008175 1 76
        per-op statistics
                NULL: 16 16 0 640 384 320 11 331 0
             GETATTR: 512052 512052 0 79823516 57349824 107131 612667 751847 0
             SETATTR: 0 0 0 0 0 0 0 0 0
              LOOKUP: 16713 16713 0 3040536 3706344 560 17488 20232 346
              ACCESS: 211705 211705 0 33860920 25404600 37059 229754 283822 0
            READLINK: 0 0 0 0 0 0 0 0 0
                READ: 2654501510 2654501510 0 445911966900 2013540728551504 6347457114 31407021389 37927280438 0
               WRITE: 0 0 0 0 0 0 0 0 0
              CREATE: 0 0 0 0 0 0 0 0 0
               MKDIR: 0 0 0 0 0 0 0 0 0
             SYMLINK: 0 0 0 0 0 0 0 0 0
               MKNOD: 0 0 0 0 0 0 0 0 0
              REMOVE: 0 0 0 0 0 0 0 0 0
               RMDIR: 0 0 0 0 0 0 0 0 0
              RENAME: 0 0 0 0 0 0 0 0 0
                LINK: 0 0 0 0 0 0 0 0 0
             READDIR: 0 0 0 0 0 0 0 0 0
         READDIRPLUS: 0 0 0 0 0 0 0 0 0
              FSSTAT: 56356 56356 0 6243572 9467808 82068 74356 159001 0
              FSINFO: 2 2 0 184 328 0 0 0 0
            PATHCONF: 1 1 0 92 140 0 0 0 0
              COMMIT: 0 0 0 0 0 0 0 0 0

What did you see instead?

mountstats parse error, no detailed nfs mount client stats available
Jul 12 23:51:30 sn3htcw06 node_exporter[2431]: ts=2023-07-12T13:51:30.827Z caller=collector.go:169 level=error msg="collector failed" name=mountstats duration_seconds=0.003460234 err="failed to parse mountstats: invalid NFS protocol \"rdma\" in stats 1.1 statement: [0 0 1093 11 0 442101286 442101259 27 8269562384 1239649 98584262 1057117501 2009661 115761925121512 115760829802743 202123752 57245792 0 4652 0 0 0 0 44440 1157693890 0 78]"
@discordianfish
Copy link
Member

Yeah this seems like something we should support

@dongjiang1989 dongjiang1989 mentioned this issue Aug 8, 2023
Merged
@dongjiang1989
Copy link
Contributor

dongjiang1989 commented Aug 8, 2023 •
edited

@miloaissatu procfs add support nfs rdma protocal parse. prometheus/procfs#558
PTAL!

@dongjiang1989 dongjiang1989 mentioned this issue Oct 17, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
accepted help wanted
Projects
None yet
Milestone
No milestone
3 participants
@discordianfish @dongjiang1989 @miloaissatu