Crashes on Mac M1 arm64

[timja]

Host operating system: output of uname -a

Darwin Tims-MacBook-Pro.local 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:27 PDT 2021; root:xnu-7195.141.2~5/RELEASE_ARM64_T8101 arm64

node_exporter version: output of node_exporter --version

crashes if I try to run the command

➜  node_exporter-1.2.2.darwin-arm64 ./node_exporter --version
[1]    95714 killed     ./node_exporter --version

(but it's 1.2.2 downloaded from github release at https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.darwin-arm64.tar.gz)

node_exporter command line flags

none

Are you running node_exporter in Docker?

no

What did you do that produced an error?

./node_exporter

What did you expect to see?

it to work

What did you see instead?

[1]    95634 killed     ./node_exporter

It works on 1.2.1. The diff is very small: v1.2.1...v1.2.2

prometheus/procfs#402 is likely the cause

[discordianfish]

@timja Can you build prometheus/procfs#404 and see if that fixes it?

[timja]

Is it possible to get a binary? or some instructions on how to get node_exporter built with a different version?

The tests fail on my platform:

>> extracting fixtures
./ttar -C ./ -x -f fixtures.ttar
WARNING sed unable to handle null bytes, using Python (slow).
touch fixtures/.unpacked
>> running all tests
GO111MODULE=on go test   ./...
--- FAIL: TestVM (0.00s)
    vm_test.go:85: unexpected power supply class (-want +got):
          &procfs.VM{
        - 	AdminReserveKbytes:        &8192,
        + 	AdminReserveKbytes:        nil,
        - 	BlockDump:                 &0,
        + 	BlockDump:                 nil,
        - 	CompactUnevictableAllowed: &1,
        + 	CompactUnevictableAllowed: nil,
        - 	DirtyBackgroundBytes:      &0,
        + 	DirtyBackgroundBytes:      nil,
        - 	DirtyBackgroundRatio:      &10,
        + 	DirtyBackgroundRatio:      nil,
        - 	DirtyBytes:                &0,
        + 	DirtyBytes:                nil,
        - 	DirtyExpireCentisecs:      &3000,
        + 	DirtyExpireCentisecs:      nil,
        - 	DirtyRatio:                &20,
        + 	DirtyRatio:                nil,
        - 	DirtytimeExpireSeconds:    &43200,
        + 	DirtytimeExpireSeconds:    nil,
        - 	DirtyWritebackCentisecs:   &500,
        + 	DirtyWritebackCentisecs:   nil,
        - 	DropCaches:                &0,
        + 	DropCaches:                nil,
        - 	ExtfragThreshold:          &500,
        + 	ExtfragThreshold:          nil,
        - 	HugetlbShmGroup:           &0,
        + 	HugetlbShmGroup:           nil,
        - 	LaptopMode:                &5,
        + 	LaptopMode:                nil,
        - 	LegacyVaLayout:            &0,
        + 	LegacyVaLayout:            nil,
        - 	LowmemReserveRatio:        []*int64{&256, &256, &32, &0, &0},
        + 	LowmemReserveRatio:        nil,
        - 	MaxMapCount:               &65530,
        + 	MaxMapCount:               nil,
        - 	MemoryFailureEarlyKill:    &0,
        + 	MemoryFailureEarlyKill:    nil,
        - 	MemoryFailureRecovery:     &1,
        + 	MemoryFailureRecovery:     nil,
        - 	MinFreeKbytes:             &67584,
        + 	MinFreeKbytes:             nil,
        - 	MinSlabRatio:              &5,
        + 	MinSlabRatio:              nil,
        - 	MinUnmappedRatio:          &1,
        + 	MinUnmappedRatio:          nil,
        - 	MmapMinAddr:               &65536,
        + 	MmapMinAddr:               nil,
        - 	NrHugepages:               &0,
        + 	NrHugepages:               nil,
        - 	NrHugepagesMempolicy:      &0,
        + 	NrHugepagesMempolicy:      nil,
        - 	NrOvercommitHugepages:     &0,
        + 	NrOvercommitHugepages:     nil,
        - 	NumaStat:                  &1,
        + 	NumaStat:                  nil,
        - 	NumaZonelistOrder:         "Node",
        + 	NumaZonelistOrder:         "",
        - 	OomDumpTasks:              &1,
        + 	OomDumpTasks:              nil,
        - 	OomKillAllocatingTask:     &0,
        + 	OomKillAllocatingTask:     nil,
        - 	OvercommitKbytes:          &0,
        + 	OvercommitKbytes:          nil,
        - 	OvercommitMemory:          &0,
        + 	OvercommitMemory:          nil,
        - 	OvercommitRatio:           &50,
        + 	OvercommitRatio:           nil,
        - 	PageCluster:               &3,
        + 	PageCluster:               nil,
        - 	PanicOnOom:                &0,
        + 	PanicOnOom:                nil,
        - 	PercpuPagelistFraction:    &0,
        + 	PercpuPagelistFraction:    nil,
        - 	StatInterval:              &1,
        + 	StatInterval:              nil,
        - 	Swappiness:                &60,
        + 	Swappiness:                nil,
        - 	UserReserveKbytes:         &131072,
        + 	UserReserveKbytes:         nil,
        - 	VfsCachePressure:          &100,
        + 	VfsCachePressure:          nil,
        - 	WatermarkBoostFactor:      &15000,
        + 	WatermarkBoostFactor:      nil,
        - 	WatermarkScaleFactor:      &10,
        + 	WatermarkScaleFactor:      nil,
        - 	ZoneReclaimMode:           &0,
        + 	ZoneReclaimMode:           nil,
          }
FAIL
FAIL	github.com/prometheus/procfs	0.493s
ok  	github.com/prometheus/procfs/bcache	0.931s
--- FAIL: TestBlockDevice (0.00s)
    stats_test.go:149: not supported on this platform
FAIL
FAIL	github.com/prometheus/procfs/blockdevice	0.799s
--- FAIL: TestFSBtrfsStats (0.00s)
    get_test.go:37: failed to parse Btrfs stats: not supported on this platform
FAIL
FAIL	github.com/prometheus/procfs/btrfs	0.547s
ok  	github.com/prometheus/procfs/internal/fs	0.674s
ok  	github.com/prometheus/procfs/internal/util	0.415s
ok  	github.com/prometheus/procfs/iscsi	0.157s
ok  	github.com/prometheus/procfs/nfs	0.283s
ok  	github.com/prometheus/procfs/xfs	0.919s
FAIL
make: *** [common-test] Error 1

[IonatanMocan]

@timja I had the same problem. The solution is to download node_exporter with Homebrew: https://formulae.brew.sh/formula/node_exporter

brew install node_exporter
node_exporter --version

# node_exporter, version 1.2.2 (branch: , revision: )
#  build user:       Homebrew
#  build date:       
#  go version:       go1.16.6
#  platform:         darwin/arm64

Then simply run node_exporter. Node Exporter will run on http://localhost:9100 and will expose metrics at http://localhost:9100/metrics

I hope this solves your problem.

[discordianfish]

Just tested it on my new Mac M1 and master seems to work, so I'll consider this done.
You can try building master by running 'go build' directly to get around tests. There are some known issues with tests on macos due to the build flags #1545

[joeblubaugh]

I'm experiencing this problem on release 1.3.1 @discordianfish . Would you like me to open a new issue?

[sdarwin]

Here is something that I observe today on a mac arm64 machine.
Is it the same issue?

# uname -a
Darwin 61145.local 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:47:26 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T8101 arm64

61145:bin root# ./node_exporter --version
Killed: 9

[discordianfish]

Are you sure this is the same problem? Or is this just mac preventing unknown binaries to run?

[sdarwin]

@discordianfish

Or is this just mac preventing unknown binaries to run?

Not sure.

• On previous versions of Mac OSX, node_exporter ran fine.
• On version 12.4 of Mac OSX and x86_64 architecture, node_exporter runs fine.
  uname -a Darwin mac7 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64 x86_64
• on arm64, the error appeared. Do you have contradictory evidence, because it doesn't show errors for you on arm64?

[discordianfish]

Unfortunately my M1 is in repair right now and I can't try.. I'll do if it's back

[milljm]

We seem to be experiencing the same issue (node_exporter-1.3.1.darwin-arm64.tar.gz):

$ ./node_exporter --version
Killed: 9

$ uname -a
Darwin miniarm4 21.6.0 Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:35 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T8101 arm64

[milljm]

The fix for us, was to install a go environment and build node_exporter manually. For sake of completeness, these were our steps:

• Install Conda Mambaforge
• Using Conda, install go, and the necessary compilers
  • mamba install go clangxx_osx-arm64 clang_osx-arm64
• build node_exporter

Because building node_exporter was simple, have you guys thought to provide this binary via conda-forge? I might decide to do so if y'all feel like helping maintain it :)