Alert rule panic; runtime error: invalid memory address or nil pointer dereference

[robinkanters]

What happened:

• Grafana quits unexpectedly from time to time, logging alert rule panic (stacktrace below)

What you expected to happen:

• For it not to crash 😉

How to reproduce it (as minimally and precisely as possible):
I have the alert rule that I pasted below, if I click the Refresh button, it shows an error popup in the top right, I think this might be the cause. The screenshot below is of the rule with the uid from the stack trace.

Anything else we need to know?:

• Might be related to Failed to evaluate queries and expressions: alert rule panic #58703
• Stack trace

Environment:

• Grafana version: v9.3.2 (21c1d14)
• Data source type & version: AlertManager, Athena, cloudwatch, ElasticSearch, OpenSearch, Prometheus
• OS Grafana is installed on: Linux (AWS ECS)
• User OS & Browser: EndeavourOS; Firefox 108.0.2
• Grafana plugins:
  • Amazon Athena
  • Direct Input
  • Grafana OnCall
  • OpenSearch
• Others:

[mellieA]

@robinkanters Does this same query complete as expected in the explore view? If so, are you able to share a sample of what the return data looks like?

[yuri-tceretian]

I can confirm the bug. The Cloudwatch plugin returns no data result as a data frame with a single value. After #55347 that value gets converted to a single value NoData

grafana/pkg/expr/commands.go

Lines 175 to 176 in 2d20c8d

	case mathexp.NoData:
	newRes.Values = append(newRes.Values, v.New())

then when math expression tree is executed, the is_nan method walkFunc produces a slice with a nil element.

grafana/pkg/expr/mathexp/exp.go

Lines 491 to 517 in 05fd7eb

	func (e *State) walkFunc(node *parse.FuncNode) (Results, error) {
	var res Results
	var err error
	var in []reflect.Value
	for _, a := range node.Args {
	var v interface{}
	switch t := a.(type) {
	case *parse.StringNode:
	v = t.Text
	case *parse.VarNode:
	v = e.Vars[t.Name]
	case *parse.ScalarNode:
	v = NewScalarResults(e.RefID, &t.Float64)
	case *parse.FuncNode:
	v, err = e.walkFunc(t)
	case *parse.UnaryNode:
	v, err = e.walkUnary(t)
	case *parse.BinaryNode:
	v, err = e.walkBinary(t)
	default:
	return res, fmt.Errorf("expr: unknown func arg type: %T", t)
	}
	if err != nil {
	return res, err
	}
	in = append(in, reflect.ValueOf(v))
	}

because functin isNill cannot handle NoData result correctly

grafana/pkg/expr/mathexp/funcs.go

Lines 100 to 114 in 040ce40

	func isNaN(e *State, varSet Results) (Results, error) {
	newRes := Results{}
	for _, res := range varSet.Values {
	newVal, err := perFloat(e, res, func(f float64) float64 {
	if math.IsNaN(f) {
	return 1
	}
	return 0
	})
	if err != nil {
	return newRes, err
	}
	newRes.Values = append(newRes.Values, newVal)
	}
	return newRes, nil

Then that result is provided to unary operator ! that crashes because nil it does not match any type in walkUnary matcher

grafana/pkg/expr/mathexp/exp.go

Line 124 in 05fd7eb

}

and in the default branch it tries to call method Type() on the nil, which causes a crash.

cc @kylebrandt

[Roberto6969]

Update from 9.4.0-960xxx to 9.4.0-96993pre caused same error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x27a2d8c]

How can I resolve this?

Thank You.

[Roberto6969]

Update: Update from 9.4.0-96993pre to version=9.4.0-97045pre didn't solve problem.

Is this error somehow in relation with:

logger=ngalert.multiorg.alertmanager t=2023-01-18T11:35:46.792823325Z level=error msg="unable to create Alertmanager for org" org=6 error="unable to initialize the notification log component of alerting: proto: wrong wireType = 2 for field Idx"

[yuri-tceretian]

Update: Update from 9.4.0-96993pre to version=9.4.0-97045pre didn't solve problem.

Is this error somehow in relation with:

logger=ngalert.multiorg.alertmanager t=2023-01-18T11:35:46.792823325Z level=error msg="unable to create Alertmanager for org" org=6 error="unable to initialize the notification log component of alerting: proto: wrong wireType = 2 for field Idx"

@Roberto6969 this is not related. Main seems to be broken (panics when starts with alertmanager configuration that were created by previous version). We're fixing this right now

[Roberto6969]

@yuri-tceretian Great - I appreciate your effort.