Querying managed Prometheus metrics error: json: cannot unmarshal object into Go struct field HttpBody.data of type string

[veggiemonk]

Maybe it is something I did wrong, I couldn't find an example on how to query Google Managed Prometheus programmatically.

The error occurs because the Data field of HttpBody of type string which cause decoding JSON to fail while the request has succeeded (status 200). See https://github.com/googleapis/google-api-go-client/blob/main/monitoring/v1/monitoring-gen.go#L1518
After checking the header of the response, Content-Type is application/json

Could you recommend a way to query metrics without a front-end ?

Much appreciated

Environment details

• Programming language: Go
• OS: linux/amd64
• Language runtime version: go1.21.5
• Package version: google.golang.org/api v0.154.0

Steps to reproduce

1. Replace "XXX_project_id_XXX" with a project ID where Google Managed Prometheus is enabled.

package main

import (
	"context"
	"fmt"
	"testing"
	"time"

	monv1 "google.golang.org/api/monitoring/v1"
)

func main() {
	fmt.Println("Hello, 世界")
}

func TestPromMetrics(t *testing.T) {
	ctx := context.Background()
	svc, err := monv1.NewService(ctx)
	if err != nil {
		t.Fatal(err)
	}
	s := monv1.NewProjectsLocationPrometheusApiV1Service(svc)
	q := &monv1.QueryRangeRequest{
		Query:   "agones_gameservers_total",
		Start:   time.Now().UTC().Add(-5 * time.Minute).Format(time.RFC3339),
		End:     time.Now().UTC().Format(time.RFC3339),
		Step:    "60s",
		Timeout: "60s",
	}
	resp, err := s.QueryRange("projects/XXX_project_id_XXX", "global", q).Do()
	if err != nil {
		t.Fatal(err, resp)
	}
	t.Logf("%#v", resp)
}

2. Set up Application Default Credentials
3. Run go test -run TestPromMetrics

The full error message:

--- FAIL: TestPromMetrics (0.23s)
    metrics_test.go:39: json: cannot unmarshal object into Go struct field HttpBody.data of type string <nil>
FAIL
exit status 1

[veggiemonk]

By forming the query manually it works:

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"net/url"
	"path"
	"testing"
	"time"

	monv1 "google.golang.org/api/monitoring/v1"
	"google.golang.org/api/option"
	apihttp "google.golang.org/api/transport/http"
)

func main() {
	fmt.Println("Hello, 世界")
}

const projectID = "XXX"

func TestPromMetrics(t *testing.T) {
	opts := []option.ClientOption{
		option.WithScopes("https://www.googleapis.com/auth/monitoring.read"),
	}
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	targetx := fmt.Sprintf("https://monitoring.googleapis.com/v1/projects/%s/location/global/prometheus", projectID)
	transport, err := apihttp.NewTransport(ctx, http.DefaultTransport, opts...)
	if err != nil {
		t.Fatal("create HTTP transport", "err", err)
	}
	client := http.Client{Transport: transport}
	u, err := url.Parse(targetx)
	if err != nil {
		t.Fatal("parse target URL", "err", err)
	}
	u.Path = path.Join(u.Path, "/api/v1/query_range")
	q := &monv1.QueryRangeRequest{
		Query:   "up",
		Start:   time.Now().UTC().Add(-5 * time.Minute).Format(time.RFC3339),
		End:     time.Now().UTC().Format(time.RFC3339),
		Step:    "60s",
		Timeout: "60s",
		// ForceSendFields: nil,
		// NullFields:      nil,
	}
	body, err := json.Marshal(q)
	if err != nil {
		t.Fatal("marshal query", "err", err)
	}

	newReq, err := http.NewRequestWithContext(ctx, http.MethodPost, u.String(), bytes.NewReader(body))
	resp, err := client.Do(newReq)
	if err != nil {
		t.Fatal("do request", "err", err)
	}
	defer resp.Body.Close()
	res, err := io.ReadAll(resp.Body)
	if err != nil {
		t.Fatal("read response", "err", err)
	}
	t.Log(string(res))
}

[quartzmo]

@veggiemonk,

Thank you for reporting this issue.

I couldn't find an example on how to query Google Managed Prometheus programmatically.

I looked through the samples in https://github.com/GoogleCloudPlatform/golang-samples/tree/main/monitoring but couldn't find anything for queries.

[quartzmo]

@veggiemonk,

Thank you for providing the manual workaround.

Question: Can https://pkg.go.dev/cloud.google.com/go/monitoring be used for this use case?

Unfortunately, due to the maintenance mode status of this project, I won't be able to prioritize reproducing this issue right now. If you can still reproduce it, can you by any chance provide a deeper analysis of where the json error is occurring in the client library and what the fix might be?

[veggiemonk]

@quartzmo

Question: Can https://pkg.go.dev/cloud.google.com/go/monitoring be used for this use case?

That package is for dealing with metrics stored in Cloud Operations as far as I understood. The metrics I used are store in Google Managed Prometheus which, apparently, is a different APIs.

can you by any chance provide a deeper analysis of where the json error is occurring in the client library and what the fix might be?

The error occurs because the Data field of HttpBody of type string which cause decoding JSON to fail while the request has succeeded (status 200). See https://github.com/googleapis/google-api-go-client/blob/main/monitoring/v1/monitoring-gen.go#L1518
The field should be of type []byte as the error states:
json: cannot unmarshal object into Go struct field HttpBody.data of type string

That's the fix as far as I can tell.

[quartzmo]

@veggiemonk Thank you for the link to the incorrect type. This is helpful, I will continue to investigate.