gRPC service scraping support

[jvmlet]

Proposal

Nowadays, gRPC services are getting more adopted and preferred over traditional REST services.
To expose gRPC service metrics, it usually requires embedded web server running aside to host REST endpoint configured to be scrapped by Prometheus.
This propose is to develop proto file that describes the rpc method, it's request and response messages.
Prometheus team will use the gRPC client to scrap the metrics exposed by gRPC server developed based on the same proto file.

[brian-brazil]

See #4039, scraping over standard HTTP is the way we do things and OpenMetrics also is intended to work over HTTP. Creating a brand new protocol to be integrated not just in Prometheus but effectively in all other metrics monitoring vendors would be a lot of effort and complexity, as now everyone would have to support a brand new standard.

I would suggest exposing a HTTP endpoint.

[jvmlet]

There are always proprietary features to provide your clients with in addition to standards . This is what distinguishes THE product from just another standard implementation , no ? ;-) May be some day (in the nearest future? ), it will become standard...
Have a look at Hibernate validator specific annotations... they are deprecated in favor of standard @NotBlank etc...

[ST-DDT]

I really like the idea of directly publishing grpc metrics instead of serving another port just for the http metrics.

If you are hesitant about adding additional maintenance overhead, then maybe an officially agreed on grpc interface specification would be a good way to start.
With that interface in place there could be sidecar-containers/translating proxies/aggregators that translate the grpc output to http, and if it gains widespread popularity then the grpc support could be added directly to prometheus.
However without an official "approval" these projects are doomed to be personal/per company only.
(Another alternative would be supporting scrape plugins)

[roidelapluie]

Hello,

OpenMetrics has an official proto: https://github.com/OpenObservability/OpenMetrics/blob/master/proto/openmetrics_data_model.proto

For the general discussion about gRPC vs HTTP, the developers mailing list is better fitted than this issue as the consequences are way above the single prometheus server: https://groups.google.com/forum/#!forum/prometheus-developers

[bwplotka]

Interesting discussion I love the overall idea TBH.

As Prometheus maintainer, I would love to continue this discussion - channel does not matter much for me (: Up to whatever is more convenient to everyone.

Per gRPC metric scraping:

I am also sometimes sad that I have to add an HTTP port to my gRPC services. But the truth is that even if we have Prometheus on gRPC there is a need for HTTP port. The reason is that it's useful to have various other debug endpoints (sometimes human readable) like go pprof, index page, or even debug UI (e.g we have that in Thanos). Wonder if you don't have such problem? 🤔

Also what's the harm in adding another port, especially if you other sources?

Additionally, have you looked on option to reuse same socket/port to handle both HTTP and gRPC? It go it's super simple, I would expect same from Java etc Have you tried that? 🤗 So then you can have single port with both HTTP and gRPC handlers!

[ST-DDT]

The reason is that it's useful to have various other debug endpoints

You can do the same with grpc, you just need a grpc viewer instead of a browser.
You can discover the endpoints and contents using protobuf's/grpc's reflection api (And you have the additional (grpc-native) watch option).

Also what's the harm in adding another port, especially if you other sources?

The second port is not the problem.
It forces me to add another server library (e.g. tomcat) in addition to grpc's netty (=two server libraries).
Which in combination wastes a lot of precious resources (startup time, cpu/threads, ram, image size).
I know that this is rather "our" problem than this project's, but still troublesome.

Additionally, have you looked on option to reuse same socket/port to handle both HTTP and gRPC? It go it's super simple, I would expect same from Java etc Have you tried that? 🤗 So then you can have single port with both HTTP and gRPC handlers!

I haven't found a way to serve the http-servlet endpoints via netty...
( I also haven't found a way to get the netty server from/into grpc-java )
Still our problem, but grpc+protobuf is extremely fast and less prone to parsing errors/path mismatch issues and a well known internet api standard.

OpenMetrics has an official proto: https://github.com/OpenObservability/OpenMetrics/blob/master/proto/openmetrics_data_model.proto

Is there a grpc interface for/using that yet?

[roidelapluie]

OpenMetrics has an official proto: https://github.com/OpenObservability/OpenMetrics/blob/master/proto/openmetrics_data_model.proto

Is there a grpc interface for/using that yet?

I think that openmetrics specs do not deal with the transport.

cc @RichiH for confirmation.

[jvmlet]

@bwplotka, bloomrpc is postman-like alternative for grpc.

[bwplotka]

I don't mean debugging gRPC. I mean debugging my binary.

For example:

• UI that shows the configuration flags
• Pprof endpoints (e.g in Go): heap, cpu etc
• fgprof endpoints
• Events/Tracers

The second port is not the problem.
It forces me to add another server library (e.g. tomcat) in addition to grpc's netty (=two server libraries).
Which in combination wastes a lot of precious resources (startup time, cpu/threads, ram, image size).

Thanks for mentioning this, I was not aware. This is crazy indeed. In Go it's much simple: You can create such a simple HTTP server by just using a standard library.

But I still cannot believe you need in Java, Tomcat for sending Status Code 200 and few bytes as HTTP response? 🤔

I can see many server integrations: https://github.com/prometheus/client_java - all of them are that heavy? I am pretty sure you can find one that is lighter than gRPC lib itself.

I know that this is rather "our" problem than this project's, but still troublesome.

No no, no one says it's only your problem. Java Ecosystem is a major player here, and gRPC is popular too, let's make sure this is smooth 🤗

However, do you think introducing gRPC endpoint only because HTTP is heavy is a good reason? gRPC is using HTTP directly so this sounds wrong 🤔 Maybe we can improve "heaviness" in https://github.com/prometheus/client_java itself? Can we create an issue about this there?

The problem with gRPC endpoint is that it might be familiar for us and you but the case of forcing everything gRPC is quite rare. And if we would add gRPC support then people with GraphQL only and thousands other protocol might want to add their too. OpenMetrics is simple HTTP on purpose. (:

[bwplotka]

Still our problem, but grpc+protobuf is extremely fast and less prone to parsing errors/path mismatch issues and a well known internet api standard.

I wished that was true btw, but it's not as well known as we might want. ):

[jvmlet]

In java you can do web development by referencing j2ee standard libraries, but in runtime you need to provide the j2ee servlet container implementation : some of them lightweight, some of them not. Another option - create socket and parse/format HTTP traffic by your own (no one does it)
The problem with HTTP is not because HTTP is heavy (only), but because of the selected technology stack.
The proposal is to allow user to be able to provide HTTP or GRPC endpoint and not to force him to use only one of them.

[roidelapluie]

Thank for your input. We have looked at this during our bug scrub. We have added this to the dev summit agenda. We also welcome people to create a discussion about this on the developers mailing list.

[beorn7]

Sorry for the long delay. This topic was finally discussed at the dev-summit, but without conclusive results. At the summit, there was nobody present championing gRPC scrapes, but neither was there strong opposition. Mostly, the mood was that this is indeed "not as easy as it looks" and we need more input and good knowledge of both gRPC and the implications for standardized exposition (OpenMetrics and such).

To get the people involved that actually have an informed opinion either way, we need a wider discussion. Therefore, this issue was converted into a GH discussion. Also, feel free to discuss the issue at the developers mailing list: https://groups.google.com/forum/#!forum/prometheus-developers

[jvmlet]

Thanks for the update. May be someone form grpc-java authors will be able to provide their input and thoughts. Tagging @ejona86

[jvmlet]

BTW, grpc already provides health check standard service definition. Consul, for example. allows to register http or grpc endpoint for health checking ....