Allow Prometheus to use ReadRequest_STREAMED_XOR_CHUNKS when calling remote read API.

[bwplotka]

#5703 extended remote read protocol and added ReadRequest_STREAMED_XOR_CHUNKS

Prometheus implements remote read API, but it can also be configured to use/call other remote read API. Enable streaming remote read if the server supports it.

AC:

• Prometheus remote read client is capable to use ReadRequest_STREAMED_XOR_CHUNKS

Similar client code for inspiration: thanos-io/thanos#1268

[YaoZengzeng]

I'd like to fix this issue 😃 @bwplotka

Maybe we can solve this problem by following steps:

1. Extend the Read method of Client in storage/remote/client.go to support both SAMPLE and STREAM, the return type changed to *prompb.QueryResult to *http.Response would be better.

2. Refactor FromQueryResult to decode both type of response and expose unified storage.SeriesSet.

3. Encapsulate the chunks into some struct to make it implement storage.SeriesSet. Actually the implementation method is similar to chunkSeries stuff in tsdb/query.go. In addition, this step may easy to be implemented if Add ChunksIterator method to Series interface. #5882 being merged :)
   ( At first I‘d encapsulate the chunks to make the code run)

My initial idea is probably like this, any comments? @bwplotka

Test is also needed and now I'm trying to figure out how to do it properly.

[bwplotka]

Sorry for lag and thanks @YaoZengzeng

Yes that makes sense to me 👍

I think step 2 would be the most work, especially packing SAMPLED method to SeriesSet, but I guess we can't really avoid this.

For Streaming be careful as we might need to implement something like this we did in Thanos in SeriesSet.

[YaoZengzeng]

@bwplotka Thanks for your comments and I'll fix this issue step by step 😃

[YaoZengzeng]

@bwplotka Please review #5974 again, otherwise the fixing of this issue can't move on 😂

[bwplotka]

Sorry for massive lag, proposed solution (: Also reach me on IRC bwplotka or CNCF slack channel -> bwplotka if you need to ping me - bit spammed on email.

[bwplotka]

Hi @YaoZengzeng this would be nice to have (: Any progress? What are the blockers? 🤗

[Sudhar287]

Hello @YaoZengzeng. If you're not going to continue this, can I please take your work forward?

[YaoZengzeng]

@Sudhar287 Yes, Please :)

[Sudhar287]

Thank you for the/your quick response. :)

[bwplotka]

@Sudhar287 any update maybe? (: We kind of slowly need that for various tooling. (:

If not I think it might be good Community Bridge issue. cc @khyatisoneji

[Sudhar287]

Hello @bwplotka! It seems that this PR #6042 cannot proceed until #6747 is completed according to this. I was waiting for that to be finished first...Do you have a different proposal? :)

[bwplotka]

Wow, I think there is misunderstanding, I don't think anything is blocked and we can start working on it right way.

[bwplotka]

I commented #6042 (comment)

I don't think there was anything blocked... Sorry for confusion 🤦 We are definitely good to go.

We kind of need this as we wrote on Thanos tool for serving CSV / JSON as some our API and we want to extend it to remote-read. Would be nice if we can chunks to Prometheus though 🤗

[Sudhar287]

Thanks for the response and clarification. I'll start working on it! :)

[Sudhar287]

fyi, working on this actively. Will have the PR next week or so. :)

[bwplotka]

Awesome! (((:

…

On Thu, 28 May 2020 at 19:03, Sudhar287 ***@***.***> wrote: fyi, working on this actively. Will have the PR next week or so. :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5926 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3O5RQLK54J7EUIX2K33RT2RPHANCNFSM4INFY42A> .

[Tanemahuta]

Hello there,

I have an implementation ready for which I refactored to ReadClient as follows:

type ReadClient interface {
	Read(ctx context.Context, stream bool, queries ...*prompb.Query) (ReadResponse, error)
}

// ReadResponse is the interface for an iterable response
type ReadResponse interface {
	// Next returns true in case there is another time series
	Next() bool
	// At returns a partial or full TimeSeries or nil
	At() *TimeSeries
	// Err denotes the error that iteration as failed with. When an error occurs, set cannot continue to iterate.
	Err() error
	// IsPartial returns true if the TimeSeries in this response are chunked
	IsPartial() bool
}

// TimeSeries represents a partial or complete time series from a query
type TimeSeries struct {
	// Labels denote the identifier of the TimeSeries
	Labels []prompb.Label
	// Samples denote the time stamps and values
	Samples []prompb.Sample
	// QueryIdx denotes the index of the original query
	QueryIdx int64
}

This also solves // TODO: Support batching multiple queries into one read request, as the protobuf interface allows for it.

Is the sketched solution suitable for you? - If so, I will prepare a PR.

[bwplotka]

It looks great @Tanemahuta, but we already have another PR #8351 (review)

Maybe you can help us review it and maybe give your design as suggestions in the comments? 🤗

[Tanemahuta]

Will do. This changes the complete design and breaks the API though.

[detailyang]

@bwplotka any update? read STREAMED_XOR_CHUNKS response is important to reduce memory overhead and query latency:)

[hpple]

Hi, any updates on this issue?

[beorn7]

Hello from the bug scrub.

@bwplotka we believe the ball is in your court reviewing #11379 .