Need a way to expose Loki on a read only basis to a separate cluster in the same vpc

[sarasensible]

It looks like there is no way to configure Loki to run as a lightweight read only client from S3. My use case is I have a main cluster that is sending logs to s3 with its own Loki instance and I would like to use Loki in a separate cluster to read those logs from the same s3 bucket to be a datasource for Grafana. Since I can't really do that - see https://community.grafana.com/t/read-only-cluster-separated-from-main-cluster/77219/7 which states that you have to enable the writers for the readers to function - and I don't have the ability to configure a remote read endpoint - this PR is stale grafana/loki#7324 - I need to expose the Loki query frontend service as an internal load balancer so I can query it from the other cluster as a Grafana datasource. Unfortunately the Loki helm chart does not have the ability to configure service-specific annotations or the service type.

Would love to have some input from the community on this topic - short of deploying a multi cluster service mesh for this very simple use case, this use case looks very unsupported but it seems like a reasonable topology to me.

[sarasensible]

Just want to confirm an internal load balancer for the query-frontend does enable this.

[sarasensible]

Update: We are going to go the route of keeping the logging infrastructure in the same cluster for now since it seems like there are many hoops like this one to jump through to separate it out. I'll close it out with the understanding that this would still be a desirable feature to support.