Storage: Allow setting a constant prefix for all created keys

[aschleck]

What this PR does / why we need it:

Adds a new option under the aws stanza named key_prefix. This is useful if you have many Loki installations and do not want to create a million buckets. This is different than compactor.shared_store_key_prefix because it also affects eg the loki_cluster_seed.json file.

Which issue(s) this PR fixes:
Fixes #5889

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• CHANGELOG.md updated
  • If the change is worth mentioning in the release notes, add add-to-release-notes label
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR

[CLAassistant]

All committers have signed the CLA.

[salvacorts]

@aschleck Thank you for your PR! We really appreciate it.

I think being able to set a prefix may make sense for other storages than just S3. IMO, if we want to implement this, we should support it for all (or most) storage backends. Would you be open to giving that a try? Happy to change my opinion tho :)

Here's a PR for Mimir doing something similar. From what I can see, looks like Mimir only support alpha-numeric prefixes, not path-like prefixes (e.g. some/prefix)
grafana/mimir#1686

[salvacorts]

To add to my comment above, looks like (similarly to mimir), we already have a prefixed client:
https://github.com/grafana/loki/blob/main/pkg/storage/stores/indexshipper/storage/prefixed_object_client.go#L16-L18
(It's currently in the storage pkg, we may need/want to move it to the client pkg along with object_client.go)

I think we can take the same approach as Mimir and use it when creating the object client as they do here:
https://github.com/grafana/mimir/pull/1686/files#diff-a17652466a6d455977112abb7636d8ded2f5a2e66fd8727d9e1da6759ef3a886R143-R145

[aschleck]

Thank you Salva for looking at this! To make sure I understand, the request is:

• move the prefixed_object_client.go into pkg/storage/chunk/client
• wrap the clients created in

  loki/pkg/storage/factory.go

  Line 578 in 174ef44

  func NewObjectClient(name string, cfg Config, clientMetrics ClientMetrics) (client.ObjectClient, error) {

  with the prefixed object client
• add top level key object_prefix to https://grafana.com/docs/loki/latest/configuration/#storage_config
  Is that correct? I've updated the PR.

[aschleck]

I can't figure out why this integration test is failing. I know it's something in this PR, it passes on the parent commit, but I'm not seeing the issue. Current theory is maybe the compactor is stuck but no idea why. Any suggestions?

[jcdauchy-moodys]

Thanks, I am currently testing on a 2.8.3 branch + your changes, it builds and run.

[aschleck]

@salvacorts do you know why the integration test is failing?

[cstyan]

Not sure how this is intended to work, but if it circumvents the schema code and stores all data under a single prefix then this is a giant potential footgun for s3 users that should come with documentation and a warning.

[aschleck]

@cstyan thank you for thinking about this! My expectation is that changing this would be equivalent to changing the bucket name in the S3 config, so it's just as much as a footgun to the schema stuff as switching the bucket option would be. Does that make sense? I'm certainly happy to make the documentation clearer on that front.

In my original PR this was actually an S3-specific option and lived at the same level as the bucket option, which in light of your point may actually be a better solution. It also passed the CI tests so I kind of like reverting back to it. What do you think of the approach in aschleck@8a71b0e?

[cstyan]

@aschleck Would you mind elaborating as to your use case for wanting to have a constant prefix?

Am I misunderstanding that this would mean all objects are stored under just a single prefix, such as bucket-name/<chunk id> instead of using the schema config to store things like bucket-name/<tenant-id>/<stream-fingerprint>/<chunk-id>.

[aschleck]

@cstyan I am not an expert on Loki's codebase, so please correct me if my change is not doing what I intend. I am not trying to force everything into bucket-name/<chunk id>. I really appreciate you looking at this PR.

My particular problem is that I have a ton of Kubernetes clusters, and I want to run a separate, isolated Loki instance in each of them. However, given there are only a finite number of AWS S3 buckets allowed per account, I want to make all of these different Loki instances share the same bucket but in isolated parts of it. So my goal is to make objects write into bucket-name/<configurable prefix, eg cluster name>/<tenant-id>/<stream-fingerprint>/<chunk-id>. Does this make sense?

My understanding, and the understanding of others in #5889, is that currently one Loki instance basically takes over an entire bucket. And you get eg loki_cluster_seed.json in the root of your bucket.

[aschleck]

I am trying to do for Loki what grafana/mimir#1686 did for Mimir and thanos-io/thanos#5337 did for Thanos.

[aschleck]

I had some time today and fixed the integration test problem. @salvacorts and @cstyan please let me know if you have further concerns or if we can merge this.

[aschleck]

Oh! That's super interesting, thank you for following up. I would never have thought about that case so appreciate you finding it.

What do you think is the best mitigation? Should we error out if there's a leading / or silently strip it? Personally I think stripping it makes sense. @ningyougang @slim-bean

[aschleck]

Changed the logic to strip leading /s. @slim-bean what do you think?

[aschleck]

Thank you @slim-bean! Do you think you can approve this PR and merge it? What is left to do here?

[aschleck]

@slim-bean @salvacorts @JStickler can we get this PR approved and merged?

[DimitriosNaikopoulos]

Hey all!! Great work on this feature 👏 Do we know if this solution is ready to be merged? Happy to assist if there is anything left to do 😄

[sandeepsukhani]

nice work @aschleck!
The changes look good to me. I think we need to also make the changes in the code initializing storage for ruler?

[chaudum]

I think we need to add some documentation around what the caveats are when you use a single storage bucket with multiple Loki installations under different prefixes.
The main problem is that rate limits on the bucket caused by one Loki installation affects all other Loki installations that use the same cluster.

[chaudum]

An area where the global prefix could cause issues is the caching object client of the index shipper, because it splits object keys by delimiter and checks the length of the resulting parts.

https://github.com/grafana/loki/blob/main/pkg/storage/stores/indexshipper/storage/cached_client.go#L82-L120

Maybe you could add a test case with a configured global prefix to see how it behaves.

[aschleck]

I rebase this PR. CI says "Build is blocked, please, contact repo admin in order to proceed", I'm not sure why but I don't think it's my code.

An area where the global prefix could cause issues is the caching object client of the index shipper, because it splits object keys by delimiter and checks the length of the resulting parts.

https://github.com/grafana/loki/blob/main/pkg/storage/stores/indexshipper/storage/cached_client.go#L82-L120

Maybe you could add a test case with a configured global prefix to see how it behaves.

I wouldn't expect this to cause any problems. The prefix is internal to the prefixed client so this caching client shouldn't see it. I added this test case just in case and it passes.

I think we need to add some documentation around what the caveats are when you use a single storage bucket with multiple Loki installations under different prefixes. The main problem is that rate limits on the bucket caused by one Loki installation affects all other Loki installations that use the same cluster.

Most users aren't going to hit the S3 concurrency limits where this matters, so I'm not quite sure where you envision putting this. I would imagine it goes in a "Loki tuning guide" but I don't think one exists. This feels like something that might belong in a follow-up PR?

[JStickler]

CI says "Build is blocked, please, contact repo admin in order to proceed", I'm not sure why but I don't think it's my code.

@aschleck Our security team has added a layer of protection to the repo so that any PRs coming from a fork needs to have their CI builds approved by an admin every time CI is run. It has slowed things down considerably, and we're trying to find a different solution to the security problem. In the meantime, the team has to manually trigger CI builds. I triggered one just now, before I saw that your branch has conflicts that need to be resolved.

[aschleck]

Ahhh another merge conflict, you folks are prolific :) I just rebased it again. Thank you for triggering the CI build!

[slim-bean]

Thank you everyone for all the involvement here and huge thanks to @aschleck for your patience and persistence with this change ❤️ 🎉