Feature: allow to configure s3 subpath

[oneacl]

Is your feature request related to a problem? Please describe.

Currently a full S3 bucket is required for Mimir - however, in some situations a bucket may need to be shared with other apps (one example would be aws lightsail where there's a fixed cost per bucket).

Describe the solution you'd like

Ability to specify a folder where all mimir data will be stored in a bucket

Additional context

This feature request will also apply for Loki but I'm not sure if I should create a (duplicate) issue there as well - so guidance here would be appreciated.

Thank you.

[dimitarvdimitrov]

I think that's doable and a reasonable request. We already have some abstractions for bucket prefixes. I can work on a draft PR to show how it will look like. It will increase config surface and I'll leave the decision of whether that's ok to a maintainer

edit: opened #1686 to show how this might be done

[oneacl]

wow, that was fast - thank you so much.

should I raise a similar feature request for Loki or would that create duplication?

[pracucci]

should I raise a similar feature request for Loki or would that create duplication?

The two projects currently don't share the storage layer implementation, so you should open an issue to Loki too.

[oneacl]

Raised grafana/loki#5889 - thank you again for all your work, really appreciate it.

[oneacl]

Hi - I noticed that the slash character is not supported so we can only use one single 'directory' (slightly incorrect terminology in the S3 world) - was wondering if it's a technical limitation or just to keep things simple?
I have an environment in which I'm allocated a subpath (a few "directories" down) - so I can't currently use it for mimir.

Thank you.

[dimitarvdimitrov]

the limitation is enforced by this regex. IIRC it was only to keep things simple. I think this can be extended to support nested prefixes as well without any negative implications. PRs are welcome :)

[rpartapsing]

Is there any update on this one?

[dimitarvdimitrov]

@rpartapsing this was released in Mimir 2.2. The config parameter is blocks_storage.storage_prefix same for ruler_storage and alertmanager_storage

[rpartapsing]

@dimitarvdimitrov . I am using Image: grafana/mimir:2.7.1
tried this (what ideally should be the config)

    blocks_storage:
      s3:
        bucket_name: mimir-k8s-dev-aws-dsh
        storage_prefix: blocks

But does not work: field storage_prefix not found in type s3.Config

I also tried what you suggested:

    blocks_storage:
      storage_prefix: blocks
      s3:
        bucket_name: mimir-k8s-dev-aws-dsh

Then I get: unable to successfully send a request to object storage: Access Denied"

while my resource policy should allow it. (it works e.g. for the blocks if I remove the storage_prefix

            "Resource": [
                "arn:aws:s3:::mimir-k8s-dev-aws-dsh/*",
                "arn:aws:s3:::mimir-k8s-dev-aws-dsh/ruler/*",
                "arn:aws:s3:::mimir-k8s-dev-aws-dsh/blocks/*",
                "arn:aws:s3:::mimir-k8s-dev-aws-dsh/alertmanager/*"
            ]

[dimitarvdimitrov]

I don't see anything pointing to the storage prefix feature not working in what you posted. I think opening a discussion here on GitHub or dropping a message in our community slack channel may be better suited for this.

[rpartapsing]

@dimitarvdimitrov
Hi, It seemed my config map grafana-mimir-config didnt get the creds in there correctly. Added them manually and now it works :)
Also because we use IAM role in service accounts (IRSA) and dont allow any user with aws access keys.
I was hoping/assuming that by adding in the sa like this:

serviceAccount:
  # -- Whether to create a service account or not. In case 'create' is false, do set 'name' to an existing service account name.
  create: true
  # -- Override for the generated service account name.
  name: mimir-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${PlatformAwsAccountId}:role/${platformName}-mimir_role

i would not have to enter any access id/secret. But that doesn't work.
Where should I post this feature request, since we basically want to get rid of my manually created user?

[dimitarvdimitrov]

Where should I post this feature request, since we basically want to get rid of my manually created user?

you can open an issue in this repo and add the helm label https://github.com/grafana/mimir/issues/new

[rpartapsing]

@dimitarvdimitrov I wasnt able to put the label, But here it is: #4914