Add MaxNoncurrentVersions to NoncurrentExpiration action #13580

krisis · 2021-11-04T06:33:26Z

Description

This unit allows users to limit the maximum number of noncurrent versions of an
object.

Motivation and Context

To limit number of noncurrent versions of an object.

How to test this PR?

#!/bin/bash

ALIAS=$1

./mc mb $ALIAS/mybucket
./mc version enable $ALIAS/mybucket
for i in {1..10}
do
   ./mc cp /etc/issue $ALIAS/mybucket/obj-1
done
cat <<EOF | ./mc ilm import $ALIAS/mybucket
{
    "Rules": [
        {
            "ID": "test-max-noncurrent",
            "Status": "Enabled",
            "Filter": {},
            "NoncurrentVersionExpiration": {
                "MaxNoncurrentVersions": 5
            }
        }
    ]
}
EOF

bash -x <above-script.sh> myminio

Now, perform mc cp to upload a new version and trigger the removal of excess noncurrent versions as configured.

This requires mc compiled with minio/minio-go#1580

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Optimization (provides speedup with no functional changes)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation updated
Unit tests added/updated

vadmeste

Some comments

internal/bucket/lifecycle/lifecycle.go

cmd/object-handlers-common.go

internal/bucket/lifecycle/expiration.go

cmd/object-handlers-common.go

harshavardhana · 2021-11-09T16:24:20Z

@klauspost the main idea here was to deliver this fix first and then subsequently take time to merge the format change and optimize this part further.

This is also an opt-in where Max value is only set this should tax the Put() with reasonable reasoning.

krisis · 2021-11-10T04:41:00Z

@klauspost To summarize the changes I have made in this PR since your last comment, I have implemented a scanner callback (as suggested) which removes noncurrent versions keeping only most recent MaxNoncurrentVersions configured.

I have retained the less performant codepath invoked by the upload handlers. This is not active by default. It can be enabled via an env var (MINIO_API_ILM_INLINE). This is for those workloads where the same object is updated many times over within a full namespace scan (16 cycles). This shall be used only when we observe such a workload.

Once #13573 is merged, we can improve performance of this codepath, or even remove it.

cmd/object-handlers-common.go

cmd/data-scanner.go

cmd/object-handlers-common.go

PTAL

harshavardhana · 2021-11-18T20:24:39Z

@krisis can you rebase this PR on top of the newer changes?

cmd/data-scanner.go

cmd/storage-datatypes.go

cmd/api-errors.go

cmd/storage-rest-common.go

cmd/xl-storage-format-utils.go

harshavardhana

One comment everything else LGTM and tested

This unit allows users to limit the maximum number of noncurrent versions of an object. To enable this rule, ``` cat <<EOF | ./mc ilm import myminio/mybucket { "Rules": [ { "ID": "test-max-noncurrent", "Status": "Enabled", "Filter": { "Prefix": "user-uploads/" }, "NoncurrentVersionExpiration": { "MaxNoncurrentVersions": 5 } } ] } EOF

minio-trusted · 2021-11-20T01:40:56Z

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-gateway-nas.sh	✔️
mint-compress-encrypt-dist-erasure.sh	✔️
mint-pools.sh	✔️
Deleting image on docker hub
Deleting image locally

This unit allows users to limit the maximum number of noncurrent versions of an object. To enable this rule you need the following *ilm.json* ``` cat >> ilm.json <<EOF { "Rules": [ { "ID": "test-max-noncurrent", "Status": "Enabled", "Filter": { "Prefix": "user-uploads/" }, "NoncurrentVersionExpiration": { "MaxNoncurrentVersions": 5 } } ] } EOF mc ilm import myminio/mybucket < ilm.json ```

krisis force-pushed the feat-noncurrent-expiry branch from e4f500e to 46b4df1 Compare November 4, 2021 06:34

krisis mentioned this pull request Nov 4, 2021

Add MaxNoncurrentVersions to NoncurrentVersionExpiration minio/minio-go#1580

Merged

krisis force-pushed the feat-noncurrent-expiry branch from 46b4df1 to a0f8d0f Compare November 4, 2021 22:26

krisis marked this pull request as ready for review November 4, 2021 22:26

krisis requested a review from harshavardhana November 5, 2021 16:22

vadmeste reviewed Nov 8, 2021

View reviewed changes

internal/bucket/lifecycle/lifecycle.go Show resolved Hide resolved

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

krisis force-pushed the feat-noncurrent-expiry branch from a0f8d0f to 77cbdc4 Compare November 8, 2021 18:24

kannappanr requested a review from poornas November 8, 2021 23:20

poornas requested changes Nov 9, 2021

View reviewed changes

internal/bucket/lifecycle/expiration.go Outdated Show resolved Hide resolved

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

krisis force-pushed the feat-noncurrent-expiry branch from cc4a832 to b74119f Compare November 9, 2021 00:10

harshavardhana approved these changes Nov 9, 2021

View reviewed changes

poornas approved these changes Nov 9, 2021

View reviewed changes

klauspost previously requested changes Nov 9, 2021

View reviewed changes

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

krisis force-pushed the feat-noncurrent-expiry branch from d9c5789 to 4941b74 Compare November 10, 2021 00:03

krisis added the do-not-close label Nov 10, 2021

harshavardhana requested a review from klauspost November 10, 2021 00:05

harshavardhana approved these changes Nov 10, 2021

View reviewed changes

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

harshavardhana requested changes Nov 10, 2021

View reviewed changes

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

harshavardhana mentioned this pull request Nov 10, 2021

Improve performance on multiple versions #13573

Merged

6 tasks

krisis force-pushed the feat-noncurrent-expiry branch from 7f75864 to 8b91e90 Compare November 11, 2021 00:10

harshavardhana requested changes Nov 11, 2021

View reviewed changes

cmd/data-scanner.go Outdated Show resolved Hide resolved

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

cmd/object-handlers-common.go Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

krisis force-pushed the feat-noncurrent-expiry branch 3 times, most recently from b0c3e0d to 0cc60d9 Compare November 11, 2021 05:02

harshavardhana approved these changes Nov 11, 2021

View reviewed changes

krisis force-pushed the feat-noncurrent-expiry branch from 51c3fed to 82a8352 Compare November 19, 2021 00:33

klauspost reviewed Nov 19, 2021

View reviewed changes

cmd/data-scanner.go Outdated Show resolved Hide resolved

cmd/storage-datatypes.go Show resolved Hide resolved

harshavardhana requested changes Nov 19, 2021

View reviewed changes

cmd/api-errors.go Outdated Show resolved Hide resolved

harshavardhana requested changes Nov 19, 2021

View reviewed changes

cmd/storage-rest-common.go Show resolved Hide resolved

cmd/xl-storage-format-utils.go Show resolved Hide resolved

krisis force-pushed the feat-noncurrent-expiry branch from 923bb99 to 77e35e0 Compare November 19, 2021 18:03

harshavardhana requested changes Nov 19, 2021

View reviewed changes

krisis and others added 7 commits November 19, 2021 16:05

Combine error logging and notifications

8419b8a

Fix lint errors

05de47f

remove conflict resolution issues

3e507f1

Group expiry by days and max-noncurrent-versions under expiryState

93b350e

Address review comments

d627c75

Set fi.NumVersions for legacy object (xlv1)

f7830b9

krisis force-pushed the feat-noncurrent-expiry branch from 77e35e0 to b1418e9 Compare November 20, 2021 00:05

Add unit test for getFileInfoVersions

d1d4588

krisis force-pushed the feat-noncurrent-expiry branch from b1418e9 to d1d4588 Compare November 20, 2021 00:17

harshavardhana approved these changes Nov 20, 2021

View reviewed changes

harshavardhana merged commit 3da9ee1 into minio:master Nov 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MaxNoncurrentVersions to NoncurrentExpiration action #13580

Add MaxNoncurrentVersions to NoncurrentExpiration action #13580

krisis commented Nov 4, 2021 •

edited

vadmeste left a comment

harshavardhana commented Nov 9, 2021

krisis commented Nov 10, 2021

This comment has been minimized.

harshavardhana commented Nov 18, 2021

harshavardhana left a comment

minio-trusted commented Nov 20, 2021

Add MaxNoncurrentVersions to NoncurrentExpiration action #13580

Add MaxNoncurrentVersions to NoncurrentExpiration action #13580

Conversation

krisis commented Nov 4, 2021 • edited

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

vadmeste left a comment

Choose a reason for hiding this comment

harshavardhana commented Nov 9, 2021

krisis commented Nov 10, 2021

This comment has been minimized.

harshavardhana commented Nov 18, 2021

harshavardhana left a comment

Choose a reason for hiding this comment

minio-trusted commented Nov 20, 2021

Mint Automation

krisis commented Nov 4, 2021 •

edited