Replace custom Redis config struct with go-redis UniversalOptions (adds sentinel & cluster support)

[andsens]

Less code more features! :-D

This PR replaces the custom Redis config struct with UniversalOptions from go-redis, which implicitly adds support for both redis sentinel and redis clustering.

I have two failing tests that I can't quite figure out how to fix:

    --- FAIL: TestConfigSuite/TestMarshalRoundtrip (0.00s)
    --- FAIL: TestConfigSuite/TestParseInvalidVersion (0.00s)

Associated stack trace

    suite.go:87: test panicked: cannot marshal type: func(context.Context, string, string) (net.Conn, error)
        goroutine 18 [running]:
        runtime/debug.Stack()
        	/usr/lib/go-1.21/src/runtime/debug/stack.go:24 +0x5e
        github.com/stretchr/testify/suite.failOnPanic(0xc0002b8820, {0x8ebe40, 0xc0002c5850})
        	/home/aim/Workspace/forks/distribution/vendor/github.com/stretchr/testify/suite/suite.go:87 +0x39
        github.com/stretchr/testify/suite.Run.func1.1()
        	/home/aim/Workspace/forks/distribution/vendor/github.com/stretchr/testify/suite/suite.go:183 +0x24c
        panic({0x8ebe40?, 0xc0002c5850?})
        	/usr/lib/go-1.21/src/runtime/panic.go:914 +0x21f
        gopkg.in/yaml%2ev2.handleErr(0xc0001513c8)
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/yaml.go:249 +0x6d
        panic({0x8ebe40?, 0xc0002c5850?})
        	/usr/lib/go-1.21/src/runtime/panic.go:914 +0x21f
        gopkg.in/yaml%2ev2.(*encoder).marshal(0xc000220b00, {0x0, 0x0}, {0x913e00?, 0xc0002de618?, 0x0?})
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:183 +0xa34
        gopkg.in/yaml%2ev2.(*encoder).structv.func1()
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:226 +0x1fc
        gopkg.in/yaml%2ev2.(*encoder).mappingv(0xc000220b00, {0x0?, 0x0}, 0xc000150c58)
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:256 +0x14a
        gopkg.in/yaml%2ev2.(*encoder).structv(0xc000220b00, {0x0, 0x0}, {0x9778a0?, 0xc0002de5e8?, 0xc0002de6c8?})
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:213 +0xe5
        gopkg.in/yaml%2ev2.(*encoder).marshal(0xc000220b00, {0x0, 0x0}, {0x9778a0?, 0xc0002de5e8?, 0x91b7e0?})
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:160 +0x945
        gopkg.in/yaml%2ev2.(*encoder).structv.func1()
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:226 +0x1fc
        gopkg.in/yaml%2ev2.(*encoder).mappingv(0xc000220b00, {0x0?, 0x0}, 0xc00013d028)
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:256 +0x14a
        gopkg.in/yaml%2ev2.(*encoder).structv(0xc000220b00, {0x0, 0x0}, {0x96d600?, 0xc0002de400?, 0x18?})
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:213 +0xe5
        gopkg.in/yaml%2ev2.(*encoder).marshal(0xc000220b00, {0x0, 0x0}, {0x96d600?, 0xc0002de400?, 0x96d7c0?})
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:160 +0x945
        gopkg.in/yaml%2ev2.(*encoder).marshal(0xc000220b00, {0x0, 0x0}, {0x8dcb80?, 0xc0002de400?, 0xc00013d348?})
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:154 +0x74b
        gopkg.in/yaml%2ev2.(*encoder).marshalDoc(0xc000220b00, {0x0, 0x0}, {0x8dcb80?, 0xc0002de400?, 0xc00013d3c8?})
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/encode.go:93 +0xcb
        gopkg.in/yaml%2ev2.Marshal({0x8dcb80, 0xc0002de400})
        	/home/aim/Workspace/forks/distribution/vendor/gopkg.in/yaml.v2/yaml.go:203 +0x319
        github.com/distribution/distribution/v3/configuration.(*ConfigSuite).TestParseInvalidVersion(0xc00007e4b0)
        	/home/aim/Workspace/forks/distribution/configuration/configuration_test.go:397 +0xfa
        reflect.Value.call({0xc00007de80?, 0xc0000550b8?, 0x0?}, {0x99472c, 0x4}, {0xc000065e70, 0x1, 0xc000065d78?})
        	/usr/lib/go-1.21/src/reflect/value.go:596 +0xce7
        reflect.Value.Call({0xc00007de80?, 0xc0000550b8?, 0xc00007e4b0?}, {0xc000065e70?, 0x5214cd?, 0xc69e78?})
        	/usr/lib/go-1.21/src/reflect/value.go:380 +0xb9
        github.com/stretchr/testify/suite.Run.func1(0xc0002b8820)
        	/home/aim/Workspace/forks/distribution/vendor/github.com/stretchr/testify/suite/suite.go:197 +0x467
        testing.tRunner(0xc0002b8820, 0xc00019c630)
        	/usr/lib/go-1.21/src/testing/testing.go:1595 +0xff
        created by testing.(*T).Run in goroutine 6
        	/usr/lib/go-1.21/src/testing/testing.go:1648 +0x3ad

As v3 is still in alpha I was hoping that there would still be time for this to make it into the release.

[Jamstah]

Please sign your commits as per https://github.com/distribution/distribution/blob/main/CONTRIBUTING.md#contributing-code

[milosgajdos]

The problem with this change is, it's tying the code to a specific Go module, which I'm not entirely sure is a good idea.

[andsens]

@Jamstah done.
@milosgajdos, I understand the issue. I mean, we could just replicate the entire struct I guess? In practice the options struct seems to be quite stable and well thought out. The tests will catch any incompatible changes when bumping and then we can decide from there what to do about it, rather than doing a bunch of work that might not be necessary.

p.s.: I figured out the test failure: The marshaller is trying to serialize the Dialer function I believe, but I'm not familiar enough with the rest of the code to determine how to fix that.

[andsens]

Also, it looks like redis.tls.enabled had no effect previously? I don't see it used anywhere...

[andsens]

@milosgajdos, replicating the struct actually fixes the marshalling sooo 🤷‍♂️ fixed :-D

[Jamstah]

We're already using the go module to provide the client itself, so why not also use it to provide the config struct?

If we ever change go module it would need rewriting anyway.

I like the idea of matching our config to be more "standard".

[andsens]

@Jamstah I agree. In that case the marshalling (or the test) needs some work though. I don't know how to exclude the functions that are referenced in those configs, any ideas?

[andsens]

*ping after the holidays. I would love if we could figure out how to progress with this.

[milosgajdos]

We are still missing doc updates here:

distribution/docs/content/about/configuration.md

Lines 953 to 1020 in 0d1792f

	## `redis`
	```yaml
	redis:
	addr: localhost:6379
	password: asecret
	db: 0
	dialtimeout: 10ms
	readtimeout: 10ms
	writetimeout: 10ms
	pool:
	maxidle: 16
	maxactive: 64
	idletimeout: 300s
	tls:
	enabled: false
	```
	Declare parameters for constructing the `redis` connections. Registry instances
	may use the Redis instance for several applications. Currently, it caches
	information about immutable blobs. Most of the `redis` options control
	how the registry connects to the `redis` instance. You can control the pool's
	behavior with the [pool](#pool) subsection. Additionally, you can control
	TLS connection settings with the [tls](#tls) subsection (in-transit encryption).
	You should configure Redis with the **allkeys-lru** eviction policy, because the
	registry does not set an expiration value on keys.
	| Parameter | Required | Description |
	|-----------|----------|-------------------------------------------------------|
	| `addr` | yes | The address (host and port) of the Redis instance. |
	| `password`| no | A password used to authenticate to the Redis instance.|
	| `db` | no | The name of the database to use for each connection. |
	| `dialtimeout` | no | The timeout for connecting to the Redis instance. |
	| `readtimeout` | no | The timeout for reading from the Redis instance. |
	| `writetimeout` | no | The timeout for writing to the Redis instance. |
	### `pool`
	```yaml
	pool:
	maxidle: 16
	maxactive: 64
	idletimeout: 300s
	```
	Use these settings to configure the behavior of the Redis connection pool.
	| Parameter | Required | Description |
	|-----------|----------|-------------------------------------------------------|
	| `maxidle` | no | The maximum number of idle connections in the pool. |
	| `maxactive`| no | The maximum number of connections which can be open before blocking a connection request. |
	| `idletimeout`| no | How long to wait before closing inactive connections. |
	### `tls`
	```yaml
	tls:
	enabled: false
	```
	Use these settings to configure Redis TLS.
	| Parameter | Required | Description |
	|-----------|----------|-------------------------------------- |
	| `enabled` | no | Whether or not to use TLS in-transit. |

Also, it looks like redis.tls.enabled had no effect previously? I don't see it used anywhere...

Yes, that was an oversight on my side when switching to the new module. There is an issue someone opened recently: #4284

My other comment is...there is now a boat load of new config items. What are they initialized to when omitted from the config file? Is not having a config value for each of them going to make the user confused? I guess what I'm trying to say is: what's the behaviour like when they're initialized to their default Go vals when omitted.

[milosgajdos]

What does Protocol stand for?

[andsens]

Protocol 2 or 3. Use the version to negotiate RESP version with redis-server. Default is 3.

It seems to have been removed in the latest v9.

[squizzi]

Probably would be useful to carry the comments from here https://github.com/redis/go-redis/blob/v9.1.0/options.go#L31-L139 into this struct as well in case questions like these come up again.

[andsens]

That's not the UniversalOptions struct though :-/
I carried over all the comments from that one.

Again, it's probably a lot smarter to simply use that struct directly. The indirection just adds a lot of confusion. If the go-redis struct is referenced directly you know there isn't any tomfoolery happening.

[andsens]

Here are the structs with all the options specified:

• Simple mode: https://github.com/redis/go-redis/blob/v9.1.0/options.go#L31-L139
• Sentinel/failover mode: https://github.com/redis/go-redis/blob/v9.1.0/sentinel.go#L21-L81
• Cluster mode: https://github.com/redis/go-redis/blob/v9.1.0/cluster.go#L28-L87

I'm starting to think that using the redis-go UniversalOptions struct directly is the better option here. It cuts down on maintenance and documentation. We can simply link out to e.g. https://pkg.go.dev/github.com/go-redis/redis/v9#UniversalOptions.

All that is needed for that is figuring out how to get around the marshalling issue.

[milosgajdos]

I'm starting to think that using the redis-go UniversalOptions struct directly is the better option here. It cuts down on maintenance and documentation. We can simply link out to e.g. https://pkg.go.dev/github.com/go-redis/redis/v9#UniversalOptions.

I think that might be the best course of action should we decide to go forth with this.

[milosgajdos]

@andsens can you sign your commit, please

[andsens]

@milosgajdos ah crap, sry. I reverted to the previous version and forgot to amend. Will do when I get back to my work PC. In the meantime, what do we do about the failed marshalling of the config?
It's not used anywhere except during testing. I guess it's a useful feature for debugging.

[milosgajdos]

There is some silly reflection stuff in configuration package to enable the ENV var shortcuts via cli...we should try fixing it