Read replica support for Postgres and MySQL datastores #1878
Conversation
This is accomplished by adding a datastore proxy that selects a read replica for each SnapshotReader, in a round robin fashion, and ensures that the revision being requested is available on the replica. This extra check does add some latency to the overall operation, but it should provide for the safest approach for using read replicas Fixes authzed#1321 Fixes authzed#1320
github-actions
bot
added
area/CLI
…d of using read replicas
josephschorr
force-pushed
the
read-replica-support
branch
from
1d5612b
to
aa51f2a
Compare
|
Just to clarify, this update will allow spice to use multiple reader nodes for postgres and mysql? So if I have 3 readers I would be able to pass in the entries for all of them to be used? |
|
@benny-yamagata Yes, the replica URI parameter is a list of URIs and the system will round robin between them |
There was a problem hiding this comment.
The implementation works with the assumption individual hosts will be listed, but folks most of the time would use replicas behind a load-balancer in order to be able to scale read traffic when needed. I don't think the implementation will work for that more common use case because the datastore snapshot reader does not use a single transaction.
| @@ -97,15 +97,30 @@ type sqlFilter interface { | |||
| // URI: [scheme://][user[:[password]]@]host[:port][/schema][?attribute1=value1&attribute2=value2... | |||
| // See https://dev.mysql.com/doc/refman/8.0/en/connecting-using-uri-or-key-value-pairs.html | |||
| func NewMySQLDatastore(ctx context.Context, uri string, options ...Option) (datastore.Datastore, error) { | |||
| ds, err := newMySQLDatastore(ctx, uri, options...) | |||
| ds, err := newMySQLDatastore(ctx, uri, -1 /* primary */, options...) | |||
There was a problem hiding this comment.
magic number -> use constant
please replace it everywhere it's used
| index uint32, | ||
| options ...Option, | ||
| ) (datastore.ReadOnlyDatastore, error) { | ||
| ds, err := newMySQLDatastore(ctx, url, int(index), options...) |
There was a problem hiding this comment.
I searched online if there is any URI query parameter we can add to enforce read only, but didn't find any. Users should make sure the credentials provided for the replica have read-only permissions.
| @@ -48,7 +48,7 @@ type datastoreTester struct { | |||
| func (dst *datastoreTester) createDatastore(revisionQuantization, gcInterval, gcWindow time.Duration, _ uint16) (datastore.Datastore, error) { | |||
| ctx := context.Background() | |||
| ds := dst.b.NewDatastore(dst.t, func(engine, uri string) datastore.Datastore { | |||
| ds, err := newMySQLDatastore(ctx, uri, | |||
| ds, err := newMySQLDatastore(ctx, uri, -1, | |||
There was a problem hiding this comment.
magic number, use constant
| @@ -116,7 +117,24 @@ func NewPostgresDatastore( | |||
| url string, | |||
| options ...Option, | |||
| ) (datastore.Datastore, error) { | |||
| ds, err := newPostgresDatastore(ctx, url, options...) | |||
| ds, err := newPostgresDatastore(ctx, url, -1 /* is primary */, options...) | |||
There was a problem hiding this comment.
magic number, use constant
| func NewReadOnlyPostgresDatastore( | ||
| ctx context.Context, | ||
| url string, | ||
| index uint32, |
There was a problem hiding this comment.
I don't think this is how typically folks deploy MySQL read replicas. There is usually a URI for the primary, and a URI for replicas behind a load balancer. Why did you choose to enumerate the replicas?
There was a problem hiding this comment.
Because there may be cases where there are multiple replicas
| if len(replicas) == 0 { | ||
| log.Debug().Msg("No replicas provided, using primary as read source.") | ||
| return primary, nil | ||
| } |
| return | ||
| } | ||
| } | ||
| finalError = err |
There was a problem hiding this comment.
| finalError = err | |
| finalError = err |
| rr.chosenReader = rr.replica.SnapshotReader(rr.rev) | ||
| rr.chosePrimary = false | ||
| }) | ||
| return finalError |
There was a problem hiding this comment.
| return finalError | |
| return finalError |
| return rr.chosenReader.LookupNamespacesWithNames(ctx, nsNames) | ||
| } | ||
|
|
||
| func (rr *replicatedReader) chooseSource(ctx context.Context) error { |
There was a problem hiding this comment.
I think returning the chosen reader in chooseSource would better encapsulate this logic.
| var finalError error | ||
| rr.choose.Do(func() { | ||
| // If the revision is not known to the replica, use the primary instead. | ||
| if err := rr.replica.CheckRevision(ctx, rr.rev); err != nil { |
There was a problem hiding this comment.
I'm not entirely certain whether this works with replicas behind a load-balancer.
If the user inputs a load balancer as the single replica URI, it will only check the revision once. However, it cannot guarantee that it will hit the same replica every time, as the connection pool would have opened connections against multiple replicas.
Even if you called CheckRevision each time it still cannot be guaranteed since checking the revision and then performing the other datastore API call is not using the same transaction. This worked when we were hitting the primary and doing our own MVCC, but does not when introducing load-balanced replicas.
There was a problem hiding this comment.
It should hold to a single connection and therefore be safe; the only other approach would be to add the revision check into every transaction, and that would likely make things too slow.
It uses a single connection, which means it should stay connected to the same replica |
| options ...Option, | ||
| ) (datastore.Datastore, error) { | ||
| isPrimary := replicaIndex < 0 |
There was a problem hiding this comment.
I think this flag makes the rest of the function harder to follow. Maybe something like:
newBasePostgresDatastore()
newPostgresDatastore(index) {
ds := newBasePostgresDatastore()
if index < 0 {
return makeIntoRWDS(ds)
}
return makeIntoRODS(ds)
}or
newBasePostgresDatastore()
newReadWriteDatastore(base) {}
newReadReplicaDatastore(base) {}
I possibly just missed it, but it looked like you were using a |
Yeah, I traced it and it does use the pool. We'll have to do something else |
No description provided.