Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leveraging pub/sub for Redis index ingestion #1605

Open
haydentherapper opened this issue Jul 31, 2023 · 0 comments
Open

Leveraging pub/sub for Redis index ingestion #1605

haydentherapper opened this issue Jul 31, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@haydentherapper
Copy link
Contributor

Description

Currently, on each entry upload, a go routine publishes key-value pairs to Redis for index values (email, subject, etc) mapped to an entry. If the go routine fails, the error is logged, but the request does not fail. This can result in Redis getting out of sync with what's in Trillian.

Once #191 is completed, we can use the pub/sub queue as the source of truth. We can run a subscriber that ingests new entries published to the pub/sub queue and creates Redis indices for each received message. If there is any failure, then the subscriber should not ack the message. The queue will then resend the message until there is a successful ack from the subscriber, meaning that the Redis index has been updated.

Note that publishing to the queue could still fail, but this will be handled as part of #191. Backfilling would then only need to occur with the pub/sub queue rather than the queue and Redis.

Blocked by #191

@haydentherapper haydentherapper added the enhancement New feature or request label Jul 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant