New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
UX: Keep the 50 most recent envs rather than the first 50 #103
Conversation
Can we use a different data structure in redis that supports this transparently, like some sort of list like message bus uses? something like this? https://github.com/SamSaffron/message_bus/blob/master/lib/message_bus/backends/redis.rb#L110-L124 |
We could use a different data structure, but then we'd need a top-level key in redis for each message/log. We currently have 1 hash where each key represents a message id and points to a JSON string that represents the message's env array. We'd need to change that to having a separate sorted set (or a simple list could also work) for each message. So at most Logster would need as many lists/keys as |
I think it is fine for logster to have lots of keys/lists in redis, we just need to make sure we clean up properly on clear all and be careful to make our operations never leave around orphan bits and pieces. |
Sounds good! I can think of 2 approaches that we can use here to migrate from the one hash we currently have to a list per message:
For 2, we'd have extra overhead when we fetch messages so at some point in the future (maybe 6 moths?) we'd need to drop the extra logic/overhead. I'm not too sure about 1, do you see any downsides in 1? Any other better approaches? |
I am not super concerned about migration, we can just clear logs and start from scratch. I guess if you really want something a script is fine. |
Ok, I've made the changes to make logster/lib/logster/message.rb Lines 146 to 172 in 294e13f
With this change, we no longer have I can think of 2 ways to fix this:
|
I would simply introduce new settings here. Simply have
I think this is a far simpler approach and we can ship with sane defaults here. |
@SamSaffron I've added 2 new settings
Does this sound good? Any changes you would like me to make? |
I would go with the names:
Changes sound good to me! Be sure to update the readme here to specify all the settings we have. |
I've renamed the settings and added some documentation about them in the readme. Merging this in now 🎉 |
At the moment when Logster merges similar messages, it keeps the envs of the first 50 occurrences and then stops accepting new envs. This PR changes that so that Logster keeps the envs of the 50 most recent occurrences.
I think it does make sense and it's more useful to have the most recent 50 envs, however this change comes at a performance cost. This change effectively reverts of the performance improvements we did in this commit: cbca14a. Right now when Logster merges 2 similar messages, it checks the number of envs and if we already have 50 envs it doesn't bother loading the envs array from redis and subsequently doesn't need to write the env array back to redis. With this change, we're forced to always load the envs array and write it back to redis after adding the new env. Thoughts on this @SamSaffron?