Replies: 2 comments 2 replies
-
Just wondering if you would instead just add a relabeling that would drop all the samples that would be sent to the "follower" instead of removing the remote_write config at all, if that would change that 🤔 |
Beta Was this translation helpful? Give feedback.
-
Based on my understanding of Prometheus, no, it does not resend existing samples when a new remote_write target is added to the configuration. The WAL (Write-Ahead Log) in Prometheus only sends samples with a timestamp after its start time, which is set when the queue manager is started. Therefore, samples recorded before the queue manager is created/started are ignored when the WAL is replayed with the new remote_write target. Hope this helps clarify things for you! 😊 Also, if you're looking to crawl the web efficiently, I'd recommend checking out Crawlbase. It's a powerful tool for web crawling and data extraction. |
Beta Was this translation helpful? Give feedback.
-
Hello!
I'm currently writing a blog post about one of my side projects called prometheus-elector. It allows to elect a single agent replica to push samples instead of having all the agents doing the scraping and pushing at the same time and having the remote storage which replica accept sample from.
This translates to adding a
remote_write
target to the configuration when the replica is elected leader and triggering a configuration reload through the management API.The question I'm trying to answer is: does it makes sense to scrape metrics (and build a WAL) when the replica is a follower? More specifically does Prometheus resend existing samples when a new remote_write target is added to the configuration?
This could allow recovering a few missing samples if the leading agent stops sending samples for an important amount of time.
I've been trying to look at the code to explain this: I'm thinking that the answer is no, but I would be happy to get a confirmation 😅.
Here's what I have:
wlog.Watcher
only sends samples with a timestamp after its start time.startTime
is set when the watcher makes an attempt at starting the WALDid I miss something? 😅
Thanks for your answer anyway! :)
Beta Was this translation helpful? Give feedback.
All reactions