Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reattempt of informer start() triggers all previous events #1598

Open
jobcespedes opened this issue Mar 6, 2024 · 3 comments
Open

Reattempt of informer start() triggers all previous events #1598

jobcespedes opened this issue Mar 6, 2024 · 3 comments

Comments

@jobcespedes
Copy link

Describe the bug
Redo informer start() triggers all previous events. This is being done inside setTimeout periodically as a workaround since informer silently stop receiving events (see: #596)

** Client Version **
1.0.0-rc4

** Server Version **
e.g. 1.26.5

To Reproduce

  1. MakeInformer
  2. Start informer
  3. Set timeout starting informer in 30s
  4. Trigger an update event in k8s for the informer to catch it
  5. Wait for timeout to start informer again
  6. See the update event in step 4 being reproduce again

Expected behavior
A clear and concise description of what you expected to happen.

** Example Code**

const timeoutDelay = 1000 * 30;
const timeout = setTimeout(() => {
  if (informer) {
    informer
      .start()
      .then(() => {
        console.log("Informer refresh connection");
      })
      .catch((err) => {
        console.log("Informer refresh error");
        console.error(err);
      });
  }
}, timeoutDelay);

Environment (please complete the following information):

  • OS: linux
  • NodeJS Version: v18.14.2
  • Cloud runtime: EKS

Additional context
This is probably a secudary issue, being the primary that informer silently stop reproducing events, like in #596

@mstruebing
Copy link
Member

mstruebing commented Mar 6, 2024

I'm not an expert on how informers work but I guess this is because we have this data cached already and you do not instantiate a new one? As far as I know informers cache their data so it would make sense that they are replaying the events in case of a restart from my point of view.

As a workaround you could also try to save the last event you processed and skip until the event occurs and only process the newer ones?

Or can you try something like this and instantiate a new informer?:

const timeoutDelay = 1000 * 30;
let informer;
const timeout = setTimeout(() => {
    informer = ...; // instantiate a new informer
    informer
      .start()
      .then(() => {
        console.log("Informer refresh connection");
      })
      .catch((err) => {
        console.log("Informer refresh error");
        console.error(err);
      });
}, timeoutDelay);

@jobcespedes
Copy link
Author

Hi @mstruebing,

Thank you for the suggestion. I thought about the option of starting a new informer. In this regard, I was wondering about the implications of not doing it correctly, reliably, and transparently, since the application using the informer cannot lose events. Another possible implication that I also wanted to avoid is memory leaks.

Regarding the mention of the cache, I also suspected that something has to do with the cache. I am reviewing the code in cache.ts to better understand what happens when using start(). I found the Bookmark option. According to the description, the functionality of the option could be used to avoid the current behavior. However, I believe it is not yet programmed in the informer.

The fact that the informer/watcher stops silently is also something that I would like to understand and avoid.

@mstruebing
Copy link
Member

If you find the root cause why this is happening we are more than happy to accept PRs.

If the informer replays all your events you could save the last processed event, at least I believe the event + resourceVersion should be unique so you are sure you start processing at the right event in case of a restart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants