Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add resilience in case the cluster falls bellow 1 node #53

Closed
wants to merge 3 commits into from
Closed

feat: Add resilience in case the cluster falls bellow 1 node #53

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
96a3072
feat: Add resilience in case the cluster falls bellow 1 node
narvikd Feb 27, 2023
5add21c
Merge branch 'dev' into feat/recover_from_just_one_node
narvikd Feb 27, 2023
b9d3359
Call reinstall
narvikd Feb 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
20 changes: 18 additions & 2 deletions cluster/consensus/consensus.go
View file
Open in desktop
Expand Up @@ -2,6 +2,7 @@ package consensus

import (
"errors"
"fmt"
"github.com/hashicorp/go-hclog"
"github.com/hashicorp/raft"
"github.com/hashicorp/raft-boltdb/v2"
Expand Down Expand Up @@ -267,12 +268,27 @@ func (n *Node) waitForClusterReadiness() error {
if currentTry > maxRetryCount {
return errors.New("quorum retry max reached")
}

if n.IsQuorumPossible(true) {
n.logger.Info("quorum possible.")
break
}
n.logger.Error("it is not possible to reach Quorum due to lack of nodes. Retrying...")
time.Sleep(sleepTime)

_, errLeader := discover.SearchLeader(n.ID)
if errLeader != nil {
msg := fmt.Sprintf("it isn't possible to reach Quorum due to lack of nodes. "+
"Tried to search for a leader to join an existent consensus. "+
"Leader not found: %v",
errLeader.Error(),
)
n.logger.Error(msg)
n.logger.Error("it isn't possible to reach Quorum due to lack of nodes and leader not available. Retrying...")
time.Sleep(sleepTime)
continue
}

n.logger.Warn("existent leader found, reinstalling....")
n.ReinstallNode()
}
return nil
}