Ensure all indexers broadcast standalone compactors before starting m…#6542
Ensure all indexers broadcast standalone compactors before starting m…#6542nadav-govari wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 389a849075
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| info!( | ||
| "all indexers report standalone compactors enabled, starting compaction scan loop" | ||
| ); | ||
| ctx.send_self_message(ScanAndPlan).await?; |
There was a problem hiding this comment.
When a non-migrated indexer becomes live after this branch sends ScanAndPlan—for example, an old indexer temporarily missed by gossip during a rolling enableStandaloneCompactors rollout—the planner never checks all_indexers_migrated() again; ScanAndPlan keeps scanning and assigning tasks every interval. That reintroduces the split-brain this change is meant to prevent, because that indexer can still run in-process merge pipelines while standalone compactors receive new work.
Useful? React with 👍 / 👎.
|
|
||
| async fn all_indexers_migrated(&self) -> bool { | ||
| self.cluster | ||
| .live_nodes() |
There was a problem hiding this comment.
explain why live nodes but not ready nodes
2 participants
Description
When initially turning on
enableStandaloneCompactors, the janitor running the merge planner and the compactors come online basically immediately and start performing merges. Indexers take a long time (minutes) to decommission, and deploy one by one. That means that we can end up with a split brain situation where there's merge pipelines running on the old indexers, and compactors from the new deploy are up and running. They then end up with a bunch of publish failures.This addresses that by adding a standalone compactors value to ClusterMember, so that the merge planner can await for all indexers to show that they have standalone compactors enabled. This necessarily means that no merge pipelines are running on those indexers, so that starting standalone compaction is safe.
How was this PR tested?
Unit tests, local cluster.