Add a metastore read replica role for read-only routing#6548
Draft
shuheiktgw wants to merge 11 commits into
Draft
Add a metastore read replica role for read-only routing#6548shuheiktgw wants to merge 11 commits into
shuheiktgw wants to merge 11 commits into
Conversation
Introduce the `metastore_read_replica` service role and the `metastore_read_replica_uri` node config option: the foundation for routing read-only metastore traffic to a PostgreSQL read replica. - Add `QuickwitService::MetastoreReadReplica`, parsed from `metastore_read_replica` / `metastore-read-replica`. - Split `QuickwitService::default_services()` from `supported_services()` so the new role is opt-in and never enabled implicitly on all-in-one nodes. - Add the optional `metastore_read_replica_uri` field (env `QW_METASTORE_READ_REPLICA_URI`), redacted alongside `metastore_uri`. - Validate that the role requires a PostgreSQL `metastore_read_replica_uri` and cannot be co-located with the `metastore` role.
Add the resolution plumbing for connecting to a PostgreSQL read replica
over a read-only connection.
- Add `MetastoreFactoryOptions { read_only }` and thread it through
`MetastoreFactory::resolve`.
- Add `MetastoreResolver::resolve_read_only`, which rejects any non-
PostgreSQL backend.
- Key the PostgreSQL factory cache on `(uri, options)` so the read-write
and read-only clients get distinct connection pools.
- Add `PostgresqlMetastore::new_read_only`: a read-only connection pool
with migrations skipped (the replica is migrated by the primary).
Resolve and expose a metastore gRPC server when the `metastore_read_replica` role is enabled, backed by a read-only connection to `metastore_read_replica_uri`. - The read replica server reuses the metrics + load-shed layers but omits the control-plane event layers, which only wrap write RPCs. - A read-replica node is exempted from the control-plane connectivity wait, like a primary metastore node, so dedicated replica pods start independently. - Extract `metastore_max_in_flight_requests` shared by both roles.
Add `ReadReplicaRoutingMetastore`, a `MetastoreService` wrapper that routes the stale-tolerant reads issued by the search and analytics paths (`index_metadata`, `indexes_metadata`, `list_indexes_metadata`, `list_splits`, `list_metrics_splits`, `list_sketch_splits`) to read replica nodes when any are connected, and everything else (writes and non-hot-path reads) to the primary. - Routing is decided per request from the read replica balance channel's live connection set, so it degrades to the primary when no replica is deployed. The check is a synchronous `watch` read with no borrow held across an await. - Wire it into the searcher service and the DataFusion session builder, so all search (REST, Elasticsearch, gRPC) and metrics analytics benefit, while REST admin handlers keep read-your-writes against the primary.
- Document `metastore_read_replica_uri` in the node config reference and the example `quickwit.yaml`. - Add a PostgreSQL-gated test that resolves a read-only metastore against a real database and verifies it serves read RPCs.
Replace the full `MetastoreService` implementation on `ReadReplicaRoutingMetastore` (which forced ~45 delegating methods, most of them writes) with a narrow, read-only `MetastoreReadService` trait — the read-only subset of the metastore RPCs (`index_metadata`, `list_indexes_metadata`, `list_splits`, `list_metrics_splits`, `list_sketch_splits`). - `MetastoreServiceClient` implements `MetastoreReadService`; `MetastoreReadServiceClient = Arc<dyn MetastoreReadService>`. - `ReadReplicaRoutingMetastore` now implements only the 5-method trait, so writes are excluded at the type level rather than delegated. - The search and DataFusion read paths take `MetastoreReadServiceClient` / `&dyn MetastoreReadService`; `list_parquet_splits_*` take `&dyn MetastoreReadService`. `single_node_search` keeps its concrete `MetastoreServiceClient` parameter and adapts internally. Addresses review feedback that the wrapper should expose a Go-style read-only interface instead of reimplementing the whole service.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is an alternative to #6538. Instead of running a single metastore deployment and routing read-only requests to a PostgreSQL read replica via a per-request gRPC header, this PR introduces a dedicated
metastore_read_replicanode role: read-only metastore pods are deployed separately (connected to the replica), discovered through the cluster, and searcher/analytics reads are routed to them at the service-discovery layer.Routing by which node you talk to (rather than a per-request header) means no proto/codegen/interceptor changes, and it isolates read load onto separate processes that can scale and fail independently of the write-critical primary metastore.
How it works
--service metastore_read_replicaandmetastore_read_replica_uripointing at the PostgreSQL read replica (one shared config; the role selects which URI is used).metastore_read_replica_uriover a read-only connection (migrations skipped — the replica is migrated by the primary) and serves the samequickwit.metastore.MetastoreServicegRPC service, advertising themetastore_read_replicarole in the cluster.ReadReplicaRoutingMetastore(given to the searcher service + DataFusion) routes the stale-tolerant search/analytics reads (index_metadata,indexes_metadata,list_indexes_metadata,list_splits,list_metrics_splits,list_sketch_splits) to read-replica nodes when any are connected, and everything else (writes, non-hot-path reads) to the primary. When no replica is deployed it degrades to the primary, so the feature is fully opt-in.Trade-offs
Testing
quickwit-config: role parsing, opt-in default set, and all validation paths.ReadReplicaRoutingMetastore: read routing (replica when connected, primary when not) through the publicMetastoreServiceinterface.clippy --workspace --all-features --tests,cargo +nightly fmt --check, log-format / license / typos all clean.Known limitation / follow-up
There is no automated end-to-end multi-node test (searcher → read-replica node over gRPC): the
ClusterSandboxharness uses an in-memory metastore, and this feature is PostgreSQL-only. Covering it needs a PostgreSQL-backed sandbox variant (proposed as a follow-up). Manual production-path verification via two metastore roles against a real DB is the interim check.🤖 Generated with Claude Code