fix(managed_client): wait for result ready on sync query path#28
Merged
Conversation
fetch_table fetched the persisted result as Arrow using the result_id from a synchronous QueryResponse without waiting for it to reach 'ready'. Against the live async backend the result is often still 'processing', so Arrow fetches failed on every read-modify-write (merge/append) and state read. The async path already waited; the sync path now does too. Adds a regression test driving the sync path with a 'processing' -> 'ready' result, asserting Arrow is fetched only after readiness.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Extract a single typed _poll() helper shared by the query-run and result-ready waits (removing two near-identical loops and the magic 0.5/0.3 sleep intervals), flatten _query_database_scoped via _await_query_run, and hoist the class constants to the top. No behavior change; verified by the unit suite and a full prod e2e re-run.
There was a problem hiding this comment.
Correct, well-scoped fix. The sync QueryResponse path now waits for result readiness before the caller fetches Arrow, the polling logic is cleanly deduplicated into _poll, and the regression test exercises the actual processing -> ready path. No blocking issues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A real prod end-to-end run of
hotdata-dlt-destinationfailed on every read-modify-write (merge/append) and state read:_query_database_scopedwaited for readiness on the async path but returned theresult_idimmediately on the synchronousQueryResponsepath.fetch_table()then fetched it as Arrow while stillprocessing. Shared client, sohotdata-airflowwas exposed too. The in-memory test masked it (fake Arrow API always ready).Fix
The sync path now also calls
_wait_result_ready()before returning. Adds a regression test that drives the sync path throughprocessing -> readyand asserts Arrow is fetched only after readiness.Verified by re-running the full prod e2e (replace+nested, merge dedup, incremental state-sync) — all green.