feat: add serial crash-safe mutation recovery#142
Open
gwokhou wants to merge 1 commit into
Open
Conversation
5346945 to
4ca33c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
This PR makes document import fail more cleanly. If
openkb addis interrupted or errors halfway through, OpenKB should no longer leave half-written KB files behind; it can recover the previous stable state before accepting the next mutation.Summary
Adds crash-safe mutation recovery for serial KB mutations without bringing back the concurrent add pipeline from #104.
This PR keeps the scope focused on the first split-out piece from the abandoned PR: journaled snapshot/rollback for
addand PageIndex Cloud import, plus the atomic writer prerequisites needed for hardlink-backed rollback.What changed
Added
openkb/mutation.pywith:Wired recovery into serial mutation paths:
openkb add <file>converts into staging, snapshots final KB paths, publishes, compiles, then commits via registry write + journal markopenkb add --from-pageindex-cloudprepares cloud data read-only, snapshots doc-specific paths, writes/compiles/registers under the same recovery contractKept conversion serial while adding staging support:
convert_document(..., staging_dir=...)writes raw/source artifacts into isolated staging before commitMade wiki writers atomic where hardlink snapshots depend on temp+replace semantics:
Explicitly out of scope
This does not reintroduce the concurrency half of #104:
ThreadPoolExecutorfile_processing_jobsValidation
UV_CACHE_DIR=/tmp/uv-cache UV_PYTHON=3.13 uv run --extra dev pytest -q898 passed, 5 warningsUV_CACHE_DIR=/tmp/uv-cache UV_PYTHON=3.13 uv run --extra dev pytest tests/test_add_command.py::TestImportFromPageindexCloud tests/test_mutation.py -q25 passedUV_CACHE_DIR=/tmp/uv-cache UV_PYTHON=3.13 uv run --extra dev pytest tests/test_converter.py tests/test_add_command.py tests/test_mutation.py -q72 passedUV_CACHE_DIR=/tmp/uv-cache UV_PYTHON=3.13 uv run --extra dev ruff check openkb/converter.py openkb/mutation.py tests/test_mutation.pyNotes
Full-project
ruffandtyare not clean on currentmain; this PR fixes the new diagnostics introduced by the recovery changes but does not widen scope to existing lint/type debt.