Fix repeated-prefix-chain data loss in join_on_conjunctions#209
Merged
Conversation
test_leading_conjunction_joins_to_following_piece and test_trailing_conjunction_at_list_end were exact duplicates of the pre-existing test_starts_with_conjunction and test_ends_with_conjunction tests (same input, same assertions). They added no new coverage of the join_on_conjunctions restructuring and only inflated the test file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace value-based pieces.index() lookup with a position-tracked forward scan. The old code re-found each prefix by searching for its string value after the list had already been mutated, which silently dropped one repetition of a chain when the same prefix word appeared more than once (e.g. 'Juan de la de la Vega' -> last 'de la Vega', missing the first 'de la'). Fixes #208.
Code review noted the comment said "more than 1 rootnames" but the actual condition is total_length >= 1, which is true for essentially all real input. Clarify wording to match the condition.
PR review flagged that the current tests only cover the narrowest case (a single prefix phrase repeated twice). Add two stronger guards for the exact code path #208's fix touches: 3+ chained repeats, and a repeated chain immediately followed by a suffix (the prefix-run absorption loop and the suffix-boundary loop share the same index variable, so this combination is worth pinning down explicitly).
Code review noted the compound condition conflated "is this piece a prefix candidate at all" with "is it exempted as a leading first name" into one boolean expression. Name the second clause so its rationale is legible at the call site.
group_contiguous_integers already returns non-overlapping ranges, so there's no need to flatten each range into individual indices and delete them one at a time. Process the ranges themselves in reverse and slice-assign each one directly, matching the same pattern already used by the two other loops in this method.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
i == 0/elsebranch pair injoin_on_conjunctions's conjunction-joining loop into a single block. Pure simplification, no behavior change (verified via characterization tests).pieces.index(prefix)lookup re-found the wrong occurrence when the same prefix word repeated in a chain, silently dropping part of the name (e.g.HumanName("Juan de la de la Vega").lastreturned"de la Vega"instead of"de la de la Vega"). Replaced with a position-tracked forward scan that absorbs contiguous prefix runs, then extends to the next prefix/suffix boundary — no more.index()/try-except ValueError.total_length >= 1condition it describes./simplifyreview): extracted the bundledis_prefix(...) or (i == 0 and total_length >= 1)guard into a namedleading_first_namelocal, so the "exempt a leading piece as a first name" rationale is legible at the call site instead of buried in a compound boolean.group_contiguous_integersalready returns non-overlapping ranges, this is now a single loop over the ranges themselves (in reverse), slice-assigning each range directly — consistent with the same pattern the other two loops in this method already use.Developed via TDD with subagent-driven review (implementer + independent spec-compliance review + independent code-quality review per commit, plus a final whole-branch review). One round of review caught and fixed two duplicate tests along the way. A follow-up
/simplifypass (4 parallel cleanup reviews: reuse, simplification, efficiency, altitude) found the rewrite already efficient and free of reuse gaps, and applied the one legible altitude fix above; a couple of other suggestions (merging two nested while-loops, forcing reuse of an unrelated helper) were evaluated and skipped as not actual improvements. One more simplification (the reverse-index-delete loop above) was found afterward on manual inspection.Known minor side-effect of the fix (found in post-merge-readiness review, not a regression to worry about): the old exception-driven code had an inconsistent suffix boundary — a stale, reused loop index from its
except ValueError: passpath could let a prefix run "absorb" past where a suffix token should have stopped it. The new position-tracked scan enforces that suffix-stop boundary uniformly, so a small class of inputs where a suffix-like token sits between prefix words (not at the true trailing position) now classify differently betweenfirst/middlethan before — e.g.HumanName("aen Sr. aan abu")wasfirst="aen Sr.", middle=""on master, is nowfirst="aen", middle="Sr."on this branch. No words are lost in either version (.suffixis empty in both cases either way, since "Sr." isn't in trailing position). This only showed up in adversarial/synthetic token fuzzing, not in realistic names, and is more consistent with the documented suffix-stop design — but it's an undocumented side effect of the refactor worth calling out for anyone diffing behavior closely.Fixes #208
Test plan
python -m pytest -q— 1256 passed, 4 skipped, 22 xfailed (up from the 1248 baseline by exactly the 8 new regression test cases; zero regressions)tests/test_prefixes.py:test_repeated_prefix_chain_de_la,test_repeated_prefix_chain_van_der,test_triple_repeated_prefix_chain,test_repeated_prefix_chain_followed_by_suffix