Skip to content

Fix repeated-prefix-chain data loss in join_on_conjunctions#209

Merged
derek73 merged 7 commits into
masterfrom
fix/join-on-conjunctions-restructure
Jul 4, 2026
Merged

Fix repeated-prefix-chain data loss in join_on_conjunctions#209
derek73 merged 7 commits into
masterfrom
fix/join-on-conjunctions-restructure

Conversation

@derek73

@derek73 derek73 commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • Unify the i == 0/else branch pair in join_on_conjunctions's conjunction-joining loop into a single block. Pure simplification, no behavior change (verified via characterization tests).
  • Fix Repeated prefix chain silently drops part of the name in join_on_conjunctions #208: the prefix-joining loop's value-based pieces.index(prefix) lookup re-found the wrong occurrence when the same prefix word repeated in a chain, silently dropping part of the name (e.g. HumanName("Juan de la de la Vega").last returned "de la Vega" instead of "de la de la Vega"). Replaced with a position-tracked forward scan that absorbs contiguous prefix runs, then extends to the next prefix/suffix boundary — no more .index()/try-except ValueError.
  • Small follow-up: corrected a stale comment ("more than 1 rootnames") to match the actual total_length >= 1 condition it describes.
  • Cleanup pass (/simplify review): extracted the bundled is_prefix(...) or (i == 0 and total_length >= 1) guard into a named leading_first_name local, so the "exempt a leading piece as a first name" rationale is legible at the call site instead of buried in a compound boolean.
  • Further simplification: the contiguous-conjunction join at the top of the method flattened each range into individual indices and deleted them one at a time in reverse. Since group_contiguous_integers already returns non-overlapping ranges, this is now a single loop over the ranges themselves (in reverse), slice-assigning each range directly — consistent with the same pattern the other two loops in this method already use.

Developed via TDD with subagent-driven review (implementer + independent spec-compliance review + independent code-quality review per commit, plus a final whole-branch review). One round of review caught and fixed two duplicate tests along the way. A follow-up /simplify pass (4 parallel cleanup reviews: reuse, simplification, efficiency, altitude) found the rewrite already efficient and free of reuse gaps, and applied the one legible altitude fix above; a couple of other suggestions (merging two nested while-loops, forcing reuse of an unrelated helper) were evaluated and skipped as not actual improvements. One more simplification (the reverse-index-delete loop above) was found afterward on manual inspection.

Known minor side-effect of the fix (found in post-merge-readiness review, not a regression to worry about): the old exception-driven code had an inconsistent suffix boundary — a stale, reused loop index from its except ValueError: pass path could let a prefix run "absorb" past where a suffix token should have stopped it. The new position-tracked scan enforces that suffix-stop boundary uniformly, so a small class of inputs where a suffix-like token sits between prefix words (not at the true trailing position) now classify differently between first/middle than before — e.g. HumanName("aen Sr. aan abu") was first="aen Sr.", middle="" on master, is now first="aen", middle="Sr." on this branch. No words are lost in either version (.suffix is empty in both cases either way, since "Sr." isn't in trailing position). This only showed up in adversarial/synthetic token fuzzing, not in realistic names, and is more consistent with the documented suffix-stop design — but it's an undocumented side effect of the refactor worth calling out for anyone diffing behavior closely.

Fixes #208

Test plan

  • python -m pytest -q — 1256 passed, 4 skipped, 22 xfailed (up from the 1248 baseline by exactly the 8 new regression test cases; zero regressions)
  • Repeated-prefix-chain bug independently reproduced against the pre-fix commit and confirmed fixed post-fix
  • New regression tests in tests/test_prefixes.py: test_repeated_prefix_chain_de_la, test_repeated_prefix_chain_van_der, test_triple_repeated_prefix_chain, test_repeated_prefix_chain_followed_by_suffix
  • Two independent fuzz sweeps (20,000 random synthetic tokens; 6,254 name-shaped permutations) against master found zero data-loss regressions — all divergences are either the intended fix or the suffix-boundary reclassification noted above
  • Re-ran full suite after each cleanup commit — still 1256 passed, 4 skipped, 22 xfailed

derek73 and others added 4 commits July 4, 2026 13:44
test_leading_conjunction_joins_to_following_piece and
test_trailing_conjunction_at_list_end were exact duplicates of the
pre-existing test_starts_with_conjunction and test_ends_with_conjunction
tests (same input, same assertions). They added no new coverage of the
join_on_conjunctions restructuring and only inflated the test file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace value-based pieces.index() lookup with a position-tracked
forward scan. The old code re-found each prefix by searching for its
string value after the list had already been mutated, which silently
dropped one repetition of a chain when the same prefix word appeared
more than once (e.g. 'Juan de la de la Vega' -> last 'de la Vega',
missing the first 'de la'). Fixes #208.
Code review noted the comment said "more than 1 rootnames" but the
actual condition is total_length >= 1, which is true for essentially
all real input. Clarify wording to match the condition.
@derek73 derek73 self-assigned this Jul 4, 2026
@derek73 derek73 added the bug label Jul 4, 2026
@derek73 derek73 added this to the v1.3.0 milestone Jul 4, 2026
derek73 added 3 commits July 4, 2026 14:11
PR review flagged that the current tests only cover the narrowest case
(a single prefix phrase repeated twice). Add two stronger guards for the
exact code path #208's fix touches: 3+ chained repeats, and a repeated
chain immediately followed by a suffix (the prefix-run absorption loop
and the suffix-boundary loop share the same index variable, so this
combination is worth pinning down explicitly).
Code review noted the compound condition conflated "is this piece a
prefix candidate at all" with "is it exempted as a leading first
name" into one boolean expression. Name the second clause so its
rationale is legible at the call site.
group_contiguous_integers already returns non-overlapping ranges, so
there's no need to flatten each range into individual indices and
delete them one at a time. Process the ranges themselves in reverse
and slice-assign each one directly, matching the same pattern already
used by the two other loops in this method.
@derek73 derek73 merged commit 1f0a6e0 into master Jul 4, 2026
8 checks passed
@derek73 derek73 deleted the fix/join-on-conjunctions-restructure branch July 4, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Repeated prefix chain silently drops part of the name in join_on_conjunctions

1 participant