perf(SDK-6463): prune excluded dirs from zip/md5 walks — minutes-long 'Creating tests.zip' stall on monorepos#1142
Open
Bhargavi-BS wants to merge 1 commit into
Open
perf(SDK-6463): prune excluded dirs from zip/md5 walks — minutes-long 'Creating tests.zip' stall on monorepos#1142Bhargavi-BS wants to merge 1 commit into
Bhargavi-BS wants to merge 1 commit into
Conversation
…emetry size walk (SDK-6463) On large monorepos the CLI stalled for minutes before uploading: 1. archiver.glob and the spec-md5 walk (hashUtil via checkUploaded) pass excludes as readdir-glob 'ignore', which filters entries only AFTER the walker has descended into and lstat'ed every file under node_modules/.git/dist/etc. The customer's log showed ~86s in 'Creating tests.zip' and a similar hidden cost in the md5 step. 2. utils.fetchFolderSize(node_modules) — telemetry only — recursively stats the entire node_modules between archiving and uploading. Fix: reuse every ignore pattern ending in '/**' as readdir-glob's 'skip' option (getDirectorySkipPatterns), which prevents descending into matching directories. This is provably safe: a directory matching '<x>/**' means every descendant also matches, so pruning cannot change the archive contents or the md5. Verified on a synthetic 144k-file monorepo with the customer's exact 130 exclude patterns: zip 7.5s -> 0.1s and md5 5.5s -> 0.0s, with byte-identical zip entry lists and an UNCHANGED md5 (upload cache keys unaffected). Also adds a 5s cooperative deadline to the folder-size telemetry walk so it can never stall a run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SDK-6463 (follow-up) — minutes-long stall creating tests.zip on large monorepos
Customer (Sapiens, NX monorepo, ~120 exclude globs,
home_directory: "./") reported very slow test-setup upload. Their log shows the time is NOT the upload (3s, small zip) — it's spent before it:plus a similar hidden cost earlier in the spec-md5 cache check.
Root cause (verified)
Three full-tree filesystem walks of the monorepo root, none of which prune excluded directories:
archiver.glob(pattern, { ignore })(bin/helpers/archiver.js). archiver delegates toreaddir-glob, whoseignorefilters entries after the walker has alreadyreaddir+lstat-ed them. It therefore descends intonode_modules,.git,dist, … visiting every file just to discard it. The library's separateskipoption is the pruning mechanism — the CLI never used it.checkSpecsMd5→hashUtilusesreaddir-globthe same way: a second identical full-tree walk.utils.fetchFolderSize(node_modules)(runs.js) recursively stats the entirenode_modules, blocking between archive and upload, only to report a folder size.On the customer's Windows machine (per-file stat + AV scanning) with a real NX monorepo this is minutes per run.
Fix
utils.getDirectorySkipPatterns(ignoreFiles)— reuse every ignore pattern ending in/**as areaddir-globskippattern; wired into both the archive glob and the md5 walk.Safety proof:
skipprunes a directory whose relative path matches the pattern. A directory matching<x>/**is inside<x>, so every one of its descendants also matches<x>/**and was going to be ignored anyway — pruning cannot change which files are included. (Pruning engages one level below each excluded root: e.g.node_modules/**skips every child ofnode_modules, paying a single readdir ofnode_modulesitself.)fetchFolderSize— cooperative 5s deadline; folder size is best-effort telemetry and must never stall a run.Verification (real CLI code, synthetic 144k-file monorepo, customer's exact 130 exclude patterns)
archiveSpecs(tests.zip)checkSpecsMd5aeceb90f…) → upload-cache keys unaffected, no forced re-uploadsDirect
readdir-globmeasurement on the same tree: no-skip walk 3.04s / 140,883 entries visited vs skip 0.08s / 2,083 — the excluded trees are simply never entered.Unit tests added for
getDirectorySkipPatterns; suite: 675→677 passing, 0 new failures (16 pre-existing on master).Scope
Separate from PR #1131 (config/a11y) and #1139 (a11y fail-guard): this addresses the customer's "test setup zip upload taking a lot of time" report independently.
🤖 Generated with Claude Code