Skip to content

PDX-516: feat(mcp): validate UiAssert screen context (action=New vs View)#228

Merged
mrdailey99 merged 2 commits into
developfrom
feature/PDX-516-uiassert-screen-context-rule
Jun 24, 2026
Merged

PDX-516: feat(mcp): validate UiAssert screen context (action=New vs View)#228
mrdailey99 merged 2 commits into
developfrom
feature/PDX-516-uiassert-screen-context-rule

Conversation

@mrdailey99

Copy link
Copy Markdown
Collaborator

Summary

  • Adds UI-SCREEN-CONTEXT-001 (check.type: uiAssertScreenContext) to the best-practices validator, registered in MULTI_VALIDATOR_REGISTRY so it can emit one violation per offending step.
  • Severity major, weight 5 (consistent with the major runtime-anti-pattern tier).
  • Closes the gap where a create-and-verify test that leaves the UiAssert nested under the action=New create screen scored a clean is_valid:true / 100 / valid with no warning, while asserting against a page that no longer exists at runtime.

Two heuristics (both over data already in the XML)

  1. A UiAssert / UiRead whose nearest enclosing UiWithScreen has a target URI with action=New (the create screen). Salesforce tears down the create form on Save, so verifying there is the wrong context.
  2. Any UI action that appears after a Save within the same UiWithScreen <clause name="substeps"> block. Save is detected as a UiDoAction whose locator binding contains action=save (percent-encoded action%3Dsave in the URI) and/or name=save with an action interaction. Save navigates away, so trailing steps run in stale context.

Both messages name the offending step (title + testItemId) and recommend moving it into a new UiWithScreen targeting action=View (or the appropriate post-save screen). The rule must NOT fire for the correct structure (assert under an action=View screen with nothing problematic after Save).

Jira

https://provartesting.atlassian.net/browse/PDX-516

Test plan

  • POSITIVE: constructed bad fixture (opportunity-create-verify-bad.testcase — UiAssert moved into the action=New substeps after Save, standalone View screen removed) → rule fires; reported as major; quality_score < 100.
  • NEGATIVE: good fixture (opportunity-create-verify-good.testcase — assert under action=View) → rule does NOT fire.
  • Save-then-step: minimal inline fixture with a step after Save inside the same New screen → fires (names the trailing step).
  • Clean minimal: assert under action=View, nothing after Save → no violation from this rule.
  • yarn compile clean; full mocha suite 1496 passing / 1 pending / 0 failing; yarn lint clean.
  • node scripts/mcp-smoke.cjs 30 PASS (4 pre-existing environmental timeouts on the globally-installed package; smoke does not exercise this rule). TOTAL_EXPECTED unchanged — this PR adds a rule, not a tool.

Changes

  • src/mcp/tools/bestPracticesEngine.ts — new validator validateUiAssertScreenContext + helpers (getUiWithScreenTargetUri, getScreenAction, getArgUri, isSaveAction, getScreenSubstepCalls, describeStep, flagAssertsOnNewScreen, flagStepsAfterSave); registered in MULTI_VALIDATOR_REGISTRY.
  • src/mcp/rules/provar_best_practices_rules.json — new rule entry UI-SCREEN-CONTEXT-001.
  • docs/mcp.md — documents the new rule, fix guidance, and a coverage-limit note under the runtime anti-pattern rules.
  • docs/VALIDATION_RULE_REGISTRY.md — regenerated via scripts/build-validation-rule-registry.cjs (count 178 → 179, major 67 → 68).
  • test/unit/mcp/bestPracticesEngine.test.ts — 5 new tests.
  • test/fixtures/opportunity-create-verify-{good,bad}.testcase — negative + positive fixtures.

Note for the Provar team

This adds a new customer-facing validator rule (UI-SCREEN-CONTEXT-001) that changes provar_testcase_validate behaviour for create-and-verify Salesforce tests. The external/customer-facing docs (docs/provar-mcp-public-docs.md, docs/university-of-provar-mcp-course.md) are maintained separately and should be updated to describe the new rule.

🤖 Generated with Claude Code

RCA: Create-and-verify tests that leave the UiAssert nested under the action=New create screen scored a clean 100 with no warning, because no validator checked whether a verification step sits in the correct post-save screen context.
Fix: Added UI-SCREEN-CONTEXT-001 (check.type uiAssertScreenContext, MULTI_VALIDATOR_REGISTRY) with two heuristics over existing XML: UiAssert/UiRead nested under a UiWithScreen whose target URI carries action=New, and any UI action following a Save (locator binding action%3Dsave) within the same substeps block.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Quality Orchestrator

🟢 LOW · 2 / 100 · All changed files have mapped tests.


🧪 Tests to Run · Running 1 of 57 tests

  • unit/mcp/bestPracticesEngine.test.ts
▶ Run command
npx vitest run \
  unit/mcp/bestPracticesEngine.test.ts

⚡ quality-orchestrator  ·  /qo stub <file>  ·  qo analyze-local

RCA: The first cut of UI-SCREEN-CONTEXT-001 matched Save via raw substring (action%3dsave) so Save & New false-matched, ran heuristic 2 on every UiWithScreen so post-save steps under Edit/View false-fired, and emitted two violations for one step (assert after Save under New) double-penalising the local score (92.5 not 96.25).
Fix: isSaveAction now parses the locator binding and compares the decoded action token exactly to save; heuristic 2 is scoped to action=New only; validateUiAssertScreenContext de-dups to at most one violation per offending testItemId. Added Edit and Save & New false-positive guard tests, locked the bad fixture to one violation and score 96.25, refreshed docs/notes, and bumped version 1.6.2 to 1.6.3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mrdailey99 mrdailey99 merged commit 3a6901d into develop Jun 24, 2026
4 checks passed
@mrdailey99 mrdailey99 deleted the feature/PDX-516-uiassert-screen-context-rule branch June 24, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant