Design only, 2 October 2026; nothing is implemented, installed or reviewed. This page proposes sf-ui-observation/1, the record a planned UI collector (roadmap X4) would store for each acceptance check at its pinned stage, and the rules that turn those records into a Verdict. It answers #77 for E31 (#9). Every rule below is a proposal until an independent review accepts it.
The ideas come from TesterArmy e2e, an Apache-2.0 TypeScript framework over Playwright. The Factory takes the ideas, not the framework: whether to install Playwright, axe-core or e2e is owner decision PD-25, still open.
Where it fits
The UI collector would be a sibling of the Repository collector, and its judge a sibling of Repository Assurance. The collector runs each acceptance check on its pinned stage (base or Candidate), stores one sf-ui-observation/1 Evidence record per acceptance check at its pinned stage, and never judges: a record states what the runner saw, never whether that was wanted. A pure judge reads the records back from the Evidence store and returns Verified, Failed or Inconclusive under the proposed scope local-ui.
Exploration is a separate, upstream activity. It belongs in the design debt register and whole-product review (X5) or in post-development validation, never inside Verification. Rule U12 covers how its findings can later count.
What the frozen acceptance pins
The UI acceptance reuses the four parts of Repository Assurance's acceptance, with the same stage rules. Check names are unique across all four.
| Part | Stage | Pinned expectation |
|---|---|---|
before | base | One fixed check that must fail at a named, pinned expect |
after | candidate | One fixed check, named differently, that must pass |
guardrails | candidate | Fixed checks, including pinned axe rule sets, that must pass |
reviews | candidate, if run | None: always inconclusive (rule U2); every model check belongs here |
An acceptance that places a model check outside reviews is refused when parsed. The acceptance also pins, by digest, the recipe, the replay file, the site allowlist, the secret-name list, the per-step and run budgets, and a frozen environment record (rule U14). Expected results exist only here, never in the recipe the Candidate could edit.
What a record holds
One record describes one acceptance check at its pinned stage (base or Candidate) under one Attempt. Its Evidence metadata names exactly that check; its observation holds exactly one result. Shared run artefacts may be referenced by several check records, but a stage-level bundle never substitutes for a check record. Artefacts are stored as their own Evidence records and named here by id, so a record can be checked without trusting the runner's file layout.
| Field | Contents |
|---|---|
format | "sf-ui-observation/1" |
| Run identity | The eight Evidence pins (candidateDigest, scenarioRevision, recipeRevision, runId, attemptId, epoch, environment, collector), the unique acceptance check name, its stage (base or candidate) and the run's start and end times |
| Start | Start and end times of this check; an attempt refused before launch has no qualifying check record |
| Pinned inputs | Digests of the frozen acceptance, the frozen environment, the recipe (test files as one snapshot), the replay file (or null when the recipe has no agent actions), the site allowlist and the secret-name list |
| Check | Exactly one acceptance check: name, kind (fixed or model), one closed observation code, the id of the expect that ended the check (or null), tries, modelCalls, durationMs and the artefact ids it produced. There is no outcome field, as in the Evidence module's process observation |
| Artefacts | Per artefact: role (trace, screenshot, video, report, junit, accessibility-tree or console), Evidence id (64 lower-case hexadecimal characters), byte count and media type |
| Budgets | Pinned per-step maxModelCalls and stepTimeoutMs, run timeoutMs, maxArtefactBytes and maxOutputBytes, and the measured use of each |
| Environment | Platform, runner name and version digest, browser name and build, viewport, locale, time zone, colour scheme, the confinement profile and its digest, and the observed telemetry state |
| Process | Exit, termination (timeout, cancelled, output-limit or none), output overflow and whether the run was attributable |
Closed codes
Each check carries exactly one observation code, a runner fact. Whether it passes is decided only by the judge, against the check's pinned stage expectation (rule U5). An unknown code makes the record unreadable, which the judge treats as missing Evidence.
| Code | What the runner saw |
|---|---|
PASSED | Every expect in a fixed check held on its first and only try |
EXPECT_FAILED | A fixed expect observed its condition false, or still false at the end of its pinned wait; the record names that expect |
MODEL_JUDGED | A model-judged step (assert, waitFor or extract) ran, whatever the model said |
REPLAY_STALE | A pinned replay action no longer matches the page |
REPLAY_MISSING | An agent action has no pinned replay entry |
RETRY_PASS | The check passed only after an earlier try failed |
STEP_TIMEOUT | A step ran past its pinned step timeout outside an expect wait |
MODEL_BUDGET | A step reached its pinned model-call budget |
RUN_TIMEOUT | The run ran past its pinned run timeout |
SITE_REFUSED | The page tried to reach a site outside the allowlist; the request was blocked |
SECRET_REFUSED | A fill asked for a secret name not listed, or on a site not allowed to receive it |
ARTEFACT_MISSING | An artefact the check names is absent from the Evidence store or its bytes do not match its id |
RUNNER_ERROR | The runner crashed, or its output could not be attributed to this check |
Four conditions stop a run before any check. The proposed collector reports the attempt as not started and creates no qualifying check record; missing check Evidence is Inconclusive: NOT_CONFINED (no reviewed boundary), TELEMETRY_ON (telemetry not confirmed off), INPUT_DRIFT and ENVIRONMENT_DRIFT (rule U14).
Judging rules
Each rule is written so a test can plant its counter-example.
- U1. Fixed checks decide. Only a
fixedcheck (a deterministic locator andexpect, or a pinned axe rule set at a pinned axe version) can be observed asPASSEDorEXPECT_FAILED. A record giving either code for amodelcheck does not qualify. - U2. Model-judged checks are review checks. Every
modelcheck maps to the reserved review check that Repository Assurance already holds, which is always Inconclusive. A model's answer, however confident, cannot turn a Verdict into Verified or Failed. - U3. Strict pinned replay. An agent action enters the replay file only after a later fixed check in the same exploratory run passes. The replay file is pinned by digest in the frozen acceptance. In Verification the runner replays strictly: a mismatch gives
REPLAY_STALE, a missing entryREPLAY_MISSING, and neither falls back to a live model. A verification record withmodelCallsabove zero on a replayed step does not qualify. - U4. Flaky is Inconclusive. The recipe pins retries to zero, and a recipe that allows retries is refused when parsed. If a record still shows
triesabove one, a final pass isRETRY_PASS, neverPASSED. - U5. Judged against the pinned stage. The judge decides each check from its acceptance part, using only qualifying records for that check's own stage. A
beforecheck (base) passes only onEXPECT_FAILEDat its pinnedexpectand fails onPASSED, because the base already behaves as the Candidate should. Anafteror guardrail check (Candidate) passes only onPASSEDand fails onEXPECT_FAILEDat anyexpect. Every other code, abeforethat fails at a differentexpect, and a check with no qualifying record are inconclusive; reviews are always inconclusive. The Verdict then follows Repository Assurance's rule 2: any failed check givesFailed, even beside inconclusive checks; otherwise any inconclusive check or rejected record givesInconclusive; otherwiseVerified. So a correct repro, which fails on the base, can reachVerified. - U6. Evidence under pins. Every trace, screenshot, video and report is hashed and stored as Evidence under the run's eight pins. The judge reads each one back through the Evidence store and checks its id. The collector writes into a fresh, empty directory per Attempt and never deletes artefacts at the start of a run.
- U7. Bounded steps, distinct codes. Each step has its own model-call budget and timeout; exceeding them gives
MODEL_BUDGETandSTEP_TIMEOUTrespectively, never one shared code. A Budget is never extended during a run. - U8. Limited model view. Where a model takes part, it sees accessibility-tree text only, with password fields masked and secrets shown by name as
<secret:name>. The collector substitutes the value at fill time. A planted canary value must be absent from every stored record, artefact and log. Model text is never run as code or used as a selector; it can only choose among element references the runner offered. - U9. Site allowlist. The recipe pins the sites a run may visit and, separately, the sites that may receive each named secret. Anything else is blocked and recorded as
SITE_REFUSEDorSECRET_REFUSED. This follows the names-only rule of Service access: records hold secret names, never values. - U10. Telemetry off. The runner's telemetry is turned off in its pinned configuration, and the boundary denies any connection outside the allowlist. A run that cannot confirm both does not start (
TELEMETRY_ON). - U11. Confined execution first. The runner works only inside the Confined executor or an equivalent, independently reviewed host boundary; otherwise it does not start (
NOT_CONFINED). The current profile denies spawn, fork and network, so it cannot host a browser as it stands. - U12. Exploration yields untrusted findings. A bug-bash finding is never Evidence and never a Verdict. It counts only once it becomes a fixed repro pinned as the
beforecheck (failing at its pinnedexpecton the base) and theaftercheck (passing on the Candidate), judged under U5. Until then it stays a proposal. - U13. No relabelling. UI records never qualify in the fixture or repository judges, and fixture or repository records never qualify here. A
local-uiVerdict is local browser evidence for one pinned environment, not accessibility conformance, release or customer value. - U14. Drift blocks. A record qualifies only when its pinned inputs (acceptance, recipe, replay file, site allowlist, secret-name list and budgets) and its environment (platform, runner name and version digest, browser name and build, viewport, locale, time zone, colour scheme and confinement profile digest) equal the frozen acceptance's pins. Before launch, the collector compares what it is about to run with the pins; any difference gives a not-started attempt outcome (no qualifying check record),
INPUT_DRIFTorENVIRONMENT_DRIFT, naming each field that differs. After the run, the judge re-checks the same fields against the values the run observed and rejects a record that differs. Nothing re-pins, upgrades the browser or accepts a near match; a new pin needs a new frozen acceptance.
What TesterArmy e2e does that the Factory does not adopt
- It runs test code without a sandbox; U11 makes confinement a prerequisite.
- Its telemetry is on by default; U10 requires it off and enforced by the boundary.
- Its model-driven steps can pass or fail a test; U1 and U2 keep that power with fixed checks only.
Not established
- No UI collector, judge, record parser or code exists. Nothing is installed: no Playwright, axe-core or e2e, pending PD-25.
- No browser has run under the Confined executor. Its tested profile denies the spawn, fork and loopback network a browser needs, so a browser profile would need its own boundary review. Confinement of browsers is unproved.
- The codes, field names and the
local-uiscope are proposals with no independent review. - Roadmap X2 (
sf-design/1) is not built, and X4 depends on it. - E31's acceptance is unchanged. The proposed
local-uijudge is not supported by the current Verification Workflow contract. W31 needs a separately versioned and independently reviewed collector/judge binding before an external host can substitute it; this design satisfies no E31 criterion on its own.
Next step
After PD-25 and X2, a spike in its own worktree runs e2e with no model against the local dashboard fixture inside a confined boundary. It records sf-ui-observation/1 drafts so an independent review can test U1 to U14 against real runs. No model use in CI is planned.
