Factory docs, home
Page navigation

Pre-release documentation. Proposed designs are labelled in the page; acceptance evidence is maintained in the build review. Local paths appear as placeholders. Current state: Build review.

Design only, 2 October 2026; nothing is implemented, installed or reviewed. This page proposes sf-ui-observation/1, the record a planned UI collector (roadmap X4) would store for each acceptance check at its pinned stage, and the rules that turn those records into a Verdict. It answers #77 for E31 (#9). Every rule below is a proposal until an independent review accepts it.

The ideas come from TesterArmy e2e, an Apache-2.0 TypeScript framework over Playwright. The Factory takes the ideas, not the framework: whether to install Playwright, axe-core or e2e is owner decision PD-25, still open.

Where it fits

The UI collector would be a sibling of the Repository collector, and its judge a sibling of Repository Assurance. The collector runs each acceptance check on its pinned stage (base or Candidate), stores one sf-ui-observation/1 Evidence record per acceptance check at its pinned stage, and never judges: a record states what the runner saw, never whether that was wanted. A pure judge reads the records back from the Evidence store and returns Verified, Failed or Inconclusive under the proposed scope local-ui.

Exploration is a separate, upstream activity. It belongs in the design debt register and whole-product review (X5) or in post-development validation, never inside Verification. Rule U12 covers how its findings can later count.

What the frozen acceptance pins

The UI acceptance reuses the four parts of Repository Assurance's acceptance, with the same stage rules. Check names are unique across all four.

PartStagePinned expectation
beforebaseOne fixed check that must fail at a named, pinned expect
aftercandidateOne fixed check, named differently, that must pass
guardrailscandidateFixed checks, including pinned axe rule sets, that must pass
reviewscandidate, if runNone: always inconclusive (rule U2); every model check belongs here

An acceptance that places a model check outside reviews is refused when parsed. The acceptance also pins, by digest, the recipe, the replay file, the site allowlist, the secret-name list, the per-step and run budgets, and a frozen environment record (rule U14). Expected results exist only here, never in the recipe the Candidate could edit.

What a record holds

One record describes one acceptance check at its pinned stage (base or Candidate) under one Attempt. Its Evidence metadata names exactly that check; its observation holds exactly one result. Shared run artefacts may be referenced by several check records, but a stage-level bundle never substitutes for a check record. Artefacts are stored as their own Evidence records and named here by id, so a record can be checked without trusting the runner's file layout.

FieldContents
format"sf-ui-observation/1"
Run identityThe eight Evidence pins (candidateDigest, scenarioRevision, recipeRevision, runId, attemptId, epoch, environment, collector), the unique acceptance check name, its stage (base or candidate) and the run's start and end times
StartStart and end times of this check; an attempt refused before launch has no qualifying check record
Pinned inputsDigests of the frozen acceptance, the frozen environment, the recipe (test files as one snapshot), the replay file (or null when the recipe has no agent actions), the site allowlist and the secret-name list
CheckExactly one acceptance check: name, kind (fixed or model), one closed observation code, the id of the expect that ended the check (or null), tries, modelCalls, durationMs and the artefact ids it produced. There is no outcome field, as in the Evidence module's process observation
ArtefactsPer artefact: role (trace, screenshot, video, report, junit, accessibility-tree or console), Evidence id (64 lower-case hexadecimal characters), byte count and media type
BudgetsPinned per-step maxModelCalls and stepTimeoutMs, run timeoutMs, maxArtefactBytes and maxOutputBytes, and the measured use of each
EnvironmentPlatform, runner name and version digest, browser name and build, viewport, locale, time zone, colour scheme, the confinement profile and its digest, and the observed telemetry state
ProcessExit, termination (timeout, cancelled, output-limit or none), output overflow and whether the run was attributable

Closed codes

Each check carries exactly one observation code, a runner fact. Whether it passes is decided only by the judge, against the check's pinned stage expectation (rule U5). An unknown code makes the record unreadable, which the judge treats as missing Evidence.

CodeWhat the runner saw
PASSEDEvery expect in a fixed check held on its first and only try
EXPECT_FAILEDA fixed expect observed its condition false, or still false at the end of its pinned wait; the record names that expect
MODEL_JUDGEDA model-judged step (assert, waitFor or extract) ran, whatever the model said
REPLAY_STALEA pinned replay action no longer matches the page
REPLAY_MISSINGAn agent action has no pinned replay entry
RETRY_PASSThe check passed only after an earlier try failed
STEP_TIMEOUTA step ran past its pinned step timeout outside an expect wait
MODEL_BUDGETA step reached its pinned model-call budget
RUN_TIMEOUTThe run ran past its pinned run timeout
SITE_REFUSEDThe page tried to reach a site outside the allowlist; the request was blocked
SECRET_REFUSEDA fill asked for a secret name not listed, or on a site not allowed to receive it
ARTEFACT_MISSINGAn artefact the check names is absent from the Evidence store or its bytes do not match its id
RUNNER_ERRORThe runner crashed, or its output could not be attributed to this check

Four conditions stop a run before any check. The proposed collector reports the attempt as not started and creates no qualifying check record; missing check Evidence is Inconclusive: NOT_CONFINED (no reviewed boundary), TELEMETRY_ON (telemetry not confirmed off), INPUT_DRIFT and ENVIRONMENT_DRIFT (rule U14).

Judging rules

Each rule is written so a test can plant its counter-example.

  • U1. Fixed checks decide. Only a fixed check (a deterministic locator and expect, or a pinned axe rule set at a pinned axe version) can be observed as PASSED or EXPECT_FAILED. A record giving either code for a model check does not qualify.
  • U2. Model-judged checks are review checks. Every model check maps to the reserved review check that Repository Assurance already holds, which is always Inconclusive. A model's answer, however confident, cannot turn a Verdict into Verified or Failed.
  • U3. Strict pinned replay. An agent action enters the replay file only after a later fixed check in the same exploratory run passes. The replay file is pinned by digest in the frozen acceptance. In Verification the runner replays strictly: a mismatch gives REPLAY_STALE, a missing entry REPLAY_MISSING, and neither falls back to a live model. A verification record with modelCalls above zero on a replayed step does not qualify.
  • U4. Flaky is Inconclusive. The recipe pins retries to zero, and a recipe that allows retries is refused when parsed. If a record still shows tries above one, a final pass is RETRY_PASS, never PASSED.
  • U5. Judged against the pinned stage. The judge decides each check from its acceptance part, using only qualifying records for that check's own stage. A before check (base) passes only on EXPECT_FAILED at its pinned expect and fails on PASSED, because the base already behaves as the Candidate should. An after or guardrail check (Candidate) passes only on PASSED and fails on EXPECT_FAILED at any expect. Every other code, a before that fails at a different expect, and a check with no qualifying record are inconclusive; reviews are always inconclusive. The Verdict then follows Repository Assurance's rule 2: any failed check gives Failed, even beside inconclusive checks; otherwise any inconclusive check or rejected record gives Inconclusive; otherwise Verified. So a correct repro, which fails on the base, can reach Verified.
  • U6. Evidence under pins. Every trace, screenshot, video and report is hashed and stored as Evidence under the run's eight pins. The judge reads each one back through the Evidence store and checks its id. The collector writes into a fresh, empty directory per Attempt and never deletes artefacts at the start of a run.
  • U7. Bounded steps, distinct codes. Each step has its own model-call budget and timeout; exceeding them gives MODEL_BUDGET and STEP_TIMEOUT respectively, never one shared code. A Budget is never extended during a run.
  • U8. Limited model view. Where a model takes part, it sees accessibility-tree text only, with password fields masked and secrets shown by name as <secret:name>. The collector substitutes the value at fill time. A planted canary value must be absent from every stored record, artefact and log. Model text is never run as code or used as a selector; it can only choose among element references the runner offered.
  • U9. Site allowlist. The recipe pins the sites a run may visit and, separately, the sites that may receive each named secret. Anything else is blocked and recorded as SITE_REFUSED or SECRET_REFUSED. This follows the names-only rule of Service access: records hold secret names, never values.
  • U10. Telemetry off. The runner's telemetry is turned off in its pinned configuration, and the boundary denies any connection outside the allowlist. A run that cannot confirm both does not start (TELEMETRY_ON).
  • U11. Confined execution first. The runner works only inside the Confined executor or an equivalent, independently reviewed host boundary; otherwise it does not start (NOT_CONFINED). The current profile denies spawn, fork and network, so it cannot host a browser as it stands.
  • U12. Exploration yields untrusted findings. A bug-bash finding is never Evidence and never a Verdict. It counts only once it becomes a fixed repro pinned as the before check (failing at its pinned expect on the base) and the after check (passing on the Candidate), judged under U5. Until then it stays a proposal.
  • U13. No relabelling. UI records never qualify in the fixture or repository judges, and fixture or repository records never qualify here. A local-ui Verdict is local browser evidence for one pinned environment, not accessibility conformance, release or customer value.
  • U14. Drift blocks. A record qualifies only when its pinned inputs (acceptance, recipe, replay file, site allowlist, secret-name list and budgets) and its environment (platform, runner name and version digest, browser name and build, viewport, locale, time zone, colour scheme and confinement profile digest) equal the frozen acceptance's pins. Before launch, the collector compares what it is about to run with the pins; any difference gives a not-started attempt outcome (no qualifying check record), INPUT_DRIFT or ENVIRONMENT_DRIFT, naming each field that differs. After the run, the judge re-checks the same fields against the values the run observed and rejects a record that differs. Nothing re-pins, upgrades the browser or accepts a near match; a new pin needs a new frozen acceptance.

What TesterArmy e2e does that the Factory does not adopt

  • It runs test code without a sandbox; U11 makes confinement a prerequisite.
  • Its telemetry is on by default; U10 requires it off and enforced by the boundary.
  • Its model-driven steps can pass or fail a test; U1 and U2 keep that power with fixed checks only.

Not established

  • No UI collector, judge, record parser or code exists. Nothing is installed: no Playwright, axe-core or e2e, pending PD-25.
  • No browser has run under the Confined executor. Its tested profile denies the spawn, fork and loopback network a browser needs, so a browser profile would need its own boundary review. Confinement of browsers is unproved.
  • The codes, field names and the local-ui scope are proposals with no independent review.
  • Roadmap X2 (sf-design/1) is not built, and X4 depends on it.
  • E31's acceptance is unchanged. The proposed local-ui judge is not supported by the current Verification Workflow contract. W31 needs a separately versioned and independently reviewed collector/judge binding before an external host can substitute it; this design satisfies no E31 criterion on its own.

Next step

After PD-25 and X2, a spike in its own worktree runs e2e with no model against the local dashboard fixture inside a confined boundary. It records sf-ui-observation/1 drafts so an independent review can test U1 to U14 against real runs. No model use in CI is planned.

Source: docs/ui-observation.md