Proposed E37 Module and E38 Workflow. No operational connector or autonomous production repair is implemented by this design. The owner requested Sentry crashes, PostHog context and Datadog latency deterioration to lead to investigation, repair, independent validation, production verification and source feedback. Other services must fit the same replaceable contracts. This draft authorises design and tracking; it grants no service access, deployment or issue-writeback permission.
O37.1 bounded implementation candidate: adoption contract exposes caller-authenticated redacted intake and atomic Signal/checkpoint coverage. Corrected source/package implementation passed independent review; the original corrupting candidate and additive correction remain retained. Combined local checks passed 1,926; separate final documentation delta review passed. The finite ledger refuses growth and finite-erasure data; provider adapters, grouping/metrics and Repair remain unimplemented.
Existing parts and missing boundaries
The design already defines Signals and domain ownership. The current Factory model records the bounded independently reviewed intake candidate separately from unimplemented provider and Repair work. Discovery collects attributed public text, Service access inventories names and permissions, and Portfolio projects supplied facts. None is a production telemetry connector. Command journal, Evidence, Execution, Repository, Assurance and Product gate provide local primitives; their fixture Verdicts and local integration Receipts cannot establish production recovery. A general Action gateway and customer-outcome evaluator remain unimplemented.
| Existing contract | Preserve its responsibility |
|---|---|
| E13 incident recovery | Operations restores required customer service first, including when the planner/provider is unavailable; causal repair follows |
| E14 prioritisation | Compare affected customers, business value, urgency, alternatives and capacity; do not let alert count become priority |
| E16 performance | Realistic workload, frozen thresholds, before/after and customer observation; a lower cost alone is insufficient |
| E03–E05 delivery and measurement | Recovery readiness, authorised exposure, customer-path availability and independent benefit assessment |
| E24 / E32 Research | Bounded useful investigation and attributable findings; research does not grant release authority |
| E30 / E31 Implementation and Verification | Produce an immutable Candidate, then obtain an independent Verdict under separately owned acceptance |
| E36 Temporal Planning | Optional timing/capacity inputs and shared event-delivery contract; no service-specific connector, incident or repair ownership |
The proposed provider event integration publishes scoped Factory notifications to subscribed assistant hosts. Source-to-Factory ingestion still belongs here, with its own sender authentication and reconciliation. Neither direction grants a model commission, repair authority or permission for source writeback.
Add Operational Signal Intake, a Module that preserves source semantics, and Repair, a Workflow that connects independently owned stages. The fixed 23-family audit and original four-Workflow programme stay historical scope. These future parts must meet the same independent adoption gate; E35 does not acquire their implementation as a prerequisite.
View diagram source (Mermaid)
flowchart TD
S[Commissioned service adapters] --> I[Operational Signal Intake]
I --> C[Durable Repair case and episode]
C --> U{Urgent customer harm?}
U -->|yes| O[Independent Operations restoration]
O --> R[Investigate and reproduce]
U -->|no| R
R --> P[Select ready bounded Slice]
P --> B[Implementation]
B --> V[Independent Verification]
V --> A[Authorised release and recovery gates]
A --> M[Release and cohort specific production assessment]
M --> F[Separate source feedback Action and readback]
M -->|new regression| C
T[Optional temporal and business Signals] --> C
Arrows carry typed proposals and accepted facts, never implied Authority. Production failure can invoke separately commissioned restoration without waiting for this Workflow or a model. An unresolved alert may require observation or a decision instead of a code change.
Operational Signal Intake contract
Entry point: software-factory/operational-signals, exported as the O37.1 intake candidate (adoption contract); live source adapters and detector policy beyond it remain proposed. A caller supplies commissioned source adapters, identity mappings, storage, clock, Evidence validation and detector policy. Import is passive; no Factory, global Portfolio, scheduler, credential lookup or provider contact starts implicitly. Keep normalisation deterministic. A model may later propose an interpretation as untrusted data; it cannot set identity, coverage, severity policy or permission.
| Record or port | Required information |
|---|---|
SourceBinding | Provider/adapter version, organisation/project/account, Product, service/environment, permitted reads and writes, credential references, redaction/retention and collection bounds |
OperationalObservation | Stable source event/issue or metric-query identity, source revision, event/observed/received times, release/build/cohort where known, redacted content digest, provenance and coverage |
SourceCheckpoint | Bound source/store identity, cursor/watermark and revision, overlap/reconciliation window, durable accepted items and unresolved gaps |
OperationalSignal | Accepted observation references, symptom/metric and affected scope, detector version/reason, uncertainty and current disposition; no release permission |
Correlation | Explicit evidence linking observations or cases, confidence and rule version; separate from duplicate identity or causal proof |
CollectionResult | Exact, partial or unknown coverage, cursor progress, omissions/rate limits/auth failures and retry/wake bounds; never infer zero failures from missing data |
Adapters authenticate the actual sender or API response using their reviewed provider contract and current binding. A webhook is a wake-up or attributed observation, not a trusted instruction; signatures, replay limits and subsequent authoritative reads must match the chosen provider's supported semantics. Refuse wrong tenants, environments and mismatched identifiers. Limit destinations and retrieval to commissioned endpoints; never follow a payload-supplied arbitrary URL or execute source text. Distinguish unauthorised, unavailable, expired and corrupt input without leaking credentials.
Redact before persistent journals, outboxes, prompts and diagnostic logs. Stack traces, exception messages, request fields, user properties and session replay can contain secrets or personal data. Preserve the minimum useful provenance through bounded approved references/digests; fetching sensitive detail or replay needs separate read/retention authority. Redaction must preserve its policy/version and uncertainty, not manufacture a successful reproduction. Deletion and expired-source obligations apply to derived evidence and recovery copies under existing policy.
Deduplicate by bound source identity and stable event/revision, not text alone. An unchanged delivery replays; conflicting bytes under the same immutable identity refuse or require an explicit provider revision. Commit accepted observations and checkpoint progress atomically within the owned store; acknowledge only durable acceptance. Handle pagination, duplicate/out-of-order/late events, expired cursors, missed webhooks, bounded overlap and source reconciliation. Do not advance past a rejected or unaccounted page silently. A replaced store, changed credentials/scope or source retention gap requires explicit reconciliation and partial/unknown coverage, never an empty healthy result.
Issue grouping and fingerprints are mutable source projections. Retain underlying event IDs and explicit merge/split/alias revisions. Re-evaluate affected correlations without rewriting previous cases; an issue split or a changed grouping rule must not duplicate a repair or conceal a new failure. Never join Products on matching messages. Cross-service correlation keeps every source's provenance and scope; one Sentry error and one latency graph may share a cause, but correlation alone proves neither cause nor resolution.
Slow deterioration and measurement semantics
A metric observation retains the exact query and its digest, service/endpoint/environment/release dimensions, units, distribution/aggregation/rollup, time window and zone, numerator and denominator where applicable, sample count, sampling policy/rate and collection coverage. Keep the baseline, workload/population, seasonality rule, detector version, threshold, persistence/hysteresis, recovery criterion and minimum evidence. Changes to instrumentation, sampling, query or traffic mix can invalidate comparison and must be visible.
A p99 is a percentile of a defined distribution. Do not average separate p99 values and call the result a combined p99. Use a provider-supported distribution aggregation or retain distinct populations. Low volume, missing windows, partial telemetry and dropped samples remain insufficient evidence; zero alerts or a falling request count cannot establish improved latency. Client/server endpoints, old/new releases and customer segments must not be silently mixed.
Detect slow drift against commissioned customer/SLO limits and a declared trend or change rule as well as abrupt incidents. Do not let an adapting baseline quietly redefine worsening performance as healthy. Record why a deterioration crossed the rule, its persistence and customer impact; retain seasonal/reference windows. Suppression, noise limits and cooldowns have an owner, reason, expiry and escalation condition. They cannot erase the observation or suppress urgent harm beyond policy. Any forecast or detection model is advisory; policy and evidence decide routing.
Datadog documents distribution-based global percentile aggregation and distinct anomaly trigger/recovery windows, with seasonality choices. These inform adapter validation, not a universal detector configuration. Distributions · Anomaly monitors.
Repair Workflow contract
Proposed entry point: software-factory/repair, not exported today. An external host supplies Signal intake/read, investigation/reproduction, Implementation, independent Verification, authority/admission, release/recovery, production observation/evaluation and optional source-feedback ports. Each has versioned typed inputs/results, identity/pins, current permission checks, bounds, cancellation and unknown-effect semantics. Substitute conforming ports without consumer changes. Use E30/E31 contracts where supplied; external implementations are equally valid. Factory registration, Temporal Planning and the Research Workflow are optional callers/collaborators.
| Record | Required meaning |
|---|---|
RepairCase | Stable Product/service/symptom scope, linked source observations and correlations, owner, current priority and episode references |
RepairEpisode | One bounded recurrence/attempt to resolve a case, with frozen scope, state/version, causal occurrence, Run/Action IDs, authority/Budget pins and prior episode link |
ResolutionContract | Affected error/fingerprint or metric, release/environment/cohort, precondition/reproduction, baseline, target, guardrails, exposure mapping, observation window, minimum traffic/sample/coverage and independent evaluator |
ProductionResolution | Independently evaluated resolved, unresolved or inconclusive result against that exact contract, with release Receipt and observations, limitations and follow-up |
SourceFeedback | Separate proposed/authorised Action, exact provider target/version/intent, payload digest, dispatch/reconciliation state, Receipt and readback |
Keep episode progress durable: received, investigating, ready, implementing, verifying, awaiting-release, observing and closing. A wait records its previous stage, reason, owner, attempted remedies and wake condition/deadline. Terminal dispositions distinguish verified resolution, justified dismissal, duplicate, inconclusive and cancellation. A finding that cannot be reproduced is not a verified fix. A duplicate points to the owning case/episode; it does not discard its evidence. Independently validate any final conclusion. All lists, attempts, retained data, time, provider use and pending effects have commissioned bounds.
Urgent harm goes to the existing independent Operations restoration contract first. Restore safe required/replacement behaviour using current recovery authority, including during retirement; do not revive deliberately retired harmful behaviour. Continue causal investigation after restoration. Ordinary or gradual deterioration enters bounded prioritisation/investigation and respects incident reserves. Within commissioned access, authority and Budget, advance autonomously; do not add a manual approval at each stage. Missing authority, access, policy or qualified capability creates a durable wait or the existing owner queue, with the precise unavailable decision.
Investigation can consume PostHog context, Sentry evidence, logs, traces and research through permitted ports. Establish a falsifiable hypothesis and reproduce the observed failure on the affected pre-change version with relevant environment, inputs and realistic workload where evidence permits. Freeze acceptance before the implementer repairs it. Preserve an independently owned regression scenario that fails before and passes after; a crash needs the affected path, and a performance issue needs representative before/after distributions and guardrails. An unrepresentative synthetic benchmark, newly weakened threshold or success report from the implementer cannot verify repair. Insufficient evidence remains inconclusive and may commission a bounded instrumentation Slice.
Implementation returns an immutable Candidate and Evidence. Verification independently checks the exact Candidate, acceptance, environment and relevant unchanged behaviours. A changed Candidate or stale/revoked input needs fresh Verification. Release uses the caller's proved Action/reconcile protocol, actual artefact/release identity and current deployment authority, with applicable alpha, rollback and customer-data protections. Local repository integration alone is not a release adapter. The absent general Action gateway must be supplied/proved at this boundary; the Workflow cannot invent one by treating a callback return as a Receipt.
Production verification and source closure
Freeze the Resolution contract before repair depends on it. Bind deployed artefact and release IDs to the affected service, environment, endpoint, feature flags and exposed cohort. Begin the declared production observation after confirmed exposure, not at commit time. Require independently checked instrumentation, adequate traffic/sample and coverage, baseline comparability, required duration, guardrails and adverse segment checks. For a rare crash, silence without enough exposure remains inconclusive. For p99 deterioration, an aggregate improvement cannot hide a harmed segment or changed traffic denominator. Delayed telemetry and remaining old releases must be accounted for before concluding anything about the repaired release.
Production resolution, provider triage and customer benefit are separate facts. A release, green test, no alert or manually resolved service issue is not production resolution. Resolve the episode only under its declared evidence rule; retain unresolved or inconclusive outcomes and their next decision. A scoped resolution does not claim unrelated releases, cohorts or services are fixed. A later qualifying regression opens a new episode linked to the previous evidence; terminal Runs and past assessments remain immutable. Identify whether a late event is from an old release before calling it a new regression. Recovery follows existing Operations policy without expanding authority.
Sentry permits manual resolution and labels recurring resolved issues as regressed; issue details depend on environment and can identify affected releases. Therefore, source status is triage data and release scope must be checked separately. Issue states · Issue details.
The requested Sentry path commissions source feedback as required: verified readback precedes episode closure. Other external consumers may declare feedback optional where their service has no writable status. After a scoped production resolution, propose the commissioned source feedback Action such as resolving the appropriate Sentry issue for its supported release scope or adding an evidence-linked status. Recheck current source grouping, issue revision, Product/environment, grant and payload at dispatch. Preserve one logical Action ID and intent across retries and replacement workers. If the API lacks native idempotency or a conditional update, prove an adapter-specific execute/reconcile contract; do not promise exactly-once remote effects. A lost acknowledgement stays unknown until readback/reconciliation proves the result or proves it safe to retry.
Read back the intended target/status/release link and retain the attributable Receipt plus any concurrent update or disagreement. A generic successful response is insufficient. Never overwrite a new regression, different release or human triage change silently. If the source cannot express the proven scope, keep the Factory result and use an authorised narrower comment/status or an explicit pending feedback reason. Feedback is separate from the Production resolution record. Required feedback that remains unknown leaves closing in a durable wait; optional feedback remains explicitly pending without inventing source closure.
PostHog groups exception events into issues and offers analytics/session context and webhook alert integration. These are useful inputs, subject to their own scope and privacy permission; they do not authenticate a release or fix. Error tracking.
Delivery, authority and bounded recurrence
Reuse the shared delivery contract: stable causal occurrence/Command IDs, immutable payload pins, inbox acceptance and verified acknowledgement, per-consumer checkpoints, ownership fencing, current dispatch checks and unknown-effect reconciliation. This is a neutral replaceable port, not a dependency on a planning engine. Its generic implementation/proof is counted once in E36; E37/E38 add source reconciliation and repair-specific state/composition. If delivery is implemented first here, move that shared estimate explicitly rather than count it again. Journal pending Events alone are not a dispatcher.
Source delivery, repair episode, worker Attempt and outward Action each have different identities. Duplicate webhooks or matching research/timing Signals converge on an existing logical episode operation, not a new release. Cancellation, revocation and Budget exhaustion stop new dispatch while independent reconciliation continues for admitted unknown effects. A receiver must enforce current authority at its documented linearisation boundary; cross-store reads do not create a transaction. Cap polling, retries, fan-out, causal depth, episodes per commissioned window and work per tick. Repeated self-caused notifications are recognised; unresolved harm is retained and escalated rather than erased by a dedupe or rate limit.
Independent proof and work plan
E37's external consumer must ingest useful crash/context/metric fixtures through replaceable commissioned adapters and reproduce identity, redaction, cursor, grouping and detector results. E38's separate consumer must complete a useful repair episode using public contracts and independently owned verification/measurement collaborators. Neither may import private Factory code or require unrelated Portfolio/scheduling state. Exact package/adapter/environment evidence must distinguish synthetic contract proof from a real authorised service exercise and actual production proof.
The independent matrix includes duplicate/reordered/missed webhooks, expired cursor and changed scope, forged source identity and hostile metadata, secret/retention canaries, grouping merge/split, cross-Product correlation refusal, rate limits and partial telemetry; sparse/mixed percentile populations, changed sampling, gradual drift and seasonal confounders; failed reproduction, stale Candidate and revoked authority; crash at each write/dispatch/ack boundary, unknown release/writeback, cancellation and exhausted Budget; insufficient production traffic, harmed segments, old-release events, new regression episodes and concurrent source triage. Restore-first recovery must still work when planning, research and this Workflow are unavailable.
| Epic | Bounded increments | Additional estimate |
|---|---|---|
| E37 Operational Signal Intake | Neutral records and durable ingestion; replaceable source bindings/reconciliation; metric/detector semantics and independent external fault/privacy proof | 3–5 |
| E38 Repair Workflow | Case/episode and autonomous waits; reproduce/frozen acceptance; Implementation/Verification composition; authorised release plus production evaluator; source feedback and full fault/substitution proof | 5–8 |
Low confidence: 8–13 additional reviewed implementation increments. Original E28–E35 remains 24–38, E36 remains 5–8, and the combined programme is 37–59. Increments include their external example, independent review and docs; they are not days or equal throughput units. Re-estimate after a redacted multi-service intake consumer and a complete disposable repair episode with a real acknowledgement-loss experiment.
These estimates assume caller-supplied or separately proved release, recovery, authority, Evidence and provider-access facilities. Building a universal deployment platform, credential manager or all provider integrations is excluded; any newly discovered missing foundation must receive explicit scope and an updated estimate before depending on it. E37 should first prove one crash source and one metric source, then a substitute context/source adapter; no service brand is mandatory for other consumers. Completing E37/E38 adoption does not complete E13/E16 or claim the user's real production systems are repaired.
