docs: propose run runtime recovery

2026-08-07 11:36:13 +08:00
parent 87e903998e
commit 3818fa0344
7 changed files with 271 additions and 0 deletions
@@ -0,0 +1,2 @@
 schema: spec-driven
 created: 2026-08-07
@@ -0,0 +1,78 @@
 ## Context
 Platform persists a server lifecycle projection after Run reports `process.start` as `running`. Generated Run only reports the autonomous bootstrap terminal result; its process supervisor subsequently records a child exit locally without issuing another lifecycle report. When Run is stopped, Platform consequently retains `running` even though it has no fresh process observation. The existing UI displays that projection as current status.
 Run also emits autonomous process logs to a stable server-bound stream ID. Its `SpoolLogSink` currently owns one in-memory counter that starts at zero for every Run process, while Platform accepts only a contiguous range after its latest acknowledged sequence or an exact duplicate. A restarted Run therefore sends a different entry in an old sequence range, receives a conflict, and retries the blocked spool segment indefinitely. SSE correctly delivers only persisted Platform logs, so it has no new event to show.
 The independent `run` repository remains machine lifecycle authority. Platform owns authentication, audit, desired lifecycle requests, durable ingest, and projections. Plugins remain the sole owner of game-specific lifecycle actions and log-source declarations.
 ## Goals / Non-Goals
 **Goals:**
 - Keep Platform lifecycle projections convergent with Run-observed managed-process transitions, including process exit and Run restart recovery.
 - Distinguish the last observed server lifecycle fact from whether its dedicated Run endpoint is currently fresh enough to vouch for it.
 - Make each durable Run log stream resume with a monotonically increasing sequence after an ordinary Run restart.
 - Turn unreconcilable log overlap into an explicit, observable quarantine outcome rather than an infinite retry loop.
 - Preserve the existing signed Run channel, bounded log spool, and browser SSE delivery model.
 **Non-Goals:**
 - Infer a stopped game process merely because the Run executable or heartbeat stopped.
 - Add a game-specific status probe, file tail, executable name, or child-process rule to Platform or Run.
 - Give Platform Web direct host, shell, socket, credential, or log-spool access.
 - Rewrite historical log retention or make browser SSE a source of truth.
 - Re-add the independent Run source tree to this repository.
 ## Decisions
 ### Report managed process transitions through the lifecycle channel
 Run will retain enough autonomous-assignment identity to turn a local supervised process transition into a signed lifecycle report. The initial `running` observation, a later `stopped` observation, and an `exited` observation use the existing report route with process state, exit classification, and bounded audit summary. On Run startup, supervisor reconciliation will report any persisted managed process identity after checking its actual OS process state. Reports are idempotent by observed state/version so a restart cannot repeatedly create misleading transitions.
 This retains Platform's existing authorization and projection route rather than adding process data to heartbeat. Heartbeat answers endpoint freshness, while lifecycle reports carry process facts. Reusing job result messages was rejected because autonomous lifecycle work has no Platform job lease.
 ### Model freshness separately from lifecycle state
 `serverInstances.state` remains the last Run-reported lifecycle projection. Platform will derive a server runtime observation view from the bound endpoint's last heartbeat and endpoint status, with a documented freshness threshold derived from the negotiated heartbeat interval. A current observation can be `fresh`; one beyond the threshold is `stale` or `unreachable` without changing the stored lifecycle state. The management UI displays both: for example, `Last observed: running` and `Run offline / state unverified`.
 Platform must not project `stopped` only because heartbeat expires, because a Run or its descendants may continue locally. Adding a game-specific lifecycle state was rejected: freshness is a generic observation attribute, not game setup state.
 ### Persist watermarks per durable log stream
 Run's spool will store an atomic per-stream watermark containing at least the highest locally allocated sequence and highest Platform-acknowledged sequence. Sequence allocation is keyed by full `logStreamId`, not a worker-global counter. The spool updates the acknowledged watermark only after a valid Platform response covers the segment, then retains that watermark even after deleting the acknowledged segment.
 On restart, Run opens the same spool root, restores watermarks and pending segments, and allocates the next sequence above both. A clean spool root that lacks a watermark for a preexisting Platform stream needs an explicit reconciliation result before it may reuse that stable stream ID. The preferred contract is a signed lightweight Run log-stream progress endpoint returning only the latest acknowledged sequence for the Run-bound stream; no log bodies, host paths, or browser access are exposed. A nonempty local spool remains the source for retrying unacknowledged content.
 Stable stream IDs are retained so the UI can continue to show a coherent stream across Run restarts. Rotating the stream ID on every restart was rejected because it fragments operator history and masks durability failures.
 ### Quarantine irreconcilable batches and keep delivery moving
 Platform responses distinguish a sequence gap from a conflicting acknowledged range. For a conflict, Run must not resend the exact same segment indefinitely. It records a redacted diagnostic, moves the affected segment to a durable rejected area, and advances only after reconciling its next allocation watermark. A gap that indicates local data loss follows the same safe isolation path until progress reconciliation completes; it must not be silently skipped.
 Quarantining protects control and live log delivery from a permanently poisoned spool segment, but the operator receives an audit/diagnostic signal. Platform does not accept replacement log bodies for an already acknowledged range.
 ### Define Run shutdown separately from server stop
 Run graceful shutdown will perform a generic supervisor shutdown procedure: preserve its journal and flush bounded durable channels, and report only process facts it actually observes. An explicit operator stop remains a plugin-declared lifecycle action executed by Run. Windows process-tree containment or child cleanup must be generic and can be implemented only when it preserves the declared lifecycle action semantics; closing Run must never fabricate a stopped projection without observation.
 ## Risks / Trade-offs
 - [Risk] A Run update can restart before its last log ACK is durably recorded. → Mitigation: write the pending segment before sending it and atomically persist the ACK watermark before deleting the segment.
 - [Risk] A lost or manually deleted spool root cannot prove the next sequence for a stable Platform stream. → Mitigation: require signed progress reconciliation before allocation and quarantine conflicting data rather than overwriting history.
 - [Risk] Endpoint heartbeat is temporarily delayed while a healthy server runs. → Mitigation: show freshness as an observation qualifier, keep the last lifecycle fact visible, and use a threshold based on the negotiated heartbeat interval rather than one missed beat.
 - [Risk] Reporting process exits can race with an explicit stop result. → Mitigation: include observed state identity/timestamp and make Platform reject stale transition regressions while accepting equivalent terminal facts idempotently.
 - [Risk] Platform and Run releases are deployed out of order. → Mitigation: version the progress endpoint/response capability, retain compatible ingest behavior, and make Run fail closed for missing reconciliation rather than reuse a stale sequence.
 ## Migration Plan
 1. Add Platform domain/DTO/protocol support for Run log-stream progress and runtime observation freshness while preserving existing lifecycle and log ingest routes.
 2. Deploy Platform compatibility first; it accepts existing uploads and serves progress to capable signed Runs.
 3. Release Run with persisted per-stream watermarks, progress reconciliation, conflict quarantine, and managed-process transition reporting.
 4. Deploy Platform Web to present observation freshness separately from lifecycle projection and show log ingest recovery diagnostics.
 5. Exercise a Windows generated Run restart with retained spool, an empty/recreated spool, supervised process exit, explicit stop, and offline endpoint scenarios.
 Rollback is code-only. A Platform rollback must keep accepting the existing signed lifecycle and ingest routes. A Run rollback retains spool segments and watermarks; operators must not delete spool state as a rollback step.
 ## Open Questions
 - The precise heartbeat freshness multiplier should be standardized alongside the existing negotiated heartbeat interval; the implementation must choose one documented value and cover it with tests.
 - The final generic Windows containment primitive must be validated against graceful plugin-declared stop behavior before it is enabled for generated Runs.
@@ -0,0 +1,28 @@
 ## Why
 Stopping or restarting a generated Run can leave Platform displaying a stale `running` server projection, while newly produced supervised-process logs are rejected because the restarted Run reuses an already acknowledged sequence range. Operators then see neither a trustworthy runtime state nor new terminal output, even though control registration and SSE remain available.
 ## What Changes
 - Make generated Run report later supervised-process exits and recovered process facts through the existing signed lifecycle report channel, not only the initial autonomous bootstrap result.
 - Define a durable per-log-stream sequence and acknowledgement recovery contract so a Run restart can continue an existing stream without reusing conflicting sequence numbers; explicitly isolate irreconcilable local spool segments instead of retrying them forever.
 - Expose runtime observation freshness separately from the persisted lifecycle projection, so Platform Web does not present an unverified historical `running` state as current process truth after the bound Run is offline.
 - Ensure direct Run shutdown has explicit generic process-supervision semantics; it must not silently imply that a game process stopped unless Run observed and reported that fact.
 ## Capabilities
 ### New Capabilities
 - `run-supervised-runtime-observation`: Run reports observed transitions for generated autonomous processes, and Platform preserves their audited lifecycle projection.
 - `restart-safe-run-log-ingest`: Generated Run restores durable per-stream log progress or isolates conflicts so restarted log delivery resumes without sequence overlap.
 - `runtime-observation-freshness`: Management views distinguish a current Run-observed state from a stale lifecycle projection when the bound endpoint is offline or its heartbeat is overdue.
 ### Modified Capabilities
 - None.
 ## Impact
 - Affects the independent `run` repository's process supervisor, autonomous lifecycle monitor, log spool, and Run-Platform protocol client.
 - Affects `platform/` lifecycle projection, log ingest contracts, DTO/API types, and endpoint freshness data.
 - Affects `platform_web/` server list/detail and management-terminal status presentation; SSE remains the browser log transport.
 - Requires synchronized contract changes and end-to-end restart/recovery tests. It does not add plugin-specific lifecycle behavior, new product areas, or direct browser-to-host access.
@@ -0,0 +1,48 @@
 ## ADDED Requirements
 ### Requirement: Run log sequences survive restart per stream
 Generated Run SHALL allocate durable log sequence numbers independently for each full log stream ID. It MUST persist the locally allocated and Platform-acknowledged watermarks before deleting acknowledged spool segments.
 #### Scenario: Run restarts with an existing spool
 - **WHEN** generated Run restarts using a spool root that contains acknowledged watermarks or pending log segments
 - **THEN** it SHALL restore the watermark for every affected log stream
 - **AND** the next emitted entry for each stream MUST use a sequence greater than all restored allocated and acknowledged values
 #### Scenario: Multiple declared streams emit output
 - **WHEN** declared stdout and stderr streams emit interleaved log lines
 - **THEN** Run SHALL maintain a monotonic sequence independently within each stream
 - **AND** one stream's activity MUST NOT create a sequence gap in another stream
 ### Requirement: Run reconciles a missing local watermark
 Before reusing a stable server-bound log stream ID without a local watermark, generated Run SHALL obtain the latest acknowledged sequence through a signed Platform Run-channel progress query. The query response MUST contain only stream progress metadata needed for sequence recovery.
 #### Scenario: Spool root was recreated
 - **WHEN** a generated Run finds no local watermark for a stable stream that Platform already knows
 - **THEN** Run SHALL obtain the stream's latest acknowledged sequence before allocating the next entry
 - **AND** it MUST NOT restart that stream at sequence one
 #### Scenario: Platform has no existing stream progress
 - **WHEN** the signed progress query reports no acknowledged sequence for a valid new Run-bound stream
 - **THEN** Run SHALL initialize that stream at its first sequence
 - **AND** Platform SHALL continue to create the bound log stream from the signed ingest request
 ### Requirement: Conflicting durable batches are isolated
 Run SHALL classify an acknowledged-range conflict or sequence gap as a non-retryable spool recovery condition. It MUST durably quarantine the affected segment with a redacted reason, emit an operator-visible diagnostic, and continue only after its allocation watermark has been reconciled safely.
 #### Scenario: Platform rejects replacement content in an acknowledged range
 - **WHEN** Platform rejects a Run log batch because the range conflicts with acknowledged history
 - **THEN** Run MUST NOT retry the same segment indefinitely
 - **AND** it SHALL retain an auditable quarantined copy outside the active upload queue
 #### Scenario: Platform reports a sequence gap
 - **WHEN** Platform rejects a Run log batch because its first sequence does not follow the acknowledged stream sequence
 - **THEN** Run SHALL isolate the batch and reconcile stream progress
 - **AND** it MUST NOT silently skip the missing range or overwrite acknowledged entries
 ### Requirement: Browser log delivery reflects accepted durable entries
 Platform SHALL publish browser SSE log events only after a batch is durably accepted. An ingest recovery failure MUST NOT block control heartbeat or lifecycle reporting.
 #### Scenario: A spool segment is quarantined
 - **WHEN** Run quarantines an irreconcilable log segment
 - **THEN** Platform Web MUST NOT present the rejected body as a live log event
 - **AND** the management terminal SHALL continue receiving subsequently accepted log entries
@@ -0,0 +1,39 @@
 ## ADDED Requirements
 ### Requirement: Run reports observed managed-process transitions
 Generated Run SHALL report each material observed transition of an autonomous supervised server process through the signed lifecycle report channel. A report MUST identify the bound server and Run endpoint and include the generic process state and bounded exit classification where applicable.
 #### Scenario: Autonomous process starts
 - **WHEN** an autonomous generated Run starts its declared supervised process
 - **THEN** Run SHALL report `running` after supervision is established
 - **AND** Platform SHALL project the server lifecycle state from that Run-owned fact
 #### Scenario: Supervised process exits
 - **WHEN** a process supervised by generated Run exits after its initial start report
 - **THEN** Run SHALL report the observed `exited` process state and exit classification
 - **AND** Platform SHALL project a requested-stop classification as stopped and an unexpected exit as failed
 #### Scenario: Run recovers a managed process journal
 - **WHEN** generated Run starts with a persisted managed-process journal
 - **THEN** it SHALL check the actual operating-system process state before reporting it
 - **AND** it MUST NOT preserve a historical running projection when the recovered process is not alive
 ### Requirement: Lifecycle reports are idempotent observed facts
 Platform SHALL accept equivalent Run lifecycle observations idempotently and SHALL reject a stale report that would regress a newer observed process transition for the same managed process.
 #### Scenario: Run retries an exit report
 - **WHEN** Run retries the same observed exit report after a transport failure
 - **THEN** Platform SHALL preserve one equivalent terminal lifecycle projection
 - **AND** the retry MUST NOT change the projection back to running
 #### Scenario: Older running observation arrives late
 - **WHEN** Platform has accepted a newer terminal observation for a managed process
 - **THEN** an older `running` report for that same process MUST NOT overwrite the terminal projection
 ### Requirement: Run shutdown does not invent server state
 Run shutdown handling SHALL report only process facts that Run has observed through its generic supervisor. Loss of a Run process or heartbeat MUST NOT by itself be reported or projected as a stopped game server.
 #### Scenario: Run becomes unavailable while process state is unknown
 - **WHEN** Platform stops receiving heartbeats from a bound Run
 - **THEN** Platform MUST retain the last observed lifecycle fact
 - **AND** it MUST NOT replace that fact with stopped solely because the endpoint is unavailable
@@ -0,0 +1,35 @@
 ## ADDED Requirements
 ### Requirement: Platform exposes observation freshness with lifecycle projection
 Platform SHALL expose the last Run-projected server lifecycle state together with a generic observation freshness derived from the bound Run endpoint's status and heartbeat age. Freshness MUST be distinct from the server lifecycle state.
 #### Scenario: Bound Run is fresh
 - **WHEN** the bound Run endpoint has an accepted recent heartbeat within the configured freshness interval
 - **THEN** Platform SHALL expose the server runtime observation as fresh
 - **AND** the lifecycle projection MAY be presented as currently observed
 #### Scenario: Bound Run heartbeat is overdue
 - **WHEN** the bound Run endpoint is offline, disabled, or beyond the configured heartbeat freshness interval
 - **THEN** Platform SHALL expose the runtime observation as unverified or unreachable
 - **AND** it MUST retain the last lifecycle projection rather than convert it to stopped
 ### Requirement: Management views label unverified runtime state
 Server list, server detail, and management terminal views SHALL show when a displayed lifecycle state is not currently vouched for by a fresh bound Run. They MUST NOT label a stale lifecycle projection as current process truth.
 #### Scenario: Last observation was running but Run is offline
 - **WHEN** the last lifecycle projection is running and the bound Run is unreachable
 - **THEN** the management UI SHALL show the last observed running state with an offline or unverified qualifier
 - **AND** it MUST NOT display the server as confirmed online solely from the persisted projection
 #### Scenario: Management terminal opens while Run is unverified
 - **WHEN** an operator opens the management terminal for a server whose bound Run is unverified
 - **THEN** the terminal SHALL show that live delivery depends on Run recovery
 - **AND** it SHALL continue to display accepted historical logs through the bounded SSE replay
 ### Requirement: Explicit server stop remains Run-observed
 An operator-requested stop SHALL remain a Platform-authorized intent executed by Run's generic supervision and plugin-declared lifecycle action. Platform SHALL project stopped only from the resulting Run observation or accepted lifecycle result.
 #### Scenario: Operator stops a server while Run is available
 - **WHEN** an authorized operator requests a server stop and the bound Run completes the declared stop action
 - **THEN** Platform SHALL project the reported stopped process state
 - **AND** the UI SHALL present the result as a current observation while the Run remains fresh
@@ -0,0 +1,41 @@
 ## 0. Task Boundaries
 正向提示词：为服务器管理第一方区域建立可靠的 Run 运行状态与重启后日志恢复能力。成功标准是：Run 观察到的启动、停止、退出和恢复状态能安全投影到 Platform；Run 重启后管理终端继续收到新日志；Run 离线时界面明确显示状态未验证而不伪造“已停止”。
 方向提示词：在 `platform/` 实现签名 Run 契约、运行观测投影和日志进度查询，在独立 `git@git.npc0.com:admin343/run.git` 实现通用进程观察与 spool 水位恢复，在 `platform_web/` 展示状态新鲜度。保留插件声明的生命周期和日志源、Platform-Run 通道隔离、现有 SSE 历史回放。验证必须覆盖两个仓库的定向测试、`scripts/check-structure.sh`，以及严格 OpenSpec 校验。
 任务边界：不得把 `run/` 源码加入本仓库；不得在 Platform 或 Run 写入 SCUM 可执行文件、端口、Steam、路径或文件尾随特例；不得提供浏览器到主机的直接 shell、socket、路径或凭据访问；不得将 Run 失联直接投影为服务器已停止；不得混入账单、云主机销售或第三方平台代理工作流。
 ## 1. Platform Contracts And Projections
 - [ ] 1.1 Add typed domain, DTO, API, validation, and protocol contracts for signed Run log-stream progress queries scoped to the authenticated Run endpoint and bound server instance.
 - [ ] 1.2 Implement Platform log-stream progress lookup that returns only the latest acknowledged sequence and cannot disclose log bodies, host paths, credentials, or another server's stream metadata.
 - [ ] 1.3 Extend Run lifecycle observations with stable managed-process identity and ordering data, then make lifecycle projection idempotent and reject stale state regressions.
 - [ ] 1.4 Project autonomous Run recovered and exit observations through the existing signed lifecycle channel, preserving requested-stop versus unexpected-exit classification.
 - [ ] 1.5 Expose a server runtime observation view that combines the persisted lifecycle projection with generic bound-endpoint heartbeat freshness without changing lifecycle state solely because Run is unavailable.
 - [ ] 1.6 Add focused Platform tests for report ordering/idempotency, authorization and scope of log progress, progress values after durable ingest, and fresh versus unverified runtime observation.
 ## 2. Independent Run Recovery
 - [ ] 2.1 Update the shared/copyable Run-Platform protocol types and API client in `git@git.npc0.com:admin343/run.git` for lifecycle observation ordering and signed log-stream progress reconciliation.
 - [ ] 2.2 Add atomic per-stream allocated and acknowledged watermark persistence to the Run log spool, including restart loading and acknowledgement-before-segment-deletion ordering.
 - [ ] 2.3 Replace the worker-global in-memory log counter with stream-specific allocation restored from the spool watermark and pending durable segments.
 - [ ] 2.4 Reconcile signed Platform stream progress before a stable Run-bound stream with no local watermark emits new entries; cover newly created and recreated-spool cases.
 - [ ] 2.5 Classify acknowledged-range conflicts and sequence gaps as durable recovery failures, quarantine the affected spool segment with redacted diagnostics, and resume only after safe watermark reconciliation.
 - [ ] 2.6 Make autonomous process supervision report observed exit and startup-recovery transitions through the lifecycle channel, with retry-safe process identity and ordering metadata.
 - [ ] 2.7 Define and test graceful Run shutdown behavior that preserves durable state and never reports a server stop unless its generic supervisor observed that process state.
 - [ ] 2.8 Add Run unit tests for per-stream interleaving, restart continuity, missing-watermark progress lookup, conflict quarantine, process exit reporting, and Windows supervisor recovery.
 ## 3. Management Runtime Presentation
 - [ ] 3.1 Extend Platform Web API types and server-management contracts to consume lifecycle projection and runtime observation freshness separately.
 - [ ] 3.2 Update server list and server detail status UI so stale `running` is presented as last observed with a Run offline/unverified qualifier, not confirmed online.
 - [ ] 3.3 Update the management terminal header and empty/error states to show that live output awaits Run recovery while preserving accepted bounded SSE history.
 - [ ] 3.4 Add focused frontend tests for fresh, stale, offline, and recovered Run observations plus terminal presentation during log recovery.
 ## 4. Cross-Repository Verification And Release
 - [ ] 4.1 Run Platform and Run contract compatibility tests for signed progress recovery and lifecycle observation ordering.
 - [ ] 4.2 Perform a Windows generated Run scenario covering normal start, supervised process exit, direct Run restart with retained spool, recreated spool reconciliation, quarantined conflict, and operator-requested stop.
 - [ ] 4.3 Run targeted Go and frontend test suites, `scripts/check-structure.sh`, and `openspec validate repair-run-runtime-state-and-log-recovery --strict`; record the evidence before completing tasks.
 - [ ] 4.4 Deploy Platform compatibility before the Run release, then verify runtime freshness and terminal delivery in an environment with no direct browser-to-host access.