9.0 KiB
Context
Platform persists a server lifecycle projection after Run reports process.start as running. Generated Run only reports the autonomous bootstrap terminal result; its process supervisor subsequently records a child exit locally without issuing another lifecycle report. When Run is stopped, Platform consequently retains running even though it has no fresh process observation. The existing UI displays that projection as current status.
Run also emits autonomous process logs to a stable server-bound stream ID. Its SpoolLogSink currently owns one in-memory counter that starts at zero for every Run process, while Platform accepts only a contiguous range after its latest acknowledged sequence or an exact duplicate. A restarted Run therefore sends a different entry in an old sequence range, receives a conflict, and retries the blocked spool segment indefinitely. SSE correctly delivers only persisted Platform logs, so it has no new event to show.
The independent run repository remains machine lifecycle authority. Platform owns authentication, audit, desired lifecycle requests, durable ingest, and projections. Plugins remain the sole owner of game-specific lifecycle actions and log-source declarations.
Goals / Non-Goals
Goals:
- Keep Platform lifecycle projections convergent with Run-observed managed-process transitions, including process exit and Run restart recovery.
- Distinguish the last observed server lifecycle fact from whether its dedicated Run endpoint is currently fresh enough to vouch for it.
- Make each durable Run log stream resume with a monotonically increasing sequence after an ordinary Run restart.
- Turn unreconcilable log overlap into an explicit, observable quarantine outcome rather than an infinite retry loop.
- Preserve the existing signed Run channel, bounded log spool, and browser SSE delivery model.
Non-Goals:
- Infer a stopped game process merely because the Run executable or heartbeat stopped.
- Add a game-specific status probe, file tail, executable name, or child-process rule to Platform or Run.
- Give Platform Web direct host, shell, socket, credential, or log-spool access.
- Rewrite historical log retention or make browser SSE a source of truth.
- Re-add the independent Run source tree to this repository.
Decisions
Report managed process transitions through the lifecycle channel
Run will retain enough autonomous-assignment identity to turn a local supervised process transition into a signed lifecycle report. The initial running observation, a later stopped observation, and an exited observation use the existing report route with process state, exit classification, and bounded audit summary. On Run startup, supervisor reconciliation will report any persisted managed process identity after checking its actual OS process state. Reports are idempotent by observed state/version so a restart cannot repeatedly create misleading transitions.
This retains Platform's existing authorization and projection route rather than adding process data to heartbeat. Heartbeat answers endpoint freshness, while lifecycle reports carry process facts. Reusing job result messages was rejected because autonomous lifecycle work has no Platform job lease.
Model freshness separately from lifecycle state
serverInstances.state remains the last Run-reported lifecycle projection. Platform will derive a server runtime observation view from the bound endpoint's last heartbeat and endpoint status, with a documented freshness threshold derived from the negotiated heartbeat interval. A current observation can be fresh; one beyond the threshold is stale or unreachable without changing the stored lifecycle state. The management UI displays both: for example, Last observed: running and Run offline / state unverified.
Platform must not project stopped only because heartbeat expires, because a Run or its descendants may continue locally. Adding a game-specific lifecycle state was rejected: freshness is a generic observation attribute, not game setup state.
Persist watermarks per durable log stream
Run's spool will store an atomic per-stream watermark containing at least the highest locally allocated sequence and highest Platform-acknowledged sequence. Sequence allocation is keyed by full logStreamId, not a worker-global counter. The spool updates the acknowledged watermark only after a valid Platform response covers the segment, then retains that watermark even after deleting the acknowledged segment.
On restart, Run opens the same spool root, restores watermarks and pending segments, and allocates the next sequence above both. A clean spool root that lacks a watermark for a preexisting Platform stream needs an explicit reconciliation result before it may reuse that stable stream ID. The preferred contract is a signed lightweight Run log-stream progress endpoint returning only the latest acknowledged sequence for the Run-bound stream; no log bodies, host paths, or browser access are exposed. A nonempty local spool remains the source for retrying unacknowledged content.
Stable stream IDs are retained so the UI can continue to show a coherent stream across Run restarts. Rotating the stream ID on every restart was rejected because it fragments operator history and masks durability failures.
Quarantine irreconcilable batches and keep delivery moving
Platform responses distinguish a sequence gap from a conflicting acknowledged range. For a conflict, Run must not resend the exact same segment indefinitely. It records a redacted diagnostic, moves the affected segment to a durable rejected area, and advances only after reconciling its next allocation watermark. A gap that indicates local data loss follows the same safe isolation path until progress reconciliation completes; it must not be silently skipped.
Quarantining protects control and live log delivery from a permanently poisoned spool segment, but the operator receives an audit/diagnostic signal. Platform does not accept replacement log bodies for an already acknowledged range.
Define Run shutdown separately from server stop
Run graceful shutdown will perform a generic supervisor shutdown procedure: preserve its journal and flush bounded durable channels, and report only process facts it actually observes. An explicit operator stop remains a plugin-declared lifecycle action executed by Run. Windows process-tree containment or child cleanup must be generic and can be implemented only when it preserves the declared lifecycle action semantics; closing Run must never fabricate a stopped projection without observation.
Risks / Trade-offs
- [Risk] A Run update can restart before its last log ACK is durably recorded. → Mitigation: write the pending segment before sending it and atomically persist the ACK watermark before deleting the segment.
- [Risk] A lost or manually deleted spool root cannot prove the next sequence for a stable Platform stream. → Mitigation: require signed progress reconciliation before allocation and quarantine conflicting data rather than overwriting history.
- [Risk] Endpoint heartbeat is temporarily delayed while a healthy server runs. → Mitigation: show freshness as an observation qualifier, keep the last lifecycle fact visible, and use a threshold based on the negotiated heartbeat interval rather than one missed beat.
- [Risk] Reporting process exits can race with an explicit stop result. → Mitigation: include observed state identity/timestamp and make Platform reject stale transition regressions while accepting equivalent terminal facts idempotently.
- [Risk] Platform and Run releases are deployed out of order. → Mitigation: version the progress endpoint/response capability, retain compatible ingest behavior, and make Run fail closed for missing reconciliation rather than reuse a stale sequence.
Migration Plan
- Add Platform domain/DTO/protocol support for Run log-stream progress and runtime observation freshness while preserving existing lifecycle and log ingest routes.
- Deploy Platform compatibility first; it accepts existing uploads and serves progress to capable signed Runs.
- Release Run with persisted per-stream watermarks, progress reconciliation, conflict quarantine, and managed-process transition reporting.
- Deploy Platform Web to present observation freshness separately from lifecycle projection and show log ingest recovery diagnostics.
- Exercise a Windows generated Run restart with retained spool, an empty/recreated spool, supervised process exit, explicit stop, and offline endpoint scenarios.
Rollback is code-only. A Platform rollback must keep accepting the existing signed lifecycle and ingest routes. A Run rollback retains spool segments and watermarks; operators must not delete spool state as a rollback step.
Open Questions
- The precise heartbeat freshness multiplier should be standardized alongside the existing negotiated heartbeat interval; the implementation must choose one documented value and cover it with tests.
- The final generic Windows containment primitive must be validated against graceful plugin-declared stop behavior before it is enabled for generated Runs.