docs: propose run runtime recovery
This commit is contained in:
npc0-hue
+48
@@ -0,0 +1,48 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Run log sequences survive restart per stream
|
||||
Generated Run SHALL allocate durable log sequence numbers independently for each full log stream ID. It MUST persist the locally allocated and Platform-acknowledged watermarks before deleting acknowledged spool segments.
|
||||
|
||||
#### Scenario: Run restarts with an existing spool
|
||||
- **WHEN** generated Run restarts using a spool root that contains acknowledged watermarks or pending log segments
|
||||
- **THEN** it SHALL restore the watermark for every affected log stream
|
||||
- **AND** the next emitted entry for each stream MUST use a sequence greater than all restored allocated and acknowledged values
|
||||
|
||||
#### Scenario: Multiple declared streams emit output
|
||||
- **WHEN** declared stdout and stderr streams emit interleaved log lines
|
||||
- **THEN** Run SHALL maintain a monotonic sequence independently within each stream
|
||||
- **AND** one stream's activity MUST NOT create a sequence gap in another stream
|
||||
|
||||
### Requirement: Run reconciles a missing local watermark
|
||||
Before reusing a stable server-bound log stream ID without a local watermark, generated Run SHALL obtain the latest acknowledged sequence through a signed Platform Run-channel progress query. The query response MUST contain only stream progress metadata needed for sequence recovery.
|
||||
|
||||
#### Scenario: Spool root was recreated
|
||||
- **WHEN** a generated Run finds no local watermark for a stable stream that Platform already knows
|
||||
- **THEN** Run SHALL obtain the stream's latest acknowledged sequence before allocating the next entry
|
||||
- **AND** it MUST NOT restart that stream at sequence one
|
||||
|
||||
#### Scenario: Platform has no existing stream progress
|
||||
- **WHEN** the signed progress query reports no acknowledged sequence for a valid new Run-bound stream
|
||||
- **THEN** Run SHALL initialize that stream at its first sequence
|
||||
- **AND** Platform SHALL continue to create the bound log stream from the signed ingest request
|
||||
|
||||
### Requirement: Conflicting durable batches are isolated
|
||||
Run SHALL classify an acknowledged-range conflict or sequence gap as a non-retryable spool recovery condition. It MUST durably quarantine the affected segment with a redacted reason, emit an operator-visible diagnostic, and continue only after its allocation watermark has been reconciled safely.
|
||||
|
||||
#### Scenario: Platform rejects replacement content in an acknowledged range
|
||||
- **WHEN** Platform rejects a Run log batch because the range conflicts with acknowledged history
|
||||
- **THEN** Run MUST NOT retry the same segment indefinitely
|
||||
- **AND** it SHALL retain an auditable quarantined copy outside the active upload queue
|
||||
|
||||
#### Scenario: Platform reports a sequence gap
|
||||
- **WHEN** Platform rejects a Run log batch because its first sequence does not follow the acknowledged stream sequence
|
||||
- **THEN** Run SHALL isolate the batch and reconcile stream progress
|
||||
- **AND** it MUST NOT silently skip the missing range or overwrite acknowledged entries
|
||||
|
||||
### Requirement: Browser log delivery reflects accepted durable entries
|
||||
Platform SHALL publish browser SSE log events only after a batch is durably accepted. An ingest recovery failure MUST NOT block control heartbeat or lifecycle reporting.
|
||||
|
||||
#### Scenario: A spool segment is quarantined
|
||||
- **WHEN** Run quarantines an irreconcilable log segment
|
||||
- **THEN** Platform Web MUST NOT present the rejected body as a live log event
|
||||
- **AND** the management terminal SHALL continue receiving subsequently accepted log entries
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Run reports observed managed-process transitions
|
||||
Generated Run SHALL report each material observed transition of an autonomous supervised server process through the signed lifecycle report channel. A report MUST identify the bound server and Run endpoint and include the generic process state and bounded exit classification where applicable.
|
||||
|
||||
#### Scenario: Autonomous process starts
|
||||
- **WHEN** an autonomous generated Run starts its declared supervised process
|
||||
- **THEN** Run SHALL report `running` after supervision is established
|
||||
- **AND** Platform SHALL project the server lifecycle state from that Run-owned fact
|
||||
|
||||
#### Scenario: Supervised process exits
|
||||
- **WHEN** a process supervised by generated Run exits after its initial start report
|
||||
- **THEN** Run SHALL report the observed `exited` process state and exit classification
|
||||
- **AND** Platform SHALL project a requested-stop classification as stopped and an unexpected exit as failed
|
||||
|
||||
#### Scenario: Run recovers a managed process journal
|
||||
- **WHEN** generated Run starts with a persisted managed-process journal
|
||||
- **THEN** it SHALL check the actual operating-system process state before reporting it
|
||||
- **AND** it MUST NOT preserve a historical running projection when the recovered process is not alive
|
||||
|
||||
### Requirement: Lifecycle reports are idempotent observed facts
|
||||
Platform SHALL accept equivalent Run lifecycle observations idempotently and SHALL reject a stale report that would regress a newer observed process transition for the same managed process.
|
||||
|
||||
#### Scenario: Run retries an exit report
|
||||
- **WHEN** Run retries the same observed exit report after a transport failure
|
||||
- **THEN** Platform SHALL preserve one equivalent terminal lifecycle projection
|
||||
- **AND** the retry MUST NOT change the projection back to running
|
||||
|
||||
#### Scenario: Older running observation arrives late
|
||||
- **WHEN** Platform has accepted a newer terminal observation for a managed process
|
||||
- **THEN** an older `running` report for that same process MUST NOT overwrite the terminal projection
|
||||
|
||||
### Requirement: Run shutdown does not invent server state
|
||||
Run shutdown handling SHALL report only process facts that Run has observed through its generic supervisor. Loss of a Run process or heartbeat MUST NOT by itself be reported or projected as a stopped game server.
|
||||
|
||||
#### Scenario: Run becomes unavailable while process state is unknown
|
||||
- **WHEN** Platform stops receiving heartbeats from a bound Run
|
||||
- **THEN** Platform MUST retain the last observed lifecycle fact
|
||||
- **AND** it MUST NOT replace that fact with stopped solely because the endpoint is unavailable
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Platform exposes observation freshness with lifecycle projection
|
||||
Platform SHALL expose the last Run-projected server lifecycle state together with a generic observation freshness derived from the bound Run endpoint's status and heartbeat age. Freshness MUST be distinct from the server lifecycle state.
|
||||
|
||||
#### Scenario: Bound Run is fresh
|
||||
- **WHEN** the bound Run endpoint has an accepted recent heartbeat within the configured freshness interval
|
||||
- **THEN** Platform SHALL expose the server runtime observation as fresh
|
||||
- **AND** the lifecycle projection MAY be presented as currently observed
|
||||
|
||||
#### Scenario: Bound Run heartbeat is overdue
|
||||
- **WHEN** the bound Run endpoint is offline, disabled, or beyond the configured heartbeat freshness interval
|
||||
- **THEN** Platform SHALL expose the runtime observation as unverified or unreachable
|
||||
- **AND** it MUST retain the last lifecycle projection rather than convert it to stopped
|
||||
|
||||
### Requirement: Management views label unverified runtime state
|
||||
Server list, server detail, and management terminal views SHALL show when a displayed lifecycle state is not currently vouched for by a fresh bound Run. They MUST NOT label a stale lifecycle projection as current process truth.
|
||||
|
||||
#### Scenario: Last observation was running but Run is offline
|
||||
- **WHEN** the last lifecycle projection is running and the bound Run is unreachable
|
||||
- **THEN** the management UI SHALL show the last observed running state with an offline or unverified qualifier
|
||||
- **AND** it MUST NOT display the server as confirmed online solely from the persisted projection
|
||||
|
||||
#### Scenario: Management terminal opens while Run is unverified
|
||||
- **WHEN** an operator opens the management terminal for a server whose bound Run is unverified
|
||||
- **THEN** the terminal SHALL show that live delivery depends on Run recovery
|
||||
- **AND** it SHALL continue to display accepted historical logs through the bounded SSE replay
|
||||
|
||||
### Requirement: Explicit server stop remains Run-observed
|
||||
An operator-requested stop SHALL remain a Platform-authorized intent executed by Run's generic supervision and plugin-declared lifecycle action. Platform SHALL project stopped only from the resulting Run observation or accepted lifecycle result.
|
||||
|
||||
#### Scenario: Operator stops a server while Run is available
|
||||
- **WHEN** an authorized operator requests a server stop and the bound Run completes the declared stop action
|
||||
- **THEN** Platform SHALL project the reported stopped process state
|
||||
- **AND** the UI SHALL present the result as a current observation while the Run remains fresh
|
||||
Reference in New Issue
Block a user