fix: recover run runtime state and logs
This commit is contained in:
@@ -8,29 +8,29 @@
|
||||
|
||||
## 1. Platform Contracts And Projections
|
||||
|
||||
- [ ] 1.1 Add typed domain, DTO, API, validation, and protocol contracts for signed Run log-stream progress queries scoped to the authenticated Run endpoint and bound server instance.
|
||||
- [ ] 1.2 Implement Platform log-stream progress lookup that returns only the latest acknowledged sequence and cannot disclose log bodies, host paths, credentials, or another server's stream metadata.
|
||||
- [ ] 1.3 Extend Run lifecycle observations with stable managed-process identity and ordering data, then make lifecycle projection idempotent and reject stale state regressions.
|
||||
- [ ] 1.4 Project autonomous Run recovered and exit observations through the existing signed lifecycle channel, preserving requested-stop versus unexpected-exit classification.
|
||||
- [x] 1.1 Add typed domain, DTO, API, validation, and protocol contracts for signed Run log-stream progress queries scoped to the authenticated Run endpoint and bound server instance.
|
||||
- [x] 1.2 Implement Platform log-stream progress lookup that returns only the latest acknowledged sequence and cannot disclose log bodies, host paths, credentials, or another server's stream metadata.
|
||||
- [x] 1.3 Extend Run lifecycle observations with stable managed-process identity and ordering data, then make lifecycle projection idempotent and reject stale state regressions.
|
||||
- [x] 1.4 Project autonomous Run recovered and exit observations through the existing signed lifecycle channel, preserving requested-stop versus unexpected-exit classification.
|
||||
- [ ] 1.5 Expose a server runtime observation view that combines the persisted lifecycle projection with generic bound-endpoint heartbeat freshness without changing lifecycle state solely because Run is unavailable.
|
||||
- [ ] 1.6 Add focused Platform tests for report ordering/idempotency, authorization and scope of log progress, progress values after durable ingest, and fresh versus unverified runtime observation.
|
||||
- [x] 1.6 Add focused Platform tests for report ordering/idempotency, authorization and scope of log progress, progress values after durable ingest, and fresh versus unverified runtime observation.
|
||||
|
||||
## 2. Independent Run Recovery
|
||||
|
||||
- [ ] 2.1 Update the shared/copyable Run-Platform protocol types and API client in `git@git.npc0.com:admin343/run.git` for lifecycle observation ordering and signed log-stream progress reconciliation.
|
||||
- [ ] 2.2 Add atomic per-stream allocated and acknowledged watermark persistence to the Run log spool, including restart loading and acknowledgement-before-segment-deletion ordering.
|
||||
- [ ] 2.3 Replace the worker-global in-memory log counter with stream-specific allocation restored from the spool watermark and pending durable segments.
|
||||
- [ ] 2.4 Reconcile signed Platform stream progress before a stable Run-bound stream with no local watermark emits new entries; cover newly created and recreated-spool cases.
|
||||
- [ ] 2.5 Classify acknowledged-range conflicts and sequence gaps as durable recovery failures, quarantine the affected spool segment with redacted diagnostics, and resume only after safe watermark reconciliation.
|
||||
- [ ] 2.6 Make autonomous process supervision report observed exit and startup-recovery transitions through the lifecycle channel, with retry-safe process identity and ordering metadata.
|
||||
- [ ] 2.7 Define and test graceful Run shutdown behavior that preserves durable state and never reports a server stop unless its generic supervisor observed that process state.
|
||||
- [x] 2.1 Update the shared/copyable Run-Platform protocol types and API client in `git@git.npc0.com:admin343/run.git` for lifecycle observation ordering and signed log-stream progress reconciliation.
|
||||
- [x] 2.2 Add atomic per-stream allocated and acknowledged watermark persistence to the Run log spool, including restart loading and acknowledgement-before-segment-deletion ordering.
|
||||
- [x] 2.3 Replace the worker-global in-memory log counter with stream-specific allocation restored from the spool watermark and pending durable segments.
|
||||
- [x] 2.4 Reconcile signed Platform stream progress before a stable Run-bound stream with no local watermark emits new entries; cover newly created and recreated-spool cases.
|
||||
- [x] 2.5 Classify acknowledged-range conflicts and sequence gaps as durable recovery failures, quarantine the affected spool segment with redacted diagnostics, and resume only after safe watermark reconciliation.
|
||||
- [x] 2.6 Make autonomous process supervision report observed exit and startup-recovery transitions through the lifecycle channel, with retry-safe process identity and ordering metadata.
|
||||
- [x] 2.7 Define and test graceful Run shutdown behavior that preserves durable state and never reports a server stop unless its generic supervisor observed that process state.
|
||||
- [ ] 2.8 Add Run unit tests for per-stream interleaving, restart continuity, missing-watermark progress lookup, conflict quarantine, process exit reporting, and Windows supervisor recovery.
|
||||
|
||||
## 3. Management Runtime Presentation
|
||||
|
||||
- [ ] 3.1 Extend Platform Web API types and server-management contracts to consume lifecycle projection and runtime observation freshness separately.
|
||||
- [x] 3.1 Extend Platform Web API types and server-management contracts to consume lifecycle projection and runtime observation freshness separately.
|
||||
- [ ] 3.2 Update server list and server detail status UI so stale `running` is presented as last observed with a Run offline/unverified qualifier, not confirmed online.
|
||||
- [ ] 3.3 Update the management terminal header and empty/error states to show that live output awaits Run recovery while preserving accepted bounded SSE history.
|
||||
- [x] 3.3 Update the management terminal header and empty/error states to show that live output awaits Run recovery while preserving accepted bounded SSE history.
|
||||
- [ ] 3.4 Add focused frontend tests for fresh, stale, offline, and recovered Run observations plus terminal presentation during log recovery.
|
||||
|
||||
## 4. Cross-Repository Verification And Release
|
||||
|
||||
Reference in New Issue
Block a user