feat: 完整游戏运维功能
This commit is contained in:
npc0-hue
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-17
|
||||
@@ -0,0 +1,78 @@
|
||||
## Context
|
||||
|
||||
`domain.Job` is already the durable unit stored by MemoryStore, FileStore, and the repository-backed MySQLStore, but it currently contains only the user-visible lifecycle projection. `CoreService` separately owns an in-memory `jobLeases` map containing raw Run session tokens, lease tokens, attempts, cancel intent, and terminal fingerprints. Restarting the platform therefore makes accepted/running jobs impossible to authenticate or complete safely. The independent Run repository similarly uses an in-memory active assignment map and does not reconcile during startup.
|
||||
|
||||
The existing HTTP surface already separates control, jobs, logs, and artifacts and signs non-hello Run requests with the active Run session. This change must deepen those boundaries without introducing an all-in-one transport, exposing credentials to the browser/plugins, or implementing later execution/storage roadmap items.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- Make every scheduling decision recoverable from the Job repository after platform process restart or FileStore/MySQLStore reload.
|
||||
- Fence every ack, progress, result, cancel poll, and reconcile report by endpoint, active authenticated session, per-job attempt, and a lease credential whose persisted form is a hash.
|
||||
- Define bounded ack and execution leases, monotonic per-job attempts and progress sequences, exponential retry backoff, terminal idempotency, and durable cancel intent/result.
|
||||
- Make Run persist active assignments atomically under its workspace, reconcile immediately after registration, and recover or discard work according to the platform response.
|
||||
- Keep safe browser projections and owner/platform-admin authorization aligned with existing server ownership checks.
|
||||
- Preserve independent control/job/log/artifact paths and prove large transfer requests cannot block control or job requests.
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- Production-grade distributed scheduling or multi-platform-replica compare-and-swap coordination.
|
||||
- Process supervision, real config/file mutation, durable logs/artifacts/metrics/backups, dependency installation, Run self-update, client-manager lifecycle, or production scaling.
|
||||
- Persisting raw Run session tokens, lease tokens, host paths, sockets, AI keys, component keys, or other credentials.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Persist scheduling metadata on the Job aggregate
|
||||
|
||||
`domain.Job` and `model.Job` gain a nested retry policy plus queue, attempt, hashed lease, ack/lease deadline, last progress sequence, cancel, terminal, and reconciliation fields. The existing Job repository remains the only storage boundary, so MemoryStore, FileStore, and MySQLStore inherit the behavior through their existing typed snapshot/repository implementations.
|
||||
|
||||
This is preferred over retaining a service cache or introducing a second lease repository because transitions need one recoverable aggregate and the current stores do not provide cross-repository transactions. Raw lease tokens are generated with cryptographic randomness, returned only over the signed Run job channel, and stored as SHA-256 hashes.
|
||||
|
||||
### Use a deterministic per-job state machine
|
||||
|
||||
New jobs default to `queued`, attempt zero, a bounded retry policy, and an immediately eligible queue timestamp. Claim sweeps expired work for the endpoint, selects eligible `queued` or `retrying` work in stable creation order, increments that job's attempt, stores the lease hash and current Run session generation, and moves it to `accepted` with an ack deadline and execution lease deadline.
|
||||
|
||||
Ack before the deadline moves the attempt to `running` and renews its lease. Monotonic progress renews the running lease. Ack timeout, execution lease expiry, or an explicitly retryable failed result clears the lease and moves the job to `retrying` with exponential backoff when attempts remain; otherwise it records a terminal `failed` result. A pending cancel intent always resolves to `cancelled` rather than retrying when the lease expires. Old attempts and old lease tokens are rejected even after a newer attempt exists.
|
||||
|
||||
### Model cancellation as durable intent followed by durable result
|
||||
|
||||
Cancel before claim atomically records intent and a terminal cancelled result. Cancel after claim records idempotent intent for the assigned Run endpoint; polling is fenced by the current attempt and lease, and Run returns a normal terminal cancelled result. Repeated identical or compatible cancel requests return the existing intent/result projection. Cancellation never consumes another retry attempt.
|
||||
|
||||
### Reconcile with full active-attempt evidence
|
||||
|
||||
Run reports persisted active assignment evidence: job ID, attempt, and raw lease token over the signed job channel. Platform verifies the endpoint, stored attempt, lease hash, nonterminal state, and active Run session, then rebinds the lease to the current session generation and renews its deadline. The raw token is echoed only in the signed response and is never persisted.
|
||||
|
||||
Reported stale/unknown jobs are returned as discard IDs. Platform-active jobs absent from the Run journal are treated as abandoned and enter cancel resolution or retry policy. Reconciliation timestamps, counts, and outcome are persisted on every affected Job. This is preferred over recreating leases during reconciliation because recreating them would let stale attempts regain authority.
|
||||
|
||||
### Persist Run journal atomically and reconcile before claiming
|
||||
|
||||
Run stores a versioned JSON journal under `WorkspaceRoot`, using a temporary file plus rename and owner-only permissions. It writes the assignment before ack, updates it after ack/progress, and removes it only after an accepted terminal response or explicit platform discard. Worker startup registers, reconciles the loaded journal before any new claim, and re-executes platform-confirmed assignments under the same attempt. Corrupt journal data fails worker construction instead of silently forgetting active work.
|
||||
|
||||
### Keep API and UI projections credential-free
|
||||
|
||||
Platform user APIs expose attempt, max attempts, next retry time, ack/lease deadlines, cancel state/timestamps, terminal time, and last reconcile outcome. They omit lease hashes, raw lease tokens, session generation, secret refs, and host/runtime credentials. Existing `GetJobForSession`, `ListJobsForSession`, and cancel authorization continue to derive access from server owner/admin or platform-admin rules. Run-only handlers continue to require both the active session and request signature middleware.
|
||||
|
||||
### Preserve channel independence
|
||||
|
||||
Control, jobs, logs, and artifacts remain separate HTTP routes and clients. Run's worker loop executes heartbeat and scheduling work independently from spool/artifact transfer queues; neither the job protocol nor safe browser projection carries log bodies or artifact payloads. Isolation tests block log/artifact handlers while asserting heartbeat, ack, result, cancel, and reconcile complete.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [FileStore and the current MySQLStore are process-local snapshot implementations, not distributed CAS schedulers] -> Serialize transitions with the existing service mutex and explicitly keep multi-replica production scheduling out of scope.
|
||||
- [At-least-once recovery can repeat an interrupted operation] -> Preserve idempotency keys, fence attempts, and require bounded Run executors to be idempotent; later process supervision will refine resumability.
|
||||
- [Legacy persisted jobs lack new fields] -> Normalize zero-value scheduling fields when read/claimed so existing queued and terminal records remain valid without destructive migration.
|
||||
- [Hash-only lease storage means platform cannot recreate a lost Run lease credential] -> Require the durable Run journal to present the original token; otherwise the platform retries with a new attempt after reconciliation/expiry.
|
||||
- [Clock skew between Run and platform] -> Treat platform timestamps as authoritative; Run does not decide lease validity locally.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Deploy the expanded model/repository projection and zero-value normalization before relying on new states.
|
||||
2. Deploy platform protocol and scheduler behavior with compatibility for an empty legacy reconcile evidence list.
|
||||
3. Deploy Run protocol and persistent journal, which reconciles immediately after registration before claiming.
|
||||
4. Deploy the safe web projection and tests.
|
||||
5. Rollback may read the expanded JSON while ignoring unknown fields, but accepted/running jobs should be allowed to reconcile or expire before rolling back to code that lacks durable fencing.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None for this single-process scheduling milestone. Cross-replica transactional claiming and resumable supervised processes remain explicit later design work.
|
||||
@@ -0,0 +1,32 @@
|
||||
## Why
|
||||
|
||||
The platform currently persists the visible Job record but keeps leases, attempts, cancellation intent, and terminal idempotency in `CoreService` memory, so a platform restart loses fencing and recovery state. Run also keeps its active-job journal only in memory, preventing reliable reconciliation after Run restart and leaving ack timeouts, retry backoff, and late-result rejection underspecified.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Persist queue scheduling, attempt, lease hash and deadlines, retry/backoff, terminal fingerprint, cancellation intent/result, progress sequence, and reconciliation metadata through the existing MemoryStore, FileStore, and MySQLStore Job repository boundary.
|
||||
- Define claim, ack, lease renewal, retry-wait, terminal, cancel, and reconciliation transitions with attempt fencing and deterministic late-message rejection.
|
||||
- Rebind a valid persisted lease to a newly authenticated Run session generation only through endpoint-scoped reconciliation; never persist raw Run session or lease credentials.
|
||||
- Persist Run's active assignment journal locally, reconcile it after registration and restart, continue valid attempts, and discard platform-rejected or unknown work.
|
||||
- Keep control, jobs, logs, artifacts, and the optional game-client bridge as independent request paths and execution queues so log/artifact backpressure cannot block heartbeat or job acknowledgement/result traffic.
|
||||
- Extend platform APIs and the management console only with safe scheduling projections such as state, attempt, retry timing, cancel status, and reconcile status. Raw tokens, secret references, host paths, sockets, and credentials remain excluded.
|
||||
- Add cross-store, API, frontend, and independent Run regression coverage for reload recovery, deadlines, fencing, retry, cancellation, reconciliation, authorization/signature failure, and channel isolation.
|
||||
- Explicitly leave process supervision, real config/file execution, durable log/artifact/metric/backup storage, dependency installation, self-update, client-manager lifecycle, and production scaling to later changes.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `durable-job-scheduling`: Defines durable platform scheduling and Run reconciliation semantics, retry and cancellation state, security fencing, safe projections, and channel isolation.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
None.
|
||||
|
||||
## Impact
|
||||
|
||||
- `platform/`: Job domain/model/repository projections, scheduler service, Run job protocol DTOs and validators, owner/admin/Run-service authorization, API handlers, and persistence/reload tests.
|
||||
- `run/` independent repository: job protocol contracts, persistent journal, worker startup reconciliation, cancellation/result behavior, and isolated HTTP channel tests.
|
||||
- `platform_web/`: safe Job API types, schemas, task-status presentation, and 401/403 regression coverage without visual-system redesign.
|
||||
- `plugins/`: existing SDK and manifests are verified to remain platform-mediated; no raw Run credential or host access is added.
|
||||
- OpenSpec: adds a cross-repository behavioral contract and verification checklist while leaving earlier completed changes unarchived.
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Durable scheduling state
|
||||
The platform SHALL persist queue eligibility, retry policy, attempt, lease hash and deadlines, progress sequence, cancellation intent/result, terminal fingerprint, and reconciliation metadata through the Job repository used by MemoryStore, FileStore, and MySQLStore. It MUST NOT require a `CoreService` memory map to recover active scheduling state and MUST NOT persist raw Run session or lease tokens.
|
||||
|
||||
#### Scenario: Platform reload preserves active attempt
|
||||
- **WHEN** a claimed or running job is reloaded into a new platform service instance
|
||||
- **THEN** the stored endpoint, attempt, hashed lease, deadlines, cancel intent, and retry metadata remain authoritative and a correctly signed current-session request with the matching lease is accepted
|
||||
|
||||
#### Scenario: File and MySQL reload preserve queued work
|
||||
- **WHEN** queued or retry-wait work is persisted and the store is reopened
|
||||
- **THEN** the same job becomes claimable only at its persisted eligibility time with its prior attempt count intact
|
||||
|
||||
### Requirement: Lease and attempt fencing
|
||||
The platform SHALL issue cryptographically random per-attempt lease tokens, persist only their hashes, and fence job messages by Run endpoint, authenticated active session generation, job ID, monotonic per-job attempt, and matching lease token. Accepted jobs SHALL have an acknowledgement deadline and running jobs SHALL have a renewable execution lease.
|
||||
|
||||
#### Scenario: Ack deadline expires
|
||||
- **WHEN** Run does not acknowledge a claimed job before its acknowledgement deadline
|
||||
- **THEN** the platform rejects the late acknowledgement and schedules the job for a later attempt or records terminal failure when retry budget is exhausted
|
||||
|
||||
#### Scenario: Execution lease expires
|
||||
- **WHEN** a running attempt sends no accepted progress or reconciliation before its lease expires
|
||||
- **THEN** the platform clears that lease and applies retry or terminal policy durably
|
||||
|
||||
#### Scenario: Old attempt arrives late
|
||||
- **WHEN** an ack, progress update, result, cancel poll, or reconciliation entry references an older attempt or lease
|
||||
- **THEN** the platform rejects it without changing the current attempt or terminal result
|
||||
|
||||
#### Scenario: Invalid endpoint or session
|
||||
- **WHEN** otherwise valid attempt evidence is signed by another endpoint, an expired or rotated session, or an invalid signature
|
||||
- **THEN** the platform returns an authentication or authorization failure and leaves the job unchanged
|
||||
|
||||
### Requirement: Retry and terminal policy
|
||||
Each job SHALL have a bounded retry policy with a maximum attempt count and exponential backoff capped by a maximum delay. Ack timeout, lease expiry, and explicitly retryable failure SHALL enter durable `retrying` state when budget remains. Succeeded, non-retryable failed, cancelled, and exhausted jobs SHALL be terminal and terminal replay SHALL be idempotent only for the same attempt and result fingerprint.
|
||||
|
||||
#### Scenario: Retry waits for backoff
|
||||
- **WHEN** an attempt fails retryably and attempts remain
|
||||
- **THEN** the job records the next eligible time and cannot be claimed before that time
|
||||
|
||||
#### Scenario: Retry claim increments attempt
|
||||
- **WHEN** backoff has elapsed and Run claims the job again
|
||||
- **THEN** the platform increments the per-job attempt and issues a different lease token
|
||||
|
||||
#### Scenario: Retry budget is exhausted
|
||||
- **WHEN** another retryable failure occurs on the maximum attempt
|
||||
- **THEN** the job becomes terminal failed and is never returned by claim
|
||||
|
||||
#### Scenario: Terminal replay conflicts
|
||||
- **WHEN** Run replays the same terminal result fingerprint for the current terminal attempt
|
||||
- **THEN** the platform returns the existing accepted result, while a different fingerprint or attempt is rejected
|
||||
|
||||
### Requirement: Idempotent durable cancellation
|
||||
The platform SHALL authorize cancellation through existing owner/server-admin/platform-admin resource checks, persist cancellation intent, and persist its terminal result. Cancellation before claim SHALL complete immediately; cancellation after claim SHALL be delivered only to the fenced active attempt and SHALL resolve idempotently.
|
||||
|
||||
#### Scenario: Cancel before claim
|
||||
- **WHEN** an authorized user cancels queued or retry-wait work
|
||||
- **THEN** the job records both cancel intent and terminal cancelled result without being claimed
|
||||
|
||||
#### Scenario: Cancel after claim
|
||||
- **WHEN** an authorized user cancels accepted or running work
|
||||
- **THEN** matching Run cancel polling observes the durable intent and a cancelled result records durable completion
|
||||
|
||||
#### Scenario: Cancel is repeated
|
||||
- **WHEN** the same authorized cancellation is requested or polled more than once
|
||||
- **THEN** the platform returns the existing intent/result without creating another attempt or conflicting terminal state
|
||||
|
||||
#### Scenario: Cross-owner cancellation is denied
|
||||
- **WHEN** a non-admin user attempts to cancel a job for a server they do not own or administer
|
||||
- **THEN** the platform returns forbidden and does not persist cancel intent
|
||||
|
||||
### Requirement: Run restart and platform reconciliation
|
||||
Run SHALL persist active assignments atomically before acknowledgement and reconcile them immediately after every registration before claiming new work. Reconciliation SHALL report job ID, attempt, and lease evidence; platform SHALL confirm only matching active attempts, rebind them to the current authenticated session generation, persist reconciliation metadata, and direct Run to discard stale or unknown entries. Platform-active entries absent from Run's report SHALL enter cancellation resolution or retry policy.
|
||||
|
||||
#### Scenario: Run restart resumes confirmed attempt
|
||||
- **WHEN** Run restarts with a valid persisted active assignment and registers a rotated session
|
||||
- **THEN** reconciliation confirms and rebinds the same attempt before Run resumes it or claims other work
|
||||
|
||||
#### Scenario: Run reports stale journal entry
|
||||
- **WHEN** Run reports a terminal, unknown, wrong-endpoint, wrong-attempt, or wrong-lease journal entry
|
||||
- **THEN** the platform does not reactivate it and instructs Run to discard it
|
||||
|
||||
#### Scenario: Platform restart accepts reconciliation
|
||||
- **WHEN** platform restarts while Run retains a valid active journal entry
|
||||
- **THEN** the platform validates it against persisted job metadata without relying on prior process memory
|
||||
|
||||
#### Scenario: Platform active work is missing from Run
|
||||
- **WHEN** authenticated reconciliation omits an accepted or running job assigned to that endpoint
|
||||
- **THEN** the platform records reconciliation loss and applies cancel or retry policy rather than silently leaving unrecoverable active work
|
||||
|
||||
### Requirement: Credential-free user projection
|
||||
The platform user API and platform_web SHALL expose only safe scheduling projections, including state, attempt counts, retry timing, cancel status, and reconcile outcome. They MUST NOT expose raw or hashed lease tokens, Run sessions, secret references, host paths, sockets, or credentials, and SHALL preserve existing 401/403 handling and crystal-moonlight console styling.
|
||||
|
||||
#### Scenario: Authorized user reads job scheduling status
|
||||
- **WHEN** a server owner, server administrator, or platform administrator reads an accessible job
|
||||
- **THEN** the response includes safe attempt, retry, cancellation, terminal, and reconcile fields without credential material
|
||||
|
||||
#### Scenario: Unauthorized user reads another owner's job
|
||||
- **WHEN** a user without resource access requests another server's job
|
||||
- **THEN** the API returns forbidden or not found according to the existing resource policy and platform_web follows existing 401/403 handling
|
||||
|
||||
### Requirement: Independent channel priority
|
||||
Platform and Run SHALL keep control, jobs, logs, and artifacts on independent request paths and execution queues. Blocking or retrying log/artifact transfer MUST NOT block control heartbeat or job claim, ack, progress, result, cancel, or reconciliation traffic, and job/control messages MUST NOT carry log bodies or artifact payloads.
|
||||
|
||||
#### Scenario: Artifact transfer blocks
|
||||
- **WHEN** an artifact chunk request remains blocked
|
||||
- **THEN** heartbeat and job acknowledgement/result/cancel/reconcile requests still complete within their own deadlines
|
||||
|
||||
#### Scenario: Log ingest blocks
|
||||
- **WHEN** a log batch upload remains blocked or retries
|
||||
- **THEN** control heartbeat and job lifecycle traffic continue independently
|
||||
|
||||
### Requirement: Explicit roadmap boundary
|
||||
This change SHALL NOT claim production readiness for process supervision, real config/file execution, durable log/artifact/metric/backup storage, dependency installation, Run self-update, client-manager lifecycle, or production multi-replica scaling.
|
||||
|
||||
#### Scenario: Completion is reported
|
||||
- **WHEN** the durable scheduling change passes implementation and verification
|
||||
- **THEN** its handoff identifies those capabilities as remaining later-route work
|
||||
@@ -0,0 +1,43 @@
|
||||
## 1. Durable Platform Model
|
||||
|
||||
- [x] 1.1 Add Job domain/model scheduling, retry, lease, cancellation, terminal, and reconciliation fields with legacy zero-value normalization and validation.
|
||||
- [x] 1.2 Persist and reload the expanded Job aggregate through MemoryStore, FileStore, and MySQLStore repository paths without raw lease or session credentials.
|
||||
|
||||
## 2. Platform Scheduler State Machine
|
||||
|
||||
- [x] 2.1 Replace the CoreService in-memory job lease map with repository-backed claim, ack, lease renewal, progress sequencing, retry backoff, and terminal fencing transitions.
|
||||
- [x] 2.2 Implement idempotent cancel-before-claim, cancel-after-claim polling/result, deadline expiry, late-message rejection, and exhausted retry behavior.
|
||||
- [x] 2.3 Implement endpoint/session-generation reconciliation using attempt and lease evidence, including missing/stale work handling and persisted reconcile outcomes.
|
||||
|
||||
## 3. Contracts, Security, And API
|
||||
|
||||
- [x] 3.1 Extend platform Run job DTO/domain/validator/protocol contracts for deadlines, retryable result, full reconciliation evidence, and discard outcomes.
|
||||
- [x] 3.2 Preserve signed Run-service endpoint/session checks and owner/server-admin/platform-admin authorization for job read and cancellation APIs.
|
||||
- [x] 3.3 Expose only credential-free user Job scheduling projections and update API contract documentation.
|
||||
|
||||
## 4. Independent Run Recovery
|
||||
|
||||
- [x] 4.1 Extend the independent Run job protocol and client for deadlines, retry outcomes, reconciliation evidence, and discard instructions.
|
||||
- [x] 4.2 Implement an atomic, owner-only persistent Run job journal and startup reconciliation before new claims.
|
||||
- [x] 4.3 Make the worker recover confirmed attempts, handle cancellation idempotently, retain unaccepted results for later reconciliation, and discard platform-rejected entries.
|
||||
- [x] 4.4 Prove control/job requests remain independent while log or artifact transfers block.
|
||||
|
||||
## 5. Safe Console Projection
|
||||
|
||||
- [x] 5.1 Extend platform_web API types and schemas with safe attempt, retry, cancel, terminal, and reconcile fields while preserving shared visual styles and 401/403 behavior.
|
||||
- [x] 5.2 Update runtime task status presentation and tests without exposing lease/session/secret/host data or redesigning the console.
|
||||
|
||||
## 6. Regression And Completion Evidence
|
||||
|
||||
- [x] 6.1 Add platform tests for queue/store reload, lease and ack expiry, attempt fencing, retry/backoff, cancellation timing/idempotency, late messages, restart reconciliation, endpoint/owner rejection, and signature/session failure.
|
||||
- [x] 6.2 Add independent Run tests for journal reload, restart reconciliation/recovery, cancel/result retention, stale discard, and channel isolation.
|
||||
- [x] 6.3 Run plugin manifest/SDK tests, platform Go tests, platform_web tests/typecheck/build, independent Run tests, strict OpenSpec validation, structure check, shell/compose checks where affected, and both repositories' diff checks; then record evidence before checking tasks complete.
|
||||
|
||||
## Verification Evidence (2026-07-18)
|
||||
|
||||
- `platform`: `go test -count=1 ./...` passed across API, config, domain, DTO, model, repository, service, and validator packages.
|
||||
- `platform_web`: 92 tests passed across 17 files; `npm run typecheck` and production `npm run build` passed.
|
||||
- `plugins`: 17 manifest/SDK tests passed; TypeScript typecheck passed; dev, SCUM, and Minecraft example manifests validated.
|
||||
- Independent `run`: `go test -count=1 ./...` passed across API, config, protocol, runtime, and spool packages.
|
||||
- `openspec validate implement-durable-job-scheduling-and-reconciliation --strict`, `scripts/check-structure.sh`, shell syntax checks, and `docker compose config --quiet` passed.
|
||||
- `git diff --check` passed in both `/Users/tasia/Desktop/code/browser` and the independent `/Users/tasia/Desktop/code/browser/run` repositories.
|
||||
Reference in New Issue
Block a user