79 lines
8.5 KiB
Markdown
79 lines
8.5 KiB
Markdown
## Context
|
|
|
|
`domain.Job` is already the durable unit stored by MemoryStore, FileStore, and the repository-backed MySQLStore, but it currently contains only the user-visible lifecycle projection. `CoreService` separately owns an in-memory `jobLeases` map containing raw Run session tokens, lease tokens, attempts, cancel intent, and terminal fingerprints. Restarting the platform therefore makes accepted/running jobs impossible to authenticate or complete safely. The independent Run repository similarly uses an in-memory active assignment map and does not reconcile during startup.
|
|
|
|
The existing HTTP surface already separates control, jobs, logs, and artifacts and signs non-hello Run requests with the active Run session. This change must deepen those boundaries without introducing an all-in-one transport, exposing credentials to the browser/plugins, or implementing later execution/storage roadmap items.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
|
|
- Make every scheduling decision recoverable from the Job repository after platform process restart or FileStore/MySQLStore reload.
|
|
- Fence every ack, progress, result, cancel poll, and reconcile report by endpoint, active authenticated session, per-job attempt, and a lease credential whose persisted form is a hash.
|
|
- Define bounded ack and execution leases, monotonic per-job attempts and progress sequences, exponential retry backoff, terminal idempotency, and durable cancel intent/result.
|
|
- Make Run persist active assignments atomically under its workspace, reconcile immediately after registration, and recover or discard work according to the platform response.
|
|
- Keep safe browser projections and owner/platform-admin authorization aligned with existing server ownership checks.
|
|
- Preserve independent control/job/log/artifact paths and prove large transfer requests cannot block control or job requests.
|
|
|
|
**Non-Goals:**
|
|
|
|
- Production-grade distributed scheduling or multi-platform-replica compare-and-swap coordination.
|
|
- Process supervision, real config/file mutation, durable logs/artifacts/metrics/backups, dependency installation, Run self-update, client-manager lifecycle, or production scaling.
|
|
- Persisting raw Run session tokens, lease tokens, host paths, sockets, AI keys, component keys, or other credentials.
|
|
|
|
## Decisions
|
|
|
|
### Persist scheduling metadata on the Job aggregate
|
|
|
|
`domain.Job` and `model.Job` gain a nested retry policy plus queue, attempt, hashed lease, ack/lease deadline, last progress sequence, cancel, terminal, and reconciliation fields. The existing Job repository remains the only storage boundary, so MemoryStore, FileStore, and MySQLStore inherit the behavior through their existing typed snapshot/repository implementations.
|
|
|
|
This is preferred over retaining a service cache or introducing a second lease repository because transitions need one recoverable aggregate and the current stores do not provide cross-repository transactions. Raw lease tokens are generated with cryptographic randomness, returned only over the signed Run job channel, and stored as SHA-256 hashes.
|
|
|
|
### Use a deterministic per-job state machine
|
|
|
|
New jobs default to `queued`, attempt zero, a bounded retry policy, and an immediately eligible queue timestamp. Claim sweeps expired work for the endpoint, selects eligible `queued` or `retrying` work in stable creation order, increments that job's attempt, stores the lease hash and current Run session generation, and moves it to `accepted` with an ack deadline and execution lease deadline.
|
|
|
|
Ack before the deadline moves the attempt to `running` and renews its lease. Monotonic progress renews the running lease. Ack timeout, execution lease expiry, or an explicitly retryable failed result clears the lease and moves the job to `retrying` with exponential backoff when attempts remain; otherwise it records a terminal `failed` result. A pending cancel intent always resolves to `cancelled` rather than retrying when the lease expires. Old attempts and old lease tokens are rejected even after a newer attempt exists.
|
|
|
|
### Model cancellation as durable intent followed by durable result
|
|
|
|
Cancel before claim atomically records intent and a terminal cancelled result. Cancel after claim records idempotent intent for the assigned Run endpoint; polling is fenced by the current attempt and lease, and Run returns a normal terminal cancelled result. Repeated identical or compatible cancel requests return the existing intent/result projection. Cancellation never consumes another retry attempt.
|
|
|
|
### Reconcile with full active-attempt evidence
|
|
|
|
Run reports persisted active assignment evidence: job ID, attempt, and raw lease token over the signed job channel. Platform verifies the endpoint, stored attempt, lease hash, nonterminal state, and active Run session, then rebinds the lease to the current session generation and renews its deadline. The raw token is echoed only in the signed response and is never persisted.
|
|
|
|
Reported stale/unknown jobs are returned as discard IDs. Platform-active jobs absent from the Run journal are treated as abandoned and enter cancel resolution or retry policy. Reconciliation timestamps, counts, and outcome are persisted on every affected Job. This is preferred over recreating leases during reconciliation because recreating them would let stale attempts regain authority.
|
|
|
|
### Persist Run journal atomically and reconcile before claiming
|
|
|
|
Run stores a versioned JSON journal under `WorkspaceRoot`, using a temporary file plus rename and owner-only permissions. It writes the assignment before ack, updates it after ack/progress, and removes it only after an accepted terminal response or explicit platform discard. Worker startup registers, reconciles the loaded journal before any new claim, and re-executes platform-confirmed assignments under the same attempt. Corrupt journal data fails worker construction instead of silently forgetting active work.
|
|
|
|
### Keep API and UI projections credential-free
|
|
|
|
Platform user APIs expose attempt, max attempts, next retry time, ack/lease deadlines, cancel state/timestamps, terminal time, and last reconcile outcome. They omit lease hashes, raw lease tokens, session generation, secret refs, and host/runtime credentials. Existing `GetJobForSession`, `ListJobsForSession`, and cancel authorization continue to derive access from server owner/admin or platform-admin rules. Run-only handlers continue to require both the active session and request signature middleware.
|
|
|
|
### Preserve channel independence
|
|
|
|
Control, jobs, logs, and artifacts remain separate HTTP routes and clients. Run's worker loop executes heartbeat and scheduling work independently from spool/artifact transfer queues; neither the job protocol nor safe browser projection carries log bodies or artifact payloads. Isolation tests block log/artifact handlers while asserting heartbeat, ack, result, cancel, and reconcile complete.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- [FileStore and the current MySQLStore are process-local snapshot implementations, not distributed CAS schedulers] -> Serialize transitions with the existing service mutex and explicitly keep multi-replica production scheduling out of scope.
|
|
- [At-least-once recovery can repeat an interrupted operation] -> Preserve idempotency keys, fence attempts, and require bounded Run executors to be idempotent; later process supervision will refine resumability.
|
|
- [Legacy persisted jobs lack new fields] -> Normalize zero-value scheduling fields when read/claimed so existing queued and terminal records remain valid without destructive migration.
|
|
- [Hash-only lease storage means platform cannot recreate a lost Run lease credential] -> Require the durable Run journal to present the original token; otherwise the platform retries with a new attempt after reconciliation/expiry.
|
|
- [Clock skew between Run and platform] -> Treat platform timestamps as authoritative; Run does not decide lease validity locally.
|
|
|
|
## Migration Plan
|
|
|
|
1. Deploy the expanded model/repository projection and zero-value normalization before relying on new states.
|
|
2. Deploy platform protocol and scheduler behavior with compatibility for an empty legacy reconcile evidence list.
|
|
3. Deploy Run protocol and persistent journal, which reconciles immediately after registration before claiming.
|
|
4. Deploy the safe web projection and tests.
|
|
5. Rollback may read the expanded JSON while ignoring unknown fields, but accepted/running jobs should be allowed to reconcile or expire before rolling back to code that lacks durable fencing.
|
|
|
|
## Open Questions
|
|
|
|
None for this single-process scheduling milestone. Cross-replica transactional claiming and resumable supervised processes remain explicit later design work.
|