browser/openspec/changes/implement-durable-job-scheduling-and-reconciliation/specs/durable-job-scheduling/spec.md

ADDED Requirements

Requirement: Durable scheduling state

The platform SHALL persist queue eligibility, retry policy, attempt, lease hash and deadlines, progress sequence, cancellation intent/result, terminal fingerprint, and reconciliation metadata through the Job repository used by MemoryStore, FileStore, and MySQLStore. It MUST NOT require a CoreService memory map to recover active scheduling state and MUST NOT persist raw Run session or lease tokens.

Scenario: Platform reload preserves active attempt

• WHEN a claimed or running job is reloaded into a new platform service instance
• THEN the stored endpoint, attempt, hashed lease, deadlines, cancel intent, and retry metadata remain authoritative and a correctly signed current-session request with the matching lease is accepted

Scenario: File and MySQL reload preserve queued work

• WHEN queued or retry-wait work is persisted and the store is reopened
• THEN the same job becomes claimable only at its persisted eligibility time with its prior attempt count intact

Requirement: Lease and attempt fencing

The platform SHALL issue cryptographically random per-attempt lease tokens, persist only their hashes, and fence job messages by Run endpoint, authenticated active session generation, job ID, monotonic per-job attempt, and matching lease token. Accepted jobs SHALL have an acknowledgement deadline and running jobs SHALL have a renewable execution lease.

Scenario: Ack deadline expires

• WHEN Run does not acknowledge a claimed job before its acknowledgement deadline
• THEN the platform rejects the late acknowledgement and schedules the job for a later attempt or records terminal failure when retry budget is exhausted

Scenario: Execution lease expires

• WHEN a running attempt sends no accepted progress or reconciliation before its lease expires
• THEN the platform clears that lease and applies retry or terminal policy durably

Scenario: Old attempt arrives late

• WHEN an ack, progress update, result, cancel poll, or reconciliation entry references an older attempt or lease
• THEN the platform rejects it without changing the current attempt or terminal result

Scenario: Invalid endpoint or session

• WHEN otherwise valid attempt evidence is signed by another endpoint, an expired or rotated session, or an invalid signature
• THEN the platform returns an authentication or authorization failure and leaves the job unchanged

Requirement: Retry and terminal policy

Each job SHALL have a bounded retry policy with a maximum attempt count and exponential backoff capped by a maximum delay. Ack timeout, lease expiry, and explicitly retryable failure SHALL enter durable retrying state when budget remains. Succeeded, non-retryable failed, cancelled, and exhausted jobs SHALL be terminal and terminal replay SHALL be idempotent only for the same attempt and result fingerprint.

Scenario: Retry waits for backoff

• WHEN an attempt fails retryably and attempts remain
• THEN the job records the next eligible time and cannot be claimed before that time

Scenario: Retry claim increments attempt

• WHEN backoff has elapsed and Run claims the job again
• THEN the platform increments the per-job attempt and issues a different lease token

Scenario: Retry budget is exhausted

• WHEN another retryable failure occurs on the maximum attempt
• THEN the job becomes terminal failed and is never returned by claim

Scenario: Terminal replay conflicts

• WHEN Run replays the same terminal result fingerprint for the current terminal attempt
• THEN the platform returns the existing accepted result, while a different fingerprint or attempt is rejected

Requirement: Idempotent durable cancellation

The platform SHALL authorize cancellation through existing owner/server-admin/platform-admin resource checks, persist cancellation intent, and persist its terminal result. Cancellation before claim SHALL complete immediately; cancellation after claim SHALL be delivered only to the fenced active attempt and SHALL resolve idempotently.

Scenario: Cancel before claim

• WHEN an authorized user cancels queued or retry-wait work
• THEN the job records both cancel intent and terminal cancelled result without being claimed

Scenario: Cancel after claim

• WHEN an authorized user cancels accepted or running work
• THEN matching Run cancel polling observes the durable intent and a cancelled result records durable completion

Scenario: Cancel is repeated

• WHEN the same authorized cancellation is requested or polled more than once
• THEN the platform returns the existing intent/result without creating another attempt or conflicting terminal state

Scenario: Cross-owner cancellation is denied

• WHEN a non-admin user attempts to cancel a job for a server they do not own or administer
• THEN the platform returns forbidden and does not persist cancel intent

Requirement: Run restart and platform reconciliation

Run SHALL persist active assignments atomically before acknowledgement and reconcile them immediately after every registration before claiming new work. Reconciliation SHALL report job ID, attempt, and lease evidence; platform SHALL confirm only matching active attempts, rebind them to the current authenticated session generation, persist reconciliation metadata, and direct Run to discard stale or unknown entries. Platform-active entries absent from Run's report SHALL enter cancellation resolution or retry policy.

Scenario: Run restart resumes confirmed attempt

• WHEN Run restarts with a valid persisted active assignment and registers a rotated session
• THEN reconciliation confirms and rebinds the same attempt before Run resumes it or claims other work

Scenario: Run reports stale journal entry

• WHEN Run reports a terminal, unknown, wrong-endpoint, wrong-attempt, or wrong-lease journal entry
• THEN the platform does not reactivate it and instructs Run to discard it

Scenario: Platform restart accepts reconciliation

• WHEN platform restarts while Run retains a valid active journal entry
• THEN the platform validates it against persisted job metadata without relying on prior process memory

Scenario: Platform active work is missing from Run

• WHEN authenticated reconciliation omits an accepted or running job assigned to that endpoint
• THEN the platform records reconciliation loss and applies cancel or retry policy rather than silently leaving unrecoverable active work

Requirement: Credential-free user projection

The platform user API and platform_web SHALL expose only safe scheduling projections, including state, attempt counts, retry timing, cancel status, and reconcile outcome. They MUST NOT expose raw or hashed lease tokens, Run sessions, secret references, host paths, sockets, or credentials, and SHALL preserve existing 401/403 handling and crystal-moonlight console styling.

Scenario: Authorized user reads job scheduling status

• WHEN a server owner, server administrator, or platform administrator reads an accessible job
• THEN the response includes safe attempt, retry, cancellation, terminal, and reconcile fields without credential material

Scenario: Unauthorized user reads another owner's job

• WHEN a user without resource access requests another server's job
• THEN the API returns forbidden or not found according to the existing resource policy and platform_web follows existing 401/403 handling

Requirement: Independent channel priority

Platform and Run SHALL keep control, jobs, logs, and artifacts on independent request paths and execution queues. Blocking or retrying log/artifact transfer MUST NOT block control heartbeat or job claim, ack, progress, result, cancel, or reconciliation traffic, and job/control messages MUST NOT carry log bodies or artifact payloads.

Scenario: Artifact transfer blocks

• WHEN an artifact chunk request remains blocked
• THEN heartbeat and job acknowledgement/result/cancel/reconcile requests still complete within their own deadlines

Scenario: Log ingest blocks

• WHEN a log batch upload remains blocked or retries
• THEN control heartbeat and job lifecycle traffic continue independently

Requirement: Explicit roadmap boundary

This change SHALL NOT claim production readiness for process supervision, real config/file execution, durable log/artifact/metric/backup storage, dependency installation, Run self-update, client-manager lifecycle, or production multi-replica scaling.

Scenario: Completion is reported

• WHEN the durable scheduling change passes implementation and verification
• THEN its handoff identifies those capabilities as remaining later-route work