## ADDED Requirements

### Requirement: Durable scheduling state
The platform SHALL persist queue eligibility, retry policy, attempt, lease hash and deadlines, progress sequence, cancellation intent/result, terminal fingerprint, and reconciliation metadata through the Job repository used by MemoryStore, FileStore, and MySQLStore. It MUST NOT require a `CoreService` memory map to recover active scheduling state and MUST NOT persist raw Run session or lease tokens.

#### Scenario: Platform reload preserves active attempt
- **WHEN** a claimed or running job is reloaded into a new platform service instance
- **THEN** the stored endpoint, attempt, hashed lease, deadlines, cancel intent, and retry metadata remain authoritative and a correctly signed current-session request with the matching lease is accepted

#### Scenario: File and MySQL reload preserve queued work
- **WHEN** queued or retry-wait work is persisted and the store is reopened
- **THEN** the same job becomes claimable only at its persisted eligibility time with its prior attempt count intact

### Requirement: Lease and attempt fencing
The platform SHALL issue cryptographically random per-attempt lease tokens, persist only their hashes, and fence job messages by Run endpoint, authenticated active session generation, job ID, monotonic per-job attempt, and matching lease token. Accepted jobs SHALL have an acknowledgement deadline and running jobs SHALL have a renewable execution lease.

#### Scenario: Ack deadline expires
- **WHEN** Run does not acknowledge a claimed job before its acknowledgement deadline
- **THEN** the platform rejects the late acknowledgement and schedules the job for a later attempt or records terminal failure when retry budget is exhausted

#### Scenario: Execution lease expires
- **WHEN** a running attempt sends no accepted progress or reconciliation before its lease expires
- **THEN** the platform clears that lease and applies retry or terminal policy durably

#### Scenario: Old attempt arrives late
- **WHEN** an ack, progress update, result, cancel poll, or reconciliation entry references an older attempt or lease
- **THEN** the platform rejects it without changing the current attempt or terminal result

#### Scenario: Invalid endpoint or session
- **WHEN** otherwise valid attempt evidence is signed by another endpoint, an expired or rotated session, or an invalid signature
- **THEN** the platform returns an authentication or authorization failure and leaves the job unchanged

### Requirement: Retry and terminal policy
Each job SHALL have a bounded retry policy with a maximum attempt count and exponential backoff capped by a maximum delay. Ack timeout, lease expiry, and explicitly retryable failure SHALL enter durable `retrying` state when budget remains. Succeeded, non-retryable failed, cancelled, and exhausted jobs SHALL be terminal and terminal replay SHALL be idempotent only for the same attempt and result fingerprint.

#### Scenario: Retry waits for backoff
- **WHEN** an attempt fails retryably and attempts remain
- **THEN** the job records the next eligible time and cannot be claimed before that time

#### Scenario: Retry claim increments attempt
- **WHEN** backoff has elapsed and Run claims the job again
- **THEN** the platform increments the per-job attempt and issues a different lease token

#### Scenario: Retry budget is exhausted
- **WHEN** another retryable failure occurs on the maximum attempt
- **THEN** the job becomes terminal failed and is never returned by claim

#### Scenario: Terminal replay conflicts
- **WHEN** Run replays the same terminal result fingerprint for the current terminal attempt
- **THEN** the platform returns the existing accepted result, while a different fingerprint or attempt is rejected

### Requirement: Idempotent durable cancellation
The platform SHALL authorize cancellation through existing owner/server-admin/platform-admin resource checks, persist cancellation intent, and persist its terminal result. Cancellation before claim SHALL complete immediately; cancellation after claim SHALL be delivered only to the fenced active attempt and SHALL resolve idempotently.

#### Scenario: Cancel before claim
- **WHEN** an authorized user cancels queued or retry-wait work
- **THEN** the job records both cancel intent and terminal cancelled result without being claimed

#### Scenario: Cancel after claim
- **WHEN** an authorized user cancels accepted or running work
- **THEN** matching Run cancel polling observes the durable intent and a cancelled result records durable completion

#### Scenario: Cancel is repeated
- **WHEN** the same authorized cancellation is requested or polled more than once
- **THEN** the platform returns the existing intent/result without creating another attempt or conflicting terminal state

#### Scenario: Cross-owner cancellation is denied
- **WHEN** a non-admin user attempts to cancel a job for a server they do not own or administer
- **THEN** the platform returns forbidden and does not persist cancel intent

### Requirement: Run restart and platform reconciliation
Run SHALL persist active assignments atomically before acknowledgement and reconcile them immediately after every registration before claiming new work. Reconciliation SHALL report job ID, attempt, and lease evidence; platform SHALL confirm only matching active attempts, rebind them to the current authenticated session generation, persist reconciliation metadata, and direct Run to discard stale or unknown entries. Platform-active entries absent from Run's report SHALL enter cancellation resolution or retry policy.

#### Scenario: Run restart resumes confirmed attempt
- **WHEN** Run restarts with a valid persisted active assignment and registers a rotated session
- **THEN** reconciliation confirms and rebinds the same attempt before Run resumes it or claims other work

#### Scenario: Run reports stale journal entry
- **WHEN** Run reports a terminal, unknown, wrong-endpoint, wrong-attempt, or wrong-lease journal entry
- **THEN** the platform does not reactivate it and instructs Run to discard it

#### Scenario: Platform restart accepts reconciliation
- **WHEN** platform restarts while Run retains a valid active journal entry
- **THEN** the platform validates it against persisted job metadata without relying on prior process memory

#### Scenario: Platform active work is missing from Run
- **WHEN** authenticated reconciliation omits an accepted or running job assigned to that endpoint
- **THEN** the platform records reconciliation loss and applies cancel or retry policy rather than silently leaving unrecoverable active work

### Requirement: Credential-free user projection
The platform user API and platform_web SHALL expose only safe scheduling projections, including state, attempt counts, retry timing, cancel status, and reconcile outcome. They MUST NOT expose raw or hashed lease tokens, Run sessions, secret references, host paths, sockets, or credentials, and SHALL preserve existing 401/403 handling and crystal-moonlight console styling.

#### Scenario: Authorized user reads job scheduling status
- **WHEN** a server owner, server administrator, or platform administrator reads an accessible job
- **THEN** the response includes safe attempt, retry, cancellation, terminal, and reconcile fields without credential material

#### Scenario: Unauthorized user reads another owner's job
- **WHEN** a user without resource access requests another server's job
- **THEN** the API returns forbidden or not found according to the existing resource policy and platform_web follows existing 401/403 handling

### Requirement: Independent channel priority
Platform and Run SHALL keep control, jobs, logs, and artifacts on independent request paths and execution queues. Blocking or retrying log/artifact transfer MUST NOT block control heartbeat or job claim, ack, progress, result, cancel, or reconciliation traffic, and job/control messages MUST NOT carry log bodies or artifact payloads.

#### Scenario: Artifact transfer blocks
- **WHEN** an artifact chunk request remains blocked
- **THEN** heartbeat and job acknowledgement/result/cancel/reconcile requests still complete within their own deadlines

#### Scenario: Log ingest blocks
- **WHEN** a log batch upload remains blocked or retries
- **THEN** control heartbeat and job lifecycle traffic continue independently

### Requirement: Explicit roadmap boundary
This change SHALL NOT claim production readiness for process supervision, real config/file execution, durable log/artifact/metric/backup storage, dependency installation, Run self-update, client-manager lifecycle, or production multi-replica scaling.

#### Scenario: Completion is reported
- **WHEN** the durable scheduling change passes implementation and verification
- **THEN** its handoff identifies those capabilities as remaining later-route work