diff --git a/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/.openspec.yaml b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/.openspec.yaml new file mode 100644 index 0000000..ab39675 --- /dev/null +++ b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-30 diff --git a/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/design.md b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/design.md new file mode 100644 index 0000000..ec82c24 --- /dev/null +++ b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/design.md @@ -0,0 +1,91 @@ +## Context + +Platform currently persists `RunEndpoint` rows and stores `ServerInstance.RunEndpointID` as if a server is bound to a durable machine endpoint. The same identifier is then reused for control registration, heartbeat status, job scheduling, UI availability, and component revocation. That model conflicts with the intended architecture: Platform is a registry/dispatcher, while Run is a server-scoped RPC worker that proves possession of a component token, registers a live session, heartbeats, and claims work only while that session lease is current. + +The recent platform-side builder work already removed distribution builds from machine-side Run authority. This change continues that direction by removing durable run endpoint rows from first-party server runtime routing. Redis becomes the required runtime registry for local development, tests, and production so that all environments exercise the same lease and expiry behavior. + +## Goals / Non-Goals + +**Goals:** + +- Replace durable run endpoint routing with Redis-backed runtime session leases. +- Route runtime jobs by logical target (`serverInstanceId`, `componentKind`, optional `componentKey`) instead of endpoint rows. +- Keep server creation, deployment definitions, component auth keys, jobs, logs, artifacts, and audits durable in the database. +- Make runtime online state, capabilities, capacity, session token hashes, and heartbeat freshness ephemeral and TTL-backed. +- Require Redis in development and tests; avoid memory-only behavior hiding lease/routing bugs. +- Preserve platform-owned distribution builds and prevent generated Runs from receiving build authority. + +**Non-Goals:** + +- Do not introduce cloud hosting, SSH provisioning, billing, or external marketplace workflows. +- Do not store raw credentials, host paths, sockets, or plaintext component keys in Redis. +- Do not move durable job history, audit events, artifacts, or server definitions out of the database. +- Do not re-add a `run/` source tree to this repository. + +## Decisions + +### 1. Redis is the only runtime registry for dev, test, and production + +Platform SHALL require Redis for runtime registry behavior in every normal environment. Tests may start an isolated Redis instance or use an explicit test Redis database/prefix, but they SHALL NOT swap in a memory registry for ordinary execution. + +This keeps expiration, reconnect, session loss, and multi-process behavior visible during development. An in-memory registry would be simpler, but it would let tests pass with semantics that fail once Platform runs more than one process or restarts. + +### 2. Registry data is ephemeral and TTL-owned + +Redis stores only live-session data: + +- `runtime:v1:session:{serverInstanceId}:{componentKind}:{componentKey}` +- `runtime:v1:token:{tokenHash}` +- optional short-lived indexes for online summaries and capability snapshots + +Every key has a TTL derived from the heartbeat interval plus a small grace window. Platform startup does not scan or clean Redis. Stale sessions expire naturally, and a Redis flush/restart is treated like all Runs temporarily went offline until they register again. + +### 3. Component token identity replaces endpoint ownership + +A generated Run hello includes `serverInstanceId`, `componentKind=run`, component key/generation, registration proof, version, target OS/arch, capabilities, and capacity. Platform authenticates the component key from durable database state, then writes a Redis session lease. The accepted session token is scoped to that server/component and is used for heartbeat, claim, ack, progress, result, logs, and artifact operations. + +The system does not need a pre-existing endpoint row or a server-to-endpoint foreign key. If two Runs present the same component identity, the later valid registration supersedes the previous session by rotating the token lease. + +### 4. Jobs target logical components, not endpoints + +Durable jobs use a logical target: + +- `serverInstanceId` +- `targetComponentKind` such as `run`, `client-manager`, or `platform-builder` +- `targetComponentKey` for keyed components, empty for the server Run + +Run job claim authenticates the session token, derives the server/component target from Redis, and returns only eligible jobs for that target. Leases remain durable on the job record so retries and audit history survive Platform restarts. + +### 5. UI shows runtime connection health, not endpoint selection + +Platform Web replaces endpoint selection/status surfaces with server runtime connection health. The user sees whether the generated Run is registered, heartbeat freshness, version, target OS/arch, capabilities, current capacity, and safe unavailable reasons. The UI does not ask the owner to choose a run endpoint when creating, editing, or deploying a server. + +### 6. Compatibility is staged but not permanent + +Existing code paths that accept `runEndpointId` become compatibility shims during migration. They should either translate to logical targets where safe or return a clear deprecation validation error in first-party workflows. New code must not create `RunEndpoint` rows for machine-side generated Runs. + +## Risks / Trade-offs + +- [Redis unavailable blocks runtime routing] -> Treat Redis as required platform infrastructure; startup/health checks must report runtime registry unavailable and runtime operations must fail safely without dispatching jobs. +- [Redis restart marks healthy Runs offline until reconnect] -> Runs already heartbeat frequently; clients retry registration when heartbeat fails or receives an unknown-session response. +- [Large migration surface] -> Move in phases: registry interface first, job target fields second, UI/API cleanup third, repository deletion last. +- [Old endpoint-based packages reconnect] -> Compatibility can accept legacy registration only behind explicit migration rules; generated packages should be rebuilt with server/component identity. +- [Tests become slower with Redis] -> Use a test Redis prefix/database and cleanup by prefix in test setup, while still relying on TTL behavior for session expiry scenarios. + +## Migration Plan + +1. Add Redis configuration, health checks, and test harness support; fail fast when Redis is unavailable. +2. Introduce `RuntimeSessionRegistry` backed by Redis and move Run hello/heartbeat/token validation onto it. +3. Add durable job logical target fields while temporarily writing both logical target and legacy `runEndpointId`. +4. Change job claim, ack, progress, result, log, artifact, config, file, and lifecycle dispatch to authorize through registry-derived server/component targets. +5. Remove first-party server creation/edit/deployment references to run endpoint selection and project runtime connection health from Redis. +6. Migrate or deprecate legacy endpoint-based records; stop creating `RunEndpoint` rows for generated Runs. +7. Remove obsolete repository methods, DTO fields, tests, and documentation once compatibility paths are no longer used. + +Rollback is limited to keeping the compatibility shim and disabling new runtime dispatch. Durable server definitions and jobs remain in the database; Redis contains only disposable session leases. + +## Open Questions + +- Should Redis be required at Platform process startup, or can only runtime routes fail health checks while non-runtime admin pages stay available? +- Which Redis deployment profile should local scripts use by default: Docker Compose service, existing local Redis, or a repo-managed test container? +- How long should the legacy `runEndpointId` compatibility window remain before API fields are removed? diff --git a/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/proposal.md b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/proposal.md new file mode 100644 index 0000000..55a0bb9 --- /dev/null +++ b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/proposal.md @@ -0,0 +1,29 @@ +## Why + +The platform currently treats `RunEndpoint` as both a durable database resource and a live runtime connection. That creates the wrong product model: Platform should behave like a registry and dispatcher, while Run should behave like an authenticated RPC worker that registers, heartbeats, and receives work only while its token-backed session is alive. + +## What Changes + +- **BREAKING** Remove first-party server-to-`RunEndpoint` persistence as the runtime routing model; server records no longer bind to a durable run endpoint row. +- Introduce a Redis-backed runtime session registry for Run registration, heartbeat leases, capability snapshots, capacity snapshots, and token/session lookup. +- Require development, test, and production flows to use Redis for runtime registry behavior; do not add an in-memory registry fallback for normal test execution. +- Retarget durable jobs from endpoint rows to logical server/component targets such as `serverInstanceId + componentKind + componentKey`. +- Keep platform-owned distribution builds separate from machine-side Run sessions; build routing remains a platform builder responsibility, not a registered Run endpoint capability. +- Replace user-facing run endpoint selection/status with runtime connection health derived from Redis leases and durable component/server records. + +## Capabilities + +### New Capabilities + +- `redis-runtime-session-registry`: Redis-backed registration, heartbeat, token validation, routing, and capability snapshots for machine-side Run sessions. + +### Modified Capabilities + +- `platform-side-distribution-builds`: Remove requirements that bind a server instance to a generated run endpoint; preserve platform-owned builds while using runtime registration rather than durable endpoint rows. + +## Impact + +- `platform/`: domain types, DTOs, repositories, service scheduling, Run control registration, job claim/lease validation, server lifecycle dispatch, runtime action availability, tests, and local dev/test setup. +- `platform_web/`: server management API types, runtime connection status UI, run generation/deployment copy, and tests that currently reference run endpoints. +- `plugins/`: plugin bridge and companion-facing contracts where they expose or consume run endpoint identifiers. +- Infrastructure: Redis becomes a required dependency for dev, test, and production runtime registry behavior. diff --git a/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/specs/platform-side-distribution-builds/spec.md b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/specs/platform-side-distribution-builds/spec.md new file mode 100644 index 0000000..59bee12 --- /dev/null +++ b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/specs/platform-side-distribution-builds/spec.md @@ -0,0 +1,42 @@ +## MODIFIED Requirements + +### Requirement: Server creation requires only plugin type and server name +The system SHALL require only the game plugin type and the server name to create a server instance, and SHALL NOT require a deployment target, run endpoint, or runtime profile at creation time. The creation workflow MAY collect plugin-declared deployment mode, game configuration, and startup fields before submit, but those fields SHALL NOT create a durable run endpoint binding. + +#### Scenario: Creation form field set +- **WHEN** an owner opens the server creation workflow +- **THEN** the form requires plugin type and server name only, may present plugin-declared deployment/startup inputs, and presents no deployment target or run endpoint selector as a creation prerequisite + +#### Scenario: Creation without any registered runtime session +- **WHEN** an owner creates a server instance while no Run has registered for that instance +- **THEN** creation succeeds and the instance is created without a run endpoint binding + +#### Scenario: Runtime session established by run registration +- **WHEN** a generated Run for that instance registers itself with the platform +- **THEN** the platform creates or renews a Redis runtime session lease for that server Run and does not persist a server-to-run-endpoint association + +#### Scenario: Runtime profile and endpoint selection are not creation prerequisites +- **WHEN** an owner opens an already-created instance +- **THEN** runtime profile and run endpoint selection are not required to make the instance exist, generate a Run package, or show runtime connection guidance + +### Requirement: Build availability is independent of run endpoint capabilities +The system SHALL determine `generate-run` and `generate-client-manager` availability from plugin declarations, runtime bindings, and platform builder readiness, and SHALL NOT require any machine-side Run session or durable run endpoint row to advertise `distribution.build`. + +#### Scenario: Instance has only its generated Run runtime session +- **WHEN** a server instance's only live runtime session is its generated Run, which holds no distribution-build authority +- **THEN** `generate-run` remains available and a new run distribution can be generated through the platform builder + +#### Scenario: No privileged worker endpoint registered +- **WHEN** no machine-side Run advertises `distribution.build` or no legacy endpoint row exists +- **THEN** run generation still succeeds through the platform Docker builder + +#### Scenario: Builder unavailable +- **WHEN** the platform Docker builder is unavailable +- **THEN** the unavailable reason names the platform builder rather than a run endpoint capability + +### Requirement: Generated runs hold no distribution-build authority +The system SHALL continue to deny distribution-build work to component-authenticated generated Runs. This restriction is a security boundary and SHALL NOT be relaxed to unblock building. + +#### Scenario: Generated run claims a build +- **WHEN** a component-authenticated generated Run session claims work advertising `distribution.build` +- **THEN** the platform does not assign distribution build work to that Run session diff --git a/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/specs/redis-runtime-session-registry/spec.md b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/specs/redis-runtime-session-registry/spec.md new file mode 100644 index 0000000..0101e6b --- /dev/null +++ b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/specs/redis-runtime-session-registry/spec.md @@ -0,0 +1,72 @@ +## ADDED Requirements + +### Requirement: Redis-backed runtime session registry is required +The system SHALL use Redis as the runtime session registry in development, test, and production environments, and SHALL NOT use an in-memory registry fallback for normal runtime registration, heartbeat, token validation, or job routing. + +#### Scenario: Platform starts without Redis +- **WHEN** Platform starts or checks health while Redis is unavailable +- **THEN** runtime registry health is reported unavailable and runtime dispatch operations fail safely without assigning jobs to Runs + +#### Scenario: Automated tests exercise Redis registry behavior +- **WHEN** tests cover Run registration, heartbeat expiry, token validation, or job claim routing +- **THEN** those tests use an isolated Redis database or key prefix rather than a memory-only registry + +### Requirement: Run registration creates an ephemeral Redis session +The system SHALL authenticate generated Runs with durable component credentials and SHALL store only a short-lived Redis session lease for the live Run connection. + +#### Scenario: Valid generated Run registers +- **WHEN** a Run submits a hello request with a valid server instance ID, component kind, component key generation, registration proof, version, target, capabilities, and capacity +- **THEN** Platform authenticates the durable component key and writes a Redis session lease scoped to that server/component identity + +#### Scenario: Registration supersedes prior live session +- **WHEN** a second valid Run registers for the same server/component identity +- **THEN** Platform rotates the live Redis session token and the previous session token no longer authorizes heartbeat or job operations + +#### Scenario: Invalid token is rejected +- **WHEN** a Run presents an invalid registration proof or stale component key generation +- **THEN** Platform rejects registration and does not create or renew a Redis session lease + +### Requirement: Heartbeat leases expire without startup cleanup +The system SHALL represent runtime online state through Redis TTL leases that are renewed by heartbeats and naturally expire without Platform startup cleanup. + +#### Scenario: Heartbeat renews lease +- **WHEN** a registered Run heartbeats with the current session token before the Redis TTL expires +- **THEN** Platform renews the Redis lease and updates the safe capability/capacity snapshot + +#### Scenario: Run stops heartbeating +- **WHEN** a Run stops heartbeating beyond the configured expiry window +- **THEN** Redis expires the session keys and Platform reports that runtime connection as offline + +#### Scenario: Platform restarts +- **WHEN** Platform restarts while Redis still contains live session keys +- **THEN** Platform resumes token validation and routing from Redis without scanning or cleaning stale keys at startup + +#### Scenario: Redis loses session data +- **WHEN** Redis restarts or evicts runtime session keys +- **THEN** Platform treats affected Runs as offline until they register again and does not mutate durable server or job records solely because the Redis lease disappeared + +### Requirement: Runtime jobs target server components instead of run endpoints +The system SHALL route durable runtime jobs by logical server/component target and SHALL NOT require a durable run endpoint row to create, validate, claim, or complete machine-side runtime work. + +#### Scenario: Job is queued for a server Run +- **WHEN** Platform queues lifecycle, config, file, log, or protected-request work for a server Run +- **THEN** the durable job target identifies the server instance and `run` component rather than a run endpoint ID + +#### Scenario: Run claims work +- **WHEN** a registered Run claims work with its current session token +- **THEN** Platform derives the server/component target from Redis and assigns only eligible jobs for that target + +#### Scenario: Run attempts cross-server claim +- **WHEN** a Run session for one server attempts to claim, acknowledge, report progress, or complete a job targeting another server/component +- **THEN** Platform rejects the operation and preserves the durable job state + +### Requirement: Runtime connection projections are safe +The system SHALL expose runtime connection health as a safe projection derived from Redis leases and durable server/component metadata, without exposing session tokens, Redis keys, raw credentials, host paths, or direct sockets. + +#### Scenario: Owner views server runtime health +- **WHEN** an authorized owner views a server's runtime connection state +- **THEN** Platform returns safe status, last heartbeat time, version, target OS/architecture, capabilities, capacity, and unavailable reason + +#### Scenario: Runtime session secrets remain hidden +- **WHEN** Platform Web, plugin pages, or bridge actions request runtime status +- **THEN** responses exclude session tokens, token hashes, Redis key names, raw component credentials, host paths, and sockets diff --git a/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/tasks.md b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/tasks.md new file mode 100644 index 0000000..a1836fd --- /dev/null +++ b/openspec/changes/replace-run-endpoints-with-redis-runtime-registry/tasks.md @@ -0,0 +1,40 @@ +## 1. Prompt Boundaries + +- [ ] 1.1 Positive prompt: replace durable run endpoint routing with a Redis-backed runtime registry so the first-party server management area can route work to authenticated Run sessions without requiring endpoint selection or server-to-endpoint database binding. +- [ ] 1.2 Directional prompt: work inside `platform/`, `platform_web/`, `plugins/`, OpenSpec contracts, and local dev/test scripts; preserve platform-side distribution builds, component-token authentication, channel isolation, and existing magical-girl console styling. +- [ ] 1.3 Boundary prompt: do not add cloud host sales, billing, SaaS marketplace features, a `run/` source tree, host-path exposure, raw credentials, direct sockets, or any fallback that dispatches runtime work without a Redis-backed session lease. + +## 2. Redis Registry Foundation + +- [ ] 2.1 Add Redis configuration and health reporting for development, test, and production runtime registry use. +- [ ] 2.2 Add local/test Redis setup so automated tests use isolated Redis keys or databases rather than memory-only runtime registry behavior. +- [ ] 2.3 Define `RuntimeSessionRegistry` with Redis-backed register, heartbeat, lookup-by-token, lookup-by-server-component, revoke, and projection methods. +- [ ] 2.4 Implement Redis key namespaces, TTL renewal, token hashing, capability/capacity snapshots, and no-startup-cleanup semantics. + +## 3. Run Registration And Session Auth + +- [ ] 3.1 Change Run hello to authenticate server/component identity and write a Redis session lease instead of creating or updating a durable `RunEndpoint`. +- [ ] 3.2 Change heartbeat to renew Redis leases and return unknown-session responses that cause Run to re-register. +- [ ] 3.3 Move run request signature and session-token validation to registry-derived sessions while preserving nonce and clock-skew protection. +- [ ] 3.4 Update revocation and component key reset to revoke Redis sessions for the affected server/component without relying on endpoint rows. + +## 4. Job Targeting And Dispatch + +- [ ] 4.1 Add durable job target fields for `serverInstanceId`, `targetComponentKind`, and `targetComponentKey`, with compatibility for existing `runEndpointId` data during migration. +- [ ] 4.2 Update job creation and idempotency to target logical components rather than run endpoints, keeping platform builder jobs as a platform-owned target. +- [ ] 4.3 Update claim, ack, progress, result, cancel, reconcile, log ingest, artifacts, config writes, file operations, protected requests, and lifecycle dispatch to authorize through Redis session targets. +- [ ] 4.4 Remove validation that requires server jobs to match `ServerInstance.RunEndpointID`, replacing it with server/component target validation and Redis session presence where dispatch requires a live Run. + +## 5. API, UI, And Compatibility Cleanup + +- [ ] 5.1 Remove first-party creation/edit/deploy flows that ask for run endpoint selection or persist server-to-run-endpoint bindings. +- [ ] 5.2 Replace run endpoint list/status UI with runtime connection health projections derived from Redis sessions and durable server/component metadata. +- [ ] 5.3 Update API DTOs, docs, plugin bridge contracts, and tests to mark `runEndpointId` as legacy compatibility where still accepted. +- [ ] 5.4 Remove durable `RunEndpoint` repository usage for generated Runs after compatibility tests cover legacy records. + +## 6. Verification + +- [ ] 6.1 Add backend tests for Redis registration, TTL expiry, token rotation, Redis restart/loss, cross-server claim rejection, and job lease behavior. +- [ ] 6.2 Add frontend tests for runtime connection health and absence of run endpoint selectors in first-party server workflows. +- [ ] 6.3 Run `go test ./...`, `npm --prefix platform_web test -- --run`, `scripts/check-structure.sh`, and targeted Redis integration tests. +- [ ] 6.4 Run `openspec validate replace-run-endpoints-with-redis-runtime-registry --strict` before marking implementation tasks complete.