first commit
This commit is contained in:
npc0-hue
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-07
|
||||
@@ -0,0 +1,62 @@
|
||||
## Overview
|
||||
|
||||
The platform needs two storage shapes:
|
||||
|
||||
- **Metadata store** for users, AI providers, plugins, server instances, run endpoints, jobs, artifacts, log stream metadata, and audit events.
|
||||
- **Log body store** for high-volume append/query log entries.
|
||||
|
||||
The first can be MySQL or another transactional database. The second should not be a row-per-line table for large installations. This change implements a local durable metadata store and a segmented log body store using only the Go standard library, with interfaces that can later gain MySQL/PostgreSQL/ClickHouse/Loki adapters.
|
||||
|
||||
## Metadata Storage
|
||||
|
||||
Current repository interfaces stay unchanged. A new file-backed store wraps the existing in-memory store and persists a snapshot after successful create/update operations. It is suitable for local and single-node deployments, tests, and development environments where no external database is available.
|
||||
|
||||
Configuration:
|
||||
|
||||
- `PLATFORM_STORAGE_BACKEND=memory|file`
|
||||
- `PLATFORM_DATA_DIR=<path>`
|
||||
- `PLATFORM_METADATA_PATH=<path>`
|
||||
|
||||
Default startup uses file-backed storage under `.platform-data/metadata.json`, so data survives restarts. Tests can continue using `repo.NewMemoryStore()`.
|
||||
|
||||
The file store is not positioned as a multi-writer clustered database. A future MySQL adapter should implement the same `repo.Store` interfaces and keep database table models in `platform/model`.
|
||||
|
||||
## Log Body Storage
|
||||
|
||||
`CoreService` currently stores log bodies in memory maps. This change introduces a `LogBodyStore` service boundary:
|
||||
|
||||
- `AppendBatch(streamID, batch)` for validated, contiguous batch appends.
|
||||
- `GetBatch(streamID, firstSeq)` for duplicate/conflict detection.
|
||||
- `Query(streamID, afterSeq, limit)` for bounded cursor reads.
|
||||
|
||||
The local durable implementation writes JSONL segment files:
|
||||
|
||||
- Directory: `<PLATFORM_LOG_DIR>/<safe stream id>/`
|
||||
- Segment naming: `segment-00000000000000000001.jsonl` using the first sequence in that segment.
|
||||
- Each line is one `LogEntry`, keeping append and recovery simple.
|
||||
- A small in-memory index is rebuilt on startup from segment files.
|
||||
|
||||
This keeps control, jobs, and log upload channel semantics unchanged. It also avoids turning MySQL into a log body sink. For large production deployments, the same boundary can route log bodies to ClickHouse/Loki/OpenSearch/object storage and keep MySQL for metadata, stream state, retention policy, and query indexes.
|
||||
|
||||
## Security And Boundaries
|
||||
|
||||
- Storage paths are platform-owned configuration; they are never returned to plugins or frontend responses.
|
||||
- Log query APIs still return bounded entries only.
|
||||
- Run session tokens and bearer sessions remain service-side only.
|
||||
- No raw database credentials are exposed through DTOs.
|
||||
|
||||
## Failure Modes
|
||||
|
||||
- File store creation fails fast on invalid or unwritable paths.
|
||||
- Snapshot writes use temp-file then rename to avoid partial metadata files.
|
||||
- Log segment writes return service errors instead of acknowledging batches that were not durably written.
|
||||
- Existing in-memory tests remain valid; new tests cover restart/reload behavior for file metadata and log body stores.
|
||||
|
||||
## Validation
|
||||
|
||||
- Unit tests for file-backed metadata persistence across store reloads.
|
||||
- Unit tests for segmented log store append, duplicate lookup, cursor query, and reload.
|
||||
- API/service tests for default router admin persistence and log ingest behavior.
|
||||
- `go test ./...` in `platform`.
|
||||
- `scripts/check-structure.sh`.
|
||||
- `openspec validate implement-durable-platform-storage --strict`.
|
||||
@@ -0,0 +1,26 @@
|
||||
## Why
|
||||
|
||||
The platform currently uses in-memory repositories and in-memory log buffers, so users, sessions, jobs, server state, artifacts, and logs disappear when the platform process restarts. Hundreds or thousands of game servers also make row-per-log-line relational storage a poor default: platform metadata needs a durable database, while log bodies need append-friendly segmented storage with bounded cursor reads.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add configurable durable platform storage so the default local platform process no longer depends on volatile in-memory repositories.
|
||||
- Add a local file-backed metadata store for immediate durable operation without external database credentials.
|
||||
- Add a segmented log body store that persists log entries by stream and segment on disk, keeping log query semantics and duplicate detection intact.
|
||||
- Update platform startup configuration to choose memory or durable local storage with explicit data/log paths.
|
||||
- Document that MySQL/PostgreSQL-style databases are appropriate for platform metadata and log indexes, while high-volume log bodies should use segmented object/file storage or a purpose-built log backend such as ClickHouse/Loki/OpenSearch in later changes.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `durable-platform-storage`: Platform metadata and log bodies survive process restarts through configurable durable storage backends.
|
||||
|
||||
### Modified Capabilities
|
||||
- None.
|
||||
|
||||
## Impact
|
||||
|
||||
- Affected backend areas: `platform/config`, `platform/repo`, `platform/service`, `platform/api`, `platform/cmd/platform`, `platform/protocol`.
|
||||
- No frontend page changes.
|
||||
- No external paid services, real database credentials, or network downloads are required for this implementation.
|
||||
- Future MySQL or analytics log backends can be added behind the new storage boundaries without exposing raw host paths, credentials, or direct sockets to plugins or the frontend.
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Durable Metadata Store
|
||||
|
||||
The platform SHALL support a configurable durable metadata store so platform resources survive process restarts without relying on in-memory maps.
|
||||
|
||||
#### Scenario: File-backed platform metadata survives restart
|
||||
|
||||
- **GIVEN** the platform is configured with the file storage backend and a metadata file path
|
||||
- **WHEN** users, server instances, jobs, artifacts, log streams, or other platform resources are created or updated
|
||||
- **THEN** the metadata SHALL be persisted to disk
|
||||
- **AND** recreating the store from the same file SHALL restore those resources.
|
||||
|
||||
#### Scenario: Memory backend remains available for tests
|
||||
|
||||
- **GIVEN** tests or development code explicitly request the memory backend
|
||||
- **WHEN** the platform creates a store
|
||||
- **THEN** it SHALL use the existing in-memory repository behavior without filesystem persistence.
|
||||
|
||||
#### Scenario: Startup seeds local admin once
|
||||
|
||||
- **GIVEN** durable metadata already contains the local platform administrator
|
||||
- **WHEN** the platform starts again
|
||||
- **THEN** startup seeding SHALL be idempotent and SHALL NOT overwrite the existing user.
|
||||
|
||||
### Requirement: Segmented Log Body Storage
|
||||
|
||||
The platform SHALL store high-volume log bodies in an append-friendly segmented log store instead of relying on in-memory maps or a row-per-log-line metadata database.
|
||||
|
||||
#### Scenario: Log ingest persists entries durably
|
||||
|
||||
- **GIVEN** a valid contiguous log batch for an existing stream
|
||||
- **WHEN** the platform accepts the batch
|
||||
- **THEN** the log body entries SHALL be written to the configured log body store before the stream latest sequence is advanced.
|
||||
|
||||
#### Scenario: Duplicate batch detection survives reload
|
||||
|
||||
- **GIVEN** a log batch has already been accepted and the platform restarts
|
||||
- **WHEN** the same batch is submitted again
|
||||
- **THEN** the platform SHALL return a duplicate acknowledgement when the first sequence, last sequence, and checksum match.
|
||||
|
||||
#### Scenario: Cursor query stays bounded
|
||||
|
||||
- **GIVEN** a log stream has many persisted entries
|
||||
- **WHEN** a client queries after a sequence with a limit
|
||||
- **THEN** the platform SHALL return at most the requested bounded number of entries ordered by sequence
|
||||
- **AND** SHALL include next and latest sequence metadata.
|
||||
|
||||
### Requirement: Storage Backend Boundaries
|
||||
|
||||
The platform SHALL separate metadata storage from log body storage and SHALL NOT expose storage paths, database credentials, run sockets, or raw credentials through API responses.
|
||||
|
||||
#### Scenario: Storage details remain platform-owned
|
||||
|
||||
- **GIVEN** plugins or frontend clients request platform resources, logs, artifacts, or bridge actions
|
||||
- **WHEN** responses are generated
|
||||
- **THEN** they SHALL include only bounded DTO data and SHALL NOT include filesystem paths, database DSNs, direct storage backend URLs, run session tokens, or bearer tokens.
|
||||
@@ -0,0 +1,25 @@
|
||||
## 1. OpenSpec Artifacts
|
||||
|
||||
- [x] 1.1 Create proposal, design, spec, and tasks for durable metadata and log storage.
|
||||
- [x] 1.2 Validate the change with `openspec validate implement-durable-platform-storage --strict`.
|
||||
|
||||
## 2. Durable Metadata Store
|
||||
|
||||
- [x] 2.1 Add platform storage configuration for backend, metadata path, data directory, and log directory.
|
||||
- [x] 2.2 Implement a file-backed metadata store that persists repository snapshots and reloads them.
|
||||
- [x] 2.3 Wire platform startup to use the configured durable store by default while keeping memory store available.
|
||||
- [x] 2.4 Add tests proving metadata persistence and default admin idempotency across store reloads.
|
||||
|
||||
## 3. Segmented Log Body Store
|
||||
|
||||
- [x] 3.1 Add a `LogBodyStore` service boundary with append, duplicate lookup, and cursor query operations.
|
||||
- [x] 3.2 Implement an in-memory log body store for tests and a file-segment log body store for durable startup.
|
||||
- [x] 3.3 Refactor log ingest/query service code to use `LogBodyStore` and persist before advancing stream metadata.
|
||||
- [x] 3.4 Add tests for duplicate acknowledgement and cursor query after log store reload.
|
||||
|
||||
## 4. Documentation And Verification
|
||||
|
||||
- [x] 4.1 Document storage backend choices and log storage guidance in platform protocol/API docs.
|
||||
- [x] 4.2 Run `cd platform && go test ./...`.
|
||||
- [x] 4.3 Run `scripts/check-structure.sh`.
|
||||
- [x] 4.4 Run `openspec validate implement-durable-platform-storage --strict`.
|
||||
Reference in New Issue
Block a user