browser/openspec/changes/implement-durable-platform-storage/design.md

Overview

The platform needs two storage shapes:

• Metadata store for users, AI providers, plugins, server instances, run endpoints, jobs, artifacts, log stream metadata, and audit events.
• Log body store for high-volume append/query log entries.

The first can be MySQL or another transactional database. The second should not be a row-per-line table for large installations. This change implements a local durable metadata store and a segmented log body store using only the Go standard library, with interfaces that can later gain MySQL/PostgreSQL/ClickHouse/Loki adapters.

Metadata Storage

Current repository interfaces stay unchanged. A new file-backed store wraps the existing in-memory store and persists a snapshot after successful create/update operations. It is suitable for local and single-node deployments, tests, and development environments where no external database is available.

Configuration:

• PLATFORM_STORAGE_BACKEND=memory|file
• PLATFORM_DATA_DIR=<path>
• PLATFORM_METADATA_PATH=<path>

Default startup uses file-backed storage under .platform-data/metadata.json, so data survives restarts. Tests can continue using repo.NewMemoryStore().

The file store is not positioned as a multi-writer clustered database. A future MySQL adapter should implement the same repo.Store interfaces and keep database table models in platform/model.

Log Body Storage

CoreService currently stores log bodies in memory maps. This change introduces a LogBodyStore service boundary:

• AppendBatch(streamID, batch) for validated, contiguous batch appends.
• GetBatch(streamID, firstSeq) for duplicate/conflict detection.
• Query(streamID, afterSeq, limit) for bounded cursor reads.

The local durable implementation writes JSONL segment files:

• Directory: <PLATFORM_LOG_DIR>/<safe stream id>/
• Segment naming: segment-00000000000000000001.jsonl using the first sequence in that segment.
• Each line is one LogEntry, keeping append and recovery simple.
• A small in-memory index is rebuilt on startup from segment files.

This keeps control, jobs, and log upload channel semantics unchanged. It also avoids turning MySQL into a log body sink. For large production deployments, the same boundary can route log bodies to ClickHouse/Loki/OpenSearch/object storage and keep MySQL for metadata, stream state, retention policy, and query indexes.

Security And Boundaries

• Storage paths are platform-owned configuration; they are never returned to plugins or frontend responses.
• Log query APIs still return bounded entries only.
• Run session tokens and bearer sessions remain service-side only.
• No raw database credentials are exposed through DTOs.

Failure Modes

• File store creation fails fast on invalid or unwritable paths.
• Snapshot writes use temp-file then rename to avoid partial metadata files.
• Log segment writes return service errors instead of acknowledging batches that were not durably written.
• Existing in-memory tests remain valid; new tests cover restart/reload behavior for file metadata and log body stores.

Validation

• Unit tests for file-backed metadata persistence across store reloads.
• Unit tests for segmented log store append, duplicate lookup, cursor query, and reload.
• API/service tests for default router admin persistence and log ingest behavior.
• go test ./... in platform.
• scripts/check-structure.sh.
• openspec validate implement-durable-platform-storage --strict.