first commit

2026-07-11 14:56:10 +08:00
commit 7e05d0a4e7
660 changed files with 78119 additions and 0 deletions
@@ -0,0 +1,62 @@
+## Overview
+
+The platform needs two storage shapes:
+
+- **Metadata store** for users, AI providers, plugins, server instances, run endpoints, jobs, artifacts, log stream metadata, and audit events.
+- **Log body store** for high-volume append/query log entries.
+
+The first can be MySQL or another transactional database. The second should not be a row-per-line table for large installations. This change implements a local durable metadata store and a segmented log body store using only the Go standard library, with interfaces that can later gain MySQL/PostgreSQL/ClickHouse/Loki adapters.
+
+## Metadata Storage
+
+Current repository interfaces stay unchanged. A new file-backed store wraps the existing in-memory store and persists a snapshot after successful create/update operations. It is suitable for local and single-node deployments, tests, and development environments where no external database is available.
+
+Configuration:
+
+- `PLATFORM_STORAGE_BACKEND=memory|file`
+- `PLATFORM_DATA_DIR=<path>`
+- `PLATFORM_METADATA_PATH=<path>`
+
+Default startup uses file-backed storage under `.platform-data/metadata.json`, so data survives restarts. Tests can continue using `repo.NewMemoryStore()`.
+
+The file store is not positioned as a multi-writer clustered database. A future MySQL adapter should implement the same `repo.Store` interfaces and keep database table models in `platform/model`.
+
+## Log Body Storage
+
+`CoreService` currently stores log bodies in memory maps. This change introduces a `LogBodyStore` service boundary:
+
+- `AppendBatch(streamID, batch)` for validated, contiguous batch appends.
+- `GetBatch(streamID, firstSeq)` for duplicate/conflict detection.
+- `Query(streamID, afterSeq, limit)` for bounded cursor reads.
+
+The local durable implementation writes JSONL segment files:
+
+- Directory: `<PLATFORM_LOG_DIR>/<safe stream id>/`
+- Segment naming: `segment-00000000000000000001.jsonl` using the first sequence in that segment.
+- Each line is one `LogEntry`, keeping append and recovery simple.
+- A small in-memory index is rebuilt on startup from segment files.
+
+This keeps control, jobs, and log upload channel semantics unchanged. It also avoids turning MySQL into a log body sink. For large production deployments, the same boundary can route log bodies to ClickHouse/Loki/OpenSearch/object storage and keep MySQL for metadata, stream state, retention policy, and query indexes.
+
+## Security And Boundaries
+
+- Storage paths are platform-owned configuration; they are never returned to plugins or frontend responses.
+- Log query APIs still return bounded entries only.
+- Run session tokens and bearer sessions remain service-side only.
+- No raw database credentials are exposed through DTOs.
+
+## Failure Modes
+
+- File store creation fails fast on invalid or unwritable paths.
+- Snapshot writes use temp-file then rename to avoid partial metadata files.
+- Log segment writes return service errors instead of acknowledging batches that were not durably written.
+- Existing in-memory tests remain valid; new tests cover restart/reload behavior for file metadata and log body stores.
+
+## Validation
+
+- Unit tests for file-backed metadata persistence across store reloads.
+- Unit tests for segmented log store append, duplicate lookup, cursor query, and reload.
+- API/service tests for default router admin persistence and log ingest behavior.
+- `go test ./...` in `platform`.
+- `scripts/check-structure.sh`.
+- `openspec validate implement-durable-platform-storage --strict`.