63 lines
3.4 KiB
Markdown
63 lines
3.4 KiB
Markdown
## Overview
|
|
|
|
The platform needs two storage shapes:
|
|
|
|
- **Metadata store** for users, AI providers, plugins, server instances, run endpoints, jobs, artifacts, log stream metadata, and audit events.
|
|
- **Log body store** for high-volume append/query log entries.
|
|
|
|
The first can be MySQL or another transactional database. The second should not be a row-per-line table for large installations. This change implements a local durable metadata store and a segmented log body store using only the Go standard library, with interfaces that can later gain MySQL/PostgreSQL/ClickHouse/Loki adapters.
|
|
|
|
## Metadata Storage
|
|
|
|
Current repository interfaces stay unchanged. A new file-backed store wraps the existing in-memory store and persists a snapshot after successful create/update operations. It is suitable for local and single-node deployments, tests, and development environments where no external database is available.
|
|
|
|
Configuration:
|
|
|
|
- `PLATFORM_STORAGE_BACKEND=memory|file`
|
|
- `PLATFORM_DATA_DIR=<path>`
|
|
- `PLATFORM_METADATA_PATH=<path>`
|
|
|
|
Default startup uses file-backed storage under `.platform-data/metadata.json`, so data survives restarts. Tests can continue using `repo.NewMemoryStore()`.
|
|
|
|
The file store is not positioned as a multi-writer clustered database. A future MySQL adapter should implement the same `repo.Store` interfaces and keep database table models in `platform/model`.
|
|
|
|
## Log Body Storage
|
|
|
|
`CoreService` currently stores log bodies in memory maps. This change introduces a `LogBodyStore` service boundary:
|
|
|
|
- `AppendBatch(streamID, batch)` for validated, contiguous batch appends.
|
|
- `GetBatch(streamID, firstSeq)` for duplicate/conflict detection.
|
|
- `Query(streamID, afterSeq, limit)` for bounded cursor reads.
|
|
|
|
The local durable implementation writes JSONL segment files:
|
|
|
|
- Directory: `<PLATFORM_LOG_DIR>/<safe stream id>/`
|
|
- Segment naming: `segment-00000000000000000001.jsonl` using the first sequence in that segment.
|
|
- Each line is one `LogEntry`, keeping append and recovery simple.
|
|
- A small in-memory index is rebuilt on startup from segment files.
|
|
|
|
This keeps control, jobs, and log upload channel semantics unchanged. It also avoids turning MySQL into a log body sink. For large production deployments, the same boundary can route log bodies to ClickHouse/Loki/OpenSearch/object storage and keep MySQL for metadata, stream state, retention policy, and query indexes.
|
|
|
|
## Security And Boundaries
|
|
|
|
- Storage paths are platform-owned configuration; they are never returned to plugins or frontend responses.
|
|
- Log query APIs still return bounded entries only.
|
|
- Run session tokens and bearer sessions remain service-side only.
|
|
- No raw database credentials are exposed through DTOs.
|
|
|
|
## Failure Modes
|
|
|
|
- File store creation fails fast on invalid or unwritable paths.
|
|
- Snapshot writes use temp-file then rename to avoid partial metadata files.
|
|
- Log segment writes return service errors instead of acknowledging batches that were not durably written.
|
|
- Existing in-memory tests remain valid; new tests cover restart/reload behavior for file metadata and log body stores.
|
|
|
|
## Validation
|
|
|
|
- Unit tests for file-backed metadata persistence across store reloads.
|
|
- Unit tests for segmented log store append, duplicate lookup, cursor query, and reload.
|
|
- API/service tests for default router admin persistence and log ingest behavior.
|
|
- `go test ./...` in `platform`.
|
|
- `scripts/check-structure.sh`.
|
|
- `openspec validate implement-durable-platform-storage --strict`.
|