first commit

openspec/changes/implement-log-ingest-pipeline/design.md

@@ -0,0 +1,81 @@
+## Context
+
+Run control and job lifecycle routes are implemented, but logs still exist only as metadata records. The architecture requires logs to be treated as durable historical data: run writes batches to a local spool before upload, platform acknowledges accepted sequence ranges, and artifacts must not block control, job, or log traffic.
+
+This change implements the first HTTP JSON log ingest path and an in-repository run spool abstraction. It keeps platform storage in memory and updates existing `LogStream` metadata because durable database/log backend selection is a later architecture decision.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Define typed log ingest protocol payloads in `run/protocol` and matching platform DTO/domain contracts.
+- Add platform log ingest API routes for batch upload and bounded stream cursor query.
+- Validate run session continuity, stream identity, sequence ranges, checksums, and batch size.
+- Track accepted log entries and latest acknowledged sequence in platform service state and existing `LogStream.LatestSeq`.
+- Add a run-side local spool abstraction that persists unacknowledged batches to disk and removes acknowledged ranges only after platform ack.
+- Extend `run/api.PlatformClient` with a typed log batch ingest method.
+- Add tests for platform ack/query behavior, duplicate/out-of-order rejection, run spool retry retention, and client request/response behavior.
+
+**Non-Goals:**
+
+- No external log storage backends such as Loki, ClickHouse, OpenSearch, or Elasticsearch.
+- No browser live tail, log websocket, AI log analysis windows, or frontend behavior.
+- No compression codec implementation beyond typed metadata and checksum validation for the JSON payload.
+- No artifact transfer, game client bridge, billing, cloud host sales, or direct plugin-to-run access.
+- No raw host paths, raw credentials, direct sockets, or artifact chunks inside log payloads.
+
+## Decisions
+
+### Decision 1: HTTP JSON batch ingest first
+
+The initial ingest route uses `POST /api/v1/run/logs/batches` with typed JSON batches. This keeps the path testable, bounded, and independent from control, jobs, and artifacts.
+
+Alternative considered: streaming logs over the control or job channel. Rejected because logs are high-volume historical data and must not block heartbeat, claim/ack/result, or artifact traffic.
+
+### Decision 2: Platform validates contiguous sequence ranges
+
+Each batch carries `streamKey`, `firstSeq`, `lastSeq`, checksum, and entries. The platform accepts the next contiguous range, treats already-acknowledged duplicate batches as idempotent acks, and rejects sequence gaps or conflicting duplicates.
+
+Alternative considered: accepting any sequence order and sorting later. Rejected because retry/ack semantics need deterministic spool cleanup and missing ranges must be visible immediately.
+
+### Decision 3: Log body storage is in-memory for now
+
+The service stores accepted log entries in memory keyed by stream ID and updates existing `LogStream.LatestSeq`. This matches the current repository scope and lets later storage adapters replace the implementation behind service methods.
+
+Alternative considered: adding a local compressed segment storage backend now. Rejected because this change needs API semantics and run spool behavior first; backend choice remains open.
+
+### Decision 4: Run spool stores batches as JSON segment files
+
+The run-side spool writes one JSON file per unacknowledged batch in a caller-provided directory. Tests can inspect retry behavior without a daemon loop, and future uploaders can reuse the same abstraction.
+
+Alternative considered: purely in-memory spool. Rejected because the architecture requires local retention across temporary platform unavailability and restart.
+
+### Decision 5: Client stays transport-only
+
+`run/api.PlatformClient` will encode and decode log ingest requests and responses. Collector loops, file tailing, backpressure scheduling, and artifact priority throttling remain future runtime work.
+
+Alternative considered: implementing a background log uploader now. Rejected because that would expand scope beyond protocol, spool, and ack semantics.
+
+## Risks / Trade-offs
+
+- [Risk] In-memory platform log storage disappears on restart. Mitigation: keep storage behind `service.Core` and document this as early development behavior.
+- [Risk] JSON spool files are not optimized for very large log volumes. Mitigation: enforce bounded batch sizes now; later changes can swap segment encoding without changing ack semantics.
+- [Risk] Checksums only cover entries in this change. Mitigation: keep checksum metadata explicit and add compressed segment checksums when compression/chunking is implemented.
+- [Risk] No background uploader means no automatic retry loop. Mitigation: tests cover retained batches and client upload behavior; scheduling remains a later runtime concern.
+
+## Migration Plan
+
+1. Add log protocol, DTO, domain, validation, and service contracts.
+2. Add platform API handlers and tests for ingest and query.
+3. Add run local spool implementation and tests.
+4. Add run client method and tests.
+5. Update protocol/route docs.
+6. Verify with platform tests, run tests, structure check, and strict OpenSpec validation.
+
+Rollback before dependent changes is removal of the log ingest route/client/spool additions and this OpenSpec change. After artifact/server workflow changes depend on logs, rollback must use a new OpenSpec change.
+
+## Open Questions
+
+- Which production log body backend should be implemented first: local compressed segments, Loki, ClickHouse, OpenSearch, or Elasticsearch?
+- What maximum batch size and compression settings should production use?
+- How should browser live tail subscribe to stored logs without weakening durable ingest guarantees?