82 lines
5.7 KiB
Markdown
82 lines
5.7 KiB
Markdown
## Context
|
|
|
|
Run control and job lifecycle routes are implemented, but logs still exist only as metadata records. The architecture requires logs to be treated as durable historical data: run writes batches to a local spool before upload, platform acknowledges accepted sequence ranges, and artifacts must not block control, job, or log traffic.
|
|
|
|
This change implements the first HTTP JSON log ingest path and an in-repository run spool abstraction. It keeps platform storage in memory and updates existing `LogStream` metadata because durable database/log backend selection is a later architecture decision.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
|
|
- Define typed log ingest protocol payloads in `run/protocol` and matching platform DTO/domain contracts.
|
|
- Add platform log ingest API routes for batch upload and bounded stream cursor query.
|
|
- Validate run session continuity, stream identity, sequence ranges, checksums, and batch size.
|
|
- Track accepted log entries and latest acknowledged sequence in platform service state and existing `LogStream.LatestSeq`.
|
|
- Add a run-side local spool abstraction that persists unacknowledged batches to disk and removes acknowledged ranges only after platform ack.
|
|
- Extend `run/api.PlatformClient` with a typed log batch ingest method.
|
|
- Add tests for platform ack/query behavior, duplicate/out-of-order rejection, run spool retry retention, and client request/response behavior.
|
|
|
|
**Non-Goals:**
|
|
|
|
- No external log storage backends such as Loki, ClickHouse, OpenSearch, or Elasticsearch.
|
|
- No browser live tail, log websocket, AI log analysis windows, or frontend behavior.
|
|
- No compression codec implementation beyond typed metadata and checksum validation for the JSON payload.
|
|
- No artifact transfer, game client bridge, billing, cloud host sales, or direct plugin-to-run access.
|
|
- No raw host paths, raw credentials, direct sockets, or artifact chunks inside log payloads.
|
|
|
|
## Decisions
|
|
|
|
### Decision 1: HTTP JSON batch ingest first
|
|
|
|
The initial ingest route uses `POST /api/v1/run/logs/batches` with typed JSON batches. This keeps the path testable, bounded, and independent from control, jobs, and artifacts.
|
|
|
|
Alternative considered: streaming logs over the control or job channel. Rejected because logs are high-volume historical data and must not block heartbeat, claim/ack/result, or artifact traffic.
|
|
|
|
### Decision 2: Platform validates contiguous sequence ranges
|
|
|
|
Each batch carries `streamKey`, `firstSeq`, `lastSeq`, checksum, and entries. The platform accepts the next contiguous range, treats already-acknowledged duplicate batches as idempotent acks, and rejects sequence gaps or conflicting duplicates.
|
|
|
|
Alternative considered: accepting any sequence order and sorting later. Rejected because retry/ack semantics need deterministic spool cleanup and missing ranges must be visible immediately.
|
|
|
|
### Decision 3: Log body storage is in-memory for now
|
|
|
|
The service stores accepted log entries in memory keyed by stream ID and updates existing `LogStream.LatestSeq`. This matches the current repository scope and lets later storage adapters replace the implementation behind service methods.
|
|
|
|
Alternative considered: adding a local compressed segment storage backend now. Rejected because this change needs API semantics and run spool behavior first; backend choice remains open.
|
|
|
|
### Decision 4: Run spool stores batches as JSON segment files
|
|
|
|
The run-side spool writes one JSON file per unacknowledged batch in a caller-provided directory. Tests can inspect retry behavior without a daemon loop, and future uploaders can reuse the same abstraction.
|
|
|
|
Alternative considered: purely in-memory spool. Rejected because the architecture requires local retention across temporary platform unavailability and restart.
|
|
|
|
### Decision 5: Client stays transport-only
|
|
|
|
`run/api.PlatformClient` will encode and decode log ingest requests and responses. Collector loops, file tailing, backpressure scheduling, and artifact priority throttling remain future runtime work.
|
|
|
|
Alternative considered: implementing a background log uploader now. Rejected because that would expand scope beyond protocol, spool, and ack semantics.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- [Risk] In-memory platform log storage disappears on restart. Mitigation: keep storage behind `service.Core` and document this as early development behavior.
|
|
- [Risk] JSON spool files are not optimized for very large log volumes. Mitigation: enforce bounded batch sizes now; later changes can swap segment encoding without changing ack semantics.
|
|
- [Risk] Checksums only cover entries in this change. Mitigation: keep checksum metadata explicit and add compressed segment checksums when compression/chunking is implemented.
|
|
- [Risk] No background uploader means no automatic retry loop. Mitigation: tests cover retained batches and client upload behavior; scheduling remains a later runtime concern.
|
|
|
|
## Migration Plan
|
|
|
|
1. Add log protocol, DTO, domain, validation, and service contracts.
|
|
2. Add platform API handlers and tests for ingest and query.
|
|
3. Add run local spool implementation and tests.
|
|
4. Add run client method and tests.
|
|
5. Update protocol/route docs.
|
|
6. Verify with platform tests, run tests, structure check, and strict OpenSpec validation.
|
|
|
|
Rollback before dependent changes is removal of the log ingest route/client/spool additions and this OpenSpec change. After artifact/server workflow changes depend on logs, rollback must use a new OpenSpec change.
|
|
|
|
## Open Questions
|
|
|
|
- Which production log body backend should be implemented first: local compressed segments, Loki, ClickHouse, OpenSearch, or Elasticsearch?
|
|
- What maximum batch size and compression settings should production use?
|
|
- How should browser live tail subscribe to stored logs without weakening durable ingest guarantees?
|