first commit
This commit is contained in:
npc0-hue
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-03
|
||||
@@ -0,0 +1,81 @@
|
||||
## Context
|
||||
|
||||
Run control and job lifecycle routes are implemented, but logs still exist only as metadata records. The architecture requires logs to be treated as durable historical data: run writes batches to a local spool before upload, platform acknowledges accepted sequence ranges, and artifacts must not block control, job, or log traffic.
|
||||
|
||||
This change implements the first HTTP JSON log ingest path and an in-repository run spool abstraction. It keeps platform storage in memory and updates existing `LogStream` metadata because durable database/log backend selection is a later architecture decision.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- Define typed log ingest protocol payloads in `run/protocol` and matching platform DTO/domain contracts.
|
||||
- Add platform log ingest API routes for batch upload and bounded stream cursor query.
|
||||
- Validate run session continuity, stream identity, sequence ranges, checksums, and batch size.
|
||||
- Track accepted log entries and latest acknowledged sequence in platform service state and existing `LogStream.LatestSeq`.
|
||||
- Add a run-side local spool abstraction that persists unacknowledged batches to disk and removes acknowledged ranges only after platform ack.
|
||||
- Extend `run/api.PlatformClient` with a typed log batch ingest method.
|
||||
- Add tests for platform ack/query behavior, duplicate/out-of-order rejection, run spool retry retention, and client request/response behavior.
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- No external log storage backends such as Loki, ClickHouse, OpenSearch, or Elasticsearch.
|
||||
- No browser live tail, log websocket, AI log analysis windows, or frontend behavior.
|
||||
- No compression codec implementation beyond typed metadata and checksum validation for the JSON payload.
|
||||
- No artifact transfer, game client bridge, billing, cloud host sales, or direct plugin-to-run access.
|
||||
- No raw host paths, raw credentials, direct sockets, or artifact chunks inside log payloads.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Decision 1: HTTP JSON batch ingest first
|
||||
|
||||
The initial ingest route uses `POST /api/v1/run/logs/batches` with typed JSON batches. This keeps the path testable, bounded, and independent from control, jobs, and artifacts.
|
||||
|
||||
Alternative considered: streaming logs over the control or job channel. Rejected because logs are high-volume historical data and must not block heartbeat, claim/ack/result, or artifact traffic.
|
||||
|
||||
### Decision 2: Platform validates contiguous sequence ranges
|
||||
|
||||
Each batch carries `streamKey`, `firstSeq`, `lastSeq`, checksum, and entries. The platform accepts the next contiguous range, treats already-acknowledged duplicate batches as idempotent acks, and rejects sequence gaps or conflicting duplicates.
|
||||
|
||||
Alternative considered: accepting any sequence order and sorting later. Rejected because retry/ack semantics need deterministic spool cleanup and missing ranges must be visible immediately.
|
||||
|
||||
### Decision 3: Log body storage is in-memory for now
|
||||
|
||||
The service stores accepted log entries in memory keyed by stream ID and updates existing `LogStream.LatestSeq`. This matches the current repository scope and lets later storage adapters replace the implementation behind service methods.
|
||||
|
||||
Alternative considered: adding a local compressed segment storage backend now. Rejected because this change needs API semantics and run spool behavior first; backend choice remains open.
|
||||
|
||||
### Decision 4: Run spool stores batches as JSON segment files
|
||||
|
||||
The run-side spool writes one JSON file per unacknowledged batch in a caller-provided directory. Tests can inspect retry behavior without a daemon loop, and future uploaders can reuse the same abstraction.
|
||||
|
||||
Alternative considered: purely in-memory spool. Rejected because the architecture requires local retention across temporary platform unavailability and restart.
|
||||
|
||||
### Decision 5: Client stays transport-only
|
||||
|
||||
`run/api.PlatformClient` will encode and decode log ingest requests and responses. Collector loops, file tailing, backpressure scheduling, and artifact priority throttling remain future runtime work.
|
||||
|
||||
Alternative considered: implementing a background log uploader now. Rejected because that would expand scope beyond protocol, spool, and ack semantics.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Risk] In-memory platform log storage disappears on restart. Mitigation: keep storage behind `service.Core` and document this as early development behavior.
|
||||
- [Risk] JSON spool files are not optimized for very large log volumes. Mitigation: enforce bounded batch sizes now; later changes can swap segment encoding without changing ack semantics.
|
||||
- [Risk] Checksums only cover entries in this change. Mitigation: keep checksum metadata explicit and add compressed segment checksums when compression/chunking is implemented.
|
||||
- [Risk] No background uploader means no automatic retry loop. Mitigation: tests cover retained batches and client upload behavior; scheduling remains a later runtime concern.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add log protocol, DTO, domain, validation, and service contracts.
|
||||
2. Add platform API handlers and tests for ingest and query.
|
||||
3. Add run local spool implementation and tests.
|
||||
4. Add run client method and tests.
|
||||
5. Update protocol/route docs.
|
||||
6. Verify with platform tests, run tests, structure check, and strict OpenSpec validation.
|
||||
|
||||
Rollback before dependent changes is removal of the log ingest route/client/spool additions and this OpenSpec change. After artifact/server workflow changes depend on logs, rollback must use a new OpenSpec change.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Which production log body backend should be implemented first: local compressed segments, Loki, ClickHouse, OpenSearch, or Elasticsearch?
|
||||
- What maximum batch size and compression settings should production use?
|
||||
- How should browser live tail subscribe to stored logs without weakening durable ingest guarantees?
|
||||
@@ -0,0 +1,29 @@
|
||||
## Why
|
||||
|
||||
The run job channel can now execute lifecycle work, but server and process logs still have no durable ingest path. This change adds the first log pipeline so run can spool logs locally, upload bounded batches, and receive sequence acknowledgements without mixing log traffic into control, job, or artifact channels.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add typed run log ingest protocol payloads for log entries, batch ingest requests, batch acknowledgements, and stream cursors.
|
||||
- Add platform API routes that accept durable log batches, validate stream identity and sequence continuity, acknowledge accepted ranges, and expose bounded query by stream cursor.
|
||||
- Extend platform service behavior to append log batches to existing log stream metadata, track latest acknowledged sequence, and reject duplicate or out-of-order batches.
|
||||
- Add a run-side local spool/WAL abstraction that stores unacknowledged batches and removes only acknowledged sequence ranges.
|
||||
- Extend the run-side platform client with typed log batch ingest calls.
|
||||
- Add platform service/API tests and run spool/client tests covering retry, acknowledgement, duplicate/out-of-order rejection, and query behavior.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `log-ingest-pipeline`: Durable run-to-platform log batch ingest, acknowledgement, local spool retention, and stream cursor query workflow.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- None.
|
||||
|
||||
## Impact
|
||||
|
||||
- Affects `platform/` and `run/` only.
|
||||
- Adds Go protocol/DTO/domain/service/API/spool code and tests for log ingest.
|
||||
- Updates run/platform protocol and route documentation.
|
||||
- Does not implement artifact transfer, browser live tail, external log backends, AI log analysis, frontend pages, billing, cloud host sales, or direct plugin/run access.
|
||||
@@ -0,0 +1,74 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Log ingest payloads are typed and bounded
|
||||
The system SHALL define typed log ingest payloads for log entries, batch ingest requests, batch acknowledgements, and stream cursor queries without carrying artifact chunks, host paths, raw credentials, direct sockets, or unbounded inline data.
|
||||
|
||||
#### Scenario: Log payloads are used
|
||||
- **WHEN** run or platform code sends log ingest data
|
||||
- **THEN** it MUST use named protocol/DTO types from dedicated protocol or DTO packages
|
||||
|
||||
#### Scenario: Log batch stays bounded
|
||||
- **WHEN** run uploads a log batch
|
||||
- **THEN** the request MUST include run endpoint ID, session token, stream identity, sequence range, checksum, compression metadata, and bounded entries only
|
||||
|
||||
### Requirement: Platform accepts durable log batches
|
||||
The platform SHALL expose a log batch ingest endpoint that validates run session continuity, stream metadata, checksum, and sequence continuity before acknowledging accepted ranges.
|
||||
|
||||
#### Scenario: Contiguous batch succeeds
|
||||
- **WHEN** run uploads a valid batch whose first sequence follows the platform's latest acknowledged sequence for that stream
|
||||
- **THEN** platform MUST store the entries, update latest acknowledged sequence, and return an accepted acknowledgement for the range
|
||||
|
||||
#### Scenario: Duplicate acknowledged batch is retried
|
||||
- **WHEN** run uploads a batch whose range is already fully acknowledged and checksum matches the stored range
|
||||
- **THEN** platform MUST return an accepted idempotent acknowledgement without duplicating entries
|
||||
|
||||
#### Scenario: Out-of-order batch is submitted
|
||||
- **WHEN** run uploads a batch with a sequence gap or conflicting duplicate data
|
||||
- **THEN** platform MUST return a JSON validation error and MUST NOT advance the acknowledged sequence
|
||||
|
||||
### Requirement: Platform exposes bounded log stream query
|
||||
The platform SHALL expose a bounded log query endpoint that returns entries for one stream after a cursor sequence and includes the next cursor.
|
||||
|
||||
#### Scenario: Query returns entries after cursor
|
||||
- **WHEN** a caller queries a stream after an acknowledged sequence
|
||||
- **THEN** platform MUST return ordered entries after that cursor up to the requested limit and include the next cursor
|
||||
|
||||
#### Scenario: Query target is missing
|
||||
- **WHEN** a caller queries a missing stream
|
||||
- **THEN** platform MUST return a JSON not found error
|
||||
|
||||
### Requirement: Run spool retains unacknowledged batches
|
||||
The run-side log spool SHALL persist unacknowledged batches locally and remove them only after platform acknowledgement covers their sequence range.
|
||||
|
||||
#### Scenario: Platform upload fails
|
||||
- **WHEN** a batch remains unacknowledged after an upload failure
|
||||
- **THEN** the spool MUST retain the batch for retry
|
||||
|
||||
#### Scenario: Platform acknowledges batch
|
||||
- **WHEN** platform returns an acknowledgement covering a batch range
|
||||
- **THEN** the spool MUST mark that range acknowledged and remove the batch from pending retry listing
|
||||
|
||||
### Requirement: Run client uploads log batches
|
||||
The run-side platform client SHALL provide a typed log batch ingest method that calls the platform log endpoint and decodes typed acknowledgement responses.
|
||||
|
||||
#### Scenario: Run uploads log batch through client
|
||||
- **WHEN** run code calls the log ingest client method
|
||||
- **THEN** the client MUST send a JSON `POST` to `/api/v1/run/logs/batches` and decode the acknowledgement response
|
||||
|
||||
#### Scenario: Platform rejects log batch
|
||||
- **WHEN** the platform log ingest endpoint returns a non-success status
|
||||
- **THEN** the run client MUST return an error and MUST NOT treat the batch as acknowledged
|
||||
|
||||
### Requirement: Log ingest is documented separately from other channels
|
||||
The run/platform route and protocol documentation SHALL identify implemented log ingest routes and explicitly keep control, job, artifact, and game client bridge transport separate.
|
||||
|
||||
#### Scenario: Contributor inspects log docs
|
||||
- **WHEN** a contributor opens run or platform protocol docs
|
||||
- **THEN** the docs MUST show log batch ingest and query as implemented while artifact and game client channels remain separate
|
||||
|
||||
### Requirement: Log ingest pipeline is verified
|
||||
The change SHALL include platform service/API tests, run spool tests, run client tests, and retry/ack/query coverage.
|
||||
|
||||
#### Scenario: Verification commands run
|
||||
- **WHEN** the change is complete
|
||||
- **THEN** `go test ./...` from `platform/`, `go test ./...` from `run/`, `scripts/check-structure.sh`, and `openspec validate implement-log-ingest-pipeline --strict` MUST pass
|
||||
@@ -0,0 +1,35 @@
|
||||
## 1. Log Contracts
|
||||
|
||||
- [x] 1.1 Add typed run log ingest protocol payloads in `run/protocol` for entries, batch ingest, acknowledgements, and stream cursors.
|
||||
- [x] 1.2 Add matching platform DTO/domain contracts and conversion helpers for log batch ingest and query.
|
||||
- [x] 1.3 Add validation rules for bounded log batches, stream identity, sequence ranges, checksum, and query limits.
|
||||
|
||||
## 2. Platform Log Ingest
|
||||
|
||||
- [x] 2.1 Extend platform service behavior to ingest contiguous batches, acknowledge duplicates, reject out-of-order/conflicting batches, update `LogStream.LatestSeq`, and query entries after a cursor.
|
||||
- [x] 2.2 Implement platform log ingest/query HTTP routes using named DTOs and service methods.
|
||||
- [x] 2.3 Add platform service/API tests for accepted batches, duplicate ack, out-of-order rejection, missing stream, and cursor query.
|
||||
|
||||
## 3. Run Log Spool And Client
|
||||
|
||||
- [x] 3.1 Implement a run-side local spool abstraction that writes pending batches to disk, lists them for retry, and removes acknowledged ranges.
|
||||
- [x] 3.2 Extend `run/api.PlatformClient` with a typed log batch ingest method.
|
||||
- [x] 3.3 Add run spool/client tests for retry retention, acknowledgement cleanup, request path, JSON payload, response decoding, and platform error handling.
|
||||
|
||||
## 4. Documentation
|
||||
|
||||
- [x] 4.1 Update run and platform protocol/route documentation to mark log batch ingest/query implemented and keep control/job/artifact/game-client channels separate.
|
||||
|
||||
## 5. Verification
|
||||
|
||||
- [x] 5.1 Run `go test ./...` from `platform/` and record evidence.
|
||||
- [x] 5.2 Run `go test ./...` from `run/` and record evidence.
|
||||
- [x] 5.3 Run `scripts/check-structure.sh` and record evidence.
|
||||
- [x] 5.4 Run `openspec validate implement-log-ingest-pipeline --strict` and record evidence.
|
||||
|
||||
## Evidence
|
||||
|
||||
- 2026-07-03: `go test ./...` from `platform/` passed.
|
||||
- 2026-07-03: `go test ./...` from `run/` passed.
|
||||
- 2026-07-03: `scripts/check-structure.sh` passed with `structure check passed`.
|
||||
- 2026-07-03: `openspec validate implement-log-ingest-pipeline --strict` passed with `Change 'implement-log-ingest-pipeline' is valid`.
|
||||
Reference in New Issue
Block a user