first commit
This commit is contained in:
npc0-hue
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-03
|
||||
@@ -0,0 +1,86 @@
|
||||
## Context
|
||||
|
||||
Run control, job lifecycle, and durable log ingest are implemented as separate HTTP JSON channels. Artifact metadata exists in the platform, and job results can reference artifacts, but there is no transfer workflow that can move large run-produced files into platform-managed artifact records with resume and checksum semantics.
|
||||
|
||||
This change implements the first run-to-platform artifact upload channel. Platform storage remains in memory and artifact payloads are held only long enough to prove chunk ordering and final checksum behavior. The channel is intentionally separate from control, jobs, logs, and the optional game client bridge so large payloads do not share those routes or DTOs.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
|
||||
- Define typed artifact transfer payloads in `run/protocol` and matching platform DTO/domain contracts.
|
||||
- Add platform artifact transfer routes for open, chunk upload, resume/status, and complete.
|
||||
- Validate active run session, owner relationship, transfer identity, bounded chunk size, chunk checksum, byte ranges, resume state, and final checksum.
|
||||
- Update existing `Artifact` metadata from `uploading` to `available` only after every chunk is present and the final checksum matches.
|
||||
- Add a run-side artifact spool/queue that persists unacknowledged chunk upload requests and deletes them only after platform acknowledgement.
|
||||
- Extend `run/api.PlatformClient` with typed artifact transfer methods.
|
||||
- Add tests for platform service/API transfer behavior, resume, duplicate chunk acknowledgement, checksum errors, completion errors, run spool retention, and client request/response handling.
|
||||
|
||||
**Non-Goals:**
|
||||
|
||||
- No platform-to-run download flow, browser upload/download UI, external object storage backend, presigned URL flow, or streaming transport.
|
||||
- No plugin bridge file APIs, AI artifact inspection, archive extraction, or artifact lifecycle cleanup jobs.
|
||||
- No raw host paths, raw credentials, direct sockets, logs, or job result bodies inside artifact chunk requests.
|
||||
- No billing, cloud host sales, agent-provider/cloud-provider workflows, or direct plugin-to-run access.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Decision 1: HTTP JSON chunk endpoints first
|
||||
|
||||
The initial channel uses separate JSON `POST` endpoints under `/api/v1/run/artifacts/*`: `open`, `chunks`, `status`, and `complete`. Chunk payloads use JSON byte encoding, which Go represents as base64, and validators enforce a bounded maximum chunk size.
|
||||
|
||||
Alternative considered: multipart upload or object-storage signed URLs. Rejected for this change because there is no storage backend yet, and the first requirement is to prove protocol, validation, resume, and checksum semantics in tests.
|
||||
|
||||
### Decision 2: Run uploads only in this change
|
||||
|
||||
The transfer direction is explicit but only `upload` is accepted. Platform-to-run download will need separate authorization, cache, and throttling semantics after upload behavior is stable.
|
||||
|
||||
Alternative considered: implementing upload and download together. Rejected because download would add browser/plugin access questions and storage-adapter behavior that are outside this queue item.
|
||||
|
||||
### Decision 3: Existing Artifact metadata remains the public resource
|
||||
|
||||
Opening a transfer creates or validates the existing `Artifact` metadata record in `uploading` state. Completion updates that same record to `available`; failed checksum or missing chunk errors leave the artifact non-available.
|
||||
|
||||
Alternative considered: adding a separate persisted transfer model now. Rejected because current platform persistence is in-memory and the transfer session can stay behind `service.Core` until a database-backed storage change exists.
|
||||
|
||||
### Decision 4: Chunks are accepted idempotently by checksum
|
||||
|
||||
The platform records received chunk indexes, byte ranges, sizes, checksums, and payload bytes in memory. Re-uploading the same chunk with the same checksum returns a duplicate acknowledgement; re-uploading a different payload for an acknowledged index is rejected.
|
||||
|
||||
Alternative considered: allowing overwrite of existing chunk indexes. Rejected because resumable upload cleanup must be deterministic and conflicting retries should be visible immediately.
|
||||
|
||||
### Decision 5: Owner authorization is platform mediated
|
||||
|
||||
Run uploads are accepted only for job-owned or server-instance-owned artifacts that belong to the requesting run endpoint. Platform/plugin-owned artifact records can still be created through metadata APIs, but this run transfer channel does not let a run endpoint spoof unrelated owners.
|
||||
|
||||
Alternative considered: accepting any artifact owner kind. Rejected because run must not become a direct write path for platform/plugin-owned data without an explicit authorization change.
|
||||
|
||||
### Decision 6: Run spool stores chunk upload requests, not host paths
|
||||
|
||||
The run-side artifact spool writes one JSON file per pending chunk request. It stores the bounded request payload needed for retry and never stores or exposes the local host path that originally produced the bytes.
|
||||
|
||||
Alternative considered: storing file path plus offset for retry. Rejected because run must enforce scoped paths and must not expose raw host paths through platform-facing transfer state.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Risk] In-memory platform chunk storage disappears on restart. Mitigation: keep transfer state behind `service.Core`; storage adapters and durable transfer sessions can replace it later.
|
||||
- [Risk] JSON/base64 chunks are inefficient for large production artifacts. Mitigation: enforce bounded chunks now and leave streaming/object-storage transfer to a later change.
|
||||
- [Risk] No background artifact uploader exists. Mitigation: run client and spool semantics are implemented and tested; scheduling and priority throttling can build on them later.
|
||||
- [Risk] Upload-only support does not cover all artifact use cases. Mitigation: explicitly keep direction in the protocol so a future download change can extend without renaming the channel.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add artifact transfer protocol, DTO, domain, validation, and service contracts.
|
||||
2. Add platform API handlers and tests for open, chunk upload, resume/status, and complete.
|
||||
3. Add run artifact spool implementation and tests.
|
||||
4. Add run client methods and tests.
|
||||
5. Update protocol and route docs.
|
||||
6. Verify with platform tests, run tests, structure check, and strict OpenSpec validation.
|
||||
|
||||
Rollback before dependent changes is removal of the artifact transfer route/client/spool additions and this OpenSpec change. After server workflows depend on artifact transfer, rollback must use a new OpenSpec change.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Which durable artifact storage backend should be implemented first: local segments, filesystem package storage, S3-compatible object storage, or another adapter?
|
||||
- What production chunk size, concurrency limits, and backoff policy should artifact uploaders use?
|
||||
- How should platform-to-run download authorization interact with plugin pages and server management workflows?
|
||||
@@ -0,0 +1,29 @@
|
||||
## Why
|
||||
|
||||
Jobs and logs now have separate run-platform channels, but large files still only exist as artifact metadata or opaque result references. This change adds the first artifact transfer channel so run can upload and resume bounded file chunks with checksum verification without blocking control, job, or log traffic.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add typed run artifact transfer protocol payloads for transfer creation, chunk upload, resume status, and completion acknowledgements.
|
||||
- Add platform API routes that create artifact transfer sessions, accept bounded chunks, validate sequence/order/checksums, report resume state, and complete verified artifacts.
|
||||
- Extend platform service behavior to store chunk state in memory, update existing artifact metadata, and keep artifact transfer traffic separate from control, job, and log workflows.
|
||||
- Add a run-side local artifact transfer queue/spool abstraction that records pending chunk manifests and removes chunks only after platform acknowledgement.
|
||||
- Extend the run-side platform client with typed artifact transfer methods.
|
||||
- Add platform service/API tests and run queue/client tests covering chunk resume, checksum failures, completion validation, retry cleanup, and channel isolation assumptions.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
- `artifact-transfer-channel`: Chunked and resumable run-to-platform artifact transfer, checksum validation, local retry retention, and transfer completion workflow.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- None.
|
||||
|
||||
## Impact
|
||||
|
||||
- Affects `platform/` and `run/` only.
|
||||
- Adds Go protocol/DTO/domain/service/API/queue code and tests for artifact transfer.
|
||||
- Updates run/platform protocol and route documentation.
|
||||
- Does not implement browser upload/download UI, external object storage backends, plugin bridge file access, AI artifact inspection, billing, cloud host sales, or direct plugin/run access.
|
||||
+65
@@ -0,0 +1,65 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Artifact transfer channel is separate from other run channels
|
||||
|
||||
The platform SHALL expose artifact transfer behavior through dedicated run artifact routes and SHALL NOT require control, job, or log routes to carry artifact chunk payloads.
|
||||
|
||||
#### Scenario: Dedicated artifact routes handle chunks
|
||||
- **WHEN** a registered run endpoint uploads an artifact chunk
|
||||
- **THEN** the request is handled by a run artifact transfer route and no control, job, or log route accepts the chunk payload
|
||||
|
||||
### Requirement: Run opens upload transfer sessions
|
||||
|
||||
The platform SHALL allow an active run session to open an upload transfer for a job-owned or server-instance-owned artifact assigned to that run endpoint.
|
||||
|
||||
#### Scenario: Valid upload session opens
|
||||
- **WHEN** a run endpoint opens an upload transfer with a valid session token, artifact metadata, owner, total size, chunk size, checksum, and idempotency key
|
||||
- **THEN** the platform returns an accepted transfer ID, records the artifact in uploading state, and reports no received chunks
|
||||
|
||||
#### Scenario: Invalid owner is rejected
|
||||
- **WHEN** a run endpoint opens an upload transfer for an artifact owner that is not assigned to that run endpoint
|
||||
- **THEN** the platform rejects the request without marking the artifact available
|
||||
|
||||
### Requirement: Platform validates chunk upload integrity
|
||||
|
||||
The platform SHALL validate artifact transfer ID, run session, chunk index, byte range, payload size, and chunk checksum before acknowledging an uploaded chunk.
|
||||
|
||||
#### Scenario: Valid chunk is acknowledged
|
||||
- **WHEN** a run endpoint uploads a chunk whose byte range, payload size, and checksum match the opened transfer
|
||||
- **THEN** the platform records the chunk and returns an acknowledgement with the accepted chunk index and received chunk list
|
||||
|
||||
#### Scenario: Conflicting duplicate chunk is rejected
|
||||
- **WHEN** a run endpoint uploads a chunk index that was already acknowledged with different payload bytes or checksum
|
||||
- **THEN** the platform rejects the request as a validation error
|
||||
|
||||
### Requirement: Artifact transfer resume state is queryable
|
||||
|
||||
The platform SHALL report the current transfer state, received chunk indexes, next missing chunk index, total chunk count, and completion status for an active artifact transfer.
|
||||
|
||||
#### Scenario: Resume status reports missing chunk
|
||||
- **WHEN** a run endpoint queries transfer status after only part of an artifact has uploaded
|
||||
- **THEN** the platform returns the acknowledged chunk indexes and the next missing chunk index
|
||||
|
||||
### Requirement: Artifact completion verifies full checksum
|
||||
|
||||
The platform SHALL mark an artifact available only after all chunks are present and the final artifact checksum matches the opened transfer metadata.
|
||||
|
||||
#### Scenario: Complete verified artifact
|
||||
- **WHEN** every chunk has been uploaded and the run endpoint completes the transfer with the correct final checksum
|
||||
- **THEN** the platform marks the artifact available and returns the updated artifact metadata
|
||||
|
||||
#### Scenario: Missing chunk prevents completion
|
||||
- **WHEN** the run endpoint completes a transfer before every chunk is present
|
||||
- **THEN** the platform rejects completion and leaves the artifact non-available
|
||||
|
||||
### Requirement: Run retains unacknowledged artifact chunks
|
||||
|
||||
The run executor SHALL persist pending artifact chunk upload requests locally and SHALL remove a chunk from the pending queue only after platform acknowledgement for that artifact transfer and chunk index.
|
||||
|
||||
#### Scenario: Acknowledged chunk is removed from retry queue
|
||||
- **WHEN** a pending artifact chunk receives a platform acknowledgement for the same transfer ID and chunk index
|
||||
- **THEN** the run artifact queue removes that chunk from pending retry state
|
||||
|
||||
#### Scenario: Unacknowledged chunk remains pending
|
||||
- **WHEN** an artifact chunk has not received a matching platform acknowledgement
|
||||
- **THEN** the run artifact queue keeps the chunk available for retry
|
||||
@@ -0,0 +1,35 @@
|
||||
## 1. Artifact Transfer Contracts
|
||||
|
||||
- [x] 1.1 Add typed run artifact transfer protocol payloads in `run/protocol` for open, chunk upload, status/resume, completion, and acknowledgements.
|
||||
- [x] 1.2 Add matching platform DTO/domain contracts and conversion helpers for artifact transfer requests and responses.
|
||||
- [x] 1.3 Add validation rules for active upload direction, owner scope, bounded chunk size, byte ranges, chunk checksums, final checksums, and completion state.
|
||||
|
||||
## 2. Platform Artifact Transfer
|
||||
|
||||
- [x] 2.1 Extend platform service behavior to open upload transfers, accept idempotent chunks, reject conflicting chunks, report resume status, and mark artifacts available only after verified completion.
|
||||
- [x] 2.2 Implement platform artifact transfer HTTP routes using named DTOs and service methods.
|
||||
- [x] 2.3 Add platform service/API tests for successful upload, resume status, duplicate ack, checksum mismatch, invalid owner/session, and missing-chunk completion rejection.
|
||||
|
||||
## 3. Run Artifact Queue And Client
|
||||
|
||||
- [x] 3.1 Implement a run-side local artifact queue that writes pending chunk requests to disk, lists them for retry, and removes acknowledged chunks.
|
||||
- [x] 3.2 Extend `run/api.PlatformClient` with typed artifact transfer methods.
|
||||
- [x] 3.3 Add run queue/client tests for retry retention, acknowledgement cleanup, request paths, JSON payloads, response decoding, and platform error handling.
|
||||
|
||||
## 4. Documentation
|
||||
|
||||
- [x] 4.1 Update run and platform protocol/route documentation to mark artifact open/chunk/status/complete implemented and keep control/job/log/game-client channels separate.
|
||||
|
||||
## 5. Verification
|
||||
|
||||
- [x] 5.1 Run `go test ./...` from `platform/` and record evidence.
|
||||
- [x] 5.2 Run `go test ./...` from `run/` and record evidence.
|
||||
- [x] 5.3 Run `scripts/check-structure.sh` and record evidence.
|
||||
- [x] 5.4 Run `openspec validate implement-artifact-transfer-channel --strict` and record evidence.
|
||||
|
||||
## Evidence
|
||||
|
||||
- 2026-07-03: `go test ./...` from `platform/` passed.
|
||||
- 2026-07-03: `go test ./...` from `run/` passed.
|
||||
- 2026-07-03: `scripts/check-structure.sh` passed with `structure check passed`.
|
||||
- 2026-07-03: `openspec validate implement-artifact-transfer-channel --strict` passed with `Change 'implement-artifact-transfer-channel' is valid`.
|
||||
Reference in New Issue
Block a user