87 lines
6.8 KiB
Markdown
87 lines
6.8 KiB
Markdown
## Context
|
|
|
|
Run control, job lifecycle, and durable log ingest are implemented as separate HTTP JSON channels. Artifact metadata exists in the platform, and job results can reference artifacts, but there is no transfer workflow that can move large run-produced files into platform-managed artifact records with resume and checksum semantics.
|
|
|
|
This change implements the first run-to-platform artifact upload channel. Platform storage remains in memory and artifact payloads are held only long enough to prove chunk ordering and final checksum behavior. The channel is intentionally separate from control, jobs, logs, and the optional game client bridge so large payloads do not share those routes or DTOs.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
|
|
- Define typed artifact transfer payloads in `run/protocol` and matching platform DTO/domain contracts.
|
|
- Add platform artifact transfer routes for open, chunk upload, resume/status, and complete.
|
|
- Validate active run session, owner relationship, transfer identity, bounded chunk size, chunk checksum, byte ranges, resume state, and final checksum.
|
|
- Update existing `Artifact` metadata from `uploading` to `available` only after every chunk is present and the final checksum matches.
|
|
- Add a run-side artifact spool/queue that persists unacknowledged chunk upload requests and deletes them only after platform acknowledgement.
|
|
- Extend `run/api.PlatformClient` with typed artifact transfer methods.
|
|
- Add tests for platform service/API transfer behavior, resume, duplicate chunk acknowledgement, checksum errors, completion errors, run spool retention, and client request/response handling.
|
|
|
|
**Non-Goals:**
|
|
|
|
- No platform-to-run download flow, browser upload/download UI, external object storage backend, presigned URL flow, or streaming transport.
|
|
- No plugin bridge file APIs, AI artifact inspection, archive extraction, or artifact lifecycle cleanup jobs.
|
|
- No raw host paths, raw credentials, direct sockets, logs, or job result bodies inside artifact chunk requests.
|
|
- No billing, cloud host sales, agent-provider/cloud-provider workflows, or direct plugin-to-run access.
|
|
|
|
## Decisions
|
|
|
|
### Decision 1: HTTP JSON chunk endpoints first
|
|
|
|
The initial channel uses separate JSON `POST` endpoints under `/api/v1/run/artifacts/*`: `open`, `chunks`, `status`, and `complete`. Chunk payloads use JSON byte encoding, which Go represents as base64, and validators enforce a bounded maximum chunk size.
|
|
|
|
Alternative considered: multipart upload or object-storage signed URLs. Rejected for this change because there is no storage backend yet, and the first requirement is to prove protocol, validation, resume, and checksum semantics in tests.
|
|
|
|
### Decision 2: Run uploads only in this change
|
|
|
|
The transfer direction is explicit but only `upload` is accepted. Platform-to-run download will need separate authorization, cache, and throttling semantics after upload behavior is stable.
|
|
|
|
Alternative considered: implementing upload and download together. Rejected because download would add browser/plugin access questions and storage-adapter behavior that are outside this queue item.
|
|
|
|
### Decision 3: Existing Artifact metadata remains the public resource
|
|
|
|
Opening a transfer creates or validates the existing `Artifact` metadata record in `uploading` state. Completion updates that same record to `available`; failed checksum or missing chunk errors leave the artifact non-available.
|
|
|
|
Alternative considered: adding a separate persisted transfer model now. Rejected because current platform persistence is in-memory and the transfer session can stay behind `service.Core` until a database-backed storage change exists.
|
|
|
|
### Decision 4: Chunks are accepted idempotently by checksum
|
|
|
|
The platform records received chunk indexes, byte ranges, sizes, checksums, and payload bytes in memory. Re-uploading the same chunk with the same checksum returns a duplicate acknowledgement; re-uploading a different payload for an acknowledged index is rejected.
|
|
|
|
Alternative considered: allowing overwrite of existing chunk indexes. Rejected because resumable upload cleanup must be deterministic and conflicting retries should be visible immediately.
|
|
|
|
### Decision 5: Owner authorization is platform mediated
|
|
|
|
Run uploads are accepted only for job-owned or server-instance-owned artifacts that belong to the requesting run endpoint. Platform/plugin-owned artifact records can still be created through metadata APIs, but this run transfer channel does not let a run endpoint spoof unrelated owners.
|
|
|
|
Alternative considered: accepting any artifact owner kind. Rejected because run must not become a direct write path for platform/plugin-owned data without an explicit authorization change.
|
|
|
|
### Decision 6: Run spool stores chunk upload requests, not host paths
|
|
|
|
The run-side artifact spool writes one JSON file per pending chunk request. It stores the bounded request payload needed for retry and never stores or exposes the local host path that originally produced the bytes.
|
|
|
|
Alternative considered: storing file path plus offset for retry. Rejected because run must enforce scoped paths and must not expose raw host paths through platform-facing transfer state.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- [Risk] In-memory platform chunk storage disappears on restart. Mitigation: keep transfer state behind `service.Core`; storage adapters and durable transfer sessions can replace it later.
|
|
- [Risk] JSON/base64 chunks are inefficient for large production artifacts. Mitigation: enforce bounded chunks now and leave streaming/object-storage transfer to a later change.
|
|
- [Risk] No background artifact uploader exists. Mitigation: run client and spool semantics are implemented and tested; scheduling and priority throttling can build on them later.
|
|
- [Risk] Upload-only support does not cover all artifact use cases. Mitigation: explicitly keep direction in the protocol so a future download change can extend without renaming the channel.
|
|
|
|
## Migration Plan
|
|
|
|
1. Add artifact transfer protocol, DTO, domain, validation, and service contracts.
|
|
2. Add platform API handlers and tests for open, chunk upload, resume/status, and complete.
|
|
3. Add run artifact spool implementation and tests.
|
|
4. Add run client methods and tests.
|
|
5. Update protocol and route docs.
|
|
6. Verify with platform tests, run tests, structure check, and strict OpenSpec validation.
|
|
|
|
Rollback before dependent changes is removal of the artifact transfer route/client/spool additions and this OpenSpec change. After server workflows depend on artifact transfer, rollback must use a new OpenSpec change.
|
|
|
|
## Open Questions
|
|
|
|
- Which durable artifact storage backend should be implemented first: local segments, filesystem package storage, S3-compatible object storage, or another adapter?
|
|
- What production chunk size, concurrency limits, and backoff policy should artifact uploaders use?
|
|
- How should platform-to-run download authorization interact with plugin pages and server management workflows?
|