AI Build Analysis — Streaming Redesign

Date: 2026-06-05  ·  Status: Draft for review  ·  Repos: Cap-go/capgo (edge fn + CLI), Cap-go/capgo_builder (worker)

1. Problem

The AI build-debug feature times out for roughly half of all failed CI builds.

Evidence (workflow logs from Cap-go/capgo, June 3–5 2026): of 21 failed builder runs that reached the AI step, 10 ended with AI analysis failed: The operation was aborted due to timeout. at almost exactly 60s. Successful runs measured 45–54s — the analysis duration distribution straddles the timeout.

Root cause: two stacked, equal 60s wall-clock timeouts — cli/src/ai/analyze.ts:61 (CLI → API) and supabase/functions/_backend/public/build/ai_analyze.ts:195 (API → builder). The CLI's clock starts earlier and also covers auth, DB checks and uploading a multi-MB log body, so the CLI always aborts first whenever the LLM needs ~60s.

• A timeout leaves ai_analyzed = false → the one-analysis-per-job cost control is retry-bypassed (cost leak).
• If the builder finishes just after the client gives up, the flag flips and the paid-for analysis is discarded; retries get 409.

2. Decision summary

3. End-to-end flow

sequenceDiagram
    autonumber
    participant CLI as Capgo CLI
(CI or TTY)
    participant API as capgo API
/build/ai_analyze_stream
    participant DB as Postgres
build_requests
    participant B as capgo_builder
Worker
    participant AI as Workers AI
kimi-k2.6

    CLI->>API: POST {jobId, appId, logs}
accept: text/event-stream
    API->>API: checkPermission(app.build_native)
    API->>DB: SELECT ownership + status = 'failed'
    API->>DB: CLAIM: UPDATE ai_analyzed = true
WHERE ... AND ai_analyzed = false
    alt 0 rows returned
        API-->>CLI: 409 already_analyzed
    end
    API->>B: POST /jobs/{id}/ai-analyze (SSE)
    alt connection fails OR aiStarted = false
        API->>DB: REFUND: ai_analyzed = false
        API-->>CLI: 502 builder_error (retryable)
    else builder streams
        B->>AI: env.AI.run(stream: true)
        AI--)B: token deltas
        B--)API: event: chunk {text}
        API--)CLI: event: chunk {text}
(watchdogs: 90s first-byte / 30s idle)
        Note over CLI: TTY: render progressively
CI: buffer, print once
        B--)API: event: done {durationMs}
        API--)CLI: event: done
        API->>API: waitUntil: telemetry
(no analysis text, ever)
    end

4. Protocol

4.1 New endpoint: POST /build/ai_analyze_stream

POST /build/ai_analyze_stream
capgkey: <apikey>
content-type: application/json
accept: text/event-stream

{ "jobId": "...", "appId": "...", "logs": "<captured build log>" }

Pre-stream failures are plain JSON with an HTTP status (the stream has not started, so status codes still work):

Success: 200 + content-type: text/event-stream:

event: chunk
data: {"text":"<token delta>"}

event: done
data: {"durationMs":48211}

event: error
data: {"code":"ai_error" | "idle_timeout"}

• chunk — text deltas, concatenated in order by the client.
• done — terminal; analysis = concatenation of all chunk texts.
• error — terminal; mid-stream failure. Slot stays consumed (fail closed). CLI shows partial text + notice.

4.2 Old endpoint: POST /build/ai_analyze (deprecated)

426 Upgrade Required
{ "error": "AI build analysis requires a newer CLI. Please upgrade: npx @capgo/cli@latest",
  "code": "upgrade_required" }

The handler keeps the apikey middleware but performs no DB reads/writes and never contacts the builder.

4.3 Builder endpoint: POST {BUILDER_URL}/jobs/{jobId}/ai-analyze

Internal, x-api-key-authenticated, called only by the capgo edge function. Success: 200 + the same SSE event protocol (piped through). Pre-stream errors carry an explicit cost marker:

{ "error": "invalid_json" | "logs_too_big" | "ai_error", "aiStarted": true | false }

aiStarted: false — rejected before env.AI.run was invoked (validation, trim overflow). aiStarted: true — env.AI.run was invoked and threw; billing unknown → counts as started. The edge fn refunds only on aiStarted: false or a connection-level failure.

5. Flag lifecycle — claim-then-refund

flowchart TD
    A[Request arrives] --> P{Permission +
ownership + status checks}
    P -- fail --> R1[4xx — no claim]
    P -- pass --> C{Atomic claim
UPDATE ... WHERE ai_analyzed = false}
    C -- 0 rows --> R2[409 already_analyzed]
    C -- claimed --> F{fetch builder}
    F -- connection failure
never reached builder --> RF1[REFUND
502 retryable]:::refund
    F -- "non-200, aiStarted = false" --> RF2[REFUND
502 retryable]:::refund
    F -- "non-200, aiStarted = true
or malformed body" --> K1[NO REFUND
fail closed]:::keep
    F -- 200 SSE --> S{Stream}
    S -- "watchdog fires
(90s first-byte / 30s idle)" --> K2[NO REFUND
event: error idle_timeout]:::keep
    S -- mid-stream error event --> K3[NO REFUND
partial shown]:::keep
    S -- client disconnects --> K4[NO REFUND
ever]:::keep
    S -- done --> OK[Success
telemetry via waitUntil]:::good

    classDef refund fill:#1d3324,stroke:#6fcf97,color:#bfe8cf
    classDef keep fill:#33201d,stroke:#eb5757,color:#f3c2c2
    classDef good fill:#1c2a3a,stroke:#4fc3f7,color:#cfe9ff

Invariant: every refunded cycle is provably zero-AI-cost, so total AI runs per job stay ≤ 1 under any interleaving of retries, spam, races or disconnects. The conditional UPDATE serializes on the Postgres row lock: N parallel requests → exactly 1 claim, N−1 409s (this also removes the SELECT-then-flip race in the current code). If the refund write itself fails: log loudly, fail closed — a user rarely loses a retryable slot; Capgo never pays for an extra run.

6. Timeouts & watchdogs

First-byte is generous because prompt ingestion of a large trimmed log produces no tokens for a while. Watchdogs = timer reset inside a TransformStream; firing aborts the upstream fetch and emits event: error {"code":"idle_timeout"} downstream. Watchdog expiry never refunds. Values staggered so the inner layer always fires before the outer.

7. Component changes

7.1 capgo_builder — src/ai-analyze.ts

• env.AI.run(MODEL, { messages, stream: true }); normalize Workers-AI SSE deltas into the §4.1 protocol in a TransformStream (model shape absorbed here — edge fn and CLI stay model-agnostic).
• Pre-stream error responses gain aiStarted.
• Propagate upstream cancellation into the AI stream (output-token waste reduction; best-effort, not a security control).
• Contingency: if kimi-k2.6 rejects stream: true, buffer internally and emit one big chunk + done — protocol unchanged.
• extractAnalysis() + buffered path deleted at rollout step 4.

7.2 capgo edge function

• New route ai_analyze_stream: permission → ownership SELECT → atomic claim → builder fetch with watchdogs → SSE passthrough → refund matrix → telemetry in waitUntil.
• ai_analyze.ts proxy logic replaced by the static 426 responder; 60s timeout and flip-after-success deleted.
• Passthrough is a piped Response — no buffering, no tee, no storage of analysis text.

7.3 capgo CLI

• postAnalyzeStreamRequest: new endpoint, SSE parsing, chunk accumulation.
• TTY: progressive text rendering (replaces spinner wait). CI: buffer and print final block — GitHub Actions output format unchanged.
• Mid-stream error: print partial text + "AI analysis was interrupted; this job's analysis slot is used. The full log is saved at <path> for local AI."
• Result kinds: ok | already_analyzed | too_big | upgrade_required | error.

8. Telemetry

Privacy rule unchanged: analysis text never appears in any event, tag or log.

• AI Build Analysis Requested — unchanged (fired after claim succeeds).
• AI Build Analysis Result — result enum extended with mid_stream_error, refunded.
• Old-endpoint hits emit result: upgrade_required → old-CLI population visible on the CLI Tracking dashboard before the builder fallback is deleted.
• CLI events keep their enums, with mid_stream_error added.

9. Rollout

1. builder — Accept-gated streaming + aiStarted markers (buffered JSON path retained). Zero behavior change for the current edge fn.
2. capgo edge fn — ship ai_analyze_stream + switch ai_analyze to 426. From this moment all old CLIs get the upgrade message.
3. CLI release — streaming client. CI workflows on bunx @capgo/cli@latest pick it up on the next run.
4. builder cleanup — delete buffered path once upgrade_required telemetry confirms cutover (target: within a week).

Each step independently deployable and reversible; the only user-visible discontinuity is the intended one (step 2).

10. Testing

• capgo edge fn (tests/build-ai-analyze.test.ts, rewritten): claim-before-fetch ordering; atomic conditional UPDATE via service-role client (preserves RLS regression tests); full refund matrix; concurrent duplicates → 1 claim + 409; 426 + telemetry on old route; SSE passthrough fidelity; idle-watchdog error event.
• capgo_builder: stream normalization per delta shape; aiStarted correctness per error; single-chunk fallback.
• CLI: SSE parsing incl. frames split across packets; first-byte/idle/total timeouts; TTY vs CI rendering; partial-text interruption branch.
• Manual/E2E: one real failed CI build with --ai-analytics before rollout step 4.

11. Out of scope

• Persisting analysis text anywhere (rejected: liability).
• Handshake/ack protocols, WebSockets, Durable Objects (rejected: complexity without benefit).
• The log-capture truncation issue (captured log sometimes misses the actual Gradle/Fastlane error) — separate investigation; streaming neither causes nor fixes it.
• Per-org rate limiting (ops nicety; the claim already makes abuse non-economic — each job requires a paid failed build and yields ≤ 1 AI run).