T3 Code · CI performance audit

Faster feedback, less duplicated work, safer releases

A measured plan for improving pull request CI, dependency caching, test parallelism, Blacksmith usage, and the release pipeline. The recommended path favors small changes with visible timing deltas over a broad CI rewrite.

Audit date: July 28, 2026 Evidence: 38 recent CI runs, 15 full releases, relay and mobile workflows Mode: read-only audit, no repository changes

Executive summary

The main CI path is not runner-starved. It is dominated by serial test scheduling and repeated environment setup. The release path is dominated by unnecessary dependency ordering between stages that could safely overlap.

207s CI wall time, p50 The Test job is the critical path.

37% Runner time spent in setup 165 of 451 typical runner-seconds.

20m Full release wall time Validation and publication stages are over-serialized.

6-8m Likely release reduction Before optimizing the actual macOS build.

939MB Linux dependency cache A hit still costs 22 to 29 seconds to restore.

1.35GB Windows dependency cache 63 seconds restore, then 43 seconds install.

14/100 Superseded CI runs Older commits kept running after newer pushes.

3/4 Recent relay deploys were no-ops Every main push still pays full setup.

Recommended first move: cancel superseded PR runs, start server tests immediately, consolidate Release Smoke into Check, and reduce Windows release setup. These changes are independent, easy to measure, and do not require inventing new infrastructure.

Observed baseline

Times below are medians from recent completed runs. They separate the useful work from setup so optimization effort lands in the right place.

CI job	p50	p90	Dominant steps	Interpretation
Check	99s	119s	Setup 38s, typecheck 23s, desktop build 21s	Task caching and typecheck concurrency are relevant.
Test	207s	255s	Setup 38s, tests 155s	Workspace ordering adds a 39-second avoidable delay.
Mobile Native	74s	77s	Setup 51s, Brew 9s, lint 4s	Overprovisioned and should be path-gated.
Release Smoke	71s	87s	Setup 38s, smoke 18s	Can reuse the Check job environment with no wall-time cost.

Release stage	p50	Primary constraint
Preflight	214s	Repeats Check, typecheck, and the full test suite.
Relay configuration	63s	Full runner setup for a small production-state read.
WSL node-pty	59s	All platform matrix entries wait for this Windows-only artifact.
macOS arm64 / x64	422s / 437s	Packaging, signing, and notarization dominate.
Windows x64	428s	47s checkout, 129s setup, 156s artifact build.
Publish CLI	94s	Build waits for all desktop artifacts before it starts.
GitHub Release	96s	Setup, artifact download, publication.
Hosted web deploy	160s	Starts only after GitHub Release succeeds.

Ranked findings

Ranking considers feedback latency, compute cost, operational risk, and the amount of machinery required.

Remove workspace dependency ordering from tests

Package tests start at 23:32:22, but apps/server waits until 23:33:02. The server then runs for 107 seconds. Tests do not require dependency tests to complete first.

vp run -r --parallel --concurrency-limit 4 test

Preserve fileParallelism: false inside the server suite. If cross-package load reintroduces flakes, split server and non-server tests across isolated VMs instead.

ExpectedAbout 39s CI wall time

EffortSmall experiment

RiskLoad-sensitive flakes

Restructure the release dependency graph

Preflight validation blocks relay resolution, WSL preparation, every desktop build, CLI construction, release publication, and web deployment. Several of these stages are independent.

Require green CI for the exact SHA, split Windows out of the platform matrix, build the CLI package alongside desktop artifacts, then run GitHub release creation and hosted deployment in parallel after the publication gate.

Expected6 to 8 minutes

EffortMedium

RiskPublication ordering

Fix Windows checkout and dependency setup

The latest successful release restored 1.35 GB in roughly 63 seconds, then spent another 43 seconds linking all 1,826 packages. The artifact build does not need the entire workspace dependency graph or full git history.

Use a filtered install for desktop, server, web, and scripts. Remove fetch-depth: 0 from WSL and platform artifact jobs. Exclude the vendored reference repositories with sparse checkout.

TargetSetup below 60s

EffortSmall to medium

RiskMissing build dependency

Pilot Blacksmith Sticky Disks on Linux

A successful Linux dependency-cache hit still consumes 22 to 29 seconds for a 939 MB archive, followed by installation. Observed throughput is 53 to 70 MB/s, substantially below Blacksmith's documented comparison.

First ask Blacksmith to inspect cache routing and decompression. Then A/B a sticky pnpm store on one workflow. Test node_modules only after the safer store experiment.

Expected15 to 25s per Linux job

EffortExperiment

RiskBeta storage semantics

Cancel superseded pull request runs

Fourteen of the latest 100 CI runs were still active when a newer commit arrived on the same branch. At least 16.8 minutes of remaining branch wall time could have been cancelled.

Add a pull-request concurrency key and cancel-in-progress: true. Keep main pushes uncancelled if maintainers want a durable result for every merge commit.

ExpectedImmediate compute savings

EffortTiny

RiskLow

Reuse Check setup for Release Smoke

Release Smoke spends 38 seconds restoring dependencies to run an 18-second task. Appending it to Check raises that job from about 99 to 117 seconds, still well below Test's 207-second critical path.

Expected71 runner-seconds per CI

EffortSmall

RiskLow

Path-gate and downsize mobile native analysis

A 12-vCPU macOS runner spends 60 seconds preparing to lint 21 files in four seconds. Use the 6-vCPU macOS tier and run the expensive job only when native modules, linter configuration, tools, or the workflow change.

Preserve a stable aggregate required check so a path-skipped job cannot leave a pull request pending.

ExpectedAbout 50% macOS cost

EffortSmall

RiskPath coverage

Enable Vite task caching for deterministic work

Typecheck reports 0 of 15 cache hits and desktop build reports 0 of 4. Package scripts are uncached by default. Enable cache for typecheck and builds, then persist only node_modules/.vite/task-cache.

Prove immediate second-run hits before adding cross-run persistence. Avoid server-test caching in the first iteration.

ExpectedHigh on incremental pushes

EffortMedium validation

RiskIncorrect fingerprints

Stop unrelated relay deployments

Production relay deployment runs after every push to main. Three of the four newest completed deployments were explicit no-ops, including unrelated web and mobile changes.

Trigger on relay and deployment configuration paths, retain manual dispatch, and use a separate periodic reconciliation if drift checks remain desirable.

ExpectedMost deploy runs removed

EffortSmall

RiskConfig-only changes

Make release side effects independently repairable

Recent runs created a GitHub release and tag, then failed during hosted web deployment. The next scheduled run saw the tag at HEAD and skipped, leaving no automatic repair path.

Make hosted deployment rerunnable for an exact ref. Make npm and GitHub publication idempotent so stable-release retries can safely continue after partial success.

ExpectedFaster incident recovery

EffortMedium

RiskExternal state

Release topology

The goal is not maximum speculative concurrency. It is to keep safe build work parallel while preserving one explicit publication gate.

Keep final npm publication on GitHub-hosted Ubuntu. npm trusted publishing currently requires a supported cloud-hosted runner. Build the tarball on Blacksmith, then download and publish it from the small GitHub-hosted job.

Implementation plan

Each phase has a measurable stopping point. Do not stack speculative cache, runner-size, and test-parallelism changes into one benchmark.

Phase 0Instrument

Record a seven-day baseline

Capture p50 and p90 job durations, setup time, cache bytes and transfer rate, test duration, cancellation count, and runner cost. Use the Blacksmith dashboard for CPU, memory, network, p99 step breakdown, and cache inventory.

One shared baseline table with CI, release, relay, and mobile jobs.

Add structured test output

Emit unique JUnit XML output for each Vitest workspace package. Blacksmith automatically discovers files written during a job and uses JUnit as the canonical source over log parsing.

Blacksmith Test Analytics lists test cases and durations without false failures from intentional error logs.

Phase 1Remove waste

Cancel superseded PR CI

Use a workflow and PR-number concurrency group with cancellation enabled only for pull request events.

A second push cancels the older PR CI within one minute and main runs retain the chosen history policy.

Move Release Smoke into Check

Run the existing smoke script after desktop pipeline verification using the already prepared workspace.

Release Smoke coverage remains required, CI wall p50 does not increase, and one runner job disappears.

Rightsize and gate mobile native analysis

Move from 12-vCPU to 6-vCPU macOS and introduce a changed-files gate covering native modules, lint configuration, tool installation, and workflow changes.

Native changes run all three linters, unrelated changes avoid macOS compute, and the aggregate required check always resolves.

Reduce relay no-op deployments

Add relevant path triggers, manual dispatch, and an explicit drift reconciliation policy.

Unrelated main commits do not start a full relay deployment.

Phase 2Shorten tests

Trial package-level parallel mode

Run all test package tasks without workspace dependency ordering, capped at four concurrent tasks. Do not alter the server suite's internal file parallelism.

Test step p50 drops by at least 30 seconds with no statistically visible increase in failures over 30 runs.

Shard only if the simple trial is insufficient

Split server from non-server tests first. If more latency reduction is required, create two isolated server shards while keeping files serial inside each shard.

CI wall p50 is below 150 seconds, or the cost-to-latency trade is explicitly accepted.

Profile the slowest files

Start with GitManager, CursorAdapter, GrokAdapter, AcpJsonRpcConnection, stash image compression, MessagesTimeline, and oxlint rule suites.

Each slow file has an owner and a reason: irreducible integration work, removable setup, or accidental waiting.

Phase 3Fix setup

Escalate current cache throughput to Blacksmith

Provide cache key, archive size, observed throughput, extraction time, runner region, and representative run links. Compare against the colocated-cache expectation.

Blacksmith confirms the expected backend and explains or resolves the 53 to 70 MB/s transfer behavior.

A/B a sticky pnpm store

Pilot on one Linux job with an OS and architecture-scoped key. Keep the normal cache path available as a rollback comparison.

Median setup improves by at least 15 seconds without cache corruption, stale installs, or branch isolation regressions.

Enable Vite task cache for typecheck and builds

Validate local second-run hits, define missing inputs and outputs, then persist only the Vite task-cache directory across CI runs.

Incremental PR pushes show real task hits, while a source, config, environment, or lockfile change invalidates the correct tasks.

Phase 4Release faster

Use exact-SHA CI as the validation gate

Query the required CI result for the release SHA. Run validation only for arbitrary refs that lack a valid green result.

Normal releases do not repeat 214 seconds of validation, while untested refs cannot publish.

Split Windows from the platform matrix

macOS and Linux should start after metadata and relay configuration. Only Windows should depend on WSL node-pty.

WSL preparation is no longer visible on the macOS or Linux critical path.

Prepare CLI artifacts in parallel

Build and pack the CLI on Blacksmith alongside desktop artifacts. Keep the final trusted publish on GitHub-hosted Ubuntu.

The GitHub-hosted publication job contains only artifact verification and publication work.

Parallelize release creation and hosted deployment

Both may begin after the publication gate. Announcements wait for both.

Full release p50 is below 14 minutes with equivalent publication guarantees.

Phase 5Harden

Make deployments and publication idempotent

Query expected npm version, dist-tag, GitHub release, tag, and hosted deployment state before writing. Treat matching state as success and conflicting state as an actionable error.

Rerunning the same stable release after a downstream failure repairs it without publishing a new version.

Repair mobile screenshot automation

Preserve the Android environment-startup causal stack, fix the SQLite rollback failure, cache the Android system image and AVD if the workflow becomes regular, and request no more than six emulator cores.

One all-platform run completes successfully twice in succession.

Pin EAS tooling and fix label-trigger redundancy

Pin the EAS CLI and fingerprint action, add per-PR preview concurrency, and ensure unrelated label changes cannot retrigger deployment.

Only a target-label event or new source commit starts a preview deploy.

Blacksmith feature decisions

Feature	Decision	Where to use it	Guardrail
Colocated Actions cache	Keep, investigate current throughput	Existing setup-vp dependency caches	Confirm cache backend and region before redesigning keys.
Sticky Disks	Pilot	Linux pnpm store, possibly task cache	One workflow first, compare 30 runs, document key ownership.
Git checkout cache	Pilot selectively	Full-history preflight and tag-comparison jobs	Prefer shallow and sparse checkout where history is unnecessary.
Test Analytics	Use now	All Vitest jobs via JUnit XML	Unique output path per package or shard.
Metrics and CI Analytics	Use for rightsizing	Test, Check, release build, mobile lint, deploy jobs	Change one runner size at a time and compare p50 plus p90.
Container caching	Request repository opt-out	T3 Code does not use Docker in CI	Check whether other organization repositories still need it.

Representative Linux jobs carry a 1.85 GB organization-level Docker image cache and spend about 5.1 seconds stopping and syncing Docker at teardown, despite no T3 Code workflow invoking Docker. Ask Blacksmith whether this repository can opt out independently.

Checkout and repository shape

The vendored .repos directory contains 12,961 tracked files and occupies about 132 MB in the working tree. It accounts for roughly 83% of tracked file count and is explicitly excluded from normal format, lint, and test operations.

Use sparse checkout to exclude it from ordinary CI and release jobs.
Retain it only for explicit reference-repository synchronization work.
Remove full-history checkout from WSL and platform artifact builds. Keep it in tag-resolution and release-finalization jobs only where proven necessary.
Clean the recurring post-checkout submodule warning caused by the vendored Alchemy gitlink lacking matching root metadata.

Sparse checkout is a performance optimization, not a reason to hide an undeclared build dependency. Pilot it on CI and release smoke before applying it to signed release builds.

Suggested PR sequence

Keep one concern per PR. This preserves review clarity and makes regressions easy to bisect.

perf(ci): cancel superseded pull request runs

PR-only concurrency and cancellation.

Compute savings, minimal risk

perf(ci): reuse check setup for release smoke

Remove the standalone Release Smoke runner.

About 71 runner-seconds per CI

perf(ci): run package tests without dependency ordering

Parallel-mode trial with the current concurrency cap.

Target 30 to 40 seconds wall time

perf(ci): rightsize mobile native analysis

6-vCPU macOS plus a safe changed-files gate.

About half macOS cost

perf(release): install only desktop build dependencies

Filtered matrix install, shallow checkout, sparse reference repos.

Largest Windows setup win

perf(ci): persist Vite task results

Typecheck and build cache only, after local fingerprint validation.

Incremental push acceleration

perf(release): parallelize platform and CLI preparation

Split Windows and prepare the CLI artifact concurrently.

Minutes off release critical path

fix(release): make hosted deployment rerunnable

Repair an exact release without minting a replacement version.

Operational reliability

Rollout guardrails

Flake budget: roll back test scheduling if failure rate increases across at least 30 comparable runs.
Cache correctness: every cached task must miss after a relevant source, configuration, environment, toolchain, or lockfile change.
Required checks: path filtering must always produce a terminal aggregate check, including no-op cases.
External publication: release retries must compare expected immutable state before skipping or writing.
Runner rightsizing: compare both duration and cost. A smaller runner that doubles wall time may be a false economy on the critical path.
Blacksmith beta features: keep a documented fallback to upstream Actions cache and checkout behavior.

Secondary findings

Mobile EAS Preview can redeploy when any unrelated label changes while the deployment label remains present.
EAS uses a mutable CLI version and a fingerprint action from @main. Pin both.
Mobile Showcase Screenshots has no fully successful workflow run yet. The newest attempt passed iOS in 12m48s but Android failed during local environment startup with a SQLite rollback error.
Android screenshot setup recreates its system image and AVD and requests eight emulator cores although the emulator reports a six-core maximum.
Build logs repeatedly emit broken sourcemap, Node deprecation, peer dependency, and Effect suggestion noise. These are low performance priority but make new regressions harder to see.
The web build reports a 3.57 MB minified main chunk, about 1.07 MB gzip, plus several 600 to 900 KB editor and language chunks. This is product performance rather than CI performance, but it deserves a separate bundle audit.

Executive summary

Observed baseline

Ranked findings

Remove workspace dependency ordering from tests

Restructure the release dependency graph

Fix Windows checkout and dependency setup

Pilot Blacksmith Sticky Disks on Linux

Cancel superseded pull request runs

Reuse Check setup for Release Smoke

Path-gate and downsize mobile native analysis

Enable Vite task caching for deterministic work

Stop unrelated relay deployments

Make release side effects independently repairable

Release topology

Implementation plan

Record a seven-day baseline

Add structured test output

Cancel superseded PR CI

Move Release Smoke into Check

Rightsize and gate mobile native analysis

Reduce relay no-op deployments

Trial package-level parallel mode

Shard only if the simple trial is insufficient

Profile the slowest files

Escalate current cache throughput to Blacksmith

A/B a sticky pnpm store

Enable Vite task cache for typecheck and builds

Use exact-SHA CI as the validation gate

Split Windows from the platform matrix

Prepare CLI artifacts in parallel

Parallelize release creation and hosted deployment

Make deployments and publication idempotent

Repair mobile screenshot automation

Pin EAS tooling and fix label-trigger redundancy

Blacksmith feature decisions

Checkout and repository shape

Suggested PR sequence

Rollout guardrails

Secondary findings

Evidence and references