Skip to main content
Amends 0166 CI2 (the ci-box becomes the testbed; it stays advisory and is never a required check). Phase 2 (decision 12) would amend CI1 and is not decided here.

​
Context

Every heavy check of every builder session (install, build, gen, typecheck, the 2,111-test playground e2e, integration tests) runs on eight shared EC2 boxes driven by ~/.local/bin/ci-box, a 72-line shell script on the owner’s Mac. The boxes serve Academorix, Retail OS, Legal OS and Figentra alike. Facts read on 2026-10-30:
  • All eight are c7i.2xlarge (8 vCPU, 16 GiB), four on-demand and four spot, in four regions (eu-west-1, eu-central-1, us-east-1, eu-west-2), each with one 150 or 200 GB gp3 root disk holding the OS, every checkout, the pnpm store, Docker and ~/.ci-env. They were built by hand, have no instance role, and differ by region AMI.
  • Box 1’s disk reached 100%. ci-box put-env then truncated its ~/.ci-env to nothing while printing “set”, and every build token on it was lost. Boxes 3 and 4 sit at 94–97%. Nothing on a box cleans up after a job.
  • Average CPU over the last seven days is 3–17%; peaks reach 100%. Boxes 1 and 3 ran all 168 hours: the 60-minute idle stop does not hold when something keeps them warm.
  • Box 8 (spot) was reclaimed twice in the middle of a heavy run.
A box is a pet: repaired in place, slowly diverging, and one full disk away from stopping every builder. The owner asked for a standard under which testing is never interrupted: rebuilt from code, image-based, a sandbox per job.

​
Decision

  1. Code home. The testbed is the Pulumi package @stackra/infra-testbed at packages/infrastructure/infra-testbed (the plan pure in ., the Pulumi program in ./node, like the other infra-* packages), and the command stackra testbed in @stackra/cli. stackra testbed replaces ~/.local/bin/ci-box; during the migration ci-box is a shim that calls it.
  2. One golden image, built from code. One Dockerfile (images/job/) defines the job toolchain: Ubuntu 24.04, mise with the repository’s pins (Node 24.22.0, pnpm 12), the Encore CLI, Playwright’s browsers and their system libraries. One EC2 Image Builder recipe defines the host AMI: Ubuntu 24.04, Docker, the job image and the postgres image preloaded, the box agent (testbed-agent) and a warm pnpm store. The pipeline runs weekly and on any change to the recipe, the Dockerfile, mise.toml or the catalogs’ toolchain pins. The newest AMI is published to an SSM parameter; boxes are never patched by hand. A sick box is replaced, not repaired.
  3. Disks. A 40 GB gp3 root disk holds the OS and the images. A separate 200 GB gp3 data volume, formatted XFS with reflinks, is mounted at /data and holds /data/jobs, /data/seed, /data/pnpm-store, the turbo cache and Docker’s data root. Both disks are deleted with the instance.
  4. Disk guard. testbed-agent measures /data and / before every job and every five minutes. Under 70% is normal. At 70% it cleans up: finished job folders, stopped containers and dangling images, seed copies unused for seven days, the turbo cache down to 20 GB. At 85% (or under 10 GB free on /) the box refuses new jobs, reports unhealthy, and once its running jobs finish it marks itself unhealthy in the Auto Scaling group, which replaces it.
  5. A container per job. Every job runs in a fresh container from the job image, named and labelled with its job id. Its workspace is /data/jobs/<id>, cloned by reflink from /data/seed/<project> and then brought up to date by rsync from the caller, and deleted when the job ends, whatever the outcome. The shared pnpm store is mounted read-only; the agent alone writes it (pnpm fetch against the job’s lockfile, one writer at a time) before the container starts, and the job installs offline. Each job gets its own Docker network and, when asked, its own Postgres container on tmpfs. Containers a job starts (Encore’s test database, Testcontainers) carry the job label and are removed with it. There is no VM per job.
  6. Slots stay. A box runs one heavy job and one light job at a time, with today’s classification (encore test, playwright, e2e, act , build are heavy; everything else is light) and today’s Mac-side claim so parallel callers never pick the same slot. A command that calls ci-box or stackra testbed is refused, as today.
  7. Secrets from Infisical, never a file. ~/.ci-env is retired. The instance role authenticates to Infisical by AWS IAM auth as the shared gate identity (INFISICAL_GATE_IDENTITY_ID, the one the Quality Gate uses over GitHub OIDC), with AWS Auth added beside its OIDC auth; no new identity is created (amended 2026-10-30). At job start the agent obtains a short-lived access token (time to live: the job’s timeout) with read access to the paths the job names and only those (today figentra-ci /ci: HEROUI_AUTH_TOKEN, UNIWIND_AUTH_TOKEN; /ci/turbo only once the gate identity can read the turbo project); the container runs the command under infisical run. The Figentra Flow gate key is not a job grant: it lives in its own project, figentra-flow-gate (prod /ci/gate), read only by the figentra-flow-box identity through figentra-flow’s own poster (amended 2026-10-31). No value is written to a disk, printed, or passed on a command line. Which project and paths a job may read is configuration in infra-testbed, reviewed like code.
  8. Pricing and placement. One region (proposed: eu-west-1, the cheapest EU region checked). Two Auto Scaling groups from one launch template: heavy, on-demand c7i.2xlarge; light, spot across c7i.2xlarge, c6i.2xlarge and m7i.2xlarge with capacity rebalancing. A heavy job never lands on spot. A light job that loses its spot box is retried once on another box.
  9. Scaling and idle. Each group has a minimum it never goes below. stackra testbed run raises the desired count by one, up to the maximum, when every slot of its kind has been busy for two minutes. A box above the minimum that has run no job for 60 minutes terminates itself and lowers the desired count. A box running a job holds scale-in protection, so neither scale-in nor an instance refresh to a new AMI ever ends a running job.
  10. Health. Before every job the agent checks disk (decision 4), free memory, Docker, a Postgres container starting, and an Infisical token being issued. A box that fails refuses the job and the caller picks another. The groups replace boxes the agent marks unhealthy and boxes whose EC2 status checks fail.
  11. Fallback and the Mac rule. When no box is healthy, stackra testbed run --preset <name> --ref <ref> dispatches the testbed-fallback workflow on GitHub-hosted runners for a pushed ref and a named preset (verify, e2e, integration); arbitrary commands and unpushed worktrees wait for a box instead. The Mac rule stands: only oxfmt and oxlint run locally, and stackra testbed never falls back to the Mac.
  12. Later phase (not decided here). Ephemeral self-hosted GitHub Actions runners from the same AMI, so the tester’s boxes and CI run the same image. It needs its own ADR because it amends 0166 CI1.

​
Consequences

  • A full disk becomes a replaced box, not an outage, and a lost token is not possible: there is no token file.
  • Every box is the same image, so “works on box 3” means “works on every box”.
  • The data volume is 200 GB per box against today’s 150–200 GB shared disk, and job folders are deleted, so the pressure that filled box 1 has a ceiling and a cleaner.
  • The reachable cost drops: at eu-west-1 prices read on 2026-10-30 an on-demand c7i.2xlarge is $0.383 an hour (about $280 a month always on), spot about $0.215, gp3 $0.088 per GB-month (240 GB: about $21 a month a box). Today's eight boxes running around the clock are about $1,900 a month with their disks; the proposed shape (decision 8, with the minimums the plan proposes) is about $780 to $1,100.
  • New chores: the weekly image build must stay green (a red build keeps the last good AMI); the job-identity’s Infisical access list is reviewed with each new project; the instance role is the one AWS credential on a box.
  • Agents keep their verbs: run, status, unlock, ip, ssh, sync keep their meaning. put-env is removed; a missing token is an Infisical change, made by the owner.
  • stackra testbed shells out to the aws, ssh and rsync binaries the Mac already has; it adds no dependency.

​
Enforcement

planned — STA-1626:
  • infra-testbed unit tests assert the plan: two disks, the data volume at /data, on-demand heavy, spot only for light, one region, no secret in user data, scale-in protection on.
  • stackra testbed unit tests assert the heavy/light classification, the nested-call refusal and the health refusal.
  • testbed-agent tests (in the package, run in the image build) assert the 70% cleanup and the 85% refusal.
  • pulumi preview of the program in the tester’s lane; no apply without the owner.

​
Alternatives

  • Keep the boxes and add a cron cleaner. Rejected: it treats the symptom; the boxes stay hand-built and divergent, and the token file stays.
  • A VM per job (Firecracker, a fresh EC2 per job). Rejected for now: a fresh EC2 per job costs two to four minutes of boot and warm-up per run, and the container per job already gives a clean workspace and network.
  • Packer for the image. Rejected: EC2 Image Builder is declared in the same Pulumi program, needs no new tool and no licence review.
  • GitHub-hosted runners only. Rejected for agent work: unpushed worktrees cannot run there, and the e2e suite needs more than a standard runner holds within 0166’s budget.

​
Supersedes and sources

Supersedes: none. Amends: 0166 CI2. Related: 0173 (Infisical is the source), 0293 (raw infisical run on the Mac asks the owner; the box runs it under the gate identity). Amended 2026-10-30 (owner): the boxes use the shared gate identity instead of a new figentra-ci-boxes identity (decision 7). Amended 2026-10-31 (owner): the project figentra-ci-gate is renamed figentra-ci (same id), and the Figentra Flow gate key moved out of it into figentra-flow-gate, which no job grant names (decision 7). Sources: ~/.local/bin/ci-box as of 2026-10-30; EC2 and CloudWatch reads of the eight boxes on 2026-10-30; the owner’s brief for STA-1626.