> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stackra.figentra.com/llms.txt
> Use this file to discover all available pages before exploring further.

# 0484 — The testbed: disposable, image-built test boxes, a container per job, and secrets from Infisical

> Accepted · 2026-10-30

> [!NOTE]
>
> Amends [0166](https://docs.stackra.figentra.com/adr/0166-ci-policy-runners-budgets-concurrency-and-check-matrix.md) CI2 (the ci-box
> becomes the testbed; it stays advisory and is never a required check). Phase 2 (decision 12) would
> amend CI1 and is not decided here.

## Context

Every heavy check of every builder session (install, build, gen, typecheck, the 2,111-test playground e2e, integration
tests) runs on eight shared EC2 boxes driven by `~/.local/bin/ci-box`, a 72-line shell script on the owner's Mac. The
boxes serve Academorix, Retail OS, Legal OS and Figentra alike. Facts read on 2026-10-30:

* All eight are `c7i.2xlarge` (8 vCPU, 16 GiB), four on-demand and four spot, in four regions (eu-west-1,
  eu-central-1, us-east-1, eu-west-2), each with one 150 or 200 GB gp3 root disk holding the OS, every checkout, the
  pnpm store, Docker and `~/.ci-env`. They were built by hand, have no instance role, and differ by region AMI.
* Box 1's disk reached 100%. `ci-box put-env` then truncated its `~/.ci-env` to nothing while printing "set", and
  every build token on it was lost. Boxes 3 and 4 sit at 94–97%. Nothing on a box cleans up after a job.
* Average CPU over the last seven days is 3–17%; peaks reach 100%. Boxes 1 and 3 ran all 168 hours: the 60-minute
  idle stop does not hold when something keeps them warm.
* Box 8 (spot) was reclaimed twice in the middle of a heavy run.

A box is a pet: repaired in place, slowly diverging, and one full disk away from stopping every builder. The owner
asked for a standard under which testing is never interrupted: rebuilt from code, image-based, a sandbox per job.

## Decision

1. **Code home.** The testbed is the Pulumi package `@stackra/infra-testbed` at
   `packages/infrastructure/infra-testbed` (the plan pure in `.`, the Pulumi program in `./node`, like the other
   `infra-*` packages), and the command `stackra testbed` in `@stackra/cli`. `stackra testbed` replaces
   `~/.local/bin/ci-box`; during the migration `ci-box` is a shim that calls it.
2. **One golden image, built from code.** One Dockerfile (`images/job/`) defines the job toolchain: Ubuntu 24.04,
   mise with the repository's pins (Node 24.22.0, pnpm 12), the Encore CLI, Playwright's browsers and their system
   libraries. One EC2 Image Builder recipe defines the host AMI: Ubuntu 24.04, Docker, the job image and the
   `postgres` image preloaded, the box agent (`testbed-agent`) and a warm pnpm store. The pipeline runs weekly and on
   any change to the recipe, the Dockerfile, `mise.toml` or the catalogs' toolchain pins. The newest AMI is published
   to an SSM parameter; boxes are never patched by hand. A sick box is replaced, not repaired.
3. **Disks.** A 40 GB gp3 root disk holds the OS and the images. A separate 200 GB gp3 data volume, formatted XFS
   with reflinks, is mounted at `/data` and holds `/data/jobs`, `/data/seed`, `/data/pnpm-store`, the turbo cache and
   Docker's data root. Both disks are deleted with the instance.
4. **Disk guard.** `testbed-agent` measures `/data` and `/` before every job and every five minutes. Under 70% is
   normal. At 70% it cleans up: finished job folders, stopped containers and dangling images, seed copies unused for
   seven days, the turbo cache down to 20 GB. At 85% (or under 10 GB free on `/`) the box refuses new jobs, reports
   unhealthy, and once its running jobs finish it marks itself unhealthy in the Auto Scaling group, which replaces it.
5. **A container per job.** Every job runs in a fresh container from the job image, named and labelled with its job
   id. Its workspace is `/data/jobs/<id>`, cloned by reflink from `/data/seed/<project>` and then brought up to date
   by `rsync` from the caller, and deleted when the job ends, whatever the outcome. The shared pnpm store is mounted
   read-only; the agent alone writes it (`pnpm fetch` against the job's lockfile, one writer at a time) before the
   container starts, and the job installs offline. Each job gets its own Docker network and, when asked, its own
   Postgres container on tmpfs. Containers a job starts (Encore's test database, Testcontainers) carry the job label
   and are removed with it. There is no VM per job.
6. **Slots stay.** A box runs one heavy job and one light job at a time, with today's classification (`encore test`,
   `playwright`, `e2e`, `act `, `build` are heavy; everything else is light) and today's Mac-side claim so parallel
   callers never pick the same slot. A command that calls `ci-box` or `stackra testbed` is refused, as today.
7. **Secrets from Infisical, never a file.** `~/.ci-env` is retired. The instance role authenticates to Infisical by
   AWS IAM auth as the shared gate identity (`INFISICAL_GATE_IDENTITY_ID`, the one the Quality Gate uses over GitHub
   OIDC), with AWS Auth added beside its OIDC auth; no new identity is created (amended 2026-10-30). At job start the
   agent obtains a short-lived access token (time to live: the job's timeout) with read access to the paths the job
   names and only those (today `figentra-ci` `/ci`: `HEROUI_AUTH_TOKEN`, `UNIWIND_AUTH_TOKEN`; `/ci/turbo` only
   once the gate identity can read the turbo project); the container runs the command under `infisical run`. The
   Figentra Flow gate key is not a job grant: it lives in its own project, `figentra-flow-gate` (`prod` `/ci/gate`), read
   only by the `figentra-flow-box` identity through figentra-flow's own poster (amended 2026-10-31). No value is written to
   a disk, printed, or passed on a command line. Which project and paths a job may read is configuration in
   `infra-testbed`, reviewed like code.
8. **Pricing and placement.** One region (proposed: eu-west-1, the cheapest EU region checked). Two Auto Scaling
   groups from one launch template: `heavy`, on-demand `c7i.2xlarge`; `light`, spot across `c7i.2xlarge`,
   `c6i.2xlarge` and `m7i.2xlarge` with capacity rebalancing. A heavy job never lands on spot. A light job that loses
   its spot box is retried once on another box.
9. **Scaling and idle.** Each group has a minimum it never goes below. `stackra testbed run` raises the desired count
   by one, up to the maximum, when every slot of its kind has been busy for two minutes. A box above the minimum that
   has run no job for 60 minutes terminates itself and lowers the desired count. A box running a job holds scale-in
   protection, so neither scale-in nor an instance refresh to a new AMI ever ends a running job.
10. **Health.** Before every job the agent checks disk (decision 4), free memory, Docker, a Postgres container
    starting, and an Infisical token being issued. A box that fails refuses the job and the caller picks another. The
    groups replace boxes the agent marks unhealthy and boxes whose EC2 status checks fail.
11. **Fallback and the Mac rule.** When no box is healthy, `stackra testbed run --preset <name> --ref <ref>` dispatches
    the `testbed-fallback` workflow on GitHub-hosted runners for a pushed ref and a named preset (`verify`, `e2e`,
    `integration`); arbitrary commands and unpushed worktrees wait for a box instead. The Mac rule stands: only oxfmt
    and oxlint run locally, and `stackra testbed` never falls back to the Mac.
12. **Later phase (not decided here).** Ephemeral self-hosted GitHub Actions runners from the same AMI, so the tester's
    boxes and CI run the same image. It needs its own ADR because it amends 0166 CI1.

## Consequences

* A full disk becomes a replaced box, not an outage, and a lost token is not possible: there is no token file.
* Every box is the same image, so "works on box 3" means "works on every box".
* The data volume is 200 GB per box against today's 150–200 GB shared disk, and job folders are deleted, so the
  pressure that filled box 1 has a ceiling and a cleaner.
* The reachable cost drops: at eu-west-1 prices read on 2026-10-30 an on-demand `c7i.2xlarge` is $0.383 an hour
  (about $280 a month always on), spot about $0.215, gp3 $0.088 per GB-month (240 GB: about $21 a month a box). Today's
  eight boxes running around the clock are about $1,900 a month with their disks; the proposed shape (decision 8, with the minimums the
  plan proposes) is about $780 to $1,100.
* New chores: the weekly image build must stay green (a red build keeps the last good AMI); the job-identity's
  Infisical access list is reviewed with each new project; the instance role is the one AWS credential on a box.
* Agents keep their verbs: `run`, `status`, `unlock`, `ip`, `ssh`, `sync` keep their meaning. `put-env` is removed;
  a missing token is an Infisical change, made by the owner.
* `stackra testbed` shells out to the `aws`, `ssh` and `rsync` binaries the Mac already has; it adds no dependency.

## Enforcement

planned — STA-1626:

* `infra-testbed` unit tests assert the plan: two disks, the data volume at `/data`, on-demand heavy, spot only for
  light, one region, no secret in user data, scale-in protection on.
* `stackra testbed` unit tests assert the heavy/light classification, the nested-call refusal and the health refusal.
* `testbed-agent` tests (in the package, run in the image build) assert the 70% cleanup and the 85% refusal.
* `pulumi preview` of the program in the tester's lane; no apply without the owner.

## Alternatives

* **Keep the boxes and add a cron cleaner.** Rejected: it treats the symptom; the boxes stay hand-built and
  divergent, and the token file stays.
* **A VM per job (Firecracker, a fresh EC2 per job).** Rejected for now: a fresh EC2 per job costs two to four minutes
  of boot and warm-up per run, and the container per job already gives a clean workspace and network.
* **Packer for the image.** Rejected: EC2 Image Builder is declared in the same Pulumi program, needs no new tool and
  no licence review.
* **GitHub-hosted runners only.** Rejected for agent work: unpushed worktrees cannot run there, and the e2e suite needs
  more than a standard runner holds within 0166's budget.

## Supersedes and sources

Supersedes: none. Amends: [0166](https://docs.stackra.figentra.com/adr/0166-ci-policy-runners-budgets-concurrency-and-check-matrix.md) CI2. Related:
[0173](https://docs.stackra.figentra.com/adr/0173-secrets-infisical-is-the-source-of-production-secrets.md) (Infisical is the source),
[0293](https://docs.stackra.figentra.com/adr/0293-agent-permission-tiers-and-executor-pushes.md) (raw `infisical run` on the Mac asks the owner; the
box runs it under the gate identity). Amended 2026-10-30 (owner): the boxes use the shared gate identity instead of
a new `figentra-ci-boxes` identity (decision 7). Amended 2026-10-31 (owner): the project `figentra-ci-gate` is
renamed `figentra-ci` (same id), and the Figentra Flow gate key moved out of it into `figentra-flow-gate`, which no job grant
names (decision 7). Sources: `~/.local/bin/ci-box` as of 2026-10-30; EC2 and CloudWatch reads of the
eight boxes on 2026-10-30; the owner's brief for STA-1626.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.