Skip to main content
Status: written, not applied (STA-1626). The design is ADR 0484. The code is @stackra/infra-testbed and stackra testbed. Nothing is applied until the owner says so. Until the migration below finishes, ci-box and the eight old boxes stay in service.
Builder sessions never run heavy checks on the Mac. Only oxfmt and oxlint run there. Install, build, gen, typecheck, tests, e2e and integration tests all run on the testbed: identical EC2 boxes, built from one image, in one region (eu-west-1).

​
The shape

​
Running work

The command picks a free slot of the right kind and syncs the directory. Only the diff is uploaded: the box clones its last copy of the project first. Then it runs the command in a fresh container and returns its exit code. The other verbs:
  • “every box busy, waiting” means every slot of that kind is taken. After two minutes the command adds a box (up to the group’s maximum) and keeps waiting. Do not start a second copy of the job.
  • A command that calls ci-box or stackra testbed is refused. Run it from the Mac instead.
  • A job runs for at most 3 hours.

​
The disk guard

testbed-agent guard runs every five minutes and before every job. Nobody frees disk by hand. A box that keeps filling up points to a job that writes outside its workspace: find it in status and fix the job.

​
Replacing a box

Replace, don’t repair. A box that is unhealthy or acting oddly:
  1. Run stackra testbed status and note the box’s instance id.
  2. Wait until its slots are free. A running job holds scale-in protection.
  3. The owner, or the tester with the owner’s yes, marks it unhealthy:
The group starts a new box from the current image within about five minutes. Never SSH in to patch packages, edit configs or install tools: the next box would not have the change. Change the image instead.

​
Rebuilding the image

The image rebuilds every Sunday at 03:00 UTC. It also gets a new recipe version whenever the Dockerfile, the host components or the toolchain pins change. A failed build keeps the last good AMI. To roll out a new image, the owner starts an instance refresh. Boxes running a job wait, because the refresh skips protected boxes until they are free:
Repeat with stackra-testbed-light.

​
Secrets

  • Each box authenticates to Infisical by AWS IAM auth, as the shared gate identity (INFISICAL_GATE_IDENTITY_ID, the one the Quality Gate uses over GitHub OIDC). AWS Auth sits beside its OIDC auth; no new identity is created.
  • At job start the agent gets a token that lives for the job’s timeout. The container runs the command under infisical run. No value touches a disk, a log or a command line.
  • Which project paths a job may read is secretGrants in @stackra/infra-testbed, reviewed like code. The identity’s permissions in Infisical are the real limit.
  • ci-box put-env and ~/.ci-env are retired. A missing token is fixed in Infisical, by the owner. To rotate the identity’s auth, see Token rotation.

​
When no box is healthy

stackra testbed run --preset <verify|e2e|integration> --ref <ref> dispatches testbed-fallback.yml. It runs that preset for a pushed ref on a GitHub-hosted runner. A plain command, or work in an unpushed worktree, waits for a box instead. Nothing falls back to the Mac.

​
Cost

Prices read on 2026-10-30 for eu-west-1:
  • on-demand c7i.2xlarge: $0.383 an hour (about $280 a month always on);
  • spot: about $0.215 an hour;
  • gp3: $0.088 per GB-month (about $21 a month for one box’s two disks).
At the minimums (2 heavy, 1 light) the testbed costs about $780 a month, and up to about $1,100 with daytime scale-ups. The old fleet ran about $1,900 a month. Boxes above the minimum end themselves after 60 idle minutes.

​
Migration from the old fleet

The tester keeps using the eight old boxes until the new pool has passed the same work. Every step is a separate owner yes (plan).
1

Infisical

The owner adds AWS Auth to the shared gate identity, bound to the testbed instance role.
2

Stand up beside

The apply-testbed.yml workflow in eu-west-1 (Step 2 below): the image pipeline, the first AMI, heavy minimum 1, light minimum 0.
3

Canary

The tester runs the same jobs on both fleets for a day (core verify, playground e2e, an integration suite, one Academorix job) and compares the results.
4

Switch

ci-box becomes the shim (tools/scripts/ci-box-shim.sh): new boxes first, old boxes only when the new pool has no healthy box. The minimums go to 2 and 1.
5

Drain

One old region a day (us-east-1, eu-west-2, eu-central-1, eu-west-1): remove it from the shim’s fallback, wait for free slots, stop the boxes.
6

Retire

After a week stopped and unused, terminate the old boxes and their disks, delete the emptied academorix-ci security groups, and delete the shim’s fallback with every ~/.ci-env.
Rollback before the last step: the shim still names the old boxes, and aws ec2 start-instances brings them back.

​
Step 1 in detail: AWS Auth on the gate identity

The boxes use the shared machine identity figentra-ci-gate, the one the Quality Gate uses over GitHub OIDC (its id is the GitHub organization variable INFISICAL_GATE_IDENTITY_ID). It already has Viewer on the Infisical project figentra-ci (renamed from figentra-ci-gate, same id), which holds exactly what a job needs: prod /ci (HEROUI_AUTH_TOKEN, UNIWIND_AUTH_TOKEN) and prod /ci/turbo (TURBO_TOKEN, TURBO_REMOTE_CACHE_SIGNATURE_KEY). The Figentra Flow gate key is not here: it lives in the project figentra-flow-gate, read only by the figentra-flow-box identity through figentra-flow’s poster, and no job grant names it. The project is the boundary; the free plan has no custom roles. Nothing is created: one login method is added, so no identity slot is spent. The owner does this in the Infisical web app (EU, eu.infisical.com); no value is typed anywhere.
  1. Organization, Access Control, Machine Identities, then open figentra-ci-gate.
  2. In Authentication, leave OIDC Auth exactly as it is (CI depends on it) and choose Add auth method, AWS Auth.
  3. Fill in:
  4. Save. The identity page now lists two auth methods, OIDC Auth and AWS Auth.
These settings belong to the AWS Auth method only; CI’s OIDC tokens keep theirs. The role does not exist until step 2 creates it, and Infisical accepts the ARN before then. Account 591473336656 is the academorix profile’s. Step 2 needs two ids, neither of them secret: the identity’s id (INFISICAL_GATE_IDENTITY_ID) and the project’s id (INFISICAL_GATE_PROJECT_ID).

​
Step 2 in detail: the first deploy

Nothing is applied from the Mac. The testbed is applied by the manual workflow apply-testbed.yml (Actions, “Apply testbed”, Run workflow), on main only, in the GitHub environment testbed, whose required reviewer (the owner) approves every run. It signs in to AWS by GitHub OIDC, so no AWS key is stored anywhere, and keeps Pulumi’s state in the private, versioned bucket stackra-pulumi-state-591473336656 (eu-west-1), which its first run creates. Pulumi’s S3 backend locks with objects in that bucket, so there is no lock table. The stack holds ids and CIDRs only, so its passphrase is empty by design.

​
2a. One-time bootstrap (owner, with the academorix AWS profile; done 2026-10-30)

The workflow cannot create the role it signs in with, so the owner creates it once. GitHub’s OIDC provider, if the account has none yet:
The deploy role, trusted only by this repository’s testbed environment. The organization’s tokens use the immutable subject form (repo:figentra@314302700/stackra@1425140001:…, as ci.yml notes), so both forms are listed:
Its rights: everything but IAM (PowerUserAccess), plus IAM only for the testbed’s own roles and instance profiles, and never for itself (the deny keeps it from granting its own role more):
Then in GitHub, figentra/stackra, Settings, Environments, New environment testbed: deployment branches main only, required reviewer the owner, and two environment variables (neither is secret):
  • TESTBED_DEPLOY_ROLE_ARN = arn:aws:iam::591473336656:role/stackra-testbed-deploy
  • TESTBED_SSH_CIDRS = the Mac’s public IP as x.x.x.x/32 (comma-separate more). stackra testbed swaps its own entry later.
INFISICAL_GATE_IDENTITY_ID and INFISICAL_GATE_PROJECT_ID are already organization variables.

​
2b. The two passes

Each run below is Actions, “Apply testbed”, Run workflow on main, then the owner approves it in the testbed environment.
  1. action preview, groups unticked, heavy 1, light 0. Read the plan: a security group, two roles with their profiles and policies, a launch template, the build bucket and its object, the Image Builder components, recipe, infrastructure, distribution and pipeline, and no Auto Scaling group.
  2. action up, same inputs. Creates the state bucket on the first run, then the pipeline. No instance runs yet.
  3. action build-image. Starts the pipeline once: 30 to 60 minutes on one build instance that stops when done. Its log is in the Image Builder console under stackra-testbed. When it succeeds, the SSM parameter /imagebuilder/stackra-testbed/ami holds the AMI id.
  4. action preview, groups ticked, heavy 1, light 0: the two Auto Scaling groups appear.
  5. action up, same inputs. One heavy box starts (about $280 a month from here). stackra testbed status should list it healthy within about five minutes.
At the switch step of the migration, run up again with heavy 2 and light 1.