> ## Documentation Index > Fetch the complete documentation index at: https://docs.stackra.figentra.com/llms.txt > Use this file to discover all available pages before exploring further. # Test infrastructure (the testbed) > The shared test boxes every builder session uses: how jobs run, the disk guard, replacing a box, rebuilding the image, secrets, the fallback, cost, and the migration from the old ci-box fleet. > [!NOTE] > > Status: written, not applied (STA-1626). The design is [ADR > 0484](https://docs.stackra.figentra.com/adr/0484-the-testbed-disposable-image-built-test-boxes-a-container-per-job-and-secrets-from-infisical.md). > The code is `@stackra/infra-testbed` and `stackra testbed`. Nothing is applied until the owner > says so. Until the migration below finishes, `ci-box` and the eight old boxes stay in service. Builder sessions never run heavy checks on the Mac. Only oxfmt and oxlint run there. Install, build, gen, typecheck, tests, e2e and integration tests all run on the testbed: identical EC2 boxes, built from one image, in one region (eu-west-1). ## The shape | Part | What it is | | - | - | | Image | One Dockerfile for the job toolchain (mise, Node, pnpm, Encore, Playwright, Infisical) and one EC2 Image Builder AMI | | Groups | `heavy`: on-demand `c7i.2xlarge`, 2 to 6 boxes. `light`: spot `c7i`/`c6i`/`m7i.2xlarge`, 1 to 4 boxes | | Disks | 40 GB root (OS and images); 200 GB XFS data volume at `/data` (jobs, seeds, pnpm store, turbo cache, Docker) | | Slots | One heavy and one light job per box. Heavy: `encore test`, `playwright`, `e2e`, `act `, `build`, or `--on-demand`. Everything else: light. A light box with a spot reclaim or rebalance notice refuses new jobs (exit 75) | | Jobs | A fresh container per job, workspace `/data/jobs/`, read-only pnpm store, optional Postgres; all deleted after | | Secrets | Fetched per job from Infisical by the instance role (the shared gate identity). There is no token file | | Box agent | `testbed-agent` on every box: health, slots, the disk guard and the idle check | ## Running work ```bash theme={"system"} stackra testbed run " " "" ``` The command picks a free slot of the right kind and syncs the directory. Only the diff is uploaded: the box clones its last copy of the project first. Then it runs the command in a fresh container and returns its exit code. The other verbs: | Verb | Does | | - | - | | `stackra testbed status` | each box: heavy and light slot, disk use, health, image age, load | | `stackra testbed unlock --box N` | stops a stuck job on one box (its container and the containers it started) | | `stackra testbed ip\|ssh\|sync` | one box, by number from `status` or by instance id | | `stackra testbed scale` | shows each group's desired count and raises it by one | | `stackra testbed image` | the current AMI and its age | * **"every box busy, waiting"** means every slot of that kind is taken. After two minutes the command adds a box (up to the group's maximum) and keeps waiting. Do not start a second copy of the job. * A command that calls `ci-box` or `stackra testbed` is refused. Run it from the Mac instead. * A job runs for at most 3 hours. ## The disk guard `testbed-agent guard` runs every five minutes and before every job. | `/data` use | What happens | | - | - | | under 70% | normal | | 70% or more | cleanup: finished job folders, stopped containers, dangling images, seeds unused for 7 days, turbo cache to 20 GB | | 85% or more (or `/` under 10 GB free) | the box refuses new jobs and reports unhealthy. Once its jobs finish, the group replaces it | Nobody frees disk by hand. A box that keeps filling up points to a job that writes outside its workspace: find it in `status` and fix the job. ## Replacing a box Replace, don't repair. A box that is unhealthy or acting oddly: 1. Run `stackra testbed status` and note the box's instance id. 2. Wait until its slots are free. A running job holds scale-in protection. 3. The owner, or the tester with the owner's yes, marks it unhealthy: ```bash theme={"system"} aws autoscaling set-instance-health --instance-id --health-status Unhealthy --profile academorix --region eu-west-1 ``` The group starts a new box from the current image within about five minutes. Never SSH in to patch packages, edit configs or install tools: the next box would not have the change. Change the image instead. ## Rebuilding the image The image rebuilds every Sunday at 03:00 UTC. It also gets a new recipe version whenever the Dockerfile, the host components or the toolchain pins change. A failed build keeps the last good AMI. To roll out a new image, the owner starts an instance refresh. Boxes running a job wait, because the refresh skips protected boxes until they are free: ```bash theme={"system"} aws autoscaling start-instance-refresh --auto-scaling-group-name stackra-testbed-heavy --profile academorix --region eu-west-1 ``` Repeat with `stackra-testbed-light`. ## Secrets * Each box authenticates to Infisical by AWS IAM auth, as the shared gate identity (`INFISICAL_GATE_IDENTITY_ID`, the one the Quality Gate uses over GitHub OIDC). AWS Auth sits beside its OIDC auth; no new identity is created. * At job start the agent gets a token that lives for the job's timeout. The container runs the command under `infisical run`. No value touches a disk, a log or a command line. * Which project paths a job may read is `secretGrants` in `@stackra/infra-testbed`, reviewed like code. The identity's permissions in Infisical are the real limit. * `ci-box put-env` and `~/.ci-env` are retired. A missing token is fixed in Infisical, by the owner. To rotate the identity's auth, see [Token rotation](https://docs.stackra.figentra.com/runbooks/token-rotation.md). ## When no box is healthy `stackra testbed run --preset --ref ` dispatches `testbed-fallback.yml`. It runs that preset for a pushed ref on a GitHub-hosted runner. A plain command, or work in an unpushed worktree, waits for a box instead. Nothing falls back to the Mac. ## Cost Prices read on 2026-10-30 for eu-west-1: * on-demand `c7i.2xlarge`: $0.383 an hour (about $280 a month always on); * spot: about \$0.215 an hour; * gp3: $0.088 per GB-month (about $21 a month for one box's two disks). At the minimums (2 heavy, 1 light) the testbed costs about $780 a month, and up to about $1,100 with daytime scale-ups. The old fleet ran about \$1,900 a month. Boxes above the minimum end themselves after 60 idle minutes. ## Migration from the old fleet The tester keeps using the eight old boxes until the new pool has passed the same work. Every step is a separate owner yes ([plan](https://github.com/figentra/stackra/tree/main/.agents/specs/sta-1626-testbed/plan.md)). **1. Infisical** The owner adds AWS Auth to the shared gate identity, bound to the testbed instance role. **2. Stand up beside** The `apply-testbed.yml` workflow in eu-west-1 (Step 2 below): the image pipeline, the first AMI, `heavy` minimum 1, `light` minimum 0. **3. Canary** The tester runs the same jobs on both fleets for a day (core verify, playground e2e, an integration suite, one Academorix job) and compares the results. **4. Switch** `ci-box` becomes the shim (`tools/scripts/ci-box-shim.sh`): new boxes first, old boxes only when the new pool has no healthy box. The minimums go to 2 and 1. **5. Drain** One old region a day (us-east-1, eu-west-2, eu-central-1, eu-west-1): remove it from the shim's fallback, wait for free slots, stop the boxes. **6. Retire** After a week stopped and unused, terminate the old boxes and their disks, delete the emptied `academorix-ci` security groups, and delete the shim's fallback with every `~/.ci-env`. Rollback before the last step: the shim still names the old boxes, and `aws ec2 start-instances` brings them back. ## Step 1 in detail: AWS Auth on the gate identity The boxes use the shared machine identity `figentra-ci-gate`, the one the Quality Gate uses over GitHub OIDC (its id is the GitHub organization variable `INFISICAL_GATE_IDENTITY_ID`). It already has **Viewer** on the Infisical project `figentra-ci` (renamed from `figentra-ci-gate`, same id), which holds exactly what a job needs: `prod` `/ci` (`HEROUI_AUTH_TOKEN`, `UNIWIND_AUTH_TOKEN`) and `prod` `/ci/turbo` (`TURBO_TOKEN`, `TURBO_REMOTE_CACHE_SIGNATURE_KEY`). The Figentra Flow gate key is not here: it lives in the project `figentra-flow-gate`, read only by the `figentra-flow-box` identity through figentra-flow's poster, and no job grant names it. The project is the boundary; the free plan has no custom roles. Nothing is created: one login method is added, so no identity slot is spent. The owner does this in the Infisical web app (EU, `eu.infisical.com`); no value is typed anywhere. 1. Organization, **Access Control**, **Machine Identities**, then open `figentra-ci-gate`. 2. In **Authentication**, leave **OIDC Auth** exactly as it is (CI depends on it) and choose **Add auth method**, **AWS Auth**. 3. Fill in: | Field | Value | | - | - | | Allowed Principal ARNs | `arn:aws:iam::591473336656:role/stackra-testbed-instance` | | Allowed Account IDs | `591473336656` | | STS Endpoint | leave the default `https://sts.amazonaws.com/` | | Access Token TTL | `14400` (4 hours: covers the 3-hour job timeout, with the fetch before it) | | Access Token Max TTL | `14400` | | Access Token Max Number of Uses | `0` | | Access Token Trusted IPs | leave `0.0.0.0/0` (boxes have changing public IPs) | 4. Save. The identity page now lists two auth methods, OIDC Auth and AWS Auth. These settings belong to the AWS Auth method only; CI's OIDC tokens keep theirs. The role does not exist until step 2 creates it, and Infisical accepts the ARN before then. Account `591473336656` is the `academorix` profile's. Step 2 needs two ids, neither of them secret: the identity's id (`INFISICAL_GATE_IDENTITY_ID`) and the project's id (`INFISICAL_GATE_PROJECT_ID`). ## Step 2 in detail: the first deploy Nothing is applied from the Mac. The testbed is applied by the manual workflow `apply-testbed.yml` (Actions, "Apply testbed", Run workflow), on `main` only, in the GitHub environment `testbed`, whose required reviewer (the owner) approves every run. It signs in to AWS by GitHub OIDC, so no AWS key is stored anywhere, and keeps Pulumi's state in the private, versioned bucket `stackra-pulumi-state-591473336656` (eu-west-1), which its first run creates. Pulumi's S3 backend locks with objects in that bucket, so there is no lock table. The stack holds ids and CIDRs only, so its passphrase is empty by design. ### 2a. One-time bootstrap (owner, with the `academorix` AWS profile; done 2026-10-30) The workflow cannot create the role it signs in with, so the owner creates it once. GitHub's OIDC provider, if the account has none yet: ```bash theme={"system"} aws iam create-open-id-connect-provider --url https://token.actions.githubusercontent.com --client-id-list sts.amazonaws.com --profile academorix ``` The deploy role, trusted only by this repository's `testbed` environment. The organization's tokens use the immutable subject form (`repo:figentra@314302700/stackra@1425140001:…`, as `ci.yml` notes), so both forms are listed: ```bash theme={"system"} aws iam create-role --role-name stackra-testbed-deploy --profile academorix --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":"arn:aws:iam::591473336656:oidc-provider/token.actions.githubusercontent.com"},"Action":"sts:AssumeRoleWithWebIdentity","Condition":{"StringEquals":{"token.actions.githubusercontent.com:aud":"sts.amazonaws.com","token.actions.githubusercontent.com:sub":["repo:figentra/stackra:environment:testbed","repo:figentra@314302700/stackra@1425140001:environment:testbed"]}}}]}' ``` Its rights: everything but IAM (`PowerUserAccess`), plus IAM only for the testbed's own roles and instance profiles, and never for itself (the deny keeps it from granting its own role more): ```bash theme={"system"} aws iam attach-role-policy --role-name stackra-testbed-deploy --policy-arn arn:aws:iam::aws:policy/PowerUserAccess --profile academorix ``` ```bash theme={"system"} aws iam put-role-policy --role-name stackra-testbed-deploy --policy-name testbed-iam --profile academorix --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["iam:*Role","iam:*Roles","iam:*RolePolicy","iam:*RolePolicies","iam:*InstanceProfile","iam:*InstanceProfiles","iam:*RoleFromInstanceProfile","iam:*RoleToInstanceProfile","iam:TagRole","iam:UntagRole","iam:TagInstanceProfile","iam:PassRole"],"Resource":["arn:aws:iam::591473336656:role/stackra-testbed-*","arn:aws:iam::591473336656:instance-profile/stackra-testbed-*"]},{"Effect":"Allow","Action":"iam:CreateServiceLinkedRole","Resource":"*"},{"Sid":"NeverItsOwnRole","Effect":"Deny","Action":"iam:*","Resource":"arn:aws:iam::591473336656:role/stackra-testbed-deploy"}]}' ``` Then in GitHub, figentra/stackra, Settings, Environments, New environment `testbed`: deployment branches **main** only, required reviewer the owner, and two environment variables (neither is secret): * `TESTBED_DEPLOY_ROLE_ARN` = `arn:aws:iam::591473336656:role/stackra-testbed-deploy` * `TESTBED_SSH_CIDRS` = the Mac's public IP as `x.x.x.x/32` (comma-separate more). `stackra testbed` swaps its own entry later. `INFISICAL_GATE_IDENTITY_ID` and `INFISICAL_GATE_PROJECT_ID` are already organization variables. ### 2b. The two passes Each run below is Actions, "Apply testbed", Run workflow on `main`, then the owner approves it in the `testbed` environment. 1. **action** `preview`, **groups** unticked, heavy `1`, light `0`. Read the plan: a security group, two roles with their profiles and policies, a launch template, the build bucket and its object, the Image Builder components, recipe, infrastructure, distribution and pipeline, and no Auto Scaling group. 2. **action** `up`, same inputs. Creates the state bucket on the first run, then the pipeline. No instance runs yet. 3. **action** `build-image`. Starts the pipeline once: 30 to 60 minutes on one build instance that stops when done. Its log is in the Image Builder console under `stackra-testbed`. When it succeeds, the SSM parameter `/imagebuilder/stackra-testbed/ami` holds the AMI id. 4. **action** `preview`, **groups** ticked, heavy `1`, light `0`: the two Auto Scaling groups appear. 5. **action** `up`, same inputs. One heavy box starts (about \$280 a month from here). `stackra testbed status` should list it healthy within about five minutes. At the switch step of the migration, run `up` again with heavy `2` and light `1`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.