AGENTS.md · tester-army/e2e

Sections

1. Contracts
2. Layout
3. Commands
4. Definition of done
5. Non-obvious conventions
6. Testing quirks
7. Inspecting agent runs with unbox-ai
8. Cross-checking the web engine's tree
9. Measuring a change against main
10. Golden device trees
11. Gotchas

First lines (shown under the repo's Apache-2.0 license)

# AGENTS.md

`e2e` — an agentic end-to-end testing framework. pnpm monorepo, ESM only,
TypeScript 7.

## Contracts

There is no separate spec. The code is the contract, pinned in three places:

- The emitted `packages/e2e/dist/index.d.ts` (and `dist/engine/index.d.ts`,
  `dist/runner.d.ts`, `dist/oauth/*.d.ts`)
  is the public API. `packages/e2e/tests/types/sdk-types.ts` holds compile-time
  assertions (`@ts-expect-error` lines) for the parts that are easy to loosen
  by accident; it runs under the package `typecheck`, never under vitest.
- Wire formats live in `packages/e2e/schema/*.schema.json` (report-1,
  session-1, agent-judgment-2, the engine protocol's engine-manifest-1, and
  the deprecated agent-judgment-1) with a valid and an invalid fixture
  each. Integration tests validate every generated report and session envelope
  against them; `tests/unit/schema-fixtures.test.ts` checks the fixtures. A
  wire change edits the schema, both fixtures, and the producer in one review.
- Security invariants are the list under Gotchas below, enforced by tests in
  `tests/integration/agent-policy.test.ts` and the secret-ledger unit tests.
- Behavior changes update the matching `docs/**/*.mdx` page in the same
  change, including "not implemented yet" callouts, and `skills/e2e/` when the
  changed surface is described there. `scripts/check-error-codes.ts` (in
  `pnpm check`) fails when an error code in source is missing from
  `reference/errors.mdx` or `reference/engine.mdx`, or documented but raised
  nowhere.

There are no RFCs or design documents in the repo. The why lives in PR
descriptions and commit bodies; `git log` and `gh pr view` are the archive.
## Layout

`packages/` holds what publishes to npm; `apps/` holds the private apps and
suites that consume the built packages the way a user would.

- `packages/e2e` — the published `e2e` package: SDK surface, runner, CLI,
  `e2e/engine` contract. Core knows the contract and never an engine's
  internals: no engine package is imported from `src/` (grep for
  `from '@e2e-dev/` and `import('@e2e-dev/`; the presets below are the only
  hits). Engine words still appear as platform labels, capability names, and in
  error text that names a factory, such as `config/targets.ts` telling you to
  call `web()`; no path through them reaches engine code. The one exception is the
  `e2e init` scaffold presets in `src/cli/init/engines.ts`, which write the
  user's config and so name engine packages as text; the package build records
  the sibling packages' versions in `dist/cli/init/sibling-versions.json` for the
  ranges they write. Each preset owns its prompt label, dependencies, config,
  example, and run command; interactive choices derive from this list. These
  presets never import engine implementations.
  - `src/run/` runner core (scheduler, units, workers, retries, sessions;
    `standalone.ts` is the worker a host or MCP's one-shot attempt keeps
    alive, `standalone-attempt.ts` one attempt on it: reservation, launch,
    fixtures, deadline, close, `host-trace.ts` its `trace.md`, and
    `select-target.ts` the target a session names, shared with MCP;
    `open-session.ts` exposes `openHost` on `e2e/runner` for another test
    runner: worker-lifetime provisioning, app processes, and engine init,
    then `host.openSession({ name })` for each test's fixtures, deadline,
    steps, artifacts, and trace at close. Only one hosted attempt may be
    opening, live, or closing per process; close it before opening another),
    `src/collect/` registration+selection, `src/locator/` locator AST/engine,
    `src/agent/` the agent (the `act` executor socket plus the judgment
    methods), `src/mcp/` the `e2e mcp` server (a live session that rides
    the `act` socket with a queue executor so every MCP call is a harness
    action), `src/oauth/` subscription sign-in (the `e2e login`, `logout`,
    and `models` commands and the `e2e/oauth/*` model constructors; each
    constructor subpath is the place its `@ai-sdk/*` optional peers are
    imported, and the Copilot constructor loads `@ai-sdk/openai` lazily, only
    for the Responses models, so the CLI boots without any of them). The
    constructors and the CLI
    are the whole public surface: the flows, stores, and fetch behind them
    are module-private, not a library for other products. `tests/live/` holds hand-run
    checks that need a stored login and are never part of `pnpm test`.
- `packages/web` — the published `@e2e-dev/web` package: the
  browser engine, built with the public `defineEngine`, contributing the
  `browser` fixture and `expect(browser)`. It depends on `e2e` (peer), never the
  reverse; a target names it explicitly as `engine: web()`. There is
  no default engine and no well-known id registry in core. It imports from
  `e2e/engine` only: the semantics every engine must reproduce
  (error taxonomy, text and URL matching, assertion polling, JSON-value rules)
  are exported there, and there is no `e2e/internal` subpath.
- `packages/mobile` — the published `@e2e-dev/mobile` package: the
  iOS/Android engine on agent-device, built with the same public
  `defineEngine`, contributing the `device` fixture. Like the web engine it
  depends on `e2e` (peer), never the reverse; the engine implementation
  imports from `e2e/engine` only, and a target names it as `engine: mobile()`.
  The `@e2e-dev/mobile/tools` subpath holds the agent-side `open_app`, `swipe`,
  and `alert` tools; focused-field typing uses the engine's keyboard grammar.
  The main entry never loads the AI SDK.
  The `@e2e-dev/mobile/lease` subpath holds the agent-device lease plumbing
  device-cloud providers (`testmu`, `browserstack`) share: one daemon per
  slot, heartbeat, and a release that fails unless the session end is
  confirmed.
- `packages/e2e/src/engine/protocol/` - the engine protocol: the engine
  contract as LSP-style JSON-RPC (`connectEngine` client, `serveEngine`
  server, spawn/socket/loopback transports), so an engine can run in
  another process, machine, or language. The harness holds only protocol
  clients (`ResolvedTarget.engine` is an `EngineClientHandle`); a
  `defineEngine` handle is served over `loopbackTransport`, so a contract
  change that cannot cross the wire fails in every suite.
- `packages/kernel` - the published `@e2e-dev/kernel` package: Kernel hosted
  browsers for the web engine. An official integration with a hosted service
  is one package per service, named after it (`@e2e-dev/<service>`), with the
  vendor SDK and the engine it plugs into as peers. It implements that
  engine's provider seam (`BrowserProvider` for web, `DeviceProvider` for
  mobile) and imports the engine's types only; the engines never know it
  exists. A service with providers for both engines ships them in that one
  package, each engine an optional peer and each provider on its own entry
  that never imports the other engine (`@e2e-dev/testmu` and
  `@e2e-dev/testmu/web`).
  Its docs page under `docs/integrations/` ships with the vendor's logo as
  the sidebar icon and on its overview card (`authoring-docs`, "Page
  icons"); `scripts/check-docs-logos.ts` fails `pnpm docs:check` without
  them.
- `packages/lightpanda` - the published `@e2e-dev/lightpanda` package: a
  `BrowserProvider` that starts a local `lightpanda serve` per worker slot
  and stops it with the run; no vendor SDK, the binary is the dependency.
- `packages/eas` - the published `@e2e-dev/eas` package: EAS Simulators
  hosted iOS simulators and Android emulators for the mobile engine
  (`DeviceProvider`). Expo publishes no SDK for the sessions API, so it calls
  Expo's GraphQL API with `fetch`, and `@e2e-dev/mobile` is its only peer.
- `packages/smol` - the published `@e2e-dev/smol` package: Chromium in
  smol machines microVMs on the runner's own computer for the web engine
  (`BrowserProvider`), through the `smolmachines` SDK's embedded engine
  (peer). Each worker slot boots one warm browser machine from the pinned
  Playwright Ubuntu image; a Node TCP relay exposes DevTools, and SDK agent
  readiness allows machine startup without waiting for published ports. In
  `attempt` scope every attempt runs in a copy-on-write branch. With `app`,
  the app under test runs in the same machine, so a branch also copies its
  running app and data on the machine. When maintaining this integration,
  update the provider's SDK dev dependency to a published version and raise
  its peer minimum only if the provider needs new SDK behavior. Keep the
  Playwright image and Chromium version in sync with `@e2e-dev/web`. Run
  `pnpm --filter @e2e-dev/smol test:unit` and `pnpm check`, then verify live
  machines in both attempt and worker scopes with an app running in the VM.
- `packages/limrun` — the Limrun hosted mobile device provider. Each worker
  leases one remote device, installs its app, and releases the device at exit.
- `packages/decision` — the published `@e2e-dev/decision` package: a
  `StepExecutor` (`decisionExecutor()`) that drives `agent.act` and
  `agent.assert` through an AI SDK *decision* model answering `choice`
  questions with probability distributions, plus an optional small language
  model that writes field values when the decision model picks `type`.
  `minProbability` and `minConfidence` gate a chosen operation, target,
  secret, or assertion verdict. It also exports `decisionFilter()`, a
  `TestFilter` that asks a decision model whether the pull request's diff
  can affect each test, packing test files into calls under the model's
  size limit, 8 calls at a time, and leaves out the unlikely ones. It runs
  every test on an `e2e.config` or lockfile change, and asks nothing about a
  test whose test file, setup file, or stored screenshots changed. A failed
  call leaves its tests in and the rest are still judged; after 3 failed
  calls it stops asking. When every call fails, or git or a file read
  fails, it throws, and the run warns and runs every test.
- `packages/acp` — the published `@e2e-dev/acp` package: a `StepExecutor`
  (`acp()`, with `claudeCode()` and `codex()` presets) that runs
  `agent.act` and `agent.assert` on a coding agent over the Agent Client
  Protocol. The step tools (`createGrammarTools`, `createVerdictTool` from
  `e2e/agent`) reach the agent as an MCP server on 127.0.0.1; permission
  requests for anything else are rejected, and a tool of the agent's own
  that runs without asking fails the step with `POLICY_DENIED`.
- `packages/github` — the published `@e2e-dev/github` package: the reporter
  that posts the run as one pull request comment from GitHub Actions and