# Test runs Source: https://olef.dev/docs/concepts/test-runs Public guide snapshot. [View source](https://github.com/olef-dev/olef/blob/4a2fd8128ec9f6be0f6e5254e1c6b81211d2170c/docs/test-runs.md). For exact flags in your installation, use `olef commands --json`. An Olef test is a folder with a Bash script, `test.sh`, that drives the app with one `olef` command per step. Each command waits for its target and checks its own result, so the script needs no sleeps, no `jq` and no library. Running it needs no model and no API key: an agent explores once, and the test replays without AI. ```text tests/olef/mission_recovery/ test.sh the script (executable) screens/ optional: one screenshot per step (.jpg), shown next to the screen now when that step fails data/ optional: data sets for --params (*.env) README.md written by `record save`: the steps in plain English, parameters, how to run it recording.json written by `record save`: compiled steps, alternative selectors, what saving removed responses/ written by `record save` when the proxy captured the flow: the server responses expect/ written by `record save` when the proxy captured the flow: request payloads the steps sent ``` Write it by hand, or record it: [recording tests](https://olef.dev/docs/concepts/recording) turns a viewer recording or an agent's exploration into this folder (the sample's `mission_recorded` was saved from a viewer recording). The example below is the sample app's `mission_recovery` test. ## Writing a test ```bash #!/usr/bin/env bash set -euo pipefail cd "$(dirname "$0")" APP=io.github.andriyo.shadowdroid.sample CALLSIGN="${CALLSIGN:-Ada}" step() { export OLEF_STEP="$1"; } TOKEN="$(mktemp -d)/session.token" trap 'olef --session-file "$TOKEN" session close --exit-code $?' EXIT olef session open --token-out "$TOKEN" --agent test:mission_recovery --app "$APP" \ --clear-data --profile automation --launch .MainActivity --expect-rid status_text export OLEF_SESSION_FILE="$TOKEN" step open-mission olef ui tap --rid nav_mission --expect-rid mission_claim_button step claim-incident olef ui text --rid name_input --clear "$CALLSIGN" olef ui tap --rid mission_claim_button --expect-text "Incident claimed by $CALLSIGN" ``` - **Header**: `set -euo pipefail`, `cd` to the test's folder, parameters, the `step` function, the token and the `trap`, then `session open` (see [sessions](https://olef.dev/docs/concepts/sessions) for its settings, starting state, guards and journal). Arm the trap first: closing a session that never opened is a no-op, and close undoes the starting state. - **Parameters** are environment variables with defaults: `CALLSIGN=Grace bash test.sh`, or `test run --env CALLSIGN=Grace`. - **Steps**: `step `, one action, then zero or more checks (`--expect-*` on the action, or `ui wait` after it). - **Sub-flows** are other test scripts: `bash ../login/test.sh` opens a session nested under the caller's and returns to it. - **Portability**: Bash 3.2 (macOS `/bin/bash`): no arrays, `mapfile` or `${x,,}`; one command per line with `\` continuations. Literal text in single quotes, only parameters in double quotes. Any language can run the same commands: start with `session open`, stop at the first non-zero exit, set `OLEF_STEP` per step, and close the session in a `finally`. `test show test.sh --json` gives each step's argv. ## Running ```bash olef test run tests/olef olef test run tests/olef/mission_recovery --repeat 10 --junit-out build/junit.xml olef test run tests/olef --retries 1 --shard 2/4 --artifacts-dir build/sd-runs olef test run tests/olef/checkout --params 'data/*.env' ``` `test run` runs each folder's `test.sh` through its shebang (else `bash`), one at a time. Each run gets its own folder, named by `OLEF_RUN_DIR`; the script's `session open` writes the journal, failure report and `report.html` there. The runner itself holds no device lock: the scripts' sessions reserve the device, nested under yours when you hold one. Device selection (`-d`, `--target`), `--authority-dir` and your session token are passed to the scripts. | Flag | Effect | | --- | --- | | `--repeat N` | Run every test N times; tests and steps are classified `stable_pass`, `stable_fail` or `flaky` | | `--retries N` | Rerun a failed test up to N times; a pass after a retry is `flaky`, never a plain pass | | `--env K=V` | Set a parameter (repeatable) | | `--params GLOB` | Run once per data set (`KEY=VALUE` lines), matched in the test folder first | | `--capture steps\|failures\|off` | Evidence per step (default `failures`; `steps` keeps a screenshot of every step) | | `--shard I/N` | Run only shard I of N of the sorted test list | | `--junit-out FILE` | JUnit XML (default `/junit.xml`) | | `--artifacts-dir DIR` | Run folders and reports (default `.olef/test-runs/run-/`) | | `--fail-fast` | Stop after the first test that fails (after its retries) | The run writes `junit.xml`, `report.json` and an index `report.html` linking each run's own report. It exits non-zero with `code: test_failed` when a test failed, naming the first failure in plain words, its `fault`, and the `test heal` command for it. ### JUnit One `` per test run (per repetition with `--repeat`), one `` per step. A failure whose fault is `app` or `test` is a ``; `environment` or `olef` is an ``. Steps a failed run never reached are ``. A repetition that passed after a retry carries Surefire's `` for the earlier failure (`` when it still failed). Each suite names its run folder. ### Flake report `report.json` lists, per test, every run and every step's classification, pass/fail counts and median time. `flake_report` lists flaky steps and the flake risk seen in passing runs: checks that took more than 70% of their limit (`slow_check`), guard retries (`dismissed_system_ui`) and `--pace-ms` in use. ## When a test fails The failing command's error carries `detail.test_failure`, saved as `failures//report.json` in the run folder: - **Where**: test, step, command. - **Cause**: `code`, a plain sentence (for a changed text: "Expected text 'Relay calibrated at 71%' did not appear; the screen shows 'Relay calibrated at 70%' instead"), and whose fault it is: `app` (a check failed, a crash), `test` (the selector no longer matches), `environment` (system UI, no network, an injected fault) or `olef`. - **Expected vs actual**: the step's recorded screenshot next to the screen now, the closest on-screen texts, error texts the app shows. - **Near misses**: `closest` selectors (a misspelt id's real one first). - **App and device**: in front, processes alive, crashes with stacks, recent error logs; airplane mode, Wi-Fi, data, `http_proxy`, faults. - **Fix**: a selector swap, or an expectation update flagged "confirm the new text is intended". `test heal --from ` turns that fix into a one-line patch (`/heal.patch`). It never edits the script, and never rewrites a value built from a parameter. ## Reports from direct runs `bash test.sh` leaves its session folder under `.olef/sessions/`. `test report …` builds the same JUnit and flake report from such folders (or from a test run's artifacts), and `--baseline ` flags steps whose median time grew by at least 1.5× and 500 ms. ## Lint and show `test lint` reads scripts without running them and reports, with a fix for each: coordinates, text-only selectors (with the `ui audit` hint), actions without checks, `sleep`, a missing `set -euo pipefail` or `trap`, unquoted parameters, secrets in literal text (an error), Bash 4 features, and custom code it cannot read. `test show test.sh [--json]` prints the parameters, steps and argv lists. `olef test -- ` still runs instrumentation tests with the UiAutomation slot freed; only a program literally named like a subcommand (`run`, `report`, …) needs the `--`.