# The agent loop

Source: https://olef.dev/docs/concepts/agent-loop

Public guide snapshot. [View source](https://github.com/olef-dev/olef/blob/4a2fd8128ec9f6be0f6e5254e1c6b81211d2170c/docs/agent-loop.md). For exact flags in your installation, use `olef commands <path> --json`.

This is the canonical operating contract for LLMs and coding agents driving
Olef. The loop is **read → act → confirm**, and Olef is built so
each step costs as few round-trips as possible.

1. **Discover the surface once.** Start with `olef commands --json
   --depth 1`; use `commands --json --describe 'ui tap'` for one command,
   `commands --guide net|debugger|state` for a domain driving guide, or
   omit `--depth` for the full tree. Schema version 3 contains canonical paths,
   complete argument construction data (aliases, conflicts, requirements,
   groups, arity, and trailing/hyphen-value behavior), output contracts, and
   agent decision hints. Do not invent command names from memory or scrape
   `--help` prose.
2. **Put repeated context in config.** `olef config init ...` then
   `config validate --json`. Use an app alias instead of spending tokens on
   `--package`/`--project`/`--debugger` every call. See
   [configuration.md](https://olef.dev/docs/concepts/configuration).
3. **Connect.** `olef connect`; if it fails, `olef doctor --json`,
   then `doctor --fix` only when repair side effects are acceptable.
4. **Read by dumping.** `olef ui dump` returns the actionable tree as a
   compact element list plus strict content identity, actionable interaction
   identity, screen-bound handles, and freshness metadata. Act only
   from `snapshot_state: "consistent"`; a lifecycle race is retried within a
   bounded window, then returned as `transitioning` with a warning. Cache the
   hash/version pairs; invalidate a cached hash if its version changes.
5. **Act by selector, not coordinates.** Prefer `--rid`, then `--desc`/exact
   `--text`. Add `--observe` to wait for an accessibility quiet period and get
   the post-action screen in the same response. Prefer one `--expect-*`
   destination condition when the outcome is known; it implies observation and,
   once satisfied, returns a screen summary (hashes, foreground app, top texts)
   instead of every element (`--full` returns the whole screen).
   Add `--if-interaction <hash>` to stable selectors on dynamic screens, or use
   a returned `--handle` when no stable selector exists. Use `--if-screen
   <hash>` when even display-only content must remain identical. A mismatch
   returns the fresh screen — that *is* your re-read.
6. **Confirm.** `ui wait --text/--rid/--pkg` blocks until the expected state and
   echoes the matched element; a timeout returns `top_texts` so you see what the
   screen became instead of guessing.
7. **Watch when timing matters.** `olef watch` streams screen diffs,
   crashes, ANRs, toasts, watcher actions, debugger logpoint hits, and
   (with `net` running) HTTP events on one JSONL timeline.
8. **Triage failures with one read.** After any surprise, `olef why`
   returns a verdict + evidence; `olef log --last 5m` gives the structured
   logcat behind it. You rarely need both plus a screenshot — start with `why`.
9. **Go deeper only when needed.** `olef debug ...` (the standalone
   Java/Kotlin and C/C++ debugger as JSON; `debug attach --mode
   java|native|mixed`) and `olef layout ...` (Compose semantics/source/
   recompositions) when UI polling can't answer *why*. See
   [debugging.md](https://olef.dev/docs/concepts/debugging).
10. **Free the slot for instrumentation.** `olef test -- <cmd>` (or
    `disconnect` first) before Espresso / UI Automator runs — Android allows one
    `UiAutomation` owner at a time.

## Reading the screen

A typical agent reads `ui dump` once, acts by `--rid`/`--text`, and caches
strict `content_hash` and actionable `interaction_hash` together with their
versions. Legacy `screen_hash` remains the unprefixed strict identity for
backward compatibility. A hash is comparable only within the same version;
invalidate it when the version changes. `content_hash` changes for any visible
or actionable snapshot detail. `interaction_hash` ignores display-only content
(telemetry, timers, video surfaces) and a slider's current value, but includes
actionable hierarchy, bounds, stable selectors, enabled/selected/checked state,
range shape, and supported actions. Text participates when it is the control's
only selector.

`snapshot_state`, `captured_at_ms`, `current_app.sampled_at_ms`, and `ui_tree`
make lifecycle freshness explicit. Do not derive an action from a
`transitioning` snapshot; retry or wait for the expected package/activity.

Every `ui dump` also reports `accessibility_completeness`. UIAutomator cannot
prove that custom-drawn or unexported Compose controls are represented, so the
normal tree is explicitly marked `unverified` instead of looking deceptively
complete. For a debuggable app, ask for the deeper comparison (it attaches
the Olef runtime and reads the app's own Compose inspector):

```bash
olef ui dump --deep
olef ui tap --fallback-id cs:12345 --if-screen <screen-hash>
```

Missing Compose nodes are returned under `fallback.elements` with an id,
bounds/tap point, `source` (`compose_semantics` or `compose_layout`), confidence,
and `stable_selector:false`. High-confidence `cs:` semantics results can be
targeted directly by fallback id. Layout-only `cl:` results (including custom
drawing without exported semantics) require both `--coordinate-fallback` and a
same-snapshot `--if-screen` guard. OCR is not run automatically; direct `X Y`
taps remain an explicit coordinate target and support the same screen guard.

## Selectors

Selectors are consistent across commands: `--text`, `--rid` (resource id),
`--desc` (content description), and `--xpath`.

Text/desc selectors match as a **normalized, case-insensitive substring** by
default: before comparing, surrounding whitespace is collapsed, curly
quotes/apostrophes/ellipsis are folded to ASCII, and zero-width characters are
stripped — so `--text "sign in"` matches a `SIGN IN` button and `--text "Don't
allow"` matches text rendered with a typographic apostrophe. Add `--exact` (on
`ui find`/`tap`/`text`/`wait`/`focus`) to require a full match (so `--text
Allow` won't hit a label reading "Allow notifications…"), and `--clickable` to skip
non-clickable matches instead of resolving their clickable ancestor. `--rid` is
the most reliable target when a stable resource id exists. Matching is
**literal** — `*`, `.`, `?` and other symbols match themselves, with no
wildcards or regex (a value starting with `-` needs the `--text=-50%` equals
form so it isn't read as a flag).

Selector **actions** are **strict**: if `ui tap`/`text`/`focus` matches several
elements and none is an exact match, they fail with a structured
`ambiguous_match` error listing the candidates rather than guessing — narrow
with `--exact`, `--rid`, or `--clickable`. On a hit, `ui tap`/`wait`/`focus`
echo back the matched element so you can confirm the right node was targeted.

## Element handles

Each actionable element also has a screen-bound `handle` such as
`i:6b4f20feab9812c3/e:2`. Prefer a stable `--rid` or `--desc` plus
`--if-interaction`; use `--handle` when no stable selector exists. Handles are
accepted by `ui tap`, `ui set-progress`, and `ui text`. They are resolved to the
fresh numeric element id only after validating their embedded interaction hash,
so stale navigation/recomposition or reused numeric ids fail as `stale_element`
without delivering input. Agent-facing `next_actions` follow the same order:
stable selector first, handle second, and numeric ids only as a strictly
`--if-screen`-guarded compatibility fallback.

## Acting

Selector taps are semantic by default. A non-clickable match resolves to its
nearest enabled clickable ancestor and reports both `matched_element` and
`activated_element`; a disabled target fails with `element_disabled`, and a
label with no safe ancestor fails with `element_not_clickable`. Raw center
injection requires explicit `--coordinate-fallback` (or direct `X Y`
coordinates). Tap results separate `selector_matched`, `actionable_resolved`,
`input_delivered`, `screen_changed` (with `--observe`), and
`postcondition_satisfied`, so a valid no-op action is not confused with an
undelivered input.

Loop-fusion action verbs (`ui tap`, `ui set-progress`, coordinate gestures,
`ui pinch`, `ui text`, `ui key`, `ui back`, and `ui home`) accept `--observe`
(wait for a 500 ms accessibility-event quiet period, then return the stable
compact screen), `--if-screen <hash>` (strict optimistic concurrency), and
`--if-interaction <hash>` (ignore display-only volatility while guarding
actionable structure). The device server re-captures and validates that state
in the same guarded request that resolves the target and injects input. A
server too old to support this contract fails the distinct guarded route before
injecting anything. Both refuse changed state and return the fresh screen.

Compose layout output uses transformed render bounds when the inspector supplies
them, retaining the original `layout_bounds` and a `bounds_source`. Untransformed
layout bounds can include prefetched pages or oversized semantic modifiers.
Fallback confidence describes its source, not touch reachability; invalid tap
coordinates are rejected before injection.

Compact dumps cap each text/description preview at 1,024 characters and include
`text_truncated`/`desc_truncated` plus the original character count when capped.
Use `ui dump --full` for complete values. Selectors and screen hashes still use
the original text; truncated descriptions are never suggested as exact selectors.

`ui text` keeps `value` as the requested input (`value_source: requested`).
`observed_value` and `text_matches_requested` report a bounded accessibility
readback, so app filters and Unicode changes are visible. Missing/password
readbacks and older servers report verification as unavailable instead of
claiming success. A mismatch does not retry the input.

The quiet interval starts with this observation, even if the device was already
idle. Stability describes accessibility events and consistent screen identity;
it does not prove that a network request or an indeterminate loader finished.
Use a destination postcondition to verify application readiness.

A single `--expect-text`, `--expect-desc`, `--expect-rid`, `--expect-package`,
or `--expect-activity` postcondition implies observation; `--expect-exact`,
`--observe-delay-ms`, and `--timeout-ms` refine matching and timing. An unmet
condition fails with `postcondition_timeout`; a screen that never settles fails
with `observation_unstable`. Both preserve the freshest screen under
`detail.screen` as diagnostic evidence only, so start a new interaction cycle
instead of reusing its element ids. Both are `retryable: false` with
`input_delivered: true`: observe what the action did before acting again,
because repeating it can toggle a switch back or submit a form twice.

`ui wait` also syncs on the foreground app, not just elements: `--pkg <package>`
blocks until that app reaches the foreground (e.g. a Custom Tab or share sheet
opened), and `--pkg-not <package>` blocks until the screen leaves it.

## One command per step

A test step is one command that waits for its target, acts, and checks the
result, with no `sleep`, `jq` or second command. The verbs that act on one
element (`ui tap`, `ui text`, `ui set-progress`, `ui long-tap`,
`ui double-tap`, `ui swipe`) share these flags:

| Flag | What it does |
| --- | --- |
| `--wait-ms N` | Retries while the target is absent, ambiguous, disabled, covered or off screen. Those refusals happen on the device before any input, so a retry never acts twice; anything that may have been delivered is never retried. The result carries `target_wait` (`attempts`, `waited_ms`, `waited_for`). Default: one attempt. |
| `--reveal` | Before acting, if the target isn't visible: waits for the screen to settle, hides the keyboard, scrolls the nearest list down then up (12 swipes at most); a keyboard covering the target is hidden too. Reports `reveal`. Needs `--rid`, `--text` or `--desc`. |
| `--expect-checked [BOOL]`, `--expect-selected [BOOL]`, `--expect-enabled [BOOL]` | A postcondition on the target itself, read back by its selector on a stable screen (like `--expect-text`, and exclusive with the other `--expect-*`). Checked is read from the nearest checkable node around or inside the target. A failure says what the state was; Compose reports selectable chips as checked, and `--expect-selected` says so. |

`ui long-tap`, `ui double-tap` and `ui swipe` take `--rid/--text/--desc/--xpath/
--id/--handle` as well as coordinates. The element must be unique, enabled
and not covered (`element_disabled`, `element_occluded`, `element_offscreen`,
before any input); the touch lands on its centre, or on a clear point when part
of it is covered. An element swipe needs `--direction` (the finger's
direction) and travels `--scale` of the element (default 0.6).

`ui tap --via touch` sends a real finger tap at the element's visible point
instead of an accessibility click, for views that react to touches but not to
clicks; the result's `action` is `touch`.

`ui wait --checked/--selected/--enabled BOOL`, `--value N` (a range within half
a step) and `--has-text TEXT` add conditions to the selector's element.
`--has-text` matches what the element shows: its text, its description, and
the text drawn inside its bounds. A timeout reports the state the element had
in `detail.target`.

`ui text --value-env NAME` types the value of an environment variable. The
output names the variable instead of echoing the value, omits the readback,
and every JSON string the command prints has the value replaced by
`[secret:NAME]`. A missing variable fails with `value_env_missing` before
anything is typed.

`perm dialog allow-foreground|allow-once|deny` answers the runtime-permission
dialog on screen by meaning, whatever its button ids on this Android version:
it waits for the dialog (`--wait-ms`, default 5000), taps the answer, and waits
for the dialog to close or the next one to replace it (`next_dialog`). A dialog
without that answer fails with `dialog_choice_unavailable` and the answers it
offers. `perm grant` skips the dialog; this is for flows that must go through
it.

`samples/shadowdroid-test-app/tests/olef/mission_recovery/test.sh` is a
whole flow written this way.

## Sliders and ranges

Platform and Compose sliders expose their accessibility `range` (`type`,
`min`, `max`, `current`, and nullable `step`) plus stable `actions` in the
normal `ui dump` shape. Android's range API does not expose a declared discrete
step, so Olef leaves `step:null` unless a future platform surface can
prove it. Set a slider semantically and verify the resulting range readback:

```bash
olef ui set-progress --desc "Follow stand-off slider" --value 0.45 --observe
olef ui set-progress --rid distance_slider --percent 80 --if-interaction <interaction-hash>
olef ui set-progress --handle <handle> --percent 50 --observe
```

Out-of-range values fail unless `--clamp` is explicit. Missing range semantics
or `ACTION_SET_PROGRESS` is a typed error; `--coordinate-fallback` opts into an
approximate track click and reports when readback could not verify it.

## Entering a PIN without echoing it

Numeric system keypads can be driven without returning the secret in action
JSON: `olef ui pin "$DEVICE_PIN" --if-interaction <interaction-hash>`.
The command preflights every required digit before entering anything, activates
each label through its clickable ancestor, and presses Enter by default;
`--no-submit` leaves the digits entered for controlled validation. Its output
contains only a digit count and delivery/submission state. Unlike ordinary
loop-fusion actions, `ui pin` intentionally exposes only the pre-action
`--if-screen` / `--if-interaction` guards — not `--observe` or postconditions,
whose returned screen could reveal an unmasked keypad value. The PIN is still
present in the local process invocation, so use the host's normal shell-history
and process-visibility protections. When either guard is present, every digit
and the optional Enter revalidate the original interaction identity inside the
on-device action request; partial-entry failures remain PIN-redacted.

## Android TV / leanback

Android TV is focus + D-pad driven, not touch driven: `/v1/state` reports
`is_television: true`, each element carries a `focused` flag, and
`ui focus --text/--rid/--desc [--center]` walks the D-pad to a selector (then
optionally activates it) — the TV analog of `ui tap` / `ui scroll-to`. Prefer
it (and `ui key dpad_*`) over coordinate taps there.

## The live timeline

`watch` is the streaming workhorse — it emits debounced, hash-diffed `screen`
events plus `crash`, `toast`, `watcher_fired`, structured `logpoint` hits
from attached debug sessions (Java or native), and `http` events when a
`net` proxy is running (plus a `tls_error` when an app rejects the proxy CA,
so a failed interception is visible instead of just missing), and `fault`
events when a fault is injected, cleared, expired or rolled back. Logpoint
hits follow debug sessions as they attach and detach; an interrupted logpoint
stream or a missing proxy produces a structured warning while the other
producers continue; use `--no-logpoints` or `--no-net` when that source is
intentionally absent. `watch` holds the device
while it runs, so inject faults through its stdin
(`{"cmd":"fault","args":["inject","airplane-mode","--duration-ms","5000"]}`);
it also clears faults whose `--duration-ms` elapses while it watches.

## Faults on purpose

`fault` injects failure conditions and undoes them: process death, config
changes, Doze, low battery, full storage, clock jumps, lost or flaky network,
failing backend calls, calls/SMS and emulator crashes. Every state fault
journals its restore plan before it changes the device and verifies it took
effect; `fault clear` verifies each restore; `disconnect` clears whatever is
left. `commands --guide faults` has the details.

## The command surface

| Group | Commands |
| --- | --- |
| **Discovery/setup** | `commands --json --depth 1`, `commands --json --describe '<path>'`, `commands --guide '<topic>'`, `config paths` / `schema` / `explain` / `init` / `validate`, `skill`, `init`, `update`, `usage` |
| **Session/diagnostics** | `devices`, `connect`, `disconnect`, `test`, `doctor`, `collect`, `why`, `log` |
| **UI automation** | `ui dump`, `ui audit`, `ui gen`, `ui screenshot`, `ui find`, `ui tap`, `ui set-progress`, `ui double-tap`, `ui long-tap`, `ui swipe`, `ui drag`, `ui swipe-ext`, `ui pinch`, `ui scroll-to`, `ui focus`, `ui text`, `ui pin`, `ui key`, `ui hide-keyboard`, `ui back`, `ui home`, `ui wait`, `ui toast` (action verbs take `--observe`, `--if-screen`, and `--if-interaction`; tap/progress/text/long-tap/double-tap/swipe also accept `--handle`, `--wait-ms` and `--reveal`) |
| **Triage** | `why` (one-read verdict + evidence), `log` (structured app-scoped logcat + parsed crashes) |
| **Live timeline** | `watch` (screen changes, crashes, ANRs, toasts, watcher actions, debugger logpoint hits, and HTTP events when network capture is active) |
| **Screen video** | `video record -o DIR [--duration 30s]`; detached `video start -o DIR`, `video status`, `video mark LABEL`, `video stop` |
| **Layout / Compose** | `layout snapshot`, `layout diff`, `layout source`, `layout recompositions` |
| **Debugger** | `debug auto`, `snapshot`, `record`, `replay`, `status`, `sessions`, `clients`, `attach`, `break`, `breakpoints`, `logpoint add` / `list` / `events` / `follow` / `remove` / `clear`, `pause`, `resume`, `step-in`, `step-over`, `step-out`, `stop`, `detach`, `stack`, `threads`, `variables`, `eval`, `inspect`, `coroutines`, `continue-until`, `watch`, `step-until-screen-change`, `step-until-log`, `run-until-crash`, `native`, `tombstones` (`attach --mode java\|native\|mixed`; `--engine java\|native` picks the engine when a process has both) |
| **App lifecycle/state** | `app start`, `stop`, `install`, `reinstall`, `clear`, `info`, `wait`, `current`; `app state snapshot`, `restore`, `recover`, `cleanup` |
| **Permissions/app-ops** | `perm grant`, `revoke`, `list`, `reset`, `dialog`; `appops get`, `set` |
| **Device/system** | `device info`, `shell`, `wake`, `sleep`, `unlock`, `orientation`, `clipboard`, `notifications`, `quick-settings`, `open-url` |
| **Display profile** | `profile snapshot`, `apply`, `reset` (animations, font, density, size, rotation) |
| **Files** | `files ls`, `push`, `pull` (add `--run-as --app <pkg>` for private debuggable-app paths) |
| **Network MITM** | `net check`, `trust`, `ca import/info/reset`, `start`, `stop`, `status`, `log`, `checkpoint`, `show`, `export`, `intercept`, `resume`, `drop`, `respond`, `rule`, `rules`, `replay` |
| **Runtime** | `runtime attach` (`--startup`, `--restart`, `--coroutine-probes`), `status`, `detach`, `coroutines` |
| **Authoring/testing helpers** | `ui audit` (selector gaps), `ui gen` (Screen Object scaffold), `net export fixtures` (replayable response set + `manifest.json`, GraphQL keyed by operationName), `test` (instrumentation command with the slot freed), `debug replay --repeat --diff` (flake hunting) |

Run `olef commands --json --depth 1` for the live catalog,
`commands --json --describe '<path>'` for one command, or `--help` for a human
view.

## Related contracts

- [output-contract.md](https://olef.dev/docs/concepts/output-contract) — envelopes, exit codes, `events`,
  streaming.
- [device-state.md](https://olef.dev/docs/concepts/device-state) — app/device/permission/file/profile/video
  controls and their verified postconditions.
- [network.md](https://olef.dev/docs/concepts/network) — HTTP(S) and WebSocket capture, interception,
  rules, and replay.
- [debugging.md](https://olef.dev/docs/concepts/debugging) — `why`/`log`/`collect`, the Java/Kotlin,
  native and mixed debugger, logpoints, and layout inspection.

### Concurrent command batches

Check every command's exit status; the last successful shell command does not
make earlier failures successful. Serialize device lifecycle changes. For a
short, expected contention window, `--lock-timeout-ms 2000` waits up to two
seconds **per lifecycle lock acquisition** before returning retryable
`device_lifecycle_busy` with `waited_ms`. The default is 2000 ms; pass zero for immediate failure. The
maximum is 30000 ms. Named-AVD resolution takes this lock even for read commands. Waiting never removes a lock file or takes ownership from
another process. Long-running command orchestration should still set its own
overall deadline.

Use `set -o pipefail` around formatting pipelines and `&&` between dependent
commands. Read `commands --guide workflow` for the copyable checked Python
invocation and guarded UI recovery recipes. After a guard rejection, require a
fresh consistent snapshot and reselect the target before applying a fresh guard.
An action with delivered or unknown input must be observed before retrying.