Agent e2e verification — Wayland, browser, computer-use
Agent e2e verification — Wayland, browser, computer-use
Section titled “Agent e2e verification — Wayland, browser, computer-use”Related: Cloud agent orchestrator · 2026-09-28 Pi SDK sessions worktrees and TS coordinator
Recommended host session
Section titled “Recommended host session”Default: headless sway + grim (screenshot) + wtype (keys) + wlrctl (pointer). Optional wayvnc (human peek), cage (single-app kiosk).
WLR_BACKENDS=headless WLR_LIBINPUT_NO_DEVICES=1 sway -c minimal.conf &# discover WAYLAND_DISPLAY — do not hardcode wayland-1WAYLAND_DISPLAY=… grim /tmp/shot.pngWAYLAND_DISPLAY=… wtype "…" # or wlrctl pointer move|click| Host | Verdict |
|---|---|
| sway | Default — headless + IPC + grim/wtype proven |
| labwc | Good alt; LABWC_FALLBACK_OUTPUT=NOOP-fallback; often wayland-0 |
| cage | Single-app isolation; watch headless virtual-pointer timing |
| weston | Fine nested/CI (-Bheadless / -Bvnc); not wayvnc-compatible |
| mutter / gnome-kiosk | GNOME fidelity only; heavy D-Bus — optional profile |
| gamescope | Skip unless gaming DUT |
| wayvnc | wlroots-only RFB; not GNOME/KDE/Weston |
Browser / computer-use layers (hybrid)
Section titled “Browser / computer-use layers (hybrid)”- L1 — Playwright
chromium-headless-shell(nixpkgsplaywright-driver) for deterministic flows. - L2 — browser-use / Stagehand as skills over the same CDP session.
- L3 — Anthropic computer-use toolset executor against Wayland (or Xvfb); OpenAI CUA as optional skill.
- Cursor-style computerUse — same plane: PNG → multimodal →
wtype/wlrctl(not ydotool/uinput).
Capture: grim for screenshots; wf-recorder for traces; skip scrot (X11). Prefer grim over portal on wlroots headless.
In-core vs host vs skills
Section titled “In-core vs host vs skills”| Concern | In-core | Nix host flake | Skills |
|---|---|---|---|
| Session lifecycle | spawn/teardown, discover display, stale-socket cleanup | sway/labwc/cage | — |
| Action API | screenshot / click / type / key / scroll; ordered batches; halt-on-fail | grim, wtype, wlrctl | — |
| Coord scaling | model pixels ↔ capture; image limits | — | model resize hints |
| Model adapters | Anthropic/OpenAI tool_result shaping | — | prompts, confirm policy |
| Browser | thin CDP/Playwright connect + artifacts | playwright headless-shell only | browser-use, Stagehand |
| Observability | step log + PNG/WebM under run id | wf-recorder, ffmpeg | exporters |
| Remote peek | optional RFB metadata | wayvnc | — |
| Domain e2e | assertion hooks | — | app/site evals |
Rule: Core = contract. Host = binaries. Skills = LLM exploration + domain. Keep full Chromium / browser-use out of the default core closure (~400–650 MB headless-shell vs ~2.2 GB full set).
Maturity risks
Section titled “Maturity risks”- Stale
wayland-*sockets after hard kill — discover or clean every run. - Headless = pixman; need a DRM profile for GL/Vulkan DUTs.
- wayvnc frames stall if the nested window is covered.
- cage pointer race: create virtual pointer → settle → click.
- Classic computer-use bug: downscale screenshot without inverse-scaling clicks.
- Playwright/Nix skew — flake-pin
playwright, nevernpx. - Prompt injection — vault creds server-side; human confirm for irreversible acts.
- Pure vision loops are expensive — freeze L1 for ~10–30% of LLM steps.
Ship order (for keypoint 4)
Section titled “Ship order (for keypoint 4)”- Flake: sway / grim / wtype / wlrctl + core session supervisor
- Core computer-use executor (Anthropic toolset shape)
- Flake: playwright headless-shell + L1 harness
- Optional: wayvnc, wf-recorder, cage
- Skills: browser-use / Stagehand; mutter only if DUT needs GNOME