Earn 14 free days when your bug report or suggestion is accepted — how it works

Recording & replay

The agent does this for you. When the User Rule fires on a UI change, your agent calls codeloop_start_recording, drives the relevant flow, stops the recording, and the dashboard renders the video and key frames. You don't script anything. This page exists to explain what you'll see in the dashboard and how to drive recording manually if you want to.

Static screenshots prove the layout. Recording proves the behaviour. For interactive flows — login, checkout, drag-and-drop, multi-step wizards — CodeLoop records video of the running app while the agent drives it, extracts ~15 motion-validated key frames, and replays them in the dashboard alongside the correlated app log lines. The video is what stops the agent from claiming “I clicked Submit” when it never actually did.

The flow

codeloop_start_recording               (begin video capture)
   ↓
codeloop_interact (× many)             (agent drives the UI)
   ↓
codeloop_stop_recording                (finalize MP4)
   ↓
codeloop_interaction_replay            (extract key frames + correlate logs)
   ↓
artifacts/runs/<id>/videos/...         (MP4 + frames + replay metadata)
   ↓
gate.workflow_enforcement              (UI projects MUST have a recording)

Start a recording

# the agent calls this; you'd rarely call it manually
{ "tool": "codeloop_start_recording",
  "params": { "label": "checkout-flow", "viewport": "desktop" } }

Recording runs through ffmpeg, with per-OS capture backends:

  • macOS — avfoundation, defaults to the active display.
  • Linux — x11grab (XWayland on Wayland sessions).
  • Windows — gdigrab.
  • Android emulator — adb screenrecord.
  • iOS Simulator — xcrun simctl io booted recordVideo.

Drive interactions

While a recording is open, the agent calls codeloop_interact for every action. CodeLoop ships ~40 actions across all platforms:

FamilyExamples
Pointerclick, double_click, right_click, hover, drag
Keyboardtype, press (single key), key_combo (e.g. Cmd+S)
Formfill, select_option, check, upload_file
Navigationnavigate, back, reload, scroll
Touch (mobile)tap, swipe, long_press, pinch
Waitwait_for_selector, wait_for_url, wait_ms

Each interactcall returns the matched selector, the timestamp inside the recording, and the screenshot taken immediately after the action. That mapping is what powers the dashboard's scrubber.

Stop and validate

{ "tool": "codeloop_stop_recording" }
// → { ok: true, video_path, duration_ms, frame_count }

At stop time CodeLoop runs a quick motion check — if the recorded bytes are below a per-OS minimum or every frame hashes identically, the recording is rejected as “agent claimed it interacted but nothing moved.” This is the guardrail that catches screenshot-only fakes.

Replay (interaction_replay)

codeloop_interaction_replay post-processes the recording:

  • Extracts ~15 motion-extracted key frames sampled at the points of greatest visual change.
  • Aligns app log lines (from the build / dev server) to the timestamps of each frame.
  • Writes a replay.json describing every action, the frame index, and any failure observed.
{ "tool": "codeloop_interaction_replay",
  "params": { "video_path": "artifacts/.../checkout-flow.mp4" } }

What you see in the dashboard

Open the local dashboard, click a run, then the Interactions tab. You get:

  • The MP4 with a scrubber aligned to each interactcall — click an action and the video jumps to the moment it happened.
  • The 15 key frames as a strip below the video.
  • The correlated app log lines pinned to the frame timestamp — useful for “the click was registered but the request never fired” debugging.
  • The full replay.json as a downloadable artifact for attaching to bug reports.

Workflow enforcement

For UI projects, workflow enforcement requires a recording before codeloop_gate_check can return ready_for_review. If the agent tried to gate without recording, it gets back:

{
  "passed": false,
  "missing": ["codeloop_start_recording", "codeloop_stop_recording"],
  "next_calls": [
    { "tool": "codeloop_start_recording", "params": { "label": "main-flow" } },
    "...drive at least 3 codeloop_interact calls...",
    { "tool": "codeloop_stop_recording" },
    { "tool": "codeloop_interaction_replay", "params": { "video_path": "..." } }
  ]
}

This is what makes “agent claimed done” trustworthy — without an actual video plus matched frames, the gate refuses to signal ready.

OS prerequisites

Run npx codeloop doctor— the recording section prints a green check or a one-line install command for whatever is missing. Most common installs:

# macOS — ffmpeg + grant Screen Recording permission to the terminal/IDE
brew install ffmpeg

# Ubuntu / Debian
sudo apt-get install -y ffmpeg xdotool xwininfo

# Windows (PowerShell as admin)
winget install Gyan.FFmpeg

Common gotchas

  • 0-byte MP4 on macOS. Grant Screen Recording and Accessibility permission to your terminal / IDE under System Settings » Privacy & Security.
  • Ubuntu Wayland sessions. ffmpeg x11grab requires XWayland. Run xrandrfrom a Wayland session — if it errors, log in via the “Ubuntu on Xorg” option for recording.
  • Windows codecs. If the MP4 fails to play in the dashboard, install Gyan's ffmpeg full build which bundles libx264.
  • Recording rejected as static.The motion check requires > 1 % pixel difference between frame samples. Animate something visible (a focus state, a route change) and the check will pass.

Related