Running an iOS Simulator Inside Your Chat: DSH iOS Plugin – Practical Notes and Design Choices

It sounds odd at first: an iOS simulator running inside a web-based chat interface, responding to taps from a browser, streaming live video to a sidebar panel, and even working with a physical iPhone over USB. But after setting it up and running it through real workflows, it turns out to be less of a demo toy and more of a genuine time‑saver for iOS development and testing.

This post covers how the DSH iOS plugin actually works, where it shines, and where you’ll hit clear limits. It’s not a rehash of the README – it’s a hands‑on account from someone who installed it, configured it, and pushed the tools to see what breaks.

What Problem Does It Solve?

Picture this: you’re in a DSH (DeepSeek Harness) conversation and you ask the agent, “Boot an iPhone 17 Pro simulator, open Settings, and take a screenshot.” The sidebar instantly lights up with a live view of the simulator booting, rotating, and responding to taps. You never leave the browser. You don’t manually control the simulator or paste screenshots back into the chat. The agent drives the device, and you watch the result.

To make that work, the plugin must solve several problems at once:

  • Stream the simulator’s display to a web panel in near‑real time.
  • Relay taps, gestures, and keyboard input from the browser back to the simulator.
  • Keep every tool call as a structured card in the conversation history for later reference.
  • Support physical USB‑connected iPhones with the same toolset – otherwise the value for real testing workflows would be halved.

The plugin delivers all of this with 22 tools, a persistent sidebar panel, and two separate backends: serve-sim for simulators and WebDriverAgent for real devices.

Architecture at a Glance: 22 Tools, One Panel, Two Backends

The plugin registers 22 tools with DSH, covering device management, building and running, UI automation, logs, debugging, and SwiftUI previews. All tools return plain JSON – no inline image blocks. Visual data (screenshots, live streams) goes through DSH’s own web server via signed, time‑limited URLs that render in the sidebar panel or as compact cards.

The “iOS Simulator” panel sits on the right side of the chat. It streams an MJPEG feed of the current device. You can tap, drag, rotate, and press the Home button directly on the live view. Toolbar buttons provide Home, screenshot, rotation, and refresh. You can scale the view (fit‑to‑width, 50–125% zoom, or three preset sizes S/M/L) and choose between frameless, bezel, or realistic device frame styles. The panel width is draggable (up to 960px) and double‑click resets it; it auto‑widens when the device rotates to landscape.

The same panel works for both simulators and real iPhones. The backend changes (serve‑sim vs. WebDriverAgent), but the interaction model stays identical – you don’t need to know which device is feeding the stream.

Core Simulator Tools – The Essential Workflow

Start with ios_sim_devices. It lists every available simulator on your Mac (udid, name, runtime, state, boot status) and, separately, any USB‑connected real iPhones (udid, name, osVersion, model, state, developerMode). Use this to discover the udid or name you’ll pass to other tools.

To boot a device, call ios_sim_boot with the udid or name. The stream starts, and the sidebar panel opens automatically. The stream stays alive for the duration of the conversation (subject to idle timeout).

ios_sim_shutdown stops the device and tears down the stream if it matches.

For a quick screenshot, ios_sim_screenshot returns a JSON summary (path, byte size, dimensions, device) and renders the image in the card/panel – never as an inline image block. It works on both streaming simulators and real devices (via WebDriverAgent).

To interact, ios_sim_interact accepts normalized coordinates (0..1) for taps, text input (US keyboard layout for simulators), hardware buttons (home, lock, volumeUp), scrolls, and gestures. After each action, it waits ~300 ms and attaches a fresh screenshot to show the result.

Building and Running – Simulator vs. Real Device

ios_sim_build_run is the workhorse. Give it a path to an .xcodeproj, .xcworkspace, or Swift package, and it builds, installs, and launches the resulting .app. For simulators, it uses xcodebuild + simctl. For a real iPhone udid, it builds on the device (requires Apple Development signing) and installs there.

Full builds take minutes – the tool returns filtered xcodebuild error tails, not the full log dump.

A subtle but useful detail: if you omit the udid, the plugin automatically picks the streaming device, falling back to the first booted simulator, then the latest‑runtime iPhone, and boots it if needed. So you can just say “build and run this project” without specifying a device every time.

Physical iPhone Support – WebDriverAgent and Tunnel Forwarding

Real device support comes through ios_real_start_wda. It builds and launches WebDriverAgent (WDA) on a connected iPhone, then forwards the control (REST) and MJPEG ports through a loopback tunnel. The device must be unlocked.

The first run is a cold build – xcodebuild compiles WDA, which can take several minutes. Once running, WDA stays active; subsequent calls just check if it’s responding and reuse it. Free‑team provisioning profiles expire after 7 days, so you’ll need to re‑run the tool periodically.

If startup fails, the panel status shows a coded reason: device-locked, cert-untrusted, profile-expired, tunnel-failed, or device-unplugged. This is far more helpful than a generic “WDA failed” message.

Once WDA is up, all other tools (screenshot, interact, ui_tree, tap_element, etc.) work against the physical device using the same JSON APIs.

UI Automation – By Identity or By Text

The plugin offers two distinct UI automation paths, depending on the app’s accessibility structure.

Accessibility Tree – Tapping by Element Identity

ios_sim_ui_tree exports the accessibility element tree of the foreground app – labels, identifiers, values, and frames in points, plus screen dimensions. For simulators it uses AXe; for real devices it uses WebDriverAgent.

On real devices, the snapshot depth is limited by default because an unlimited‑depth snapshot on a complex app can take ~32 seconds and produce 751 KB – with a depth limit it drops to ~2 seconds. The output is capped at about 40 KB; deeper levels are truncated with a truncated flag.

ios_sim_tap_element taps an element by identity. It first tries an exact match on identifier or label, then a case‑insensitive substring match. If multiple distinct elements match, it lists candidates instead of guessing. The tap lands at the element’s centre. After ~300 ms it attaches a new screenshot. You can also pass expect_text or expect_gone to combine tap and validation into one round‑trip.

This works well for apps with good accessibility support. But for list‑based or feed‑style apps – where each row is a single cell with a combined label like “57 replies. 18 likes. 592 views” and no tappable sub‑buttons – you need a different approach.

List and Feed Operations – Row‑Level Actions

ios_sim_ui_rows transforms the deep accessibility snapshot into a list of rows instead of a raw tree. Each row includes:

  • a zero‑based index,
  • a frame (in points),
  • the aggregated label,
  • and parsed counters (e.g., 57 replies → replies: 57) – generic heuristics, not app‑specific dictionaries.

On real devices the default max_depth is 60, and each call takes ~15–25 seconds (WDA serialises requests). Off‑screen rows are omitted and counted in omittedOffscreen.

ios_sim_tap_row taps inside a row by relative coordinates (x/y are ratios of the row’s frame, default 0.5 = centre). The critical safety feature is expect_count: you pass a {key, delta} and the tool re‑reads the row’s label after the tap, verifying that the counter changed by exactly +1 or –1. If the key isn’t present in the parsed counters, the tap is rejected before execution. Without expect_count, the tap still runs but without verification.

This design matters because every tap on a real device has real consequences – you never want to “try” a tap to see if it works. The agent can verify the effect before committing.

OCR as a Fallback – When Accessibility Trees Are Empty

When an app has no accessibility information, or text is rendered as graphics (badge numbers, prices embedded in images), you can fall back to on‑device OCR using Vision.

ios_sim_find_text captures a screenshot, runs OCR (zh‑Hans + en‑US), and returns {device, size, items:[{text, confidence, rect}]}. The OCR helper is compiled on first use from an embedded Swift source into ~/Library/Caches/dsh-ios/bin/ocr. The output is capped at ~40 KB; you can narrow results with query or raise min_confidence.

ios_sim_tap_text runs OCR and taps the best matching text – same matching rules as tap_element (exact first, then case‑insensitive substring, ambiguity reports multiple candidates). You can pass expect_text or expect_gone to combine tap and verification.

ios_sim_wait_for polls the OCR pipeline to wait for text to appear or disappear, with a default 8‑second timeout (max 60 seconds). A timeout returns matched:false without throwing an error. Each poll on a real device costs about 1.2 seconds.

Logs and Debugging Tools

ios_sim_logs reads unified logs from the device. Two modes:

  • snapshot – runs log show --last <duration> (default 2 min) and returns the tail.
  • follow – captures live logs for a bounded period (duration_seconds, default 10, max 60) – it never hangs indefinitely.

Output is capped at ~300 lines / 30 KB with a note on how to narrow the filter.

ios_sim_processes lists running app processes – for simulators it queries launchd, for real devices it uses devicectl. ios_sim_backtrace attaches LLDB, grabs a thread backtrace, and detaches (non‑persistent). Output is capped at ~200 lines, main thread first. If macOS developer mode is off (sudo DevToolsSecurity -enable), it falls back to Xcode’s sample tool (which doesn’t suspend the process). Real devices are explicitly rejected with a clear message.

ios_sim_leaks uses Xcode’s leaks tool – either summary (leak count, total bytes, top ~30 types) or memgraph (generates a .memgraph file to open in Instruments). The app is suspended during scanning but resumed after. This only works on simulators. On iOS 26.2 simulators, even with developer mode enabled, leaks may fail with Failed to get DYLD info; the tool degrades gracefully, returns the raw diagnostic, and ensures the target process is resumed – no hang. Try mode: "memgraph" or switch to another runtime as a workaround.

ios_sim_app_info reads an installed app’s bundle path, writable data container, and key Info.plist fields. For simulators it uses simctl appinfo (with a get_app_container fallback); for real devices it uses devicectl. If the app isn’t installed, it returns installed: false and suggests using ios_sim_list_apps to discover bundle IDs.

SwiftUI Preview Hot Reload

ios_sim_preview does something unusual: it builds a minimal host app (in the plugin’s cache, never touching your package), compiles your preview into a dylib, installs the host on a simulator, and launches it. Then it watches your source files – every edit triggers a rebuild and hot‑swaps the preview without restarting the simulator or the host. The whole cycle takes about 2–5 seconds.

If compilation fails, the host keeps the last successful preview; the error tail is available via status. Only one preview session can run at a time. This tool is a major time‑saver for iterative UI work – you don’t restart the simulator or even rebuild the full app to see changes.

Security – How Traffic Is Protected

The browser never connects directly to serve-sim’s port. All traffic goes through DSH’s own web server under /_dsh/dsh-ios/* routes:

  • /stream/<token> – proxies the MJPEG feed.
  • /screenshot/<token> – serves a cached PNG.
  • /ws?token=… – forwards HID control via WebSocket.
  • plus /grant, /capture, and /status endpoints.

Tokens are HMAC‑SHA256 capability credentials (expiring in 10 minutes) signed with a private key unique to each DSH home directory. Every route applies a loopback/trusted‑transport guard: it verifies the peer address, the Host header (to prevent DNS rebinding), and Fetch‑Metadata / Origin headers. Screenshot routes only serve files inside the plugin’s cache directory – symlinks are rejected via realpath checks.

The practical effect: even if a malicious page tries to hit /_dsh/dsh-ios/, it lacks a valid token and the right Host header. Short‑lived tokens also mean URLs can’t be reused indefinitely.

Orphan Process Adoption and Keep‑Alive Strategy

If the DSH host crashes abnormally, its serve-sim child processes may survive. On restart, the plugin adopts those orphans – their handshake information is treated as authoritative. If a leftover process occupies a slot but serves a different device, the plugin uses serve-sim -k to recycle it and retry.

Crash recovery: a stream that crashes is automatically restarted after about 5 seconds. Idle stop: when no consumer is active (panel closed, no mounted cards, no active routes), the stream stops after 5 minutes of inactivity. Manual shutdown overrides auto‑recovery. Real‑device runners are exempt from idle reclamation because restarting them would trigger a lengthy xcodebuild rebuild.

This design handles edge cases gracefully – you don’t end up with orphan streams eating resources, and you don’t have to manually clean up after crashes.

Requirements and Common Setup Pitfalls

  • macOS with full Xcode – Command Line Tools alone won’t suffice; xcodebuild, xcrun simctl, and simulator runtimes come with Xcode.
  • At least one iOS simulator runtime installed in Xcode.
  • DSH ≥ 0.1.0-rc.6 with the Web UI – headless works (tools return JSON) but no visual panel.
  • The plugin bundles serve-sim as an npm dependency.
  • AXe (optional, needed for tree‑based tools): install via brew install cameroncooke/axe/axe or let the plugin auto‑download a pinned version to ~/Library/Caches/dsh-ios/bin.
  • Vision OCR (optional, needed for find_text/tap_text): compiled on first use from embedded assets/ocr.swift using swiftc.
  • macOS developer mode for LLDB attach: run sudo DevToolsSecurity -enable once. Before that, backtrace falls back to sample, and leaks runs with a warning.
  • First WDA build on a real device installs a signed WebDriverAgentRunner – trust the certificate on the device. Free‑team provisioning profiles expire after 7 days, requiring a re‑run of ios_real_start_wda.

Quick Reference – Tool Flow

  1. Install: dsh plugin --profile web add @zseven-w/dsh-ios@latest then dsh web.
  2. Discover devices: ios_sim_devices – note udid.
  3. Boot simulator: ios_sim_boot + udid/name – panel opens.
  4. Build and run a project: ios_sim_build_run + project path – auto‑selects device and boots if needed.
  5. Drive a real iPhone: first run ios_real_start_wda + udid, then use other tools.
  6. UI by identity: ios_sim_ui_tree → ios_sim_tap_element.
  7. List/feed rows: ios_sim_ui_rows → ios_sim_tap_row (prefer expect_count for verification).
  8. OCR fallback: ios_sim_find_text → ios_sim_tap_text.
  9. SwiftUI preview: ios_sim_preview start + package path – edit and watch hot‑reload in ~2‑5s.
  10. Logs: ios_sim_logs (snapshot or follow).
  11. Debug: ios_sim_processes → ios_sim_backtrace → ios_sim_leaks.
  12. Shutdown: ios_sim_shutdown + udid.

FAQ – Common Questions

ios_sim_ui_tree complains about missing AXe?
Install via brew install cameroncooke/axe/axe, or let the plugin auto‑download it. Override with DSH_IOS_AXE_BIN, or disable download with DSH_IOS_AXE_OFFLINE=1.

Why does interaction or screenshot fail on a real iPhone?
Check that ios_real_start_wda has been run successfully and the device is unlocked. The error message usually points to the exact cause (device-locked, cert-untrusted, etc.). If WDA isn’t running, the tools will return a clear error before attempting anything.

ios_sim_ui_rows returns no rows – what’s wrong?
The result explains why: depth too shallow (increase max_depth for real devices; each deeper level adds ~15‑25s), not a list view, or the app genuinely has no accessibility info after a deep scan. Shallow scans are never reported as “no accessibility” – you get a diagnostic.

Preview hot‑reload fails to compile – does it crash the session?
No – the host keeps the last successful preview. The error tail is returned via status. Fix the code and the next edit will recompile and hot‑swap.

The stream stopped by itself – is it a crash?
Most likely it’s the idle policy – 5 minutes of no consumers. The next tool call or opening the panel restarts it. If it was a crash, it auto‑restarts after ~5 seconds.

Why is tapping on a real device so slow?
WebDriverAgent serialises requests – each tap + screenshot round‑trip takes about 1‑2 seconds. This is a WDA limitation, not a plugin issue. Use expect_text or expect_gone to combine tap and validation into one round‑trip and reduce latency.

The DSH iOS plugin turns a chat interface into a functional iOS test harness. It’s not a polished commercial product – it has rough edges (WDA latency, iOS 26.2 leaks quirks, AXe installation steps) – but the design is thoughtfully built around real use cases, not just demo appeal. If you’re already in the DSH ecosystem and do iOS development, it’s worth adding to your toolkit.