fix(ios-qa): SwiftUI Button tap on iOS 17 + Documents/ boot token

Two infra fixes surfaced by a real onboarding session on iPhone SE 2020 /
iOS 17.7 with USB-direct CoreDevice tunnel.

1. Bridges.swift.template — MutationBridgeImpl.handleTap now tries
   accessibilityActivate() on the smallest button-trait AX element
   containing the tap point BEFORE falling back to DebugBridgeTouch's
   UITouch synthesis. SwiftUI Buttons on iOS 16/17 don't route through
   _UIHitTestContext (iOS 18+ only), so the original UITouch path missed
   them entirely — /tap returned {"op":"tap","ok":true} but no
   Button.action fired, screen unchanged. Walks both accessibilityElements
   (eager array) and the indexed protocol to cover SwiftUI hosting
   controllers and UIView-backed AX elements. Falls through to UITouch
   for gesture recognizers / scroll views / custom hit-test-only views.

2. StateServer.swift.template + tunnel-bootstrap.ts — bootToken file
   now lives in Documents/ instead of NSTemporaryDirectory(). iOS clears
   tmp/ aggressively (memory pressure, jetsam, between launches), so the
   daemon's `devicectl copy from tmp/gstack-ios-qa.token` returned ENOENT
   and bootstrap failed with `boot_token_unavailable` even though
   StateServer was alive and answering /healthz. Documents/ survives
   backgrounding + memory pressure and is reachable via the same
   appDataContainer domain identifier. Both ends updated to stay
   consistent (StateServer writes Documents/, tunnel-bootstrap defaults
   reads Documents/).

Repro env: macOS 14.x, Xcode 16.2, iPhone SE 2020 (iPhone12,8) iOS 17.7,
USB-direct CoreDevice tunnel, gstack v1.43.3.0 → v1.48.0.0 (issue exists
in both).

Tests: 91/91 daemon tests pass. DebugBridge SPM (swift 5.9+) builds clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
dwidoo 2026-05-28 00:52:05 +02:00
parent a6fb31726c
commit 56c8f523a8
3 changed files with 121 additions and 9 deletions

View File

@ -64,7 +64,12 @@ export type BootstrapErrorReason =
*/
export async function bootstrapTunnel(opts: BootstrapOptions): Promise<BootstrapResult> {
const port = opts.port ?? 9999;
const tokenPath = opts.bootTokenPath ?? 'tmp/gstack-ios-qa.token';
// Default Documents/ instead of tmp/. iOS clears tmp/ on memory
// pressure / between launches → `devicectl copy from tmp/...` returns
// ENOENT → bootstrap fails with `boot_token_unavailable` even though
// StateServer is alive. Documents/ persists across backgrounding.
// Pair with StateServer.swift.template `bootTokenPath`.
const tokenPath = opts.bootTokenPath ?? 'Documents/gstack-ios-qa.token';
const startupTimeoutMs = opts.startupTimeoutMs ?? 5_000;
const spawn = opts.spawnImpl;
const resolve = opts.resolveImpl;

View File

@ -210,21 +210,117 @@ enum MutationBridgeImpl {
}
}
/// Tap at (x, y) in window coordinates. Delegates to DebugBridgeTouch
/// (KIF-derived in-process touch synthesis). The Obj-C target builds a
/// real UITouch + IOHIDEvent + UIEvent and dispatches via
/// `UIApplication.sendEvent`, which is what UIKit uses for real touches.
/// This works for UIControl, SwiftUI Button (via iOS 18+
/// `_UIHitTestContext`), gesture recognizers, and anything else that
/// listens to the real event-dispatch path.
/// Tap at (x, y) in window coordinates.
///
/// Two-stage dispatch:
///
/// 1. **Accessibility activation** — walk the AX tree, find the smallest
/// element whose frame contains the tap point AND carries the
/// `.button` trait, then call `accessibilityActivate()`. This routes
/// through SwiftUI / UIKit's standard VoiceOver activation path,
/// which fires `Button.action` regardless of iOS version. Required
/// for SwiftUI Button hit-testing on iOS 16/17 where
/// `_UIHitTestContext` is unavailable (DebugBridgeTouch's
/// `sendTapAtPoint:` synthesizes a real UITouch but iOS 16/17 won't
/// route it into the SwiftUI hosting controller's gesture system).
///
/// 2. **UITouch synthesis fallback** — if the AX path doesn't match
/// (gesture recognizers, UIScrollView pan, custom hit-test-only
/// views, drag gestures, anything without an explicit button trait),
/// delegate to `DebugBridgeTouch.sendTap`. The Obj-C target builds
/// a real UITouch + IOHIDEvent + UIEvent and dispatches via
/// `UIApplication.sendEvent` — what UIKit uses for real touches.
/// Covers UIControl, SwiftUI Button on iOS 18+ (via
/// `_UIHitTestContext`), gesture recognizers, and anything else
/// that listens to the real event-dispatch path.
private static func handleTap(_ payload: JSONDict) -> Bool {
guard let x = payload["x"] as? NSNumber,
let y = payload["y"] as? NSNumber else { return false }
let point = CGPoint(x: x.doubleValue, y: y.doubleValue)
guard let scene = activeScene(), let window = activeKeyWindow(in: scene) else { return false }
// Stage 1: AX activation (SwiftUI-friendly, iOS 16+ universal).
if let element = findActivatableAXElement(at: point, in: window),
element.accessibilityActivate() {
return true
}
// Stage 2: UITouch synthesis fallback.
return DebugBridgeTouch.sendTap(at: point, in: window)
}
/// Walk the accessibility hierarchy from `window` and return the smallest
/// element whose accessibility frame contains `point` AND carries the
/// `.button` trait. "Smallest containing" mirrors how UIKit hit-testing
/// resolves nested taps — the deepest interactive element wins. Returns
/// nil if no button-trait element matches (caller falls through to
/// UITouch synthesis).
///
/// Both UIView-backed AX elements (`view.isAccessibilityElement`) and
/// synthetic NSObject AX elements (vended by SwiftUI hosting controllers
/// via `accessibilityElements` / `accessibilityElementCount`) are
/// considered. SwiftUI Buttons typically surface as the latter on iOS
/// 16/17.
private static func findActivatableAXElement(at point: CGPoint, in window: UIWindow) -> NSObject? {
var best: NSObject? = nil
var bestArea: CGFloat = .infinity
func consider(frame: CGRect, traits: UInt, element: NSObject) {
guard frame.contains(point) else { return }
guard (traits & UIAccessibilityTraits.button.rawValue) != 0 else { return }
let area = frame.width * frame.height
if area < bestArea {
best = element
bestArea = area
}
}
func visit(_ view: UIView) {
// Early-out: subtree can't contain the point.
let frame = view.convert(view.bounds, to: nil)
guard frame.contains(point) else { return }
guard !view.isHidden, view.alpha >= 0.01 else { return }
if view.isAccessibilityElement {
consider(frame: frame, traits: view.accessibilityTraits.rawValue, element: view)
}
// SwiftUI hosting controllers vend their elements lazily; force
// population by reading the count first, then prefer
// `accessibilityElements` (eager array) over the indexed
// protocol when it returns non-nil.
_ = view.accessibilityElementCount()
if let elements = view.accessibilityElements {
for case let element as NSObject in elements {
if let v = element as? UIView {
visit(v)
} else {
let f = (element.value(forKey: "accessibilityFrame") as? CGRect) ?? .zero
let t = (element.value(forKey: "accessibilityTraits") as? NSNumber)?.uintValue ?? 0
consider(frame: f, traits: t, element: element)
}
}
} else {
let count = view.accessibilityElementCount()
for i in 0..<count {
guard let element = view.accessibilityElement(at: i) as? NSObject else { continue }
if let v = element as? UIView {
visit(v)
} else {
let f = (element.value(forKey: "accessibilityFrame") as? CGRect) ?? .zero
let t = (element.value(forKey: "accessibilityTraits") as? NSNumber)?.uintValue ?? 0
consider(frame: f, traits: t, element: element)
}
}
}
for sub in view.subviews { visit(sub) }
}
visit(window)
return best
}
/// Set text on the first responder if it's a UITextField or UITextView.
private static func handleType(_ payload: JSONDict) -> Bool {
guard let text = payload["text"] as? String else { return false }

View File

@ -86,7 +86,18 @@ public final class StateServer {
private init(port: UInt16 = 9999) {
self.port = port
self.bootToken = UUID().uuidString
self.bootTokenPath = NSTemporaryDirectory() + "gstack-ios-qa.token"
// Persist in Documents/ rather than NSTemporaryDirectory().
// iOS clears tmp/ aggressively (memory pressure, between launches,
// jetsam pressure when app is backgrounded for any meaningful time).
// The daemon's `devicectl copy from tmp/gstack-ios-qa.token` then
// returns ENOENT and bootstrap fails with `boot_token_unavailable`
// — even though StateServer is alive and responding to /healthz.
// Documents/ survives backgrounding + memory pressure and is
// reachable via the same appDataContainer domain identifier.
// Pair with `tunnel-bootstrap.ts` default tokenPath.
let docs = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first
self.bootTokenPath = (docs?.appendingPathComponent("gstack-ios-qa.token").path)
?? (NSTemporaryDirectory() + "gstack-ios-qa.token")
}
public func start() {