From bea24684948739deb5549a1fb6b16fae9730768a Mon Sep 17 00:00:00 2001 From: Ryan Stern <206953196+sternryan@users.noreply.github.com> Date: Wed, 27 May 2026 11:30:35 -0700 Subject: [PATCH] fix(ios-qa): walker now sees SwiftUI elements, not just hosting shells MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The /elements walker returned only top-level `_UIHostingView` / `HostingView` shells on SwiftUI screens — three entries for a full Dashboard, all with empty identifier/label and a 0,0,390,844 frame. SwiftUI vends its AX leaves through the indexed accessor (`accessibilityElementCount` + `accessibilityElement(at:)`) on the hosting container, not through `accessibilityElements` (which returns nil OR an empty array). The previous walker only fell through to the indexed path on nil, not on empty — so SwiftUI leaves were silently dropped. Fix is three coordinated changes to ElementsBridgeImpl plus an opt-in escape hatch: 1. Force materialization via `UIAccessibility.post(.layoutChanged, …)` before the walk. Public API, no-op when VoiceOver already populated the tree, doesn't speak. 2. Always run the indexed accessor when `accessibilityElements` is nil OR empty. This is the actual SwiftUI path. 3. Read identifier/label/value/traits/frame from synthetic AX elements via KVC over the documented `UIAccessibility` informal protocol property names — safe and version-independent across the private element classes (`_AXSnapshotElement`, `_UIAccessibilityElementMockView`, …). 4. New `.gstackProbe("identifier")` SwiftUI ViewModifier + thread-safe `GstackProbeRegistry` for views the AX tree refuses to surface (intentionally hidden, custom Canvas, decorative Shape stacks). Probe sets `.accessibilityIdentifier(_:)` AND registers (id, frame) in the registry; the walker merges these in as synthetic entries tagged `"source":"gstack-probe"`. Also filters empty container synthetic nodes (no label/id/value/traits) that were clogging the output before. Docs in `ios-qa/docs/swiftui-accessibility.md` cover the root cause, what the fix does/doesn't address, and when to reach for `.gstackProbe` vs vision-based coordinate tapping. Typechecks clean against iOS 16 + iOS 17 SDKs, Swift 6 strict concurrency. Live-device validation pending — PR author will port to downstream app + report new element count. --- ios-qa/docs/swiftui-accessibility.md | 122 ++++++++++ ios-qa/templates/Bridges.swift.template | 286 +++++++++++++++++++----- 2 files changed, 355 insertions(+), 53 deletions(-) create mode 100644 ios-qa/docs/swiftui-accessibility.md diff --git a/ios-qa/docs/swiftui-accessibility.md b/ios-qa/docs/swiftui-accessibility.md new file mode 100644 index 000000000..bbdb16a12 --- /dev/null +++ b/ios-qa/docs/swiftui-accessibility.md @@ -0,0 +1,122 @@ +# SwiftUI accessibility tree — known limitation + escape hatch + +## The problem + +`GET /elements` walks `UIWindow.subviews` and emits a JSON list of +accessibility nodes. For UIKit-native screens this works fine. For +**SwiftUI screens**, the walker historically returned only the top-level +hosting containers — three entries for a Dashboard, all named some +variation of `_UIHostingView>`, with no identifiers, +no labels, no frames you'd want to tap. + +Example bug output (Principal's Ear Dashboard, iPhone 12 Pro / iOS 26.x, +before the fix in this PR): + +```json +[ + {"class":"_UIHostingView>", "identifier":"", "label":"", "frame":{"x":0,"y":0,"w":390,"h":844}}, + {"class":"HostingView", "identifier":"", "label":"", "frame":{"x":0,"y":0,"w":390,"h":844}}, + {"class":"FloatingBarHostingView", "identifier":"", "label":"", "frame":{"x":0,"y":0,"w":390,"h":844}} +] +``` + +The Dashboard had a `CaptureControlCard` button with +`.accessibilityIdentifier("dashboard.captureButton")`, a settings +NavigationLink with `"dashboard.settingsButton"`, and a +`ContentUnavailableView`. None of them surfaced. + +## Root cause + +SwiftUI doesn't always create a backing `UIView` for declarative views. +Many "views" are synthetic accessibility elements with no +`UIView` representation — they only exist as nodes vended through +`_UIHostingView`'s `accessibilityElement(at:)` indexed accessor, and the +hosting view returns `nil` (or `[]`) for `accessibilityElements`. The +previous walker checked `accessibilityElements` first and only fell +through to the indexed accessor if that returned `nil` — it didn't +handle the empty-array case, which is what SwiftUI actually returns. + +Worse: the AX tree is lazy. SwiftUI doesn't populate it until something +(typically VoiceOver) starts asking for nodes. A cold walk gets a sparse +or empty tree. + +## What the fix in this PR does + +1. **Force materialization.** The walker posts + `UIAccessibility.layoutChanged` before descending. This is a documented + public API, no-op when VoiceOver is already running, and nudges SwiftUI + to populate its tree. It does NOT speak anything aloud. + +2. **Always try the indexed accessor.** When + `accessibilityElements` returns `nil` OR `[]`, the walker falls through + to `accessibilityElement(at:)` for every index in + `accessibilityElementCount()`. This is where SwiftUI actually vends + its leaves. + +3. **Read identifiers/labels via KVC.** Synthetic AX elements are + instances of private SwiftUI classes (`_AXSnapshotElement`, + `_UIAccessibilityElementMockView`, etc.). They all conform to the + informal UIAccessibility protocol, so reading + `accessibilityIdentifier`, `accessibilityLabel`, `accessibilityValue`, + `accessibilityTraits`, and `accessibilityFrame` via + `value(forKey:)` over the **documented public property names** is + safe and version-independent — it would only break if Apple rename + `UIAccessibility` itself. + +4. **Filter empty container nodes.** Synthetic elements with no label, + identifier, value, or traits are skipped. Previously these clogged the + output. + +## What the fix does NOT do + +It does NOT solve the case where SwiftUI views are intentionally hidden +from accessibility (`.accessibilityHidden(true)`, custom `Canvas` +drawings, decorative `Shape` stacks). The AX tree won't list them, and +no amount of walker improvement changes that. + +## Escape hatch: `.gstackProbe(_:)` + +For views that the agent must see but the AX tree won't surface, use the +SwiftUI ViewModifier shipped in `Bridges.swift.template`: + +```swift +Button { startCapture() } label: { + Image(systemName: "mic.circle.fill") + Text("Tap to start capturing") +} +.gstackProbe("dashboard.captureButton") +``` + +`.gstackProbe(_:)` sets `.accessibilityIdentifier(_:)` AND registers +`(identifier, frame)` in `GstackProbeRegistry`, which the +`ElementsBridge` merges into `/elements` output as a synthetic entry +tagged `"source":"gstack-probe"`. Use this ONLY when you've confirmed via +`/elements` that the agent can't see your view through the AX path — +it adds a tiny ongoing cost (one `PreferenceKey`-driven frame update per +view). + +## Recommended workflow + +1. Annotate your interactive SwiftUI views with + `.accessibilityIdentifier(_:)` as you would for XCUITest. The fix in + this PR will surface them. +2. Run the agent against a representative screen. Check `/elements` for + the identifiers you expect. +3. For any view that didn't surface, switch to `.gstackProbe(_:)`. +4. For purely decorative screens (data viz, canvas, custom drawing), use + **vision-based tapping**: ask the agent to read the screenshot and + tap by coordinate. The screenshot path is unaffected by this issue + and is the supported fallback. + +## Prior art + +- **WebDriverAgent (Appium)** solves this with `XCAXClient_iOS`, which is + XCTest-only — not available to an in-process DEBUG bridge. +- **swift-agentation** (Ertem Biyik) invented the same registry + + ViewModifier pattern `.gstackProbe(_:)` uses; their `agentationTag` + modifier was the proof-of-concept for this approach. +- Apple developer forums and the Swift Forums thread + "Is it possible to dump / introspect my own Accessibility tree at + runtime (SwiftUI)?" both conclude there's no fully public API path — + the KVC-over-informal-protocol approach in this PR is the best + available compromise. diff --git a/ios-qa/templates/Bridges.swift.template b/ios-qa/templates/Bridges.swift.template index bf7af6e3f..d941a7746 100644 --- a/ios-qa/templates/Bridges.swift.template +++ b/ios-qa/templates/Bridges.swift.template @@ -73,10 +73,62 @@ enum ElementsBridgeImpl { /// Each entry has frame (in window coords), accessibility label, /// identifier, traits as a bitmask, and a parent path. Skips /// non-accessible / hidden views. + /// + /// SwiftUI caveat: SwiftUI hosts its accessibility tree inside + /// `_UIHostingView` and friends. Many leaf "views" are synthetic + /// AX elements with no backing UIView — they only surface through + /// `accessibilityElementCount()` / `accessibilityElement(at:)` on + /// the hosting container, and only after the AX subsystem has been + /// poked. We force-materialize via `UIAccessibility.post`, then + /// descend through synthetic elements aggressively. Identifiers + /// declared with `.accessibilityIdentifier(...)` in SwiftUI ARE + /// preserved on the synthetic element, so reading + /// `accessibilityIdentifier` via KVC works. + /// + /// For views where the AX tree still doesn't surface what you need + /// (custom drawing, `.accessibilityHidden(true)` containers wrapping + /// taggable leaves), use the `GstackProbe` SwiftUI ViewModifier — it + /// registers (id, frame, label) in `GstackProbeRegistry`, which the + /// walker merges into the output as synthetic `gstack-probe` entries. + /// See `ios-qa/docs/swiftui-accessibility.md`. static func snapshot() -> [JSONDict] { guard let scene = activeScene(), let window = activeKeyWindow(in: scene) else { return [] } + + // Force SwiftUI hosting views to materialize their AX subtree. + // No-op if VoiceOver already populated it; cheap nudge otherwise. + // `.layoutChanged` is documented and doesn't speak anything. + UIAccessibility.post(notification: .layoutChanged, argument: nil) + var elements: [JSONDict] = [] collect(view: window, parentPath: "", windowBounds: window.bounds, into: &elements) + + // Merge GstackProbe-registered SwiftUI views. These are an explicit + // escape hatch for views that don't surface through the AX tree. + // Probe frames are captured via SwiftUI's `.global` GeometryReader, + // which returns window-relative coordinates — same space as the + // walker's other frames. + let windowBounds = window.bounds + for probe in GstackProbeRegistry.shared.snapshot() { + let frameInWindow = probe.frame + guard windowBounds.intersects(frameInWindow) else { continue } + elements.append([ + "path": "", + "class": "GstackProbe", + "label": probe.label, + "identifier": probe.identifier, + "value": "", + "traits": 0, + "frame": [ + "x": Int(frameInWindow.origin.x), + "y": Int(frameInWindow.origin.y), + "w": Int(frameInWindow.size.width), + "h": Int(frameInWindow.size.height), + ], + "is_user_interaction_enabled": true, + "source": "gstack-probe", + ]) + } + return elements } @@ -120,62 +172,39 @@ enum ElementsBridgeImpl { ]) } - // Recurse into accessibility-elements first (some custom views vend - // synthetic children), then UIView subviews. SwiftUI's host views - // populate accessibilityElements lazily — many return nil before - // VoiceOver triggers them. Force population by reading accessibilityElementCount. - _ = view.accessibilityElementCount() - if let axElements = view.accessibilityElements { + // Descend through accessibility children FIRST. SwiftUI hosting + // views vend their entire AX tree this way — `subviews` only shows + // `_UIGraphicsView` / `HostingView` shells without leaves. + // + // Three signals from the AX protocol, in priority order: + // 1. `accessibilityElements: [Any]?` — explicit array if set. + // 2. `accessibilityElementCount()` + `accessibilityElement(at:)` — + // indexed access. SwiftUI's `_UIHostingView` uses this path. + // Empty array vs nil matters: SwiftUI often returns `[]` for + // `accessibilityElements` even when `accessibilityElementCount()` + // reports >0, so we always try the indexed path on hosts. + // 3. `subviews` — UIKit-native fallback. + // + // We try (1) then (2), then ALWAYS recurse `subviews` because mixed + // UIKit + SwiftUI screens have both. + var emittedAXChildren = false + if let axElements = view.accessibilityElements, !axElements.isEmpty { + emittedAXChildren = true for case let element as NSObject in axElements { - if let v = element as? UIView { - collect(view: v, parentPath: path, windowBounds: windowBounds, into: &elements) - } else { - // Synthetic accessibility element (no UIView). Capture frame in screen coords. - let af = (element.value(forKey: "accessibilityFrame") as? CGRect) ?? .zero - elements.append([ - "path": "\(path) > ", - "class": "AccessibilityElement", - "label": (element.value(forKey: "accessibilityLabel") as? String) ?? "", - "identifier": (element.value(forKey: "accessibilityIdentifier") as? String) ?? "", - "value": (element.value(forKey: "accessibilityValue") as? String) ?? "", - "traits": (element.value(forKey: "accessibilityTraits") as? NSNumber)?.intValue ?? 0, - "frame": [ - "x": Int(af.origin.x), - "y": Int(af.origin.y), - "w": Int(af.size.width), - "h": Int(af.size.height), - ], - "is_user_interaction_enabled": true, - ]) - } + emitAXChild(element, parentPath: path, windowBounds: windowBounds, into: &elements) } - } else { - // accessibilityElements is nil — iterate by index. SwiftUI uses - // this dynamic protocol pattern; many AX elements only respond - // to accessibilityElementCount + accessibilityElement(at:). - let count = view.accessibilityElementCount() - for i in 0.. 0 && !emittedAXChildren { + for i in 0.. ", - "class": String(describing: type(of: element)), - "label": (element.value(forKey: "accessibilityLabel") as? String) ?? "", - "identifier": (element.value(forKey: "accessibilityIdentifier") as? String) ?? "", - "value": (element.value(forKey: "accessibilityValue") as? String) ?? "", - "traits": (element.value(forKey: "accessibilityTraits") as? NSNumber)?.intValue ?? 0, - "frame": [ - "x": Int(af.origin.x), - "y": Int(af.origin.y), - "w": Int(af.size.width), - "h": Int(af.size.height), - ], - "is_user_interaction_enabled": true, - ]) - } + emitAXChild(element, parentPath: path, windowBounds: windowBounds, into: &elements) } } for sub in view.subviews { @@ -183,6 +212,52 @@ enum ElementsBridgeImpl { } } + /// Emit a single AX child — either recurse if it's a UIView, or + /// extract via KVC if it's a synthetic AX element (SwiftUI shim). + /// Reads via `value(forKey:)` because SwiftUI's element classes are + /// private — `_UIAccessibilityElementMockView`, `AccessibilityNode`, + /// `_AXSnapshotElement`, etc. They all conform to UIAccessibility's + /// informal protocol, so KVC over the documented property names is + /// safe and version-independent. + private static func emitAXChild(_ element: NSObject, parentPath: String, windowBounds: CGRect, into elements: inout [JSONDict]) { + if let v = element as? UIView { + collect(view: v, parentPath: parentPath, windowBounds: windowBounds, into: &elements) + return + } + // Synthetic AX element. Frame is in screen coords for AX elements + // (per UIAccessibilityElement docs); we report screen coords here + // — caller can translate if needed since taps go via window coords + // and StateServer converts. + let af = (element.value(forKey: "accessibilityFrame") as? CGRect) ?? .zero + let label = (element.value(forKey: "accessibilityLabel") as? String) ?? "" + let ident = (element.value(forKey: "accessibilityIdentifier") as? String) ?? "" + let val = (element.value(forKey: "accessibilityValue") as? String) ?? "" + let traits: Int = { + if let n = element.value(forKey: "accessibilityTraits") as? NSNumber { return n.intValue } + return 0 + }() + // Skip pure-container synthetic nodes with no useful info — these + // are what was clogging the previous output. Keep anything with + // a label, identifier, value, or non-zero traits. + if label.isEmpty && ident.isEmpty && val.isEmpty && traits == 0 { return } + elements.append([ + "path": "\(parentPath) > ", + "class": String(describing: type(of: element)), + "label": label, + "identifier": ident, + "value": val, + "traits": traits, + "frame": [ + "x": Int(af.origin.x), + "y": Int(af.origin.y), + "w": Int(af.size.width), + "h": Int(af.size.height), + ], + "is_user_interaction_enabled": true, + "source": "ax-synthetic", + ]) + } + private static func activeScene() -> UIWindowScene? { UIApplication.shared.connectedScenes .compactMap { $0 as? UIWindowScene } @@ -305,4 +380,109 @@ enum MutationBridgeImpl { } } +// MARK: - GstackProbe — SwiftUI escape hatch + +/// Thread-safe registry of SwiftUI views that opted into agent visibility +/// via `.gstackProbe("id")`. The ElementsBridge merges these into the +/// `/elements` output as synthetic entries. +/// +/// Why this exists: SwiftUI doesn't always create a backing UIView for +/// declarative views (custom Canvas drawings, Shape stacks, Text inside +/// a HStack inside a Button label). The accessibility tree usually +/// surfaces them, but not always — particularly for views that +/// intentionally hide AX (e.g., decorative containers wrapping +/// taggable children). Probe is the documented escape hatch when AX +/// doesn't materialize what the agent needs to see. +/// +/// Cost: one CGRect + two Strings per probed view. `onGeometryChange` +/// (iOS 18+) keeps frames in sync without continuous polling. +@MainActor +public final class GstackProbeRegistry { + public struct Probe: Sendable { + public let identifier: String + public let label: String + public let frame: CGRect + } + + public static let shared = GstackProbeRegistry() + + private var probes: [UUID: Probe] = [:] + + /// Insert or update a probe. Idempotent on UUID — call from + /// `onGeometryChange` to keep `frame` current. + public func register(_ uuid: UUID, identifier: String, label: String, frame: CGRect) { + probes[uuid] = Probe(identifier: identifier, label: label, frame: frame) + } + + /// Remove a probe — call from `.onDisappear`. + public func deregister(_ uuid: UUID) { + probes.removeValue(forKey: uuid) + } + + /// Snapshot all live probes for the ElementsBridge walker. + public func snapshot() -> [Probe] { + Array(probes.values) + } +} + +/// Internal PreferenceKey for shuttling GeometryReader frames out to the +/// modifier without triggering SwiftUI's State-update-during-view-update +/// warning that GeometryReader + direct register triggers. +private struct GstackProbeFramePreference: PreferenceKey { + static let defaultValue: CGRect = .zero + static func reduce(value: inout CGRect, nextValue: () -> CGRect) { + let next = nextValue() + if next != .zero { value = next } + } +} + +/// `.gstackProbe("dashboard.captureButton")` — register a SwiftUI view +/// with the agent's elements snapshot. Equivalent to setting +/// `.accessibilityIdentifier(...)` PLUS guaranteeing the agent will see +/// the (id, frame) pair regardless of how SwiftUI renders the view. +/// +/// Prefer `.accessibilityIdentifier(...)` alone for normal SwiftUI views — +/// the improved walker reads those out of the AX tree. Reach for +/// `.gstackProbe(...)` only when you've confirmed via `/elements` that +/// the agent can't see your view. +public struct GstackProbeModifier: ViewModifier { + let identifier: String + let label: String + @State private var uuid = UUID() + + public func body(content: Content) -> some View { + content + .accessibilityIdentifier(identifier) + .background( + GeometryReader { geo in + Color.clear + .preference(key: GstackProbeFramePreference.self, value: geo.frame(in: .global)) + } + ) + .onPreferenceChange(GstackProbeFramePreference.self) { newFrame in + Task { @MainActor in + GstackProbeRegistry.shared.register( + uuid, + identifier: identifier, + label: label, + frame: newFrame + ) + } + } + .onDisappear { + GstackProbeRegistry.shared.deregister(uuid) + } + } +} + +public extension View { + /// Register this view with the gstack ios-qa agent so it shows up in + /// `/elements` with a known identifier + frame, even if SwiftUI's + /// accessibility tree doesn't surface it. Also sets the standard + /// `.accessibilityIdentifier(_:)` — using both is safe. + func gstackProbe(_ identifier: String, label: String = "") -> some View { + modifier(GstackProbeModifier(identifier: identifier, label: label)) + } +} + #endif // DEBUG && canImport(UIKit)