fix(ios-qa): walker now sees SwiftUI elements, not just hosting shells

The /elements walker returned only top-level `_UIHostingView` /
`HostingView` shells on SwiftUI screens — three entries for a full
Dashboard, all with empty identifier/label and a 0,0,390,844 frame.
SwiftUI vends its AX leaves through the indexed accessor
(`accessibilityElementCount` + `accessibilityElement(at:)`) on the
hosting container, not through `accessibilityElements` (which returns
nil OR an empty array). The previous walker only fell through to the
indexed path on nil, not on empty — so SwiftUI leaves were silently
dropped.

Fix is three coordinated changes to ElementsBridgeImpl plus an opt-in
escape hatch:

1. Force materialization via `UIAccessibility.post(.layoutChanged, …)`
   before the walk. Public API, no-op when VoiceOver already populated
   the tree, doesn't speak.
2. Always run the indexed accessor when `accessibilityElements` is nil
   OR empty. This is the actual SwiftUI path.
3. Read identifier/label/value/traits/frame from synthetic AX elements
   via KVC over the documented `UIAccessibility` informal protocol
   property names — safe and version-independent across the private
   element classes (`_AXSnapshotElement`, `_UIAccessibilityElementMockView`, …).
4. New `.gstackProbe("identifier")` SwiftUI ViewModifier + thread-safe
   `GstackProbeRegistry` for views the AX tree refuses to surface
   (intentionally hidden, custom Canvas, decorative Shape stacks). Probe
   sets `.accessibilityIdentifier(_:)` AND registers (id, frame) in the
   registry; the walker merges these in as synthetic entries tagged
   `"source":"gstack-probe"`.

Also filters empty container synthetic nodes (no label/id/value/traits)
that were clogging the output before.

Docs in `ios-qa/docs/swiftui-accessibility.md` cover the root cause,
what the fix does/doesn't address, and when to reach for `.gstackProbe`
vs vision-based coordinate tapping.

Typechecks clean against iOS 16 + iOS 17 SDKs, Swift 6 strict
concurrency. Live-device validation pending — PR author will port to
downstream app + report new element count.
This commit is contained in:
Ryan Stern 2026-05-27 11:30:35 -07:00
parent a6fb31726c
commit bea2468494
2 changed files with 355 additions and 53 deletions

View File

@ -0,0 +1,122 @@
# SwiftUI accessibility tree — known limitation + escape hatch
## The problem
`GET /elements` walks `UIWindow.subviews` and emits a JSON list of
accessibility nodes. For UIKit-native screens this works fine. For
**SwiftUI screens**, the walker historically returned only the top-level
hosting containers — three entries for a Dashboard, all named some
variation of `_UIHostingView<ModifiedContent<…>>`, with no identifiers,
no labels, no frames you'd want to tap.
Example bug output (Principal's Ear Dashboard, iPhone 12 Pro / iOS 26.x,
before the fix in this PR):
```json
[
{"class":"_UIHostingView<ModifiedContent<AnyView, >>", "identifier":"", "label":"", "frame":{"x":0,"y":0,"w":390,"h":844}},
{"class":"HostingView", "identifier":"", "label":"", "frame":{"x":0,"y":0,"w":390,"h":844}},
{"class":"FloatingBarHostingView<FloatingBarContainer>", "identifier":"", "label":"", "frame":{"x":0,"y":0,"w":390,"h":844}}
]
```
The Dashboard had a `CaptureControlCard` button with
`.accessibilityIdentifier("dashboard.captureButton")`, a settings
NavigationLink with `"dashboard.settingsButton"`, and a
`ContentUnavailableView`. None of them surfaced.
## Root cause
SwiftUI doesn't always create a backing `UIView` for declarative views.
Many "views" are synthetic accessibility elements with no
`UIView` representation — they only exist as nodes vended through
`_UIHostingView`'s `accessibilityElement(at:)` indexed accessor, and the
hosting view returns `nil` (or `[]`) for `accessibilityElements`. The
previous walker checked `accessibilityElements` first and only fell
through to the indexed accessor if that returned `nil` — it didn't
handle the empty-array case, which is what SwiftUI actually returns.
Worse: the AX tree is lazy. SwiftUI doesn't populate it until something
(typically VoiceOver) starts asking for nodes. A cold walk gets a sparse
or empty tree.
## What the fix in this PR does
1. **Force materialization.** The walker posts
`UIAccessibility.layoutChanged` before descending. This is a documented
public API, no-op when VoiceOver is already running, and nudges SwiftUI
to populate its tree. It does NOT speak anything aloud.
2. **Always try the indexed accessor.** When
`accessibilityElements` returns `nil` OR `[]`, the walker falls through
to `accessibilityElement(at:)` for every index in
`accessibilityElementCount()`. This is where SwiftUI actually vends
its leaves.
3. **Read identifiers/labels via KVC.** Synthetic AX elements are
instances of private SwiftUI classes (`_AXSnapshotElement`,
`_UIAccessibilityElementMockView`, etc.). They all conform to the
informal UIAccessibility protocol, so reading
`accessibilityIdentifier`, `accessibilityLabel`, `accessibilityValue`,
`accessibilityTraits`, and `accessibilityFrame` via
`value(forKey:)` over the **documented public property names** is
safe and version-independent — it would only break if Apple rename
`UIAccessibility` itself.
4. **Filter empty container nodes.** Synthetic elements with no label,
identifier, value, or traits are skipped. Previously these clogged the
output.
## What the fix does NOT do
It does NOT solve the case where SwiftUI views are intentionally hidden
from accessibility (`.accessibilityHidden(true)`, custom `Canvas`
drawings, decorative `Shape` stacks). The AX tree won't list them, and
no amount of walker improvement changes that.
## Escape hatch: `.gstackProbe(_:)`
For views that the agent must see but the AX tree won't surface, use the
SwiftUI ViewModifier shipped in `Bridges.swift.template`:
```swift
Button { startCapture() } label: {
Image(systemName: "mic.circle.fill")
Text("Tap to start capturing")
}
.gstackProbe("dashboard.captureButton")
```
`.gstackProbe(_:)` sets `.accessibilityIdentifier(_:)` AND registers
`(identifier, frame)` in `GstackProbeRegistry`, which the
`ElementsBridge` merges into `/elements` output as a synthetic entry
tagged `"source":"gstack-probe"`. Use this ONLY when you've confirmed via
`/elements` that the agent can't see your view through the AX path —
it adds a tiny ongoing cost (one `PreferenceKey`-driven frame update per
view).
## Recommended workflow
1. Annotate your interactive SwiftUI views with
`.accessibilityIdentifier(_:)` as you would for XCUITest. The fix in
this PR will surface them.
2. Run the agent against a representative screen. Check `/elements` for
the identifiers you expect.
3. For any view that didn't surface, switch to `.gstackProbe(_:)`.
4. For purely decorative screens (data viz, canvas, custom drawing), use
**vision-based tapping**: ask the agent to read the screenshot and
tap by coordinate. The screenshot path is unaffected by this issue
and is the supported fallback.
## Prior art
- **WebDriverAgent (Appium)** solves this with `XCAXClient_iOS`, which is
XCTest-only — not available to an in-process DEBUG bridge.
- **swift-agentation** (Ertem Biyik) invented the same registry +
ViewModifier pattern `.gstackProbe(_:)` uses; their `agentationTag`
modifier was the proof-of-concept for this approach.
- Apple developer forums and the Swift Forums thread
"Is it possible to dump / introspect my own Accessibility tree at
runtime (SwiftUI)?" both conclude there's no fully public API path —
the KVC-over-informal-protocol approach in this PR is the best
available compromise.

View File

@ -73,10 +73,62 @@ enum ElementsBridgeImpl {
/// Each entry has frame (in window coords), accessibility label,
/// identifier, traits as a bitmask, and a parent path. Skips
/// non-accessible / hidden views.
///
/// SwiftUI caveat: SwiftUI hosts its accessibility tree inside
/// `_UIHostingView` and friends. Many leaf "views" are synthetic
/// AX elements with no backing UIView — they only surface through
/// `accessibilityElementCount()` / `accessibilityElement(at:)` on
/// the hosting container, and only after the AX subsystem has been
/// poked. We force-materialize via `UIAccessibility.post`, then
/// descend through synthetic elements aggressively. Identifiers
/// declared with `.accessibilityIdentifier(...)` in SwiftUI ARE
/// preserved on the synthetic element, so reading
/// `accessibilityIdentifier` via KVC works.
///
/// For views where the AX tree still doesn't surface what you need
/// (custom drawing, `.accessibilityHidden(true)` containers wrapping
/// taggable leaves), use the `GstackProbe` SwiftUI ViewModifier — it
/// registers (id, frame, label) in `GstackProbeRegistry`, which the
/// walker merges into the output as synthetic `gstack-probe` entries.
/// See `ios-qa/docs/swiftui-accessibility.md`.
static func snapshot() -> [JSONDict] {
guard let scene = activeScene(), let window = activeKeyWindow(in: scene) else { return [] }
// Force SwiftUI hosting views to materialize their AX subtree.
// No-op if VoiceOver already populated it; cheap nudge otherwise.
// `.layoutChanged` is documented and doesn't speak anything.
UIAccessibility.post(notification: .layoutChanged, argument: nil)
var elements: [JSONDict] = []
collect(view: window, parentPath: "", windowBounds: window.bounds, into: &elements)
// Merge GstackProbe-registered SwiftUI views. These are an explicit
// escape hatch for views that don't surface through the AX tree.
// Probe frames are captured via SwiftUI's `.global` GeometryReader,
// which returns window-relative coordinates — same space as the
// walker's other frames.
let windowBounds = window.bounds
for probe in GstackProbeRegistry.shared.snapshot() {
let frameInWindow = probe.frame
guard windowBounds.intersects(frameInWindow) else { continue }
elements.append([
"path": "<gstack-probe>",
"class": "GstackProbe",
"label": probe.label,
"identifier": probe.identifier,
"value": "",
"traits": 0,
"frame": [
"x": Int(frameInWindow.origin.x),
"y": Int(frameInWindow.origin.y),
"w": Int(frameInWindow.size.width),
"h": Int(frameInWindow.size.height),
],
"is_user_interaction_enabled": true,
"source": "gstack-probe",
])
}
return elements
}
@ -120,62 +172,39 @@ enum ElementsBridgeImpl {
])
}
// Recurse into accessibility-elements first (some custom views vend
// synthetic children), then UIView subviews. SwiftUI's host views
// populate accessibilityElements lazily — many return nil before
// VoiceOver triggers them. Force population by reading accessibilityElementCount.
_ = view.accessibilityElementCount()
if let axElements = view.accessibilityElements {
// Descend through accessibility children FIRST. SwiftUI hosting
// views vend their entire AX tree this way — `subviews` only shows
// `_UIGraphicsView` / `HostingView` shells without leaves.
//
// Three signals from the AX protocol, in priority order:
// 1. `accessibilityElements: [Any]?` — explicit array if set.
// 2. `accessibilityElementCount()` + `accessibilityElement(at:)` —
// indexed access. SwiftUI's `_UIHostingView` uses this path.
// Empty array vs nil matters: SwiftUI often returns `[]` for
// `accessibilityElements` even when `accessibilityElementCount()`
// reports >0, so we always try the indexed path on hosts.
// 3. `subviews` — UIKit-native fallback.
//
// We try (1) then (2), then ALWAYS recurse `subviews` because mixed
// UIKit + SwiftUI screens have both.
var emittedAXChildren = false
if let axElements = view.accessibilityElements, !axElements.isEmpty {
emittedAXChildren = true
for case let element as NSObject in axElements {
if let v = element as? UIView {
collect(view: v, parentPath: path, windowBounds: windowBounds, into: &elements)
} else {
// Synthetic accessibility element (no UIView). Capture frame in screen coords.
let af = (element.value(forKey: "accessibilityFrame") as? CGRect) ?? .zero
elements.append([
"path": "\(path) > <synthetic>",
"class": "AccessibilityElement",
"label": (element.value(forKey: "accessibilityLabel") as? String) ?? "",
"identifier": (element.value(forKey: "accessibilityIdentifier") as? String) ?? "",
"value": (element.value(forKey: "accessibilityValue") as? String) ?? "",
"traits": (element.value(forKey: "accessibilityTraits") as? NSNumber)?.intValue ?? 0,
"frame": [
"x": Int(af.origin.x),
"y": Int(af.origin.y),
"w": Int(af.size.width),
"h": Int(af.size.height),
],
"is_user_interaction_enabled": true,
])
}
emitAXChild(element, parentPath: path, windowBounds: windowBounds, into: &elements)
}
} else {
// accessibilityElements is nil — iterate by index. SwiftUI uses
// this dynamic protocol pattern; many AX elements only respond
// to accessibilityElementCount + accessibilityElement(at:).
let count = view.accessibilityElementCount()
for i in 0..<count {
}
// ALSO try the indexed path. SwiftUI hosting views often expose
// children only here, with `accessibilityElements` returning nil
// or an empty array. Safe because UIKit-native containers either
// return 0 here or vend the same elements as `.accessibilityElements`
// (dedup by object identity below would be ideal, but the cost of
// a few duplicates is acceptable vs missing every SwiftUI element).
let axCount = view.accessibilityElementCount()
if axCount > 0 && !emittedAXChildren {
for i in 0..<axCount {
guard let element = view.accessibilityElement(at: i) as? NSObject else { continue }
if let v = element as? UIView {
collect(view: v, parentPath: path, windowBounds: windowBounds, into: &elements)
} else {
let af = (element.value(forKey: "accessibilityFrame") as? CGRect) ?? .zero
elements.append([
"path": "\(path) > <ax\(i)>",
"class": String(describing: type(of: element)),
"label": (element.value(forKey: "accessibilityLabel") as? String) ?? "",
"identifier": (element.value(forKey: "accessibilityIdentifier") as? String) ?? "",
"value": (element.value(forKey: "accessibilityValue") as? String) ?? "",
"traits": (element.value(forKey: "accessibilityTraits") as? NSNumber)?.intValue ?? 0,
"frame": [
"x": Int(af.origin.x),
"y": Int(af.origin.y),
"w": Int(af.size.width),
"h": Int(af.size.height),
],
"is_user_interaction_enabled": true,
])
}
emitAXChild(element, parentPath: path, windowBounds: windowBounds, into: &elements)
}
}
for sub in view.subviews {
@ -183,6 +212,52 @@ enum ElementsBridgeImpl {
}
}
/// Emit a single AX child — either recurse if it's a UIView, or
/// extract via KVC if it's a synthetic AX element (SwiftUI shim).
/// Reads via `value(forKey:)` because SwiftUI's element classes are
/// private — `_UIAccessibilityElementMockView`, `AccessibilityNode`,
/// `_AXSnapshotElement`, etc. They all conform to UIAccessibility's
/// informal protocol, so KVC over the documented property names is
/// safe and version-independent.
private static func emitAXChild(_ element: NSObject, parentPath: String, windowBounds: CGRect, into elements: inout [JSONDict]) {
if let v = element as? UIView {
collect(view: v, parentPath: parentPath, windowBounds: windowBounds, into: &elements)
return
}
// Synthetic AX element. Frame is in screen coords for AX elements
// (per UIAccessibilityElement docs); we report screen coords here
// — caller can translate if needed since taps go via window coords
// and StateServer converts.
let af = (element.value(forKey: "accessibilityFrame") as? CGRect) ?? .zero
let label = (element.value(forKey: "accessibilityLabel") as? String) ?? ""
let ident = (element.value(forKey: "accessibilityIdentifier") as? String) ?? ""
let val = (element.value(forKey: "accessibilityValue") as? String) ?? ""
let traits: Int = {
if let n = element.value(forKey: "accessibilityTraits") as? NSNumber { return n.intValue }
return 0
}()
// Skip pure-container synthetic nodes with no useful info — these
// are what was clogging the previous output. Keep anything with
// a label, identifier, value, or non-zero traits.
if label.isEmpty && ident.isEmpty && val.isEmpty && traits == 0 { return }
elements.append([
"path": "\(parentPath) > <ax>",
"class": String(describing: type(of: element)),
"label": label,
"identifier": ident,
"value": val,
"traits": traits,
"frame": [
"x": Int(af.origin.x),
"y": Int(af.origin.y),
"w": Int(af.size.width),
"h": Int(af.size.height),
],
"is_user_interaction_enabled": true,
"source": "ax-synthetic",
])
}
private static func activeScene() -> UIWindowScene? {
UIApplication.shared.connectedScenes
.compactMap { $0 as? UIWindowScene }
@ -305,4 +380,109 @@ enum MutationBridgeImpl {
}
}
// MARK: - GstackProbe — SwiftUI escape hatch
/// Thread-safe registry of SwiftUI views that opted into agent visibility
/// via `.gstackProbe("id")`. The ElementsBridge merges these into the
/// `/elements` output as synthetic entries.
///
/// Why this exists: SwiftUI doesn't always create a backing UIView for
/// declarative views (custom Canvas drawings, Shape stacks, Text inside
/// a HStack inside a Button label). The accessibility tree usually
/// surfaces them, but not always — particularly for views that
/// intentionally hide AX (e.g., decorative containers wrapping
/// taggable children). Probe is the documented escape hatch when AX
/// doesn't materialize what the agent needs to see.
///
/// Cost: one CGRect + two Strings per probed view. `onGeometryChange`
/// (iOS 18+) keeps frames in sync without continuous polling.
@MainActor
public final class GstackProbeRegistry {
public struct Probe: Sendable {
public let identifier: String
public let label: String
public let frame: CGRect
}
public static let shared = GstackProbeRegistry()
private var probes: [UUID: Probe] = [:]
/// Insert or update a probe. Idempotent on UUID — call from
/// `onGeometryChange` to keep `frame` current.
public func register(_ uuid: UUID, identifier: String, label: String, frame: CGRect) {
probes[uuid] = Probe(identifier: identifier, label: label, frame: frame)
}
/// Remove a probe — call from `.onDisappear`.
public func deregister(_ uuid: UUID) {
probes.removeValue(forKey: uuid)
}
/// Snapshot all live probes for the ElementsBridge walker.
public func snapshot() -> [Probe] {
Array(probes.values)
}
}
/// Internal PreferenceKey for shuttling GeometryReader frames out to the
/// modifier without triggering SwiftUI's State-update-during-view-update
/// warning that GeometryReader + direct register triggers.
private struct GstackProbeFramePreference: PreferenceKey {
static let defaultValue: CGRect = .zero
static func reduce(value: inout CGRect, nextValue: () -> CGRect) {
let next = nextValue()
if next != .zero { value = next }
}
}
/// `.gstackProbe("dashboard.captureButton")` — register a SwiftUI view
/// with the agent's elements snapshot. Equivalent to setting
/// `.accessibilityIdentifier(...)` PLUS guaranteeing the agent will see
/// the (id, frame) pair regardless of how SwiftUI renders the view.
///
/// Prefer `.accessibilityIdentifier(...)` alone for normal SwiftUI views —
/// the improved walker reads those out of the AX tree. Reach for
/// `.gstackProbe(...)` only when you've confirmed via `/elements` that
/// the agent can't see your view.
public struct GstackProbeModifier: ViewModifier {
let identifier: String
let label: String
@State private var uuid = UUID()
public func body(content: Content) -> some View {
content
.accessibilityIdentifier(identifier)
.background(
GeometryReader { geo in
Color.clear
.preference(key: GstackProbeFramePreference.self, value: geo.frame(in: .global))
}
)
.onPreferenceChange(GstackProbeFramePreference.self) { newFrame in
Task { @MainActor in
GstackProbeRegistry.shared.register(
uuid,
identifier: identifier,
label: label,
frame: newFrame
)
}
}
.onDisappear {
GstackProbeRegistry.shared.deregister(uuid)
}
}
}
public extension View {
/// Register this view with the gstack ios-qa agent so it shows up in
/// `/elements` with a known identifier + frame, even if SwiftUI's
/// accessibility tree doesn't surface it. Also sets the standard
/// `.accessibilityIdentifier(_:)` — using both is safe.
func gstackProbe(_ identifier: String, label: String = "") -> some View {
modifier(GstackProbeModifier(identifier: identifier, label: label))
}
}
#endif // DEBUG && canImport(UIKit)