doc: ime.md — the IME contract, failure modes, and diagnostics
Written up from the debugging sessions that fixed the IME commit corruption, the scroll slowdown, the mid-word caps, and the distant-commit clobbering: the three-model mental model, the invariants, the stock-gioui structural behaviors the design works around (post-commit selection dedup, restartInput shadowing, focus-gated selection commands, no acks), the log signatures, the test-layer coverage table, and the operational pitfalls hit along the way.
This commit is contained in:
parent
a83cc1a72f
commit
d38a2ee05f
|
|
@ -1,6 +1,6 @@
|
||||||
# Pad documentation
|
# Pad documentation
|
||||||
|
|
||||||
Four documents, kept at the level of *what, why, and invariants* — not
|
Five documents, kept at the level of *what, why, and invariants* — not
|
||||||
line-by-line code — so they stay true as the implementation evolves.
|
line-by-line code — so they stay true as the implementation evolves.
|
||||||
|
|
||||||
| Doc | What it is |
|
| Doc | What it is |
|
||||||
|
|
@ -9,6 +9,7 @@ line-by-line code — so they stay true as the implementation evolves.
|
||||||
| [`architecture.md`](./architecture.md) | How it works: single-owner concurrency model, channel topology, Frame handoff contract, ownership rules, editor/browser/render internals. |
|
| [`architecture.md`](./architecture.md) | How it works: single-owner concurrency model, channel topology, Frame handoff contract, ownership rules, editor/browser/render internals. |
|
||||||
| [`development_plan.md`](./development_plan.md) | The active plan: completed phases, remaining work, and the on-device observation loop. |
|
| [`development_plan.md`](./development_plan.md) | The active plan: completed phases, remaining work, and the on-device observation loop. |
|
||||||
| [`release.md`](./release.md) | The release process (`scripts/release.sh`): the gates, the install-to-all-devices policy, and the rule that on-device profiling/testing is diagnostic and needs explicit approval. |
|
| [`release.md`](./release.md) | The release process (`scripts/release.sh`): the gates, the install-to-all-devices policy, and the rule that on-device profiling/testing is diagnostic and needs explicit approval. |
|
||||||
|
| [`ime.md`](./ime.md) | The IME contract: the three text models, the invariants that keep commits landing where they should, stock-gioui structural behaviors (worked around, not patched), log signatures for diagnosing IME desync, and the test layers and what each can catch. |
|
||||||
|
|
||||||
## Package inventory
|
## Package inventory
|
||||||
|
|
||||||
|
|
|
||||||
166
doc/ime.md
Normal file
166
doc/ime.md
Normal file
|
|
@ -0,0 +1,166 @@
|
||||||
|
# The IME contract and its failure modes
|
||||||
|
|
||||||
|
What the editor relies on when talking to Android's IME (Gboard), what
|
||||||
|
stock gioui does structurally, and the failure modes observed on device —
|
||||||
|
with their log signatures and the invariants that keep the system safe.
|
||||||
|
This is the map for the next time text lands in the wrong place.
|
||||||
|
|
||||||
|
## The three models
|
||||||
|
|
||||||
|
An IME session involves **three independent copies of the text** that must
|
||||||
|
be kept consistent:
|
||||||
|
|
||||||
|
1. **The file** (logic goroutine, the truth).
|
||||||
|
2. **The pushed snippet** — the 32 KB window around the caret that we push
|
||||||
|
to the IME (`key.SnippetCmd`). All IME commit positions are in *this*
|
||||||
|
coordinate space (absolute file runes, `Range.Start` = window start).
|
||||||
|
3. **Gboard's local model** — Gboard's private copy of the snippet plus its
|
||||||
|
own caret/selection/composition state, which it updates from our pushes
|
||||||
|
*and* from its own bookkeeping of the commits it sends.
|
||||||
|
|
||||||
|
Every bug in this area is a loss of consistency between model 3 and model
|
||||||
|
1. The commit that arrives names a range in model 3; applying it to model 1
|
||||||
|
is only correct if the two still agree.
|
||||||
|
|
||||||
|
## Invariants the code must maintain
|
||||||
|
|
||||||
|
- **Commits map against the whole buffer, never a stale window.**
|
||||||
|
`HandleIMECommit` converts the commit's rune range to bytes with a
|
||||||
|
whole-buffer scan (`runeToByteWhole`), so it stays exact while the
|
||||||
|
visible window moves (a fling in flight). A renderer-side IME model was
|
||||||
|
tried and abandoned: it drifts whenever the render window moves
|
||||||
|
(commit `f54ca2f`).
|
||||||
|
- **The snippet window is hysteresis-gated and decoupled from the render
|
||||||
|
window** (`State.computeIMESnippetWindow`): 32 KB, re-anchored only when
|
||||||
|
the caret comes within 4 KB of an edge. Re-anchoring reaches Gboard as a
|
||||||
|
`restartInput`, which starts a fresh input session that re-derives
|
||||||
|
auto-capitalization — a per-scroll or per-tap re-push is what caused the
|
||||||
|
random mid-word caps (commit `dd9493b`).
|
||||||
|
- **The post-commit caret is `Range.Start + len(text)`**, not
|
||||||
|
`Range.End + len(text)`. Identical for insertions; for a replacement
|
||||||
|
(autocorrect over a selection) the old formula lands `(End−Start)` runes
|
||||||
|
past the inserted text and desynchronizes the IME at the exact moment its
|
||||||
|
model is being updated (commit `51344b9`). The logic side
|
||||||
|
(`applyIMECommitBytes`) and the renderer's immediate push
|
||||||
|
(`ApplyIMECommitToModel`) must compute the *same* caret.
|
||||||
|
- **A legitimate commit is always local**: at the caret, inside the live
|
||||||
|
selection, or a correction a few runes left of the caret. The guard
|
||||||
|
(`HandleIMECommit`, `maxCommitDistance = 1024`) snaps anything else to
|
||||||
|
the caret and arms the resync. A 5-rune autocorrect 19,000 runes from
|
||||||
|
the caret is never legitimate; applying it verbatim clobbers distant
|
||||||
|
text (commit `a83cc1a`).
|
||||||
|
- **A desynchronized IME is healed only by a restart.** See the structural
|
||||||
|
fact below: selection pushes after a commit are deduplicated away, so
|
||||||
|
the *only* mechanism that makes Gboard re-read the truth is a snippet
|
||||||
|
change (`restartInput`). `IMEForceResync` arms on the **first**
|
||||||
|
anomaly, not a streak: the next frame ships the snippet trimmed by one
|
||||||
|
rune (one restart), the following frame ships the full window (one more),
|
||||||
|
then quiet. One-shot, so normal typing never pays for it
|
||||||
|
(commits `cb8ebc0`, `a83cc1a`).
|
||||||
|
|
||||||
|
## What stock gioui does structurally (worked around, not patched)
|
||||||
|
|
||||||
|
All of the following is stock v0.10.2 behavior. The fixes live entirely in
|
||||||
|
pad; the fork at `~/gioui` carries **diagnostic logging only** (the
|
||||||
|
`PADIME` lines) and can be dropped for stock gioui at any time without
|
||||||
|
behavioral change.
|
||||||
|
|
||||||
|
1. **Post-commit selection pushes are deduplicated away.** The commit
|
||||||
|
callback (`callbacks.EditorReplace`) advances the window's stored
|
||||||
|
`imeState.Selection` *before* the frame's state comparison, so a
|
||||||
|
`SelectionCmd` pushed in the same frame as a commit is "no change" and
|
||||||
|
`imm.updateSelection` is never called. Consequence: after a commit, the
|
||||||
|
IME's caret can only be corrected by a snippet restart. This is why the
|
||||||
|
resync exists and why "just re-push the selection" is not a fix.
|
||||||
|
2. **`restartInput` shadows `updateSelection`.** In
|
||||||
|
`window.EditorStateChanged`, a snippet change takes an early-return
|
||||||
|
branch: the selection update in the same state change is dropped.
|
||||||
|
Gboard re-fetches the selection on restart, so this is harmless — but
|
||||||
|
it means a restart frame is the *only* frame where selection and text
|
||||||
|
are guaranteed to be re-read together.
|
||||||
|
3. **Selection commands are focus-gated.** `keyQueue.setSelection`
|
||||||
|
silently drops a `SelectionCmd` whose tag is not the current key-focus
|
||||||
|
(`req.Tag != state.focus`). A selection push can vanish with no log and
|
||||||
|
no feedback — the IME never acks selections. If taps stop moving
|
||||||
|
Gboard's caret, check focus first.
|
||||||
|
4. **There is no ack anywhere.** Snippet pushes, selection pushes, and
|
||||||
|
commits are fire-and-forget. The system can only *detect* desync
|
||||||
|
indirectly (a commit landing where no text expects it), which is what
|
||||||
|
the drift guard does.
|
||||||
|
|
||||||
|
## Gboard behaviors observed
|
||||||
|
|
||||||
|
- Gboard keeps a word selection on its own (a tap near a word selects it)
|
||||||
|
and commits autocorrect as a **replacement over its local selection** —
|
||||||
|
the commit range is then as wide as the word, not a point insertion.
|
||||||
|
- While composing, Gboard re-sends the whole word on every keystroke
|
||||||
|
(`"t"`, `"th"`, `"thi"`, … each replacing the previous), and on
|
||||||
|
backspace it re-sends the shrinking word down to `text=""`. Both are
|
||||||
|
normal; the drift guard must not treat them as anomalies (they are at
|
||||||
|
the caret).
|
||||||
|
- When its local model desynchronizes, Gboard can enter an endless
|
||||||
|
empty-fix-up loop (`text=""`, one commit per ~150 ms) trying to
|
||||||
|
reconcile text that is not in its model. The file is never damaged (the
|
||||||
|
guard snaps each one to the caret); the resync breaks the loop.
|
||||||
|
- The phone's AICore Gboard is stricter/more aggressive than the emulator's
|
||||||
|
Gboard. A flow that is clean on the emulator can still desync on device.
|
||||||
|
|
||||||
|
## Log signatures → diagnosis
|
||||||
|
|
||||||
|
Permanent, low-volume lines (see "Diagnostics"):
|
||||||
|
|
||||||
|
| Signature in logcat | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `IME TAP … cursor=N winStart=M` | A tap moved the logic cursor (bytes). Compare with the next `IME COMMIT` ranges (runes) — they should be in the same neighborhood (rune ≈ byte − multibyte count, **compute it, don't assume a constant**). |
|
||||||
|
| `IME COMMIT range=[a,b) text=… -> [x,y)` | The logic applied the commit at bytes [x,y). The pre-`->` range is Gboard's coordinate space; the post-`->` range is where it actually landed. |
|
||||||
|
| `IME DRIFT-SNAP … snap to caret + resync` | Anomaly detected: a commit far from the caret/selection (or a stale small commit) was applied at the caret instead. One = a tap/scroll desync just happened; a run = Gboard was looping. |
|
||||||
|
| `IME PUSH snippet=[a,b) … (restart)` | We re-anchored the IME window → Gboard restarts its input session. Should be rare: app open, file switch, caret crossing a window margin. Frequent restarts = the hysteresis gate is being defeated (check render-window coupling). |
|
||||||
|
| `IME PUSH sel=[s,e)` | We pushed a caret/selection to Gboard. If a tap does not produce one (or it repeats forever), the selection pipeline is broken upstream (focus gate, dedup, float-noise re-emit). |
|
||||||
|
| `PADIME EditorStateChanged sel A -> B` / `PADIME updateSelection sel=[…]` / `PADIME restartInput` | The gio→Gboard boundary: what actually crossed into Android. If `IME PUSH` shows a value the `PADIME` lines never show, the drop is inside gioui (focus gate / dedup); if `PADIME` shows it and Gboard still misbehaves, the IME ignored it. |
|
||||||
|
|
||||||
|
**Capture the log at incident time.** Logcat rotates fast; the incident
|
||||||
|
that started this document was only reconstructable because the capture
|
||||||
|
happened minutes later.
|
||||||
|
|
||||||
|
## Test layers and what each can catch
|
||||||
|
|
||||||
|
| Layer | What it exercises | Catches |
|
||||||
|
|---|---|---|
|
||||||
|
| e2e harness (`internal/test/e2e`) | Logic + renderer headless; commits injected directly (`HandleIMECommit`) | Commit mapping math, window invariants, guard/resync logic, edit/scroll/selection state. **Never** the IME contract: Gboard is not in the loop. |
|
||||||
|
| Emulator stress loops (`/tmp/loop4.sh`, `/tmp/stress.sh`) | Real app + adb touches, real Gboard | Fling/tap/typing interplay, restart storms, caps regressions. Coarse: no boundary visibility, scenario shape matters (warm-session fling-tap-type was the original shape; it missed the cold-restore → far-scroll → tap → fast-typing shape entirely). |
|
||||||
|
| Full-path scenario (`/tmp/s8.sh`) | Cold open with restored cursor → far scroll → tap into a known word → fast Gboard typing → backspaces; asserts tap position vs commit ranges vs **exact file diff** | The desync class: text landing anywhere but the tap. The byte→rune conversion must be computed from the file, never assumed. |
|
||||||
|
|
||||||
|
Rule: **a bug in the IME boundary is invisible to every layer except the
|
||||||
|
last one.** When IME behavior is touched, run S8.
|
||||||
|
|
||||||
|
## Operational pitfalls hit in these sessions
|
||||||
|
|
||||||
|
- The launcher activity is `pad.pad/org.gioui.GioActivity`, not
|
||||||
|
`.MainActivity`. `am start` failures are silent if stderr is redirected —
|
||||||
|
check the output once, trust it after.
|
||||||
|
- `adb install`/build steps fail with "no adb device" when the emulator is
|
||||||
|
mid-reconnect; the APK may be built but not installed. Verify
|
||||||
|
`dumpsys package … lastUpdateTime` after installs.
|
||||||
|
- The "All files access" settings toggle can be a red herring: `appops
|
||||||
|
set <pkg> MANAGE_EXTERNAL_STORAGE allow` is the reliable grant, and
|
||||||
|
`appops get` output has been observed to lag; `dumpsys appops` is
|
||||||
|
authoritative.
|
||||||
|
- `difflib.SequenceMatcher` on ~1 MB strings is minutes slow; use a linear
|
||||||
|
first-diff scan for test assertions on large files.
|
||||||
|
- Float-equality re-emit is a recurring trap (LastLineY, caret px): any
|
||||||
|
"re-emit when changed" comparison on a float sum needs an epsilon gate,
|
||||||
|
or it spins a loop that re-pushes IME state every frame and starves real
|
||||||
|
work.
|
||||||
|
- When a tap's pixel→text mapping is in question, the `IME TAP` line gives
|
||||||
|
both the local point and the resulting cursor — log both, or the mapping
|
||||||
|
bug is undiagnosable from the outside.
|
||||||
|
|
||||||
|
## Diagnostics currently enabled (temporary)
|
||||||
|
|
||||||
|
- Pad-side, permanent: `IME TAP`, `IME COMMIT`, `IME DRIFT-SNAP`,
|
||||||
|
`IME PUSH snippet/sel/commit-range` (bounded: only on change).
|
||||||
|
- Fork-side, **temporary** (in `~/gioui`, gated by the `replace` in
|
||||||
|
`go.mod`): `PADIME` boundary logs in `GioView.updateSelection`,
|
||||||
|
`GioView.restartInput`, and `window.EditorStateChanged`. Remove these,
|
||||||
|
drop the `replace`, `go mod tidy` — behavior is identical on stock
|
||||||
|
gioui. Keep until the informal on-device testing is settled.
|
||||||
Loading…
Reference in New Issue
Block a user