Pad/doc/ime.md

# The IME contract and its failure modes

What the editor relies on when talking to Android's IME (Gboard), what
stock gioui does structurally, and the failure modes observed on device —
with their log signatures and the invariants that keep the system safe.
This is the map for the next time text lands in the wrong place.

## The three models

An IME session involves **three independent copies of the text** that must
be kept consistent:

1. **The file** (logic goroutine, the truth).
2. **The pushed snippet** — the 32 KB window around the caret that we push
   to the IME (`key.SnippetCmd`). All IME commit positions are in *this*
   coordinate space (absolute file runes, `Range.Start` = window start).
3. **Gboard's local model** — Gboard's private copy of the snippet plus its
   own caret/selection/composition state, which it updates from our pushes
   *and* from its own bookkeeping of the commits it sends.

Every bug in this area is a loss of consistency between model 3 and model
1. The commit that arrives names a range in model 3; applying it to model 1
is only correct if the two still agree.

## Invariants the code must maintain

- **Commits map against the whole buffer, never a stale window.**
  `HandleIMECommit` converts the commit's rune range to bytes with a
  whole-buffer scan (`runeToByteWhole`), so it stays exact while the
  visible window moves (a fling in flight). A renderer-side IME model was
  tried and abandoned: it drifts whenever the render window moves
  (commit `f54ca2f`).
- **The snippet window is hysteresis-gated and decoupled from the render
  window** (`State.computeIMESnippetWindow`): 32 KB, re-anchored only when
  the caret comes within 4 KB of an edge. Re-anchoring reaches Gboard as a
  `restartInput`, which starts a fresh input session that re-derives
  auto-capitalization — a per-scroll or per-tap re-push is what caused the
  random mid-word caps (commit `dd9493b`).
- **The post-commit caret is `Range.Start + len(text)`**, not
  `Range.End + len(text)`. Identical for insertions; for a replacement
  (autocorrect over a selection) the old formula lands `(End−Start)` runes
  past the inserted text and desynchronizes the IME at the exact moment its
  model is being updated (commit `51344b9`). The logic side
  (`applyIMECommitBytes`) and the renderer's immediate push
  (`ApplyIMECommitToModel`) must compute the *same* caret.
- **A legitimate commit is always local**: at the caret, inside the live
  selection, or a correction a few runes left of the caret. The guard
  (`HandleIMECommit`, `maxCommitDistance = 1024`) snaps anything else to
  the caret and arms the resync. A 5-rune autocorrect 19,000 runes from
  the caret is never legitimate; applying it verbatim clobbers distant
  text (commit `a83cc1a`).
- **A desynchronized IME is healed only by a restart.** See the structural
  fact below: selection pushes after a commit are deduplicated away, so
  the *only* mechanism that makes Gboard re-read the truth is a snippet
  change (`restartInput`). `IMEForceResync` arms on the **first**
  anomaly, not a streak: the next frame ships the snippet trimmed by one
  rune (one restart), the following frame ships the full window (one more),
  then quiet. One-shot, so normal typing never pays for it
  (commits `cb8ebc0`, `a83cc1a`).

## What stock gioui does structurally (worked around, not patched)

All of the following is stock v0.10.2 behavior. The fixes live entirely in
pad; the fork at `~/gioui` carries **diagnostic logging only** (the
`PADIME` lines) and can be dropped for stock gioui at any time without
behavioral change.

1. **Post-commit selection pushes are deduplicated away.** The commit
   callback (`callbacks.EditorReplace`) advances the window's stored
   `imeState.Selection` *before* the frame's state comparison, so a
   `SelectionCmd` pushed in the same frame as a commit is "no change" and
   `imm.updateSelection` is never called. Consequence: after a commit, the
   IME's caret can only be corrected by a snippet restart. This is why the
   resync exists and why "just re-push the selection" is not a fix.
2. **`restartInput` shadows `updateSelection`.** In
   `window.EditorStateChanged`, a snippet change takes an early-return
   branch: the selection update in the same state change is dropped.
   Gboard re-fetches the selection on restart, so this is harmless — but
   it means a restart frame is the *only* frame where selection and text
   are guaranteed to be re-read together.
3. **Selection commands are focus-gated.** `keyQueue.setSelection`
   silently drops a `SelectionCmd` whose tag is not the current key-focus
   (`req.Tag != state.focus`). A selection push can vanish with no log and
   no feedback — the IME never acks selections. If taps stop moving
   Gboard's caret, check focus first.
4. **There is no ack anywhere.** Snippet pushes, selection pushes, and
   commits are fire-and-forget. The system can only *detect* desync
   indirectly (a commit landing where no text expects it), which is what
   the drift guard does.

## Gboard behaviors observed

- Gboard keeps a word selection on its own (a tap near a word selects it)
  and commits autocorrect as a **replacement over its local selection** —
  the commit range is then as wide as the word, not a point insertion.
- While composing, Gboard re-sends the whole word on every keystroke
  (`"t"`, `"th"`, `"thi"`, … each replacing the previous), and on
  backspace it re-sends the shrinking word down to `text=""`. Both are
  normal; the drift guard must not treat them as anomalies (they are at
  the caret).
- When its local model desynchronizes, Gboard can enter an endless
  empty-fix-up loop (`text=""`, one commit per ~150 ms) trying to
  reconcile text that is not in its model. The file is never damaged (the
  guard snaps each one to the caret); the resync breaks the loop.
- The phone's AICore Gboard is stricter/more aggressive than the emulator's
  Gboard. A flow that is clean on the emulator can still desync on device.

## Log signatures → diagnosis

Permanent, low-volume lines (see "Diagnostics"):

| Signature in logcat | Meaning |
|---|---|
| `IME TAP … cursor=N winStart=M` | A tap moved the logic cursor (bytes). Compare with the next `IME COMMIT` ranges (runes) — they should be in the same neighborhood (rune ≈ byte − multibyte count, **compute it, don't assume a constant**). |
| `IME COMMIT range=[a,b) text=… -> [x,y)` | The logic applied the commit at bytes [x,y). The pre-`->` range is Gboard's coordinate space; the post-`->` range is where it actually landed. |
| `IME DRIFT-SNAP … snap to caret + resync` | Anomaly detected: a commit far from the caret/selection (or a stale small commit) was applied at the caret instead. One = a tap/scroll desync just happened; a run = Gboard was looping. |
| `IME PUSH snippet=[a,b) … (restart)` | We re-anchored the IME window → Gboard restarts its input session. Should be rare: app open, file switch, caret crossing a window margin. Frequent restarts = the hysteresis gate is being defeated (check render-window coupling). |
| `IME PUSH sel=[s,e)` | We pushed a caret/selection to Gboard. If a tap does not produce one (or it repeats forever), the selection pipeline is broken upstream (focus gate, dedup, float-noise re-emit). |
| `PADIME EditorStateChanged sel A -> B` / `PADIME updateSelection sel=[…]` / `PADIME restartInput` | The gio→Gboard boundary: what actually crossed into Android. If `IME PUSH` shows a value the `PADIME` lines never show, the drop is inside gioui (focus gate / dedup); if `PADIME` shows it and Gboard still misbehaves, the IME ignored it. |

**Capture the log at incident time.** Logcat rotates fast; the incident
that started this document was only reconstructable because the capture
happened minutes later.

## Test layers and what each can catch

| Layer | What it exercises | Catches |
|---|---|---|
| e2e harness (`internal/test/e2e`) | Logic + renderer headless; commits injected directly (`HandleIMECommit`) | Commit mapping math, window invariants, guard/resync logic, edit/scroll/selection state. **Never** the IME contract: Gboard is not in the loop. |
| Emulator stress loops (`/tmp/loop4.sh`, `/tmp/stress.sh`) | Real app + adb touches, real Gboard | Fling/tap/typing interplay, restart storms, caps regressions. Coarse: no boundary visibility, scenario shape matters (warm-session fling-tap-type was the original shape; it missed the cold-restore → far-scroll → tap → fast-typing shape entirely). |
| Full-path scenario (`/tmp/s8.sh`) | Cold open with restored cursor → far scroll → tap into a known word → fast Gboard typing → backspaces; asserts tap position vs commit ranges vs **exact file diff** | The desync class: text landing anywhere but the tap. The byte→rune conversion must be computed from the file, never assumed. |

Rule: **a bug in the IME boundary is invisible to every layer except the
last one.** When IME behavior is touched, run S8.

## Operational pitfalls hit in these sessions

- The launcher activity is `pad.pad/org.gioui.GioActivity`, not
  `.MainActivity`. `am start` failures are silent if stderr is redirected —
  check the output once, trust it after.
- `adb install`/build steps fail with "no adb device" when the emulator is
  mid-reconnect; the APK may be built but not installed. Verify
  `dumpsys package … lastUpdateTime` after installs.
- The "All files access" settings toggle can be a red herring: `appops
  set <pkg> MANAGE_EXTERNAL_STORAGE allow` is the reliable grant, and
  `appops get` output has been observed to lag; `dumpsys appops` is
  authoritative.
- `difflib.SequenceMatcher` on ~1 MB strings is minutes slow; use a linear
  first-diff scan for test assertions on large files.
- Float-equality re-emit is a recurring trap (LastLineY, caret px): any
  "re-emit when changed" comparison on a float sum needs an epsilon gate,
  or it spins a loop that re-pushes IME state every frame and starves real
  work.
- When a tap's pixel→text mapping is in question, the `IME TAP` line gives
  both the local point and the resulting cursor — log both, or the mapping
  bug is undiagnosable from the outside.

## Diagnostics currently enabled (temporary)

- Pad-side, permanent: `IME TAP`, `IME COMMIT`, `IME DRIFT-SNAP`,
  `IME PUSH snippet/sel/commit-range` (bounded: only on change).
- Fork-side, **temporary** (in `~/gioui`, gated by the `replace` in
  `go.mod`): `PADIME` boundary logs in `GioView.updateSelection`,
  `GioView.restartInput`, and `window.EditorStateChanged`.

### Cleanup steps (when informal on-device testing is settled)

The fork is stock v0.10.2 plus the `PADIME` lines and nothing else
(verify first: `git -C ~/gioui diff --stat` must show only
`app/GioView.java` and `app/os_android.go`); behavior is identical on
stock gioui.

```sh
# 1. Restore the fork to clean v0.10.2
git -C ~/gioui checkout -- .
git -C ~/gioui diff --stat            # must be empty

# 2. Drop the replace and re-tidy pad against stock gioui
cd ~/pad
go mod edit -dropreplace gioui.org
go mod tidy

# 3. Verify everything green on stock gioui
go build ./...
go test -race ./...
./scripts/check.sh

# 4. Rebuild both APKs and smoke-test
./scripts/build_emu.sh                # install to emulator
bash /tmp/s8.sh                       # full-path scenario must still PASS
./scripts/build_phone.sh              # install to phone, type + tap once

# 5. Commit and push the go.mod/go.sum change
git add -A && git commit -m "build: drop the temporary gioui diagnostic fork" && git push origin main

# 6. Remove this cleanup section (and the "temporary" fork-side bullet
#    above) from ime.md; commit + push.
```

If step 1 shows more than the two diagnostic files, do NOT `checkout --
.` blindly: review the diff and keep anything that is a real (non-log)
change, porting it upstream or documenting it in this file first.