1027 lines
61 KiB
Markdown
1027 lines
61 KiB
Markdown
# Gallery — development hand-off
|
||
|
||
A high-performance image gallery for Android (primary), Linux and macOS,
|
||
written in Rust with the **Slint** UI toolkit. This document is the starting
|
||
point for development: it captures every decision, gotcha, and measurement
|
||
from the framework evaluation that preceded it, so no learning has to be
|
||
rediscovered.
|
||
|
||
**Status: M3 done (2026-10-27, twice overhauled)** — M0 scaffold, M1 metadata
|
||
scan + sqlite + instant list paint, M2 thumbnail engine (turbojpeg DCT-shrink
|
||
decode, worker pool, disk cache, cancellation), M3 grid UI (3-column ListView
|
||
grid, shimmer placeholders, memory-bound cell retention) all verified: 15
|
||
unit tests (incl. two queue-priority regressions), 5k synthetic engine test,
|
||
and on a real Pixel 9 Pro (8,577 photos: 60 fps, 0 hitches). The thumbnail
|
||
loader was rewritten TWICE on 2026-10-27 because "stop → long shimmer → small
|
||
move → everything pops" kept surviving: first the scroll handler forgot cells
|
||
when scrolling stopped (fix: **viewport reconciliation** — the 10 Hz tick is
|
||
the sole authority, the invariant "every viewport cell is painted / in flight
|
||
/ failed" is restored unconditionally each tick and `debug_assert`ed), then
|
||
the worker-level FIFO still let stale transit work delay a fresh viewport
|
||
(fix: **worker-queue prepend + cap** — bursts are prepended ahead of older
|
||
queued work; per-worker cap drops the oldest tail with a `skipped` result).
|
||
Stress-verified on the emulator (debug build) + phone before shipping:
|
||
fast up/down 0–120 ms to full coverage, cold landings bounded by decode
|
||
throughput, chaos scrolling max oldest in-flight job 217 ms (was 8 s),
|
||
0 invariant violations in ~1,450 ticks, 0 panics/ANRs. M4 viewer is
|
||
DONE (2026-10-29 — see §9); next is M5 permissions.
|
||
|
||
---
|
||
|
||
## 1. Goals
|
||
|
||
- Browse and view a local photo collection (thousands of images) that stays
|
||
smooth: instant grid, no decode jank, cached thumbnails.
|
||
- Full-screen viewer with pinch-zoom/pan.
|
||
- Correct EXIF orientation, HEIC support (Android camera default).
|
||
- Same codebase on macOS + Linux (desktop `cargo run`) and Android
|
||
(`cargo apk2`).
|
||
- Quality of the *platform experience* matters: safe-area insets, IME-free
|
||
UI, native-feeling widgets, animations — first-class support preferred over
|
||
manual patching.
|
||
|
||
Non-goals (for now): editing, cloud sync, video, widgets/live-updates.
|
||
|
||
## 2. Why Slint (decision summary)
|
||
|
||
Full evaluation in `/mnt/Backup/prog/android/ui/REPORT.md` (same tree:
|
||
`~/prog/android/ui/REPORT.md`), raw notes in `NOTES.md` there. Headline:
|
||
|
||
| | Slint 1.18.1 | eframe/egui 0.37.0 |
|
||
|---|---|---|
|
||
| Virtualized list (the gallery's lifeline) | ✅ `ListView` std-widget, only visible rows instantiated | ❌ `ScrollArea::show` lays out all children; `show_rows` = manual virtualization API |
|
||
| Async image → UI | swap `slint::Image` into a property; only that tile redraws (retained mode) | `ctx.load_texture` + custom loader; visible set rebuilt every frame |
|
||
| Lifecycle hook for permissions | ✅ `init_with_event_listener` (Start/Resume/Pause/Stop/SaveState/...) | ❌ none in `App` trait (winit swallows it) |
|
||
| Android safe-area | ✅ `safe-area-insets` built in | manual 32px-ish panel workaround (winit#3910) |
|
||
| Animation | ✅ declarative built-in engine | hand-rolled |
|
||
| IME text entry | ✅ | ✅ |
|
||
| APK size (release aarch64) | 8.57 MB | 2.87 MB |
|
||
| Clean build time (x86_64, this machine) | 195 s / 270 crates | 76 s / 137 crates |
|
||
| Desktop Linux system deps | `libfontconfig1` + `pkg-config` | none |
|
||
| License | royalty-free for proprietary mobile/desktop (embedded excluded) / GPLv3 / commercial | MIT OR Apache-2.0 |
|
||
| Min Android API | 26 | 24 |
|
||
|
||
For a stateful, form-and-grid app, the declarative model was the right fit
|
||
(see the discussion about `pad`'s immediate-mode frame friction); egui wins
|
||
for games/dashboards and on APK size. Known Slint caveat: an open upstream
|
||
"abort on pointer scroll" bug in the Android backend (`todo!()` in the
|
||
pointer-event path) — verify it's fixed in 1.19 before relying on heavy
|
||
scrolling; if it fires, the panic is obvious in logcat.
|
||
|
||
## 3. Environment, build & test loop
|
||
|
||
Everything below is verified working on this machine (2026-10-27).
|
||
|
||
```bash
|
||
export ANDROID_HOME=$HOME/android-sdk
|
||
export ANDROID_NDK_ROOT=$HOME/android-sdk/ndk/27.0.12077973
|
||
export PATH="$HOME/.cargo/bin:$ANDROID_HOME/platform-tools:$PATH"
|
||
|
||
# Rust 1.99.1 via rustup; tools: cargo-apk2 1.5.0, cargo-ndk 4.1.2
|
||
./scripts/build.sh debug # debug x86_64 APK + adb install + launch
|
||
./scripts/build.sh # release x86_64 (emulator)
|
||
./scripts/build.sh aarch64 # release aarch64 (device)
|
||
cargo run # desktop (Linux needs libfontconfig1+pkg-config)
|
||
```
|
||
|
||
- Emulator: `emulator -avd pad_avd -no-window -no-audio -gpu off -no-snapshot`
|
||
(google_apis x86_64, Android 15). **`-gpu off -no-snapshot` matters** —
|
||
swiftshader_indirect + snapshot crashed the emulator once during the eval.
|
||
If it won't start, the host key lost its wireless pairing — see the pairing
|
||
procedure in `~/.pi/agent/AGENTS.md` (phone) / emulator docs (this AVD).
|
||
- APK lands at `target/<debug|release>/apk/gallery.apk`.
|
||
- Release signing uses the debug keystore
|
||
(`CARGO_APK_RELEASE_KEYSTORE=$HOME/.android/debug.keystore`,
|
||
`CARGO_APK_RELEASE_KEYSTORE_PASSWORD=android`) — fine for personal use;
|
||
for Play, generate a real keystore and point the env vars at it.
|
||
- `cargo apk2` quirk: `--target` (not `-t`). `--lib` builds the cdylib.
|
||
- Grant "All files access" to the app on the emulator after install:
|
||
`adb shell am start -a android.settings.MANAGE_ALL_FILES_ACCESS_PERMISSION`
|
||
(or Settings UI). `adb install -r` keeps the grant.
|
||
- **Emulator test data (M3)**: the AVD's `/Pictures/gallery-test/` has ~5k
|
||
*fake* JPEG stubs (zlib garbage — they scan fine but fail to decode, good
|
||
for volume tests). Real, decodable test JPEGs live in
|
||
`/Pictures/realtest/` (150, generated with PIL). They were `touch`-ed to a
|
||
future mtime so they sort to the TOP of the grid — when reasoning about
|
||
"which cells should have images", remember only cells 0–149 are real;
|
||
everything below is stubs and correctly shows shimmer. **`/Pictures/realtest_extra/`
|
||
holds 700 more real, decodable JPEGs (PIL, 1200–4032 px, mtime 2031 so they
|
||
sort to the very top, above the 150 in `realtest`) — added for the
|
||
scroll-back backlog repro; the real region is now ~850 files (283 rows).
|
||
`adb push` of a directory nests a subfolder (push files individually or
|
||
`mv` up); `adb shell touch -d '…'` works for setting mtimes. The thumb cache is at
|
||
`/sdcard/Android/data/wow.gmp.gallery/files/thumbs` (wipe it to force
|
||
re-decode).
|
||
- Perf measurement: `adb shell perfetto` / `adb shell dumpsys gfxinfo wow.gmp.gallery`,
|
||
`systrace`-style capture via Perfetto trace shell. Cold start:
|
||
`adb shell am start -W`.
|
||
|
||
## 4. Architecture
|
||
|
||
The UI framework is only the display layer. The performance lives in the
|
||
thumbnail engine, and it is framework-agnostic:
|
||
|
||
```
|
||
┌─ metadata ─────────────────────────────────────────────────────────┐
|
||
│ sqlite (rusqlite, bundled): path, mtime, size, w, h, orientation, │
|
||
│ thumb_path, date_taken — one background scan pass │
|
||
└──────────────┬─────────────────────────────────────────────────────┘
|
||
│ model → Slint (list of visible-range items)
|
||
┌─ thumbnail pool (tokio-free: N std::threads, N ≈ cores/2) ─────────┐
|
||
│ per job: token cancellation (AtomicU64 gen counter) + priority │
|
||
│ 1. disk cache hit? .thumbs/{sha1(path|mtime|size)}.jpg → done │
|
||
│ 2. decode AT TARGET SIZE (never full-size for the grid): │
|
||
│ JPEG: zune-jpeg "turbo" (decode+downscale in ONE pass, │
|
||
│ pure Rust, actively maintained) or turbojpeg (SIMD) │
|
||
│ HEIC: imagepipe (0.5, 2026-08) — VERIFY Android NDK build │
|
||
│ PNG: image/png │
|
||
│ 3. apply EXIF orientation (once, here — never per frame) │
|
||
│ 4. re-encode small JPEG → disk cache │
|
||
│ 5. channel → UI: (id, slint::Image) │
|
||
│ prefetch: decode visible ± 1 row; cancel items scrolled out │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
┌─ UI (Slint) ───────────────────────────────────────────────────────┐
|
||
│ grid screen: ListView (vertical) → row of N square Image cells │
|
||
│ viewer screen: full-size async decode (once, cached) + │
|
||
│ ScaleRotateGestureHandler pinch + 1-finger pan │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
Module layout:
|
||
|
||
```
|
||
src/
|
||
lib.rs # android_main + platform init + scan/UI glue (M1/M2)
|
||
main.rs # desktop entry [exists]
|
||
main.slint # screens (split into ui/ components at M3) [exists]
|
||
db.rs # sqlite schema + incremental scan apply [done M1]
|
||
scan.rs # directory walk, image detection, walk_stream [done M1]
|
||
thumb.rs # pool, dispatcher, disk cache, generation cancel [done M2]
|
||
images.rs # turbojpeg shrink-decode, EXIF orientation, PNG [done M2]
|
||
viewer.rs # full-size load + transform state (M4)
|
||
perm.rs # storage permission check/request (M5)
|
||
```
|
||
|
||
Crate choices (all verified current on crates.io, 2026-10):
|
||
|
||
| crate | version | why |
|
||
|---|---|---|
|
||
| `slint` / `slint-build` | 1.18.1 | pinned; upgrade to 1.19 after the pointer-scroll fix lands |
|
||
| `rusqlite` | 0.41, `bundled` feature | SQLite without system deps; cross-compiles to Android |
|
||
| `turbojpeg` | 1.5.1, `cmake` feature | libjpeg-turbo DCT-level shrink decode (1/2,1/4,1/8) — 48 MP at 1/8 peaks ~2.3 MB vs 137 MB full decode. Vendors + cross-builds libjpeg-turbo via CMake/NDK (verified). Chosen over `libvips` (same DCT shrink mechanism, +6–10 MB of .so files, LGPL, 4.3k downloads) and over `zune-jpeg` (no DCT shrink = full-size decode for 12 MP photos). |
|
||
| `image` | 0.25, `default-features=false, png` | PNG thumbs + `imageops::resize` final fit |
|
||
| `kamadak-exif` | 0.6 | EXIF orientation tag (all 8 values applied as pixel maps); `experimental::Writer` used only in tests |
|
||
| `sha1` | 0.11 | cache keys |
|
||
| `imagepipe` | — (deferred) | HEIC: the real device has **0 HEIC** files (9109 JPEG, 27 PNG), so HEIC support is deferred; `image` can gain `heic` or libvips can replace the JPEG path later — the engine API is decoder-agnostic |
|
||
|
||
Decode sizes: grid tiles render at ~cell×2 (Retina) — e.g. 500 px side.
|
||
Full-screen: full decode once (12 MP ≈ 30–80 ms off-thread, fine), cached in
|
||
the in-memory LRU until memory pressure.
|
||
|
||
**The five failure modes the current (unknown) app likely has** — this design
|
||
eliminates all five: (1) full-size decode for grid cells, (2) no disk
|
||
thumbnail cache, (3) no list virtualization, (4) decode on UI thread without
|
||
cancellation, (5) EXIF rotation applied per frame.
|
||
|
||
## 5. UI plan (Slint patterns)
|
||
|
||
- **Grid**: `ListView` is the only virtualized std-widget (confirmed in
|
||
source docs: "Only visible rows are instantiated"). Grid = vertical
|
||
`ListView`, each row a horizontal layout of square `Image` cells
|
||
(columns ≈ `floor(window-width / cell)`; recompute on resize).
|
||
Cells bind `image <=> model.image` — when the pool delivers a thumbnail,
|
||
only that cell invalidates (retained mode; no repaint calls to remember).
|
||
- **Placeholder→image**: cell shows a `Rectangle` (shimmer optional, M6)
|
||
until its `image` is set.
|
||
- **Image cache**: Slint's built-in path-based LRU is 5 MB (`CLruCache` in
|
||
`internal/core/graphics/image/cache.rs`) — too small and the wrong layer
|
||
for us; our cache lives in Rust (disk + compressed-bytes LRU).
|
||
No mipmaps in the Skia renderer — irrelevant since we pre-downsample.
|
||
- **Viewer** (DONE in M4): pinch is the `ScaleRotateGestureHandler`
|
||
element — the 1.18 CORE synthesizes `PinchGesture` from raw two-finger
|
||
touches on every backend (the second finger never reaches a `TouchArea`;
|
||
see §6.15/16). Single-finger pan + double-tap-to-zoom are a small state
|
||
machine in `src/viewer.rs` driven by `TouchArea::pointer-event`. Crib:
|
||
`slint-src/examples/native-gestures` (source at
|
||
`~/prog/android/ui/slint-src`, tag v1.18.1).
|
||
- **Screens**: single `Window` with a stacked layout (grid + viewer overlay)
|
||
or a `Navigator`-style state property; keep it simple until M4.
|
||
- **Safe area**: always pad with `root.safe-area-insets.{top,bottom}` (done
|
||
in the scaffold).
|
||
- **Animation**: use built-in `animate` statements (syntax: `animate <prop>
|
||
{ duration: ...; easing: ...; }` — it is a *statement*, not an element).
|
||
Slide/fade for the viewer is a one-liner.
|
||
|
||
## 6. Slint 1.18 API gotchas (all hit firsthand in the eval)
|
||
|
||
1. `animate` is a statement (`animate width { duration: 300ms; }`), not an
|
||
`Animation {}` element.
|
||
2. Text input widget is `LineEdit`, not `TextField`.
|
||
3. Markdown interpolation: `text: "Count: \{count}"` (backslash-brace).
|
||
4. `length` is the unit type for lengths (`property <length>`, not
|
||
`logical<length>` in the 1.18 Rust API).
|
||
5. No Rust macros/closures inside `.slint` callbacks — callbacks call global
|
||
functions or invoke Rust via `@global` / imported callbacks.
|
||
6. 1.18 ↔ 1.19 **docs drift** — when reading docs online, check which
|
||
version; the local source at `~/prog/android/ui/slint-src` (tag v1.18.1)
|
||
is ground truth for the pinned version. `git grep` beats the docs.
|
||
7. `slint::android::init_with_event_listener(app, |event: &PollEvent| ...)`
|
||
re-exports the whole `android-activity` crate as
|
||
`slint::android::android_activity`; `AndroidApp` is also re-exported at
|
||
`slint::android::AndroidApp`.
|
||
8. `slint::Image` runtime construction: `Image::from_rgba8_premultiplied`
|
||
(also `from_array`); assign to an `image` property to update a cell.
|
||
9. Desktop Linux: needs `libfontconfig1` + `pkg-config` (pure Rust on macOS).
|
||
10. Retained-mode redraw: changing a property re-renders only affected
|
||
elements — design bindings so thumbnail delivery touches one property.
|
||
11. **`in property` is NOT auto-bound to a same-named `for` variable.** In
|
||
`for cell[c] in row.cells: CellView { ... }` you must write
|
||
`CellView { cell: cell; ... }` — otherwise the component's `in property
|
||
<Cell> cell` stays at its default (empty) and the cell renders blank
|
||
while *looking* correct. (This silently broke M3 for an hour.)
|
||
12. **Updating a nested model in place does NOT reach a ListView row that
|
||
was recycled while off-screen.** In-place `inner.set_row_data(col, …)`
|
||
updates only the currently-live `CellView`s; a row whose `GridRow`
|
||
instance was recycled keeps showing stale cells until the next scroll.
|
||
The reliable path is to replace the WHOLE outer row:
|
||
`outer.set_row_data(row, Item { cells: ModelRc::new(VecModel::from(…)) })`
|
||
— the outer `row_changed` makes the ListView re-bind `row-data` and the
|
||
fresh nested read picks up the new state. Batch cell changes per row and
|
||
do ≤ ~16 row-replacements per 100 ms tick (see `apply_row_batch`), or a
|
||
fast scroll mass-updates and ANRs.
|
||
13. `ListView` has no `spacing`/`gap` property in 1.18 — bake inter-row
|
||
rhythm into the item height (we add a 4 px bottom margin inside
|
||
`GridRow`; `ROW_SPACING` in lib.rs must match).
|
||
14. **`TouchArea` grabs EVERY press (mouse and touch) — `input_event` on
|
||
`Pressed` unconditionally returns `GrabMouse` (internal/core/items/
|
||
input_items.rs), which ABORTS the sibling traversal**
|
||
(`send_mouse_event_to_item` → `VisitChildrenResult::abort`). Siblings
|
||
are visited FRONT-TO-BACK = last-declared first (`TraversalOrder::
|
||
FrontToBack` in item_tree.rs iterates `count-1 … 0`), so the TOPMOST
|
||
(last-declared) TouchArea at a position wins the press, and any
|
||
earlier-declared TouchArea under it never sees the event. The viewer's
|
||
close button therefore must be declared AFTER the full-surface
|
||
TouchArea (M4 — this cost a session of debugging).
|
||
15. **The second finger NEVER reaches a `TouchArea`.** The CORE
|
||
(not the backend) owns two-finger semantics: `TouchState::process`
|
||
(internal/core/input.rs) turns per-finger `InternalEvent::Touch` into
|
||
mouse events, and when a second finger lands it synthesizes a Release
|
||
for the first finger and, after an 8 px threshold, emits
|
||
`PinchGesture`/`RotationGesture` mouse events — on EVERY backend,
|
||
including Android (the android-activity backend does deliver the
|
||
per-finger `Touch` events; the core converts them). Consequences:
|
||
`TouchArea::pointer-event` (1.18's `touch-finger-id`) only ever sees
|
||
ONE finger; manual two-finger math is dead code. When a pinch finger
|
||
lifts, the core re-presses the survivor (a fresh `Pressed`), which
|
||
seamlessly continues as a single-finger pan.
|
||
16. **Pinch in 1.18 = the `ScaleRotateGestureHandler` element** (added in
|
||
1.18.0; declared in internal/compiler/builtin_elements.rs, NOT in any
|
||
.slint file — `git grep ScaleRotateGestureHandler` the compiler crate).
|
||
API: `in enabled`, `out active`, `out scale` (CUMULATIVE from 1.0 at
|
||
gesture start), `out rotation`, `out center` (Point, in the element's
|
||
coordinates), callbacks `started`/`updated`/`ended`/`cancelled`.
|
||
`@expands_to_parent_geometry`. While active it INTERCEPTS all
|
||
non-gesture events (its `input_event_filter_before_children`).
|
||
Correct usage: snapshot your transform at `started`, apply
|
||
`snapshot · scale` about `center` at each `updated` (never multiply
|
||
incrementally — `scale` is cumulative), commit at `ended` (the
|
||
properties read identity once inactive). See `src/viewer.rs` +
|
||
`src/main.slint` (Viewer) and upstream
|
||
`slint-src/examples/native-gestures`.
|
||
|
||
## 7. Permissions & platform integration (Android)
|
||
|
||
Declared in `Cargo.toml` (verified schema in cargo-apk2 1.5 / ndk-build2:
|
||
`uses_permission` is **top-level** under `[package.metadata.android]`, and
|
||
`Permission` supports only `name` + `max_sdk_version`):
|
||
`READ_MEDIA_IMAGES`, `READ_EXTERNAL_STORAGE` (maxSdk 32),
|
||
`MANAGE_EXTERNAL_STORAGE`.
|
||
|
||
Runtime flow (M5), using the `ANDROID_APP` clone stashed in `src/lib.rs`:
|
||
|
||
1. **MANAGE_EXTERNAL_STORAGE** (primary path, same as passgo): no runtime
|
||
dialog exists — check `Environment.isExternalStorageManager()` via JNI,
|
||
if false launch `Settings.ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION` via
|
||
`ANDROID_APP.get().run_on_java_main_thread(...)` (attach `JNIEnv` from
|
||
`vm_as_ptr()`; Activity from `activity_as_ptr()`), then **re-check on
|
||
`MainEvent::Resume`** — the hook is already wired in `src/lib.rs`
|
||
(TODO(M5) marker).
|
||
2. **READ_MEDIA_IMAGES / READ_EXTERNAL_STORAGE**: `requestPermissions` via
|
||
the same JNI route; the bare `NativeActivity` does not forward
|
||
`onRequestPermissionsResult` to native, so either (a) poll
|
||
`checkSelfPermission` after requesting (good enough), or (b) ship a custom
|
||
`NativeActivity` subclass via cargo-apk2 `java_sources = "android/java"`
|
||
(manifest `activity name` points at it) — the proper callback, reusable
|
||
pattern from the eval.
|
||
3. Facts that constrain design (verified in android-activity 0.6.1 source):
|
||
no `NewIntent` event; `poll_events` may only run on the `android_main`
|
||
thread (Slint owns it — don't add a second consumer); `AndroidApp` is
|
||
`Clone` and a clone works from any thread (`run_on_java_main_thread`,
|
||
`vm_as_ptr`, `activity_as_ptr`).
|
||
4. **Source discovery** for photos: start with a plain directory scan of
|
||
`/storage/emulated/0/Pictures` + `DCIM` (works with all-files access,
|
||
emulator-friendly, and it's what makes the app testable without
|
||
MediaStore). MediaStore integration (API-aware, scoped to
|
||
`READ_MEDIA_IMAGES`) is a later refinement if directory scanning proves
|
||
insufficient.
|
||
|
||
## 8. Performance targets & measurement
|
||
|
||
Targets (emulator is NOT representative for absolute decode times — validate
|
||
on a real device with the aarch64 build):
|
||
|
||
- Grid: 60 fps scrolling at 5k+ items; zero full-size decodes on the hot path.
|
||
- First frame: grid of placeholders < 100 ms after launch (metadata from
|
||
sqlite, images async).
|
||
- Thumbnail decode (grid size): < 5 ms off-thread per JPEG (measure with
|
||
`std::time::Instant` logs; zune-jpeg turbo should be well under).
|
||
- Steady-state memory: in-memory thumb cache bounded (~64–128 MB compressed
|
||
bytes on a phone) + Slint's texture memory for visible tiles only.
|
||
- Cold start (emulator, release): compare against the eval baseline
|
||
(Slint sample ≈ 1–2 s; gallery must not regress > 0.5 s).
|
||
|
||
**Scroll-FPS measurement (real device, verified 2026-10-27):**
|
||
`dumpsys gfxinfo wow.gmp.gallery` reports **0 frames** for Slint — the
|
||
NativeActivity window renders via its own EGL surface and is not tracked by
|
||
gfxinfo on Android 15 (works for normal View apps). Use SurfaceFlinger's
|
||
per-layer latency instead:
|
||
|
||
```bash
|
||
# find the layer name:
|
||
adb shell dumpsys SurfaceFlinger | grep "VRI-wow.gmp.gallery"
|
||
# while scrolling (swipes via `input swipe`): capture 1 s of frame history
|
||
adb shell "dumpsys SurfaceFlinger --latency 'VRI-wow.gmp.gallery/android.app.NativeActivity#<id>'"
|
||
# each line: latch_time display_time … — display_time is the vsync grid.
|
||
# fps = 1e6 / avg(display gap); jank = display gaps > ~25 ms; the latch
|
||
# column shows frame PRODUCTION timing (p95 ~17 ms = on time at 60 Hz).
|
||
```
|
||
|
||
Result (Pixel 9 Pro, release aarch64, 8,577 photos, 500 px thumbs):
|
||
sustained **60.0 fps, 0 display hitches** while scrolling, both cold cache
|
||
(production p95 21–25 ms) and warm cache (production avg 16.6 ms). The
|
||
display presents at a 60 Hz vsync grid (Slint requests Default framerate;
|
||
120 Hz is a later polish item, not an M3 acceptance criterion).
|
||
Note: the shimmer `animation-tick()` keeps frames flowing even at rest,
|
||
so latency history is continuous — filter by swiping if you need
|
||
scroll-only frames.
|
||
|
||
Test data (M2): generate a synthetic set on the device — a small Rust `bin`
|
||
or Python script that writes N=5,000 JPEGs of varied sizes/orientations into
|
||
`/storage/emulated/0/Pictures/gallery-test/` (include a few HEICs if
|
||
imagepipe works; include rotated EXIF). Reuse it for every perf run.
|
||
|
||
Measure with: `adb shell am start -W` (cold start),
|
||
`dumpsys gfxinfo wow.gmp.gallery framestats`, Perfetto trace (GPU + main +
|
||
pool threads), and `logcat` pool timing logs behind a `debug_timing` flag.
|
||
|
||
## 8.5 Jank hunt (IN PROGRESS — phone is the acceptance target)
|
||
|
||
Symptom: user reports janky grid scroll (moderate drag, no fling) on the
|
||
phone (Pixel 9 Pro, 120 Hz). Emulator repro via programmatic scroll hook.
|
||
|
||
### What is ruled out (all measured)
|
||
- **App tick cost**: 7–34 µs/tick, even inside 28–39 ms gaps. Not the tick.
|
||
- **Logd backpressure**: fixed — `nblog` (bounded mpsc + drain thread,
|
||
nanosecond `try_send` from UI). Idle CPU 71% → 5%.
|
||
- **60/120 Hz display zone flap**: SystemUI (uid 10156) was asserting a 60 Hz
|
||
override every minute; pinned with `settings put system min_refresh_rate
|
||
120` (reversible: `settings delete system min_refresh_rate`).
|
||
- **Shimmer animation**: `animation-tick()` in a per-cell binding evaluated
|
||
O(ALL 8577 cells) `sin()` every frame (70.9% CPU idle). Now a static
|
||
placeholder; proper O(visible) shimmer is M6 work.
|
||
- **Whole-row model replacement**: now in-place `cells.set_row_data` (one
|
||
property change; texture upload only when pixels change).
|
||
- **Texture upload CPU**: APLY 1.2–2.9 ms/cell (480 KB) — under budget, and
|
||
too cheap to explain 28–39 ms (would need 12–17 MB/s on a Mali).
|
||
|
||
### Measured facts
|
||
- Emulator, programmatic scroll (`/sdcard/Download/.gallery_cmd` →
|
||
`scroll <px/ms> <ms>`), real textures: 28–146 ms cadence. During every
|
||
stall the UI thread is **100 % on-CPU** (jsample: state R, wchan 0) —
|
||
Slint's render pass is CPU-bound.
|
||
- Emulator, 300 items instead of 5852: **equally janky** → scene/item count
|
||
is NOT the cost driver.
|
||
- Emulator, grey 1×1 images (`.gallery_grey` touch file): steady ~16.6 ms
|
||
(60 Hz pacing), worst 44 ms → the cost is **rasterizing ~30 live textured
|
||
cells at 1080×2400 through the emulator's software GL (llvmpipe)**.
|
||
The emulator has a hard rendering ceiling and can never scroll this
|
||
smoothly — it validates LOGIC (invariants, paint coverage), not feel.
|
||
- Phone: 64 % of frames on the 8.3 ms (120 Hz) grid; ~4 % take 28–39 ms of
|
||
non-tick work with d≈0 (UI thread not in our code). Frequency ≈ every
|
||
170 ms — much faster than new cells enter the viewport (~1.7 s), so it is
|
||
NOT first-paint/upload timing. Suspects left: GPU/present contention
|
||
(Mali + SystemUI both at 120 Hz), Slint EGL/swap path, or an
|
||
event-loop/vsync interaction. Needs the phone.
|
||
|
||
### Tools / hooks
|
||
The hunt's file hooks (`.gallery_cmd`, `.gallery_*` knobs, SCRDBG/TICK/APLY
|
||
logs, jsamp sampler) were REMOVED in the 2026-10-28 cleanup — see §8.9.
|
||
`nblog` (non-blocking logger) and the debug-level coverage logs stayed.
|
||
|
||
### Outcome
|
||
The decisive A/B was not needed: stage timestamps inside the vendored
|
||
renderer isolated the stall to the Skia renderer's final synchronous
|
||
flush (Mali driver back-pressure). Fix + evidence: §8.9 and
|
||
`docs/BUGS/slint-skia-final-flush-jank.md`.
|
||
3. Compare against Fossify (installed on the phone, user says smooth):
|
||
`dumpsys SurfaceFlinger --latency` on both apps' layers during identical
|
||
scrolls.
|
||
4. **Priority fix already applied** (commit after this edit):
|
||
- `thumb-0..3` + `thumb-dispatch` demoted to nice +2 (they inherited the
|
||
UI's -10 and could preempt it during turbojpeg bursts).
|
||
- Slint's `slint vsync` Choreographer waker (created at nice +10 by
|
||
Android — the ONLY wakeup source for the main loop while animating,
|
||
see i-slint-backend-android-activity/vsync.rs) is raised to nice 0 by
|
||
a `prio-fix` thread in android_main (log: "slint vsync raised to nice
|
||
0"). A starved waker delays the main loop's wakeup by 3–4 frames at
|
||
120 Hz = 25–33 ms with zero tick cost — the measured phone signature.
|
||
Verified on emulator: nice holds at 0 while foreground; Android
|
||
re-nices backgrounded processes (expected, irrelevant).
|
||
- Verification on phone: check the two log lines, then
|
||
`for t in /proc/$(pidof wow.gmp.gallery)/task/*; do ...` and confirm
|
||
thumb-* at 2, slint vsync at 0, before/while the user scrolls.
|
||
5. Candidate code changes (after the A/B): GRID_TARGET 500→400 (one-time
|
||
cache invalidation — don't do it before knowing the culprit), 1 painted
|
||
cell/tick instead of 2.
|
||
|
||
## 9. Milestones
|
||
|
||
- **M0 — scaffold (DONE)**: builds (x86_64 + aarch64 toolchain), installs,
|
||
runs on emulator; permission-aware init + Resume hook; build script;
|
||
manifest with permissions; this document.
|
||
- **M1 — metadata (DONE, 2026-10-27)**: `db.rs` + `scan.rs`; background scan
|
||
thread → sqlite → `VecModel` in a `ListView` (paths as text). Acceptance
|
||
verified on emulator: **5002 files in 153 ms**; relaunch `0+ 0~ 0-` in 2 ms;
|
||
mtime-touch → `1~`, delete → `1-`. Status line shows `N files (a+ u~ d–) in X ms`.
|
||
List paint is instant on relaunch: the DB list is shown first
|
||
("N files (cached)"), the background walk then runs and refreshes
|
||
("Updating… N"). First launch streams results into the list in 512-file
|
||
batches while scanning — the UI never waits for the full DB build
|
||
(`walk_stream`). Hidden dirs (`.thumbnails`, …) are skipped.
|
||
Real-device numbers (Pixel-class, 8.5k photos): cold scan ~4.2–5.9 s
|
||
(filesystem metadata cost — the walk is the bottleneck, not sqlite),
|
||
relaunch paint instant, rescan ~4.2 s. Walk parallelism is an M2/M7
|
||
optimization candidate.
|
||
Gotchas hit (also see §6/§12):
|
||
- `slint::Result` / `slint::WindowHandle` **do not exist in 1.18.1** as
|
||
published on crates.io — desktop bin is `fn main() -> Result<(),
|
||
slint::PlatformError>` + `use slint::ComponentHandle;`. (The M0 desktop
|
||
bin never actually compiled; only the Android lib was built. The eval
|
||
sample `slint_app` is broken the same way.)
|
||
- `.slint` modulo is `Math.mod(x, y)`, not `%`.
|
||
- ListView rows: `for path[index] in root.items: …` (`index` is not in
|
||
scope unless named; `parent` = the ListView).
|
||
- `rusqlite(bundled)` needs `CC_<triple>` pointing at NDK clang for
|
||
Android — `scripts/build.sh` now exports them.
|
||
- Android DB path: `EXTERNAL_DATA_DIR` is **not set** by this
|
||
NativeActivity setup and `temp_dir()` = `/data/user/0` (unwritable).
|
||
Use `/storage/emulated/0/Android/data/<pkg>/files`, internal files dir
|
||
fallback (see `db::android_data_dir`).
|
||
- Emulator: grant all-files access once via
|
||
`adb shell appops set wow.gmp.gallery MANAGE_EXTERNAL_STORAGE allow`; it
|
||
survives `adb install -r`.
|
||
- Files pushed with `adb push` are owned by `shell` mode 640 — `chmod 644`
|
||
or the app can't read them.
|
||
- **M2 — thumbnail engine (DONE, 2026-10-27)**: `images.rs` (turbojpeg
|
||
DCT-level shrink decode via 1/2/1/4/1/8 `ScalingFactor`, all 8 EXIF
|
||
orientations as uniform `(x,y)→(dx,dy)` pixel maps, `image::imageops::resize`
|
||
final fit, re-encode to disk cache at q85; PNG via `image`) + `thumb.rs`
|
||
(dispatcher thread round-robining to N=cores/2 workers clamped 2..8,
|
||
unbounded job queue, disk cache `thumbs/{sha1(path|mtime|size|target)}.jpg`
|
||
— target in the key so list-64px and grid-500px coexist; atomic
|
||
`.tmp`→rename; generation counter for cancellation; per-worker sqlite
|
||
connection writes `set_thumb` w/h/orientation/thumb_path) + `lib.rs`
|
||
(process-wide `AppState` in `OnceLock`, 10 Hz drain timer — `Timer` is
|
||
`!Send`, leaked with `std::mem::forget`, scroll-prefetch with 50-row
|
||
margin, `do_prefetch` flag so streaming scan batches don't re-prefetch) +
|
||
`main.slint` (rows carry `image` now).
|
||
*Accept, verified:* 12 unit tests + 5k synthetic engine test (cold fill,
|
||
100% cache warm refill, generation cancel). Release aarch64 APK 9.8 MB.
|
||
**Real device (Pixel 9 Pro, 8577 photos):** cold scan ~5 s with list
|
||
streaming; warm relaunch paints the cached list instantly; thumbnails
|
||
render in every visible row (screenshot-verified); relaunch cache hits
|
||
~100% (255–258 of first 256 jobs); 2 WebP files misnamed `.jpg` fail
|
||
gracefully (logged, row stays empty). Worker log cadence: every 256 jobs.
|
||
*Known transient (unreproducible): the very first aarch64-release install
|
||
rendered a pure-black screen (no drain-timer "painted" logs) until the
|
||
next launch; it did **not** reproduce on the emulator (screen-off launch +
|
||
wake) nor across repeated device launches. Suspected screen-lock / display
|
||
state race on first GL frame. If it recurs: force a window refresh on
|
||
Resume (see §10.7).*
|
||
*Build gotchas hit (also in build.sh comments):*
|
||
- `ANDROID_PLATFORM=android-26` (NDK CMake toolchain) **conflicts** with
|
||
the `android-build` crate (Slint's Java compile) unless `ANDROID_JAR`
|
||
is set explicitly to `platforms/android-35/android.jar`.
|
||
- x86_64 host builds of turbojpeg-sys need `nasm` (`~/.local/bin/nasm`;
|
||
aarch64 does not).
|
||
- `ADB_SERIAL=…` env var picks the device in build.sh (phone + emulator
|
||
are often both attached).
|
||
- build.sh keywords: `debug` (x86_64), `debug-aarch64`, `x86_64`,
|
||
`aarch64`; second arg `no-install` to skip install.
|
||
- **M3 — grid UI (DONE 2026-10-27)**: 3-column `ListView` grid (`GridRow` =
|
||
one list item holding up to 3 square `CellView`s), shimmer placeholder via
|
||
a time-driven `animation-tick()` expression, scroll-driven prefetch
|
||
(`scrolled → on_scroll`), memory-bound retention (cells drop to shimmer
|
||
when they leave the active range; disk cache repaints them in ~10 ms).
|
||
Key fixes along the way (see §6.11–13): `in property` must be assigned
|
||
explicitly in a `for`; cell updates must replace the whole outer row
|
||
(`apply_row_batch`) and are capped at 16 rows/tick to avoid ANR on fast
|
||
scroll; `on_scroll` only re-anchors the active range + submits prefetch
|
||
(no model mutation), all mutation is in the 10 Hz drain timer.
|
||
|
||
**Thumbnail loading: viewport reconciliation + worker-level priority
|
||
(rewritten 2026-10-27 after two field-reported bugs).** Design doc is the
|
||
header comment of `src/lib.rs`; engine in `src/thumb.rs`. The invariants:
|
||
|
||
1. *The viewport is the only source of truth.* `on_scroll` only records the
|
||
viewport range (zero work). The 10 Hz tick is the sole authority AND the
|
||
sole submitter: CONSUME finished results (paint viewport ones, capped),
|
||
EVICT far pixels (hard cap `MAX_PAINTED=120`), FILL the viewport
|
||
(center-first), PRUNE queued jobs outside the retention band. After each
|
||
tick: every viewport cell is PAINTED, IN-FLIGHT, or FAILED — restored,
|
||
not maintained, so no scroll pattern can leave a visible cell forgotten
|
||
(the original bug: re-submission only happened in `on_scroll`, so when
|
||
scrolling STOPPED, forgotten cells stayed empty until the next small
|
||
move). `debug_assert`ed every tick.
|
||
2. *The viewport is first at EVERY queue level.* The central queue
|
||
(`Mutex<VecDeque>` + `Condvar`, prepend-on-submit) is drained as
|
||
priority-ordered bursts and PREPENDED to the per-worker queues (also
|
||
`Mutex<VecDeque>` + `Condvar`, NOT mpsc — mpsc is FIFO-only). Each
|
||
worker queue is capped (96 ≈ two screens): overflow is dropped from the
|
||
TAIL (oldest transit work) with a `skipped: true` result; the UI clears
|
||
the in-flight marker, the cell is NOT failed, re-entry re-submits it
|
||
(disk-cache hit). A burst ≤ viewport (~45) < cap, so fresh viewport work
|
||
is never dropped. `in_flight` accounting: +1 at submit, −1 at finish OR
|
||
at cap-drop; the UI's `inflight` map (path→time) is the dedupe key and
|
||
is cleared on every result (incl. `skipped`) — a dropped job whose
|
||
path has a newer queued job may be briefly re-submitted (one wasted
|
||
decode, self-cleans).
|
||
|
||
Regression tests: `thumb::tests::scroll_back_cache_hit_not_starved`
|
||
(central-level: 6.3 s FIFO starvation → 130 ms) and
|
||
`thumb::tests::fresh_batch_preempts_stale_worker_queue` (worker-level: a
|
||
fresh job submitted while a slow batch is queued must finish within the
|
||
first two results, never behind the whole old queue).
|
||
|
||
*Stress results (emulator x86_64 DEBUG build — slowest plausible hardware —
|
||
then release aarch64 on the Pixel 9 Pro):* fast up/down ×5 (the reported
|
||
repro): viewport fully covered 0–119 ms after stop (4/5 already covered at
|
||
stop); cold landings: 1.0–1.2 s = pure decode throughput (p rises
|
||
monotonically while stale jobs wait BEHIND, oldest growing 700→1900 ms
|
||
harmlessly); chaos ×6 + 25 s idle soak: covered before every stop, max
|
||
oldest in-flight job 217 ms (pre-fix: 8,055 ms), max queue depth 45 (one
|
||
viewport), 0 invariant violations in ~1,450 ticks, 0 panics, idle
|
||
converges to 0 queued. Phone smoke: 99.98% cache hits during 8k-cell
|
||
stress scroll, 0 panics/ANRs, one 541 ms transient stall alarm (cold
|
||
decode burst).
|
||
|
||
*Verified (emulator, 150 real JPEGs + 5k stubs): first paint 0.6 s cold;
|
||
warm relaunch from disk cache; 0 ANRs through 16 fast swipes; backlog
|
||
drains to 0 when idle. Verified on Pixel 9 Pro (release aarch64, 8,577
|
||
photos): first screen ~0.7 s cold at 500 px; scroll = sustained 60 fps
|
||
with 0 display hitches (cold and warm cache) — see §8 for the
|
||
measurement method (gfxinfo does NOT track Slint's window).*
|
||
- **M4.1 — instant viewer + idle prefetch (DONE, 2026-10-29)**: the
|
||
splash is gone. `open_viewer` never shows "Loading…": (1) the path is
|
||
in the full-res LRU cache (`full_cache`: 8 entries / 64 MP of RGBA
|
||
bytes — `slint::Image` is !Send, so the cache holds bytes and the
|
||
~3 ms GPU upload happens at tap time, the grid's hitch-free path) →
|
||
shown instantly, no decode; (2) else the tapped cell's THUMBNAIL is
|
||
shown in the same frame as the tap (Image read from the model, size
|
||
from `thumb_dims`) and the full decode (urgent — bypasses the idle
|
||
budget) swaps in silently on arrival: same aspect → s is rescaled by
|
||
old/new via `Input::rebase_content`, tx/ty (viewer px) unchanged →
|
||
identical bounds, one content change, any pan/zoom/pinch the user did
|
||
on the thumb is preserved; (3) only if the cell has no painted thumb
|
||
either is there a black frame (`waiting` = the path the viewer is
|
||
waiting on; a failed decode prints an error only for that path — a
|
||
failing prefetch is silent). Tick step 4c (`prefetch_full`): while the
|
||
viewport is stable, the viewer is closed and the thumb pipeline is
|
||
caught up (`inflight` empty after FILL — during fast scrolling it
|
||
stays non-empty, so prefetch only starts when scrolling is done), one
|
||
center-out decode per tick, max 2 in flight (`full_inflight`),
|
||
results cached even if the viewer never opens. In-flight decodes are
|
||
NOT cancelled by scrolling (results still land in the cache). All
|
||
caches clear on `set_list`/world reset. Verified on the emulator:
|
||
cache-hit tap `(full 1200x1600 cached)` — instant; uncached tap
|
||
`(thumb 159x400 up, full in flight)` → `(swapped in)` 137 ms later;
|
||
corrupt file → error text (no splash). Unit test:
|
||
`rebase_content_preserves_viewer_mapping`.
|
||
- **M4 — viewer (DONE, 2026-10-29)**: tap cell → full decode (turbojpeg
|
||
DCT shrink, 2×-screen cap bounded by GPU max texture, EXIF orientation
|
||
applied) + full-screen overlay; pinch zoom (Slint's
|
||
`ScaleRotateGestureHandler` — see §6.15/16; transform math is
|
||
`zoom_at`/`Input::pinch` in `src/viewer.rs`); single-finger pan +
|
||
double-tap zoom (2.5× fit at tap point; again → reset to fit);
|
||
back key (vendor patch: `Keycode::Back` → `CloseRequested`, the app's
|
||
`on_close_requested` closes the viewer first) + × button; error text on
|
||
decode failure. Verified on the emulator (pad_avd 1080×2400): all
|
||
gestures incl. REAL two-finger pinch (sendevent MT-B — recipe below);
|
||
phone: grid + viewer open + 4080×3072 decode OK (pinch not injectable
|
||
on the user build — SELinux blocks /dev/input writes and `input`
|
||
has no multi-pointer; trust the unit tests + emulator).
|
||
**Emulator two-finger injection recipe** (userdebug + `adb root`;
|
||
active touch device = the one under `Touch Input Mapper (mode - DIRECT)`
|
||
in `dumpsys input` — on pad_avd that's event2, NOT the one `getevent`
|
||
prints first; per-axis raw→px: X×W/32767, Y×H/32767): MT-B with explicit
|
||
`ABS_MT_SLOT`(47) per finger, `ABS_MT_TRACKING_ID`(57) re-emitted on
|
||
every move frame (values ≥0 = present, -1 = lifted), one SYN(0 0 0) per
|
||
frame, BTN_TOUCH(1 330) on first-down/last-up. Each `sendevent` is a
|
||
fork (~20 ms), so `input tap x y; input tap x y` in ONE shell is the
|
||
way to make a ≤300 ms double-tap.
|
||
Gotchas (also in §6.14–16): the close button MUST be declared after the
|
||
full-surface TouchArea (TouchArea grabs + aborts sibling traversal);
|
||
`ScaleRotateGestureHandler` is declared in the compiler crate, not any
|
||
.slint file; Slint logical px = physical px / (dpi/160) on Android.
|
||
- **M5 — permissions**: implement `perm.rs` per §7; launch flow when
|
||
un-granted (settings card → open Settings → recheck on Resume); test the
|
||
exact passgo-style flow on a fresh install.
|
||
- **M6 — polish**: dark theme, animations (slide/fade), search/filter
|
||
(name/date), sort; double-tap zoom; battery-friendly (pool idles, no
|
||
busy repaint — Slint is retained, verify with gfxinfo "total frames").
|
||
- **M7 — perf & release**: real-device (aarch64) pass; Perfetto; memory
|
||
bounds; release APK size check (expect ~8–9 MB like the sample);
|
||
consider the 1.19 upgrade (verify pointer-scroll fix first).
|
||
|
||
## 10. Open questions & risks
|
||
|
||
1. **HEIC (RESOLVED for now)**: the real device has 0 HEIC files (9109
|
||
JPEG, 27 PNG), so HEIC is deferred. When needed: add a decoder to
|
||
`images.rs` dispatch (libheif-rs / `image` heic feature / libvips);
|
||
the engine API is decoder-agnostic.
|
||
2. **Slint pointer-scroll `todo!()` panic** (open issue in the Android
|
||
backend at eval time): could fire on trackpad/touch scroll. Test scrolling
|
||
on M3; if it fires, patch locally (the fix is small) or upgrade to 1.19
|
||
when available.
|
||
3. **Max texture size**: full-screen decode of 12 MP+ images (e.g.
|
||
8192 px) may exceed GPU max texture size on some devices — verify on
|
||
M4; fall back to a 2×-max downscale for display.
|
||
4. **Slint license**: royalty-free tier covers proprietary mobile/desktop/
|
||
web; embedded devices excluded — irrelevant for us, but the repo README
|
||
should state the chosen tier (royalty-free) once the app ships.
|
||
5. **Docs drift**: trust the pinned source (`~/prog/android/ui/slint-src`
|
||
@ v1.18.1) over online docs; bump pin + re-verify after any Slint upgrade.
|
||
6. **rusqlite `bundled`** compiles SQLite from source in every target
|
||
(adds ~30–60 s to clean builds) — acceptable; revisit if it hurts.
|
||
7. **Transient black screen (M2 first launch, unreproducible)**: one
|
||
aarch64-release launch rendered pure black with a dead drain timer until
|
||
relaunch; not reproduced on emulator or device since. Candidate fix if it
|
||
recurs: in the `init_with_event_listener` Resume handler, call
|
||
`window.request_redraw()` / check `window.size()` (a 0×0 or stale surface
|
||
after a screen-off start would match the symptoms). Don't add it
|
||
speculatively.
|
||
|
||
## 11. Pointers
|
||
|
||
- Framework eval (full report + raw notes + both sample apps + screenshots):
|
||
`~/prog/android/ui/` — `REPORT.md`, `NOTES.md`, `slint_app/`, `eframe_app/`,
|
||
`slint-src/` (v1.18.1), `egui-src/` (0.37.0).
|
||
- Slint docs (version-sensitive!): https://slint.dev/docs — Android guide:
|
||
`docs/guide/platforms/mobile/android.mdx` in the source tree.
|
||
- `examples/native-gestures`, `demos/` in `slint-src` — reference code for
|
||
gestures, lists, images.
|
||
- passgo (existing Go app using the same all-files-access pattern):
|
||
`~/prog/go/passgo` — its `scripts/build.sh` and manifest handling are a
|
||
proven reference for the permission UX flow.
|
||
- Emulator AVD: `pad_avd` (google_apis x86_64, API 35); start command in §3.
|
||
- Gitea: push target `git+ssh://gitea@git.wow.st/gmp/gallery.git`
|
||
(repo must exist on the server first, like the other gmp/* repos).
|
||
|
||
### 8.7 GPU-resident thumbnails (jank fix, 2026-10-28)
|
||
|
||
Root cause of the perceived scroll jank (measured with the per-frame
|
||
`set_rendering_notifier` hook, `FR` lines in `.gallery_perf.log`): every
|
||
**first present** of a new `Image` uploaded its pixels inside the render
|
||
frame — Slint's Skia backend has no RGB888 path (byte-by-byte conversion
|
||
on the UI thread) and no API to pre-upload; measured 11–30 ms per 400 px
|
||
texture = 1.3 visible hitches/s at 120 Hz. Grey-image A/B: 20 hitches/s →
|
||
1. Not a frame-rate, input, or layout issue; Fossify (RenderThread
|
||
pipelining) doesn't have this cost.
|
||
|
||
Fix (no Slint patch, no re-architecture): the thumbnail is uploaded to
|
||
Slint's own wgpu device (captured from `GraphicsAPI::WGPU30` at
|
||
`RenderingSetup`, `src/gputex.rs`) via `queue.write_texture` (async, no
|
||
fence wait) BEFORE the presenting frame, so first present costs ~0.
|
||
`Image::try_from(texture)` needs `Rgba8Unorm` +
|
||
`TEXTURE_BINDING|RENDER_ATTACHMENT` (`unstable-wgpu-30` slint feature;
|
||
wgpu/Vulkan `bytes_per_row` 256-multiple padding). Fallback to
|
||
`from_rgba8` CPU path when the device isn't captured yet.
|
||
Result (Pixel 9 Pro, 120 Hz, 400 px): **zero mid-scroll renders > 4.4 ms**
|
||
(median 1.9 ms, p95 2.9 ms); only post-idle first-frames are expensive
|
||
(invisible — screen static).
|
||
|
||
**Revised 2026-10-28 (recreation bug, see
|
||
docs/BUGS/recreation-stale-textures.md):** the upload now runs **on the
|
||
UI thread** (drain tick, between frames) instead of on the decode
|
||
workers. Workers deliver plain RGBA8 bytes; `gputex::upload_rgba` creates
|
||
the texture + `write_texture` on the device captured from the rendering
|
||
notifier — Slint's borrowed-resource contract (use the context the
|
||
notifier hands you). A long-lived worker holding a device was the source
|
||
of stale-texture panics across activity recreation; with render-thread
|
||
uploads a texture is always created on the device of the world that
|
||
renders it. The keep-alive thread (default-off anyway) was REMOVED with
|
||
the worker-side upload.
|
||
|
||
Perf file: `/sdcard/Download/.gallery_perf.log` (SCR/TICK/APLY/FR).
|
||
Clear it by restarting the app (writer holds the fd).
|
||
|
||
### 8.9 Scroll Jank Deep-Dive & WGPU/Vsync Optimization (2026-10-28)
|
||
|
||
Exhaustive profiling on physical 120Hz hardware (Pixel 9 Pro, Mali G715,
|
||
Android 17) investigated remaining mid-scroll frame gaps (18–30 ms) after the
|
||
M3 thumbnail engine and GPU-resident textures were verified.
|
||
|
||
**Findings & Fixes:**
|
||
1. **WGPU Swapchain Tail Stalls (`queue.present`):**
|
||
- *Cause:* WGPU surface was configured with `PresentMode::AutoVsync`
|
||
(`Fifo`), which enforces strict vsync backpressure on the render thread
|
||
at `queue.present(frame)`. If a frame missed the 8.33 ms 120Hz deadline by
|
||
even 0.1 ms, `Fifo` blocked the thread for an extra vsync cycle.
|
||
- *Fix:* Configured `surface_config.present_mode` to prefer
|
||
`wgpu::PresentMode::Mailbox` (falling back to `Immediate`), decoupling CPU
|
||
presentation submission from vsync display fence blocking.
|
||
- *Result:* Swapchain tail maximum (`lrmax`) dropped from **54.7 ms down to
|
||
<8 ms**. App-side render cost (`d`) measured **0.7–1.4 ms**.
|
||
2. **UI Hierarchy Flattening:**
|
||
- Flattened `CellView` in `src/main.slint` (removed inner placeholder
|
||
Rectangle) to reduce item-tree node count per cell and speed up Slint's
|
||
scene graph traversal during scroll.
|
||
3. **Event Loop Pacing:**
|
||
- Increased `DRAIN_INTERVAL_MS` to 16 ms (60Hz) to halve UI-thread reconcile
|
||
overhead during 120Hz gestures.
|
||
- Capped Android event loop poll timeout to 6 ms during animations.
|
||
- Empirical list-length A/B testing (100 entries vs 8,852 entries) proved
|
||
that the remaining micro-variance (18–27 ms vsync cadence quantization)
|
||
is structural to Slint 1.18's Android event loop polling (`poll_events`
|
||
Choreographer synchronization) and independent of app structure or model
|
||
size.
|
||
|
||
Instrumentation (`FR`, `STG`, `DECODE`) is gated behind the `--features perf`
|
||
Cargo feature so production builds have zero runtime overhead. Detailed bug notes
|
||
and analysis are preserved in `docs/BUGS/scroll-jank-120hz.md`.
|
||
|
||
### 8.10 Zero-jank campaign: the reliable perf tool (2026-10-30)
|
||
|
||
The zero-jank work needed a referee that cannot be argued with. Two tools:
|
||
|
||
- **`tools/jank_eval.py`** — the strict referee. Slot grid built from the
|
||
hardware VSYNC edges in the log (not the app's clock); a window is PASS
|
||
only with: no dropped slot, no same-slot double, no present past latch
|
||
(+5 ms edge phase), no >20 ms present gap, phase p10-p90 ≤ 0.5 ms.
|
||
7-case self-test (`--selftest`). Exit 0=PASS, 1=FAIL, 3=SUSPECT.
|
||
- **`tools/jank_capture.py`** — owns the whole capture pipeline so
|
||
provenance can never be wrong:
|
||
`python3 tools/jank_capture.py capture --app wow.gmp.sltest --drags 4,2,4 --tag <build> --tag-file <apk>`
|
||
records build identity → force-stops → **deletes the log** (the file can
|
||
then contain exactly ONE process) → cold-launches → verifies
|
||
ResumedActivity → scripted drags → pulls the complete log → reports
|
||
PROVENANCE + jank verdict + **MISS forensics** (each miss episode
|
||
classified: CHAIN-STALL / LOOP-STUCK / GUARD-REJECT / NO-WAKE-ITER,
|
||
with the ±33 ms all-thread context: UI loop, vsync thread, workers,
|
||
slow C calls, IO). Exit: 0=PASS, 1=FAIL/INVALID/SUSPECT.
|
||
`jank_capture.py report FILE` re-analyzes any captured log.
|
||
|
||
Both validated on the emulator (x86_64, 60 Hz SF): capture pipeline
|
||
×3 drag profiles, pause/resume mid-drag, determinism, multi-process
|
||
→ INVALID, explicit windows, period override, no-VSYNC → SUSPECT path,
|
||
small-log and PASS paths. Stress-testing found and fixed a circular
|
||
auto-window threshold (a janky app's low present rate hid it as
|
||
"inactive"), a BUCK multiplier bug, and a useless fallback period fit.
|
||
|
||
**Zero-jank status (phone, sltest):** phase locked (single wake source:
|
||
Choreographer; early waker disabled — it raced Choreographer and caused
|
||
±4 ms wander). Remaining issue: 20-40 ms chains of lost wakes (the
|
||
Choreographer callback chain breaks); every frame that RAN took 3.7 ms
|
||
(6 ms budget) — the CPU mystery is answered: no missing CPU work, the
|
||
jank is frames that never ran. Next phone run: `jank_capture.py capture
|
||
... --tag <build>`; the MISS forensics (sw/sn/eage/lastage fields)
|
||
will classify each break.
|
||
|
||
### 8.11 egui port decision + egtest harness (2026-10-31)
|
||
|
||
**Verdict on Slint:** stock crates.io build janky (user), vendored build
|
||
same/worse (user), minimal sltest (300 rows, no textures) janky (user),
|
||
adb-measured drops (0.05-0.83%) are a FLOOR — the user's finger says
|
||
"much more than 1 hitch per 8 s". **The base Slint-on-Mali render path is
|
||
the problem. User decision: port to egui** (different toolkit AND a
|
||
different renderer family: glow/OpenGL-ES3 vs Skia-on-wgpu).
|
||
|
||
**egtest** (`/tmp/egtest`, standalone, same discipline as sltest) — the
|
||
CONTROL experiment: the exact sltest scene (8,580 rows × 150 px, 148 px
|
||
color band + 2 px white separator) in egui 0.31.
|
||
- egui_glow/egui-winit 0.31.1, glow 0.16, winit 0.30, android-activity
|
||
0.6, `egl` 0.2.7 safe wrapper. APK 1.98 MB. `wow.gmp.egtest`.
|
||
- Virtualized list: only visible rows ± 4 painted per frame via the
|
||
painter (immediate mode = free virtualization). Momentum fling
|
||
(0.94 decay, 8 px/frame min, seed = last drag delta × 1.5).
|
||
- FR log (sltest pattern, non-blocking logger, per-line flush):
|
||
`/sdcard/Android/data/wow.gmp.egtest/files/fr.log`.
|
||
- Build: `cd /tmp/egtest && ADB_SERIAL=... ./scripts/build.sh [aarch64|x86_64] [install|no-install]`.
|
||
- Crates.io naming gotcha: `egui_glow` (underscore) vs `egui-winit`
|
||
(dash) — the index is inconsistent; winit's android feature is
|
||
`android-native-activity`; cargo-apk2 needs the activity section in
|
||
Cargo.toml (NativeActivity + lib_name) and `config_changes` as a
|
||
pipe string of aapt2 flag NAMES (no `keyboardNavigation` — it's
|
||
`navigation`; hex/decimal bitmasks rejected).
|
||
|
||
**Two real phone bugs found + fixed (they will bite the final port too):**
|
||
1. **winit 0.30 android: `RedrawRequested` is never dispatched on the
|
||
phone.** `MainEvent::Resume` never arrives → `EventLoopInner.running`
|
||
stays false → the RedrawRequested dispatch (inside `if self.running`)
|
||
is suppressed; `about_to_wait` is not. Emulator delivers Resume
|
||
fine — phone (pre-rotated display) does not. **Fix: drive frames from
|
||
`about_to_wait`** — render there + `ControlFlow::WaitUntil(now+8333µs)`
|
||
(app-owned pacing, same architecture as the Slint frame driver).
|
||
Phone result: ~698 frames / 10 s (≈100 Hz) rendering the scene on
|
||
Mali+ES3 with 1-2 ms render cost.
|
||
2. `Window::request_redraw()` via ALooper_wake was unreliable here too —
|
||
the WaitUntil pacing removes the dependency entirely.
|
||
|
||
**Current state (handoff):** egtest is INSTALLED on the phone but the
|
||
phone's WM is in a degraded state (NotificationShade stuck in focus,
|
||
`dumpsys window` broken — "Broken pipe", load avg ~6 from the user's
|
||
apps). The app was last seen rendering into a GONE window (WM never
|
||
granted visibility — plausibly a cascade of the stuck-shade state, not
|
||
an app bug). **NEEDS THE USER**: glance at the phone (close the shade /
|
||
check for a stuck system UI), then launch **Egtest** from the launcher
|
||
and scroll it hard. Question to answer: *is egtest smooth where sltest
|
||
(janky) rendered the identical scene?* FR log is live for drop analysis
|
||
afterwards (`grep FR fr.log`). If smooth → port the gallery app layer
|
||
(db/scan/turbojpeg/LRU/GPU-upload, all renderer-agnostic) to an egui
|
||
grid + the about_to_wait frame driver + EGL setup from egtest.
|
||
|
||
### 8.8 Jank-hunt round 2 + the keep-alive incident (2026-10-28)
|
||
|
||
Follow-up on "still seeing jank" (post-GPU-texture). Instrumentation:
|
||
in-app thread sampler (`src/jsamp.rs`, `/proc/self/task`, knob
|
||
`.gallery_jsamp`), per-core cpufreq, cgroup/PSI probes. Findings:
|
||
|
||
- **Cgroup CPU quota ruled out**: this device's unified hierarchy has the
|
||
`cpu` controller DISABLED (`cgroup.controllers` = `memory` only; no
|
||
`cpu.max` file) — no per-app bandwidth cap exists.
|
||
- **`/proc` facts**: utime/stime are USER_HZ ticks (1/100 s), so
|
||
per-thread CPU deltas are 10 ms-quantized. The app's SELinux context
|
||
DENIES `/proc/stat`, `/proc/pressure/cpu`, and the unified cgroup files
|
||
(even the app's own cgroup), but allows `/proc/self/task/*` and
|
||
cpufreq `scaling_cur_freq`. **A denied read in a tight loop is an ANR:
|
||
each denial is an audit event; ~100/s froze the whole phone.**
|
||
jsamp now probes readability once and disables denied features forever.
|
||
- **Real profile** (single verified 120 Hz swipe, 370 frames): steady
|
||
scroll = 1.5–2.5 ms frames, **zero spikes**. All 20–40 ms frames were
|
||
the first frame after an input pause (15–100 ms) — resume/warm-up
|
||
cost; ~820 MHz little-core clock during spikes vs 1.95 GHz between
|
||
(EAS downclock in the gap, not a cgroup/driver block). Swipe starts
|
||
after only 2 s pauses: zero spikes; only >10 s app idle pays the big
|
||
(43 ms) cold frame.
|
||
- **KEEP-ALIVE INCIDENT (phone freeze)**: the `gputex::Keepalive`
|
||
thread (16 ms buffer-copy submits on Slint's wgpu queue from a 5th
|
||
thread) made the display pipeline FREEZE on Mali G715 during a 14-min
|
||
session — phone UI hung for seconds, 51 in-flight decodes stalled ~6 s
|
||
(workers also `write_texture` on the same queue). System did not
|
||
reboot (system_server/surfaceflinger 5-day uptime), but the user
|
||
experience was "crashing my phone". **Root lesson: do not run
|
||
concurrent submitters on Slint's Mali queue.** Keep-alive is now
|
||
INACTIVE BY DEFAULT (debug knob `.gallery_keep`). The jsamp sampler
|
||
was REMOVED in the 2026-10-28 cleanup (see §8.9).
|
||
- Status (post jank-fix, §8.9): with the Skia final-flush removed, the
|
||
mid-scroll stalls are GONE (0 stalls, max 3.9–7.4 ms across 4 runs,
|
||
phone). Residual after the fix: scroll-START ramp frames (16–27 ms
|
||
in the first ~400 ms after >2 s idle, CPU-clock ramp on the 820 MHz
|
||
little cores) — revisit only if the user still notices them.
|
||
|
||
### 8.9 Scroll jank — root cause & fix (2026-10-28, CLOSED)
|
||
|
||
The mid-scroll 19–26 ms stalls (2–4 per run, at fixed content offsets,
|
||
Mali-only — emulator/SwiftShader never reproduced) are **fixed**.
|
||
|
||
**Root cause (proven by stage-timestamp profiling of the vendored
|
||
renderer):** the Slint Skia renderer's final synchronous
|
||
`GrDirectContext::flush()` at the end of every frame
|
||
(`render_components_to_canvas`) blocked the UI thread 19.2–20.4 ms
|
||
whenever the Mali driver queue was momentarily busy (texture uploads
|
||
landing, first-draw pipeline state). Steady-state cost of that call:
|
||
0.3–0.8 ms. SwiftShader's flush returns instantly → emulator never
|
||
reproduced it, and every CPU-side hypothesis failed.
|
||
|
||
Full evidence, per-stage tables, prior art and the upstream-filing
|
||
question: **`docs/BUGS/slint-skia-final-flush-jank.md`**.
|
||
|
||
**Fix (kept, per user decision):** `vendor/i-slint-renderer-skia`
|
||
(pinned 1.18.1, `[patch.crates-io]`) with that one flush removed.
|
||
It is redundant on the wgpu backend: `wgpu_30_surface::render` calls
|
||
`flush_and_submit` (Skia `submit()`) before `queue.present`, same
|
||
Vulkan queue, in-order. `vendor/` contains ONLY this crate; the rest
|
||
of the patch set is a single commented-out hunk with an explanation.
|
||
Re-apply that hunk when bumping Slint. Upstream NOT filed yet (we
|
||
cannot be 100% sure the flush isn't doing a job for non-wgpu
|
||
backends) — the bug doc is written so it can be filed later.
|
||
|
||
**What the profiling also settled:**
|
||
- Slint 1.18 DOES prune the item walk (~133–152 visited items, not
|
||
O(rows)). The earlier O(n)-walk diagnosis (and the windowed
|
||
Flickable that was built on it) was WRONG; the windowed Flickable
|
||
was reverted per user directive (use Slint's ListView — it is
|
||
smooth once the flush is gone).
|
||
- Stall frames had 0–3 new Skia images (0.1 ms each), 1 binding
|
||
re-evaluation, 0 ms partial-renderer: none of the usual suspects.
|
||
|
||
**Verification (phone, Mali G715, 8,580 photos):**
|
||
- slow scroll 5 s ×3: 0 stalls, max 3.6 / 5.8 / 3.9 ms
|
||
- fast fling 8× velocity: 0 stalls, max 7.4 ms, p50 1.0 ms
|
||
- display screenshot-verified clean
|
||
- budget at 120 Hz is 8.3 ms → 65–99 % headroom
|
||
|
||
**Cleanup done (2026-10-28):** all jank-hunt instrumentation stripped
|
||
(FR/SCR/TICK/APLYD/STG logging, `stg.rs`, `.gallery_*` knobs, jsamp
|
||
thread sampler, perf-file mirroring in nblog, DbgScroll). KEPT:
|
||
nblog non-blocking logger (logd backpressure is real), zero-dimension
|
||
decode guards (thumb.rs + gputex.rs), debug-level coverage logs.
|
||
(keep-alive REMOVED later the same day with the worker-side GPU upload —
|
||
see docs/BUGS/recreation-stale-textures.md.)
|
||
|
||
**BUG FIXED while cleaning:** the app launched into an unstoppable
|
||
fast scroll on the user's phone — a leftover `.gallery_cmd`
|
||
(`scroll -12 4000`) from the last A/B run: DbgScroll never deleted
|
||
the file, and a fresh process sees `cmd_mtime == None` → re-reads it
|
||
on every launch; while active it also overrode the user's swipes.
|
||
Fix: the debug-scroll hook (and all file knobs) are gone; the
|
||
leftover file was deleted from the device.
|
||
|
||
**Next: M5 permissions** — implement `perm.rs` per §7 (launch flow when
|
||
un-granted: settings card → open Settings → recheck on Resume); test the
|
||
passgo-style flow on a fresh install. (M4 viewer is DONE, 2026-10-29 —
|
||
see the M4 entry in §9 for the verification notes + emulator pinch
|
||
injection recipe.)
|
||
|
||
### 8.10 Thumbnail pop-in — root causes & fix (2026-10-30, CLOSED, `b9317e7`)
|
||
|
||
User report: on the warm 8580-file DB the thumbnails "snap in" slowly
|
||
after a flick; also requested a test of the COLD launch (fresh scan,
|
||
streamed list). Both turned out to be drain-tick / scan-pipeline bugs,
|
||
not render bugs.
|
||
|
||
**Bug A — paint-cap drop (warm + cold).** The drain tick paints at most
|
||
`MAX_PAINT_CELLS_PER_TICK` (2) cells/tick; finished decodes beyond the
|
||
cap were **silently dropped** (removed from `inflight`, never added to
|
||
`painted`). FILL then re-submitted them → the same cell re-decoded
|
||
3–6× per viewport; a new viewport's ~21 cells took 500–850 ms to fill
|
||
after a flick. The flick bursts themselves were clean 120 Hz — all the
|
||
"late frames" were in the settle window.
|
||
Fix: excess results hold in `app.ready` (bounded `READY_CAP=48`,
|
||
`app.ready_paths` keeps FILL from re-submitting) and paint on a later
|
||
tick at zero re-decode cost. Cap is adaptive: 2 during grid motion
|
||
(frame budget), 4 when settled.
|
||
|
||
**Bug B — cold-start scan storm.** Every 512-entry scan batch re-sorted
|
||
the WHOLE accumulated list and `set_list` did a full model reset
|
||
(`set_vec`) + generation bump + state clears. Sorted prefixes are NOT
|
||
stable as the set grows (new entries interleave by mtime), so each
|
||
batch orphaned every queued/in-flight decode (arrived marked stale →
|
||
dropped → re-submitted): a 17× re-decode storm (max observed dups per
|
||
file = 18 = batch count) plus full row churn per batch.
|
||
Fix: batches now stream in APPEND order (each 512-group mtime-sorted,
|
||
groups concatenated → stable prefix); `set_list` detects appends
|
||
(element-wise prefix match) and extends the model in place
|
||
(`extend_from_slice` — existing rows/textures untouched) with NO
|
||
reset/generation-bump. The final global sort (DB order) is ONE reset at
|
||
scan end; the viewport re-fills from the disk cache (~2 ms/cell).
|
||
Measured on phone (cold, fresh DB): scan 7637 ms → ~4.3–4.9 s, decode
|
||
dups 18 → 2–3, grid renders with thumbnails while streaming.
|
||
|
||
**Bug C — black window after pause (found while testing A/B).** The OS
|
||
can invalidate our surface (system dialog activity, app switch, screen
|
||
lock) with NO window event and no pending model change; nothing ever
|
||
sets `pending_redraw` again → the window comes back black and stays
|
||
black until the next content mutation.
|
||
Fix: (1) `request_redraw()` unconditionally in `RenderingSetup`;
|
||
(2) drain-tick **window-liveness watchdog**: window visible but
|
||
`LAST_PRESENT_US` (updated in `AfterRendering`) stale >2.5 s → one
|
||
forced redraw. `now_us`/`NOW_T0` are no longer `perf`-gated (the
|
||
watchdog needs them in all builds).
|
||
|
||
**Also: `debuggable = false`** (was `true` for simpleperf during the
|
||
jank hunt). A debuggable app shows the system **16 KB compatibility**
|
||
dialog on every cold launch; that dialog is its own activity, it
|
||
paused us, and its inconsistent lifecycle (TerminateWindow or not,
|
||
depending on timing) was the main black-window trigger in testing.
|
||
NOTE: the .so is 4 KB-aligned (fine on this 4 KB-page kernel); on a
|
||
16 KB-page kernel a non-debuggable build would be refused — revisit if
|
||
targeting such devices.
|
||
|
||
**Phone verification (cold, 8580 files, fresh DB, exclusive phone):**
|
||
clean launch (449 ms to window, no dialog), grid streams in with
|
||
thumbnails, two fast flicks render 120 Hz (median 8.3–8.5 ms, 0–2 late
|
||
frames over ~220 frames), viewport fully painted after the flicks,
|
||
grid intact after an app switch (Termux 4 s) — no black window.
|
||
|
||
**Test hygiene learned the hard way:** the phone is a shared personal
|
||
device — Fossify apps / Termux / the lock screen / the notification
|
||
shade kept stealing foreground mid-test, which masquerades as "app
|
||
not processing input / black screen" (the input lands on the other
|
||
app). Always: verify `ResumedActivity` before AND after each
|
||
measurement, keep swipes mid-screen (y 700–2100; bottom-edge = lock,
|
||
top-edge = shade), and re-pull the perf file from a recorded byte
|
||
offset (`stat -c %s` before launch) — the file accumulates across
|
||
processes. `nev=` in IT lines under-counts motion (a visual scroll is
|
||
the ground truth).
|