Greg Pomerantz
9c0525e93f
Slint's android-activity backend wakes the main loop from a dedicated "slint vsync" Choreographer thread while any animation is active (i-slint-backend-android-activity/vsync.rs). Android schedules that thread at nice +10 (lowest on device), while our 4 turbojpeg workers inherited the UI thread's nice -10. Under CPU bursts the +10 waker starves and the main loop's next wakeup lands 3-4 frames late: 25-33 ms of pure wake latency at 120 Hz, with zero cost in our tick (d=7-34 us) — exactly the phone stall signature (measured: 28-39 ms gaps, catch-up events). - prio module: setpriority FFI (self + by-tid), raise_slint_vsync() - thumb workers + dispatcher: nice +2 at spawn (ahead of background apps, behind the UI's -10) - android_main: prio-fix thread raises "slint vsync" to nice 0 with a readback log - verified on emulator: thumb-* hold nice 2, slint vsync holds 0 while foreground (Android re-nices backgrounded processes — expected) - emulator scroll cadence unchanged (llvmpipe rasterization ceiling, not a priority issue) — phone is the acceptance target, A/B test planned for when it is back online Also: drop unused android_logger dependency (nblog replaced it).
609 lines
37 KiB
Markdown
609 lines
37 KiB
Markdown
# Gallery — development hand-off
|
||
|
||
A high-performance image gallery for Android (primary), Linux and macOS,
|
||
written in Rust with the **Slint** UI toolkit. This document is the starting
|
||
point for development: it captures every decision, gotcha, and measurement
|
||
from the framework evaluation that preceded it, so no learning has to be
|
||
rediscovered.
|
||
|
||
**Status: M3 done (2026-10-27, twice overhauled)** — M0 scaffold, M1 metadata
|
||
scan + sqlite + instant list paint, M2 thumbnail engine (turbojpeg DCT-shrink
|
||
decode, worker pool, disk cache, cancellation), M3 grid UI (3-column ListView
|
||
grid, shimmer placeholders, memory-bound cell retention) all verified: 15
|
||
unit tests (incl. two queue-priority regressions), 5k synthetic engine test,
|
||
and on a real Pixel 9 Pro (8,577 photos: 60 fps, 0 hitches). The thumbnail
|
||
loader was rewritten TWICE on 2026-10-27 because "stop → long shimmer → small
|
||
move → everything pops" kept surviving: first the scroll handler forgot cells
|
||
when scrolling stopped (fix: **viewport reconciliation** — the 10 Hz tick is
|
||
the sole authority, the invariant "every viewport cell is painted / in flight
|
||
/ failed" is restored unconditionally each tick and `debug_assert`ed), then
|
||
the worker-level FIFO still let stale transit work delay a fresh viewport
|
||
(fix: **worker-queue prepend + cap** — bursts are prepended ahead of older
|
||
queued work; per-worker cap drops the oldest tail with a `skipped` result).
|
||
Stress-verified on the emulator (debug build) + phone before shipping:
|
||
fast up/down 0–120 ms to full coverage, cold landings bounded by decode
|
||
throughput, chaos scrolling max oldest in-flight job 217 ms (was 8 s),
|
||
0 invariant violations in ~1,450 ticks, 0 panics/ANRs. Next: M4 viewer.
|
||
|
||
---
|
||
|
||
## 1. Goals
|
||
|
||
- Browse and view a local photo collection (thousands of images) that stays
|
||
smooth: instant grid, no decode jank, cached thumbnails.
|
||
- Full-screen viewer with pinch-zoom/pan.
|
||
- Correct EXIF orientation, HEIC support (Android camera default).
|
||
- Same codebase on macOS + Linux (desktop `cargo run`) and Android
|
||
(`cargo apk2`).
|
||
- Quality of the *platform experience* matters: safe-area insets, IME-free
|
||
UI, native-feeling widgets, animations — first-class support preferred over
|
||
manual patching.
|
||
|
||
Non-goals (for now): editing, cloud sync, video, widgets/live-updates.
|
||
|
||
## 2. Why Slint (decision summary)
|
||
|
||
Full evaluation in `/mnt/Backup/prog/android/ui/REPORT.md` (same tree:
|
||
`~/prog/android/ui/REPORT.md`), raw notes in `NOTES.md` there. Headline:
|
||
|
||
| | Slint 1.18.1 | eframe/egui 0.37.0 |
|
||
|---|---|---|
|
||
| Virtualized list (the gallery's lifeline) | ✅ `ListView` std-widget, only visible rows instantiated | ❌ `ScrollArea::show` lays out all children; `show_rows` = manual virtualization API |
|
||
| Async image → UI | swap `slint::Image` into a property; only that tile redraws (retained mode) | `ctx.load_texture` + custom loader; visible set rebuilt every frame |
|
||
| Lifecycle hook for permissions | ✅ `init_with_event_listener` (Start/Resume/Pause/Stop/SaveState/...) | ❌ none in `App` trait (winit swallows it) |
|
||
| Android safe-area | ✅ `safe-area-insets` built in | manual 32px-ish panel workaround (winit#3910) |
|
||
| Animation | ✅ declarative built-in engine | hand-rolled |
|
||
| IME text entry | ✅ | ✅ |
|
||
| APK size (release aarch64) | 8.57 MB | 2.87 MB |
|
||
| Clean build time (x86_64, this machine) | 195 s / 270 crates | 76 s / 137 crates |
|
||
| Desktop Linux system deps | `libfontconfig1` + `pkg-config` | none |
|
||
| License | royalty-free for proprietary mobile/desktop (embedded excluded) / GPLv3 / commercial | MIT OR Apache-2.0 |
|
||
| Min Android API | 26 | 24 |
|
||
|
||
For a stateful, form-and-grid app, the declarative model was the right fit
|
||
(see the discussion about `pad`'s immediate-mode frame friction); egui wins
|
||
for games/dashboards and on APK size. Known Slint caveat: an open upstream
|
||
"abort on pointer scroll" bug in the Android backend (`todo!()` in the
|
||
pointer-event path) — verify it's fixed in 1.19 before relying on heavy
|
||
scrolling; if it fires, the panic is obvious in logcat.
|
||
|
||
## 3. Environment, build & test loop
|
||
|
||
Everything below is verified working on this machine (2026-10-27).
|
||
|
||
```bash
|
||
export ANDROID_HOME=$HOME/android-sdk
|
||
export ANDROID_NDK_ROOT=$HOME/android-sdk/ndk/27.0.12077973
|
||
export PATH="$HOME/.cargo/bin:$ANDROID_HOME/platform-tools:$PATH"
|
||
|
||
# Rust 1.99.1 via rustup; tools: cargo-apk2 1.5.0, cargo-ndk 4.1.2
|
||
./scripts/build.sh debug # debug x86_64 APK + adb install + launch
|
||
./scripts/build.sh # release x86_64 (emulator)
|
||
./scripts/build.sh aarch64 # release aarch64 (device)
|
||
cargo run # desktop (Linux needs libfontconfig1+pkg-config)
|
||
```
|
||
|
||
- Emulator: `emulator -avd pad_avd -no-window -no-audio -gpu off -no-snapshot`
|
||
(google_apis x86_64, Android 15). **`-gpu off -no-snapshot` matters** —
|
||
swiftshader_indirect + snapshot crashed the emulator once during the eval.
|
||
If it won't start, the host key lost its wireless pairing — see the pairing
|
||
procedure in `~/.pi/agent/AGENTS.md` (phone) / emulator docs (this AVD).
|
||
- APK lands at `target/<debug|release>/apk/gallery.apk`.
|
||
- Release signing uses the debug keystore
|
||
(`CARGO_APK_RELEASE_KEYSTORE=$HOME/.android/debug.keystore`,
|
||
`CARGO_APK_RELEASE_KEYSTORE_PASSWORD=android`) — fine for personal use;
|
||
for Play, generate a real keystore and point the env vars at it.
|
||
- `cargo apk2` quirk: `--target` (not `-t`). `--lib` builds the cdylib.
|
||
- Grant "All files access" to the app on the emulator after install:
|
||
`adb shell am start -a android.settings.MANAGE_ALL_FILES_ACCESS_PERMISSION`
|
||
(or Settings UI). `adb install -r` keeps the grant.
|
||
- **Emulator test data (M3)**: the AVD's `/Pictures/gallery-test/` has ~5k
|
||
*fake* JPEG stubs (zlib garbage — they scan fine but fail to decode, good
|
||
for volume tests). Real, decodable test JPEGs live in
|
||
`/Pictures/realtest/` (150, generated with PIL). They were `touch`-ed to a
|
||
future mtime so they sort to the TOP of the grid — when reasoning about
|
||
"which cells should have images", remember only cells 0–149 are real;
|
||
everything below is stubs and correctly shows shimmer. **`/Pictures/realtest_extra/`
|
||
holds 700 more real, decodable JPEGs (PIL, 1200–4032 px, mtime 2031 so they
|
||
sort to the very top, above the 150 in `realtest`) — added for the
|
||
scroll-back backlog repro; the real region is now ~850 files (283 rows).
|
||
`adb push` of a directory nests a subfolder (push files individually or
|
||
`mv` up); `adb shell touch -d '…'` works for setting mtimes. The thumb cache is at
|
||
`/sdcard/Android/data/wow.gmp.gallery/files/thumbs` (wipe it to force
|
||
re-decode).
|
||
- Perf measurement: `adb shell perfetto` / `adb shell dumpsys gfxinfo wow.gmp.gallery`,
|
||
`systrace`-style capture via Perfetto trace shell. Cold start:
|
||
`adb shell am start -W`.
|
||
|
||
## 4. Architecture
|
||
|
||
The UI framework is only the display layer. The performance lives in the
|
||
thumbnail engine, and it is framework-agnostic:
|
||
|
||
```
|
||
┌─ metadata ─────────────────────────────────────────────────────────┐
|
||
│ sqlite (rusqlite, bundled): path, mtime, size, w, h, orientation, │
|
||
│ thumb_path, date_taken — one background scan pass │
|
||
└──────────────┬─────────────────────────────────────────────────────┘
|
||
│ model → Slint (list of visible-range items)
|
||
┌─ thumbnail pool (tokio-free: N std::threads, N ≈ cores/2) ─────────┐
|
||
│ per job: token cancellation (AtomicU64 gen counter) + priority │
|
||
│ 1. disk cache hit? .thumbs/{sha1(path|mtime|size)}.jpg → done │
|
||
│ 2. decode AT TARGET SIZE (never full-size for the grid): │
|
||
│ JPEG: zune-jpeg "turbo" (decode+downscale in ONE pass, │
|
||
│ pure Rust, actively maintained) or turbojpeg (SIMD) │
|
||
│ HEIC: imagepipe (0.5, 2026-08) — VERIFY Android NDK build │
|
||
│ PNG: image/png │
|
||
│ 3. apply EXIF orientation (once, here — never per frame) │
|
||
│ 4. re-encode small JPEG → disk cache │
|
||
│ 5. channel → UI: (id, slint::Image) │
|
||
│ prefetch: decode visible ± 1 row; cancel items scrolled out │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
┌─ UI (Slint) ───────────────────────────────────────────────────────┐
|
||
│ grid screen: ListView (vertical) → row of N square Image cells │
|
||
│ viewer screen: full-size async decode (once, cached) + manual │
|
||
│ two-finger pinch transform (no built-in in 1.18) │
|
||
└────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
Module layout:
|
||
|
||
```
|
||
src/
|
||
lib.rs # android_main + platform init + scan/UI glue (M1/M2)
|
||
main.rs # desktop entry [exists]
|
||
main.slint # screens (split into ui/ components at M3) [exists]
|
||
db.rs # sqlite schema + incremental scan apply [done M1]
|
||
scan.rs # directory walk, image detection, walk_stream [done M1]
|
||
thumb.rs # pool, dispatcher, disk cache, generation cancel [done M2]
|
||
images.rs # turbojpeg shrink-decode, EXIF orientation, PNG [done M2]
|
||
viewer.rs # full-size load + transform state (M4)
|
||
perm.rs # storage permission check/request (M5)
|
||
```
|
||
|
||
Crate choices (all verified current on crates.io, 2026-10):
|
||
|
||
| crate | version | why |
|
||
|---|---|---|
|
||
| `slint` / `slint-build` | 1.18.1 | pinned; upgrade to 1.19 after the pointer-scroll fix lands |
|
||
| `rusqlite` | 0.41, `bundled` feature | SQLite without system deps; cross-compiles to Android |
|
||
| `turbojpeg` | 1.5.1, `cmake` feature | libjpeg-turbo DCT-level shrink decode (1/2,1/4,1/8) — 48 MP at 1/8 peaks ~2.3 MB vs 137 MB full decode. Vendors + cross-builds libjpeg-turbo via CMake/NDK (verified). Chosen over `libvips` (same DCT shrink mechanism, +6–10 MB of .so files, LGPL, 4.3k downloads) and over `zune-jpeg` (no DCT shrink = full-size decode for 12 MP photos). |
|
||
| `image` | 0.25, `default-features=false, png` | PNG thumbs + `imageops::resize` final fit |
|
||
| `kamadak-exif` | 0.6 | EXIF orientation tag (all 8 values applied as pixel maps); `experimental::Writer` used only in tests |
|
||
| `sha1` | 0.11 | cache keys |
|
||
| `imagepipe` | — (deferred) | HEIC: the real device has **0 HEIC** files (9109 JPEG, 27 PNG), so HEIC support is deferred; `image` can gain `heic` or libvips can replace the JPEG path later — the engine API is decoder-agnostic |
|
||
|
||
Decode sizes: grid tiles render at ~cell×2 (Retina) — e.g. 500 px side.
|
||
Full-screen: full decode once (12 MP ≈ 30–80 ms off-thread, fine), cached in
|
||
the in-memory LRU until memory pressure.
|
||
|
||
**The five failure modes the current (unknown) app likely has** — this design
|
||
eliminates all five: (1) full-size decode for grid cells, (2) no disk
|
||
thumbnail cache, (3) no list virtualization, (4) decode on UI thread without
|
||
cancellation, (5) EXIF rotation applied per frame.
|
||
|
||
## 5. UI plan (Slint patterns)
|
||
|
||
- **Grid**: `ListView` is the only virtualized std-widget (confirmed in
|
||
source docs: "Only visible rows are instantiated"). Grid = vertical
|
||
`ListView`, each row a horizontal layout of square `Image` cells
|
||
(columns ≈ `floor(window-width / cell)`; recompute on resize).
|
||
Cells bind `image <=> model.image` — when the pool delivers a thumbnail,
|
||
only that cell invalidates (retained mode; no repaint calls to remember).
|
||
- **Placeholder→image**: cell shows a `Rectangle` (shimmer optional, M6)
|
||
until its `image` is set.
|
||
- **Image cache**: Slint's built-in path-based LRU is 5 MB (`CLruCache` in
|
||
`internal/core/graphics/image/cache.rs`) — too small and the wrong layer
|
||
for us; our cache lives in Rust (disk + compressed-bytes LRU).
|
||
No mipmaps in the Skia renderer — irrelevant since we pre-downsample.
|
||
- **Viewer**: no pinch helper in Slint 1.18 (`TouchArea` has no pinch
|
||
callback; grepped the stdlib). Manual two-finger math on `TouchArea`
|
||
(distance/centroid → scale/translation, ~50 lines; crib
|
||
`slint-src/examples/native-gestures` — source checkout is at
|
||
`~/prog/android/ui/slint-src`, tag v1.18.1). Double-tap-to-zoom is a nice
|
||
extra (M6).
|
||
- **Screens**: single `Window` with a stacked layout (grid + viewer overlay)
|
||
or a `Navigator`-style state property; keep it simple until M4.
|
||
- **Safe area**: always pad with `root.safe-area-insets.{top,bottom}` (done
|
||
in the scaffold).
|
||
- **Animation**: use built-in `animate` statements (syntax: `animate <prop>
|
||
{ duration: ...; easing: ...; }` — it is a *statement*, not an element).
|
||
Slide/fade for the viewer is a one-liner.
|
||
|
||
## 6. Slint 1.18 API gotchas (all hit firsthand in the eval)
|
||
|
||
1. `animate` is a statement (`animate width { duration: 300ms; }`), not an
|
||
`Animation {}` element.
|
||
2. Text input widget is `LineEdit`, not `TextField`.
|
||
3. Markdown interpolation: `text: "Count: \{count}"` (backslash-brace).
|
||
4. `length` is the unit type for lengths (`property <length>`, not
|
||
`logical<length>` in the 1.18 Rust API).
|
||
5. No Rust macros/closures inside `.slint` callbacks — callbacks call global
|
||
functions or invoke Rust via `@global` / imported callbacks.
|
||
6. 1.18 ↔ 1.19 **docs drift** — when reading docs online, check which
|
||
version; the local source at `~/prog/android/ui/slint-src` (tag v1.18.1)
|
||
is ground truth for the pinned version. `git grep` beats the docs.
|
||
7. `slint::android::init_with_event_listener(app, |event: &PollEvent| ...)`
|
||
re-exports the whole `android-activity` crate as
|
||
`slint::android::android_activity`; `AndroidApp` is also re-exported at
|
||
`slint::android::AndroidApp`.
|
||
8. `slint::Image` runtime construction: `Image::from_rgba8_premultiplied`
|
||
(also `from_array`); assign to an `image` property to update a cell.
|
||
9. Desktop Linux: needs `libfontconfig1` + `pkg-config` (pure Rust on macOS).
|
||
10. Retained-mode redraw: changing a property re-renders only affected
|
||
elements — design bindings so thumbnail delivery touches one property.
|
||
11. **`in property` is NOT auto-bound to a same-named `for` variable.** In
|
||
`for cell[c] in row.cells: CellView { ... }` you must write
|
||
`CellView { cell: cell; ... }` — otherwise the component's `in property
|
||
<Cell> cell` stays at its default (empty) and the cell renders blank
|
||
while *looking* correct. (This silently broke M3 for an hour.)
|
||
12. **Updating a nested model in place does NOT reach a ListView row that
|
||
was recycled while off-screen.** In-place `inner.set_row_data(col, …)`
|
||
updates only the currently-live `CellView`s; a row whose `GridRow`
|
||
instance was recycled keeps showing stale cells until the next scroll.
|
||
The reliable path is to replace the WHOLE outer row:
|
||
`outer.set_row_data(row, Item { cells: ModelRc::new(VecModel::from(…)) })`
|
||
— the outer `row_changed` makes the ListView re-bind `row-data` and the
|
||
fresh nested read picks up the new state. Batch cell changes per row and
|
||
do ≤ ~16 row-replacements per 100 ms tick (see `apply_row_batch`), or a
|
||
fast scroll mass-updates and ANRs.
|
||
13. `ListView` has no `spacing`/`gap` property in 1.18 — bake inter-row
|
||
rhythm into the item height (we add a 4 px bottom margin inside
|
||
`GridRow`; `ROW_SPACING` in lib.rs must match).
|
||
|
||
## 7. Permissions & platform integration (Android)
|
||
|
||
Declared in `Cargo.toml` (verified schema in cargo-apk2 1.5 / ndk-build2:
|
||
`uses_permission` is **top-level** under `[package.metadata.android]`, and
|
||
`Permission` supports only `name` + `max_sdk_version`):
|
||
`READ_MEDIA_IMAGES`, `READ_EXTERNAL_STORAGE` (maxSdk 32),
|
||
`MANAGE_EXTERNAL_STORAGE`.
|
||
|
||
Runtime flow (M5), using the `ANDROID_APP` clone stashed in `src/lib.rs`:
|
||
|
||
1. **MANAGE_EXTERNAL_STORAGE** (primary path, same as passgo): no runtime
|
||
dialog exists — check `Environment.isExternalStorageManager()` via JNI,
|
||
if false launch `Settings.ACTION_MANAGE_ALL_FILES_ACCESS_PERMISSION` via
|
||
`ANDROID_APP.get().run_on_java_main_thread(...)` (attach `JNIEnv` from
|
||
`vm_as_ptr()`; Activity from `activity_as_ptr()`), then **re-check on
|
||
`MainEvent::Resume`** — the hook is already wired in `src/lib.rs`
|
||
(TODO(M5) marker).
|
||
2. **READ_MEDIA_IMAGES / READ_EXTERNAL_STORAGE**: `requestPermissions` via
|
||
the same JNI route; the bare `NativeActivity` does not forward
|
||
`onRequestPermissionsResult` to native, so either (a) poll
|
||
`checkSelfPermission` after requesting (good enough), or (b) ship a custom
|
||
`NativeActivity` subclass via cargo-apk2 `java_sources = "android/java"`
|
||
(manifest `activity name` points at it) — the proper callback, reusable
|
||
pattern from the eval.
|
||
3. Facts that constrain design (verified in android-activity 0.6.1 source):
|
||
no `NewIntent` event; `poll_events` may only run on the `android_main`
|
||
thread (Slint owns it — don't add a second consumer); `AndroidApp` is
|
||
`Clone` and a clone works from any thread (`run_on_java_main_thread`,
|
||
`vm_as_ptr`, `activity_as_ptr`).
|
||
4. **Source discovery** for photos: start with a plain directory scan of
|
||
`/storage/emulated/0/Pictures` + `DCIM` (works with all-files access,
|
||
emulator-friendly, and it's what makes the app testable without
|
||
MediaStore). MediaStore integration (API-aware, scoped to
|
||
`READ_MEDIA_IMAGES`) is a later refinement if directory scanning proves
|
||
insufficient.
|
||
|
||
## 8. Performance targets & measurement
|
||
|
||
Targets (emulator is NOT representative for absolute decode times — validate
|
||
on a real device with the aarch64 build):
|
||
|
||
- Grid: 60 fps scrolling at 5k+ items; zero full-size decodes on the hot path.
|
||
- First frame: grid of placeholders < 100 ms after launch (metadata from
|
||
sqlite, images async).
|
||
- Thumbnail decode (grid size): < 5 ms off-thread per JPEG (measure with
|
||
`std::time::Instant` logs; zune-jpeg turbo should be well under).
|
||
- Steady-state memory: in-memory thumb cache bounded (~64–128 MB compressed
|
||
bytes on a phone) + Slint's texture memory for visible tiles only.
|
||
- Cold start (emulator, release): compare against the eval baseline
|
||
(Slint sample ≈ 1–2 s; gallery must not regress > 0.5 s).
|
||
|
||
**Scroll-FPS measurement (real device, verified 2026-10-27):**
|
||
`dumpsys gfxinfo wow.gmp.gallery` reports **0 frames** for Slint — the
|
||
NativeActivity window renders via its own EGL surface and is not tracked by
|
||
gfxinfo on Android 15 (works for normal View apps). Use SurfaceFlinger's
|
||
per-layer latency instead:
|
||
|
||
```bash
|
||
# find the layer name:
|
||
adb shell dumpsys SurfaceFlinger | grep "VRI-wow.gmp.gallery"
|
||
# while scrolling (swipes via `input swipe`): capture 1 s of frame history
|
||
adb shell "dumpsys SurfaceFlinger --latency 'VRI-wow.gmp.gallery/android.app.NativeActivity#<id>'"
|
||
# each line: latch_time display_time … — display_time is the vsync grid.
|
||
# fps = 1e6 / avg(display gap); jank = display gaps > ~25 ms; the latch
|
||
# column shows frame PRODUCTION timing (p95 ~17 ms = on time at 60 Hz).
|
||
```
|
||
|
||
Result (Pixel 9 Pro, release aarch64, 8,577 photos, 500 px thumbs):
|
||
sustained **60.0 fps, 0 display hitches** while scrolling, both cold cache
|
||
(production p95 21–25 ms) and warm cache (production avg 16.6 ms). The
|
||
display presents at a 60 Hz vsync grid (Slint requests Default framerate;
|
||
120 Hz is a later polish item, not an M3 acceptance criterion).
|
||
Note: the shimmer `animation-tick()` keeps frames flowing even at rest,
|
||
so latency history is continuous — filter by swiping if you need
|
||
scroll-only frames.
|
||
|
||
Test data (M2): generate a synthetic set on the device — a small Rust `bin`
|
||
or Python script that writes N=5,000 JPEGs of varied sizes/orientations into
|
||
`/storage/emulated/0/Pictures/gallery-test/` (include a few HEICs if
|
||
imagepipe works; include rotated EXIF). Reuse it for every perf run.
|
||
|
||
Measure with: `adb shell am start -W` (cold start),
|
||
`dumpsys gfxinfo wow.gmp.gallery framestats`, Perfetto trace (GPU + main +
|
||
pool threads), and `logcat` pool timing logs behind a `debug_timing` flag.
|
||
|
||
## 8.5 Jank hunt (IN PROGRESS — phone is the acceptance target)
|
||
|
||
Symptom: user reports janky grid scroll (moderate drag, no fling) on the
|
||
phone (Pixel 9 Pro, 120 Hz). Emulator repro via programmatic scroll hook.
|
||
|
||
### What is ruled out (all measured)
|
||
- **App tick cost**: 7–34 µs/tick, even inside 28–39 ms gaps. Not the tick.
|
||
- **Logd backpressure**: fixed — `nblog` (bounded mpsc + drain thread,
|
||
nanosecond `try_send` from UI). Idle CPU 71% → 5%.
|
||
- **60/120 Hz display zone flap**: SystemUI (uid 10156) was asserting a 60 Hz
|
||
override every minute; pinned with `settings put system min_refresh_rate
|
||
120` (reversible: `settings delete system min_refresh_rate`).
|
||
- **Shimmer animation**: `animation-tick()` in a per-cell binding evaluated
|
||
O(ALL 8577 cells) `sin()` every frame (70.9% CPU idle). Now a static
|
||
placeholder; proper O(visible) shimmer is M6 work.
|
||
- **Whole-row model replacement**: now in-place `cells.set_row_data` (one
|
||
property change; texture upload only when pixels change).
|
||
- **Texture upload CPU**: APLY 1.2–2.9 ms/cell (480 KB) — under budget, and
|
||
too cheap to explain 28–39 ms (would need 12–17 MB/s on a Mali).
|
||
|
||
### Measured facts
|
||
- Emulator, programmatic scroll (`/sdcard/Download/.gallery_cmd` →
|
||
`scroll <px/ms> <ms>`), real textures: 28–146 ms cadence. During every
|
||
stall the UI thread is **100 % on-CPU** (jsample: state R, wchan 0) —
|
||
Slint's render pass is CPU-bound.
|
||
- Emulator, 300 items instead of 5852: **equally janky** → scene/item count
|
||
is NOT the cost driver.
|
||
- Emulator, grey 1×1 images (`.gallery_grey` touch file): steady ~16.6 ms
|
||
(60 Hz pacing), worst 44 ms → the cost is **rasterizing ~30 live textured
|
||
cells at 1080×2400 through the emulator's software GL (llvmpipe)**.
|
||
The emulator has a hard rendering ceiling and can never scroll this
|
||
smoothly — it validates LOGIC (invariants, paint coverage), not feel.
|
||
- Phone: 64 % of frames on the 8.3 ms (120 Hz) grid; ~4 % take 28–39 ms of
|
||
non-tick work with d≈0 (UI thread not in our code). Frequency ≈ every
|
||
170 ms — much faster than new cells enter the viewport (~1.7 s), so it is
|
||
NOT first-paint/upload timing. Suspects left: GPU/present contention
|
||
(Mali + SystemUI both at 120 Hz), Slint EGL/swap path, or an
|
||
event-loop/vsync interaction. Needs the phone.
|
||
|
||
### Tools / hooks in the tree (all tagged "REMOVE after the hunt")
|
||
- `src/lib.rs`: `DbgScroll` + `.gallery_cmd` file (also `stop`);
|
||
knobs `.gallery_max` (item cap) and `.gallery_grey` (1×1 grey images);
|
||
`SCRDBG`/`JANKTEST`/`TICK`/`APLY` logs at `info`.
|
||
- `jsample.c` (in /tmp, NDK-cross with
|
||
`~/android-sdk/ndk/27.0.12077973/.../x86_64-linux-android26 -static`):
|
||
per-thread `/proc stat` + state + wchan sampler at 1 ms; usage
|
||
`jsample <interval_ms> <pid> <tid...>`.
|
||
|
||
### Next (phone, morning)
|
||
1. **Decisive A/B on the phone**: `touch /sdcard/Download/.gallery_grey`,
|
||
restart app, user scrolls 30 s, capture logcat. Then remove the file,
|
||
restart, repeat. Grey smooth + textured janky → texture/GPU path is the
|
||
culprit (then: smaller decode target, upload strategy, possibly Slint
|
||
image pipeline). Grey also janky → base render/present path (then: Slint
|
||
window/EGL config, refresh-rate interaction, vsync driver).
|
||
2. Run `jsample` over one of the user's real scrolls and wchan-correlate
|
||
the 28–39 ms windows (is android_main R, S/futex, or S/do_epoll_wait?).
|
||
3. Compare against Fossify (installed on the phone, user says smooth):
|
||
`dumpsys SurfaceFlinger --latency` on both apps' layers during identical
|
||
scrolls.
|
||
4. **Priority fix already applied** (commit after this edit):
|
||
- `thumb-0..3` + `thumb-dispatch` demoted to nice +2 (they inherited the
|
||
UI's -10 and could preempt it during turbojpeg bursts).
|
||
- Slint's `slint vsync` Choreographer waker (created at nice +10 by
|
||
Android — the ONLY wakeup source for the main loop while animating,
|
||
see i-slint-backend-android-activity/vsync.rs) is raised to nice 0 by
|
||
a `prio-fix` thread in android_main (log: "slint vsync raised to nice
|
||
0"). A starved waker delays the main loop's wakeup by 3–4 frames at
|
||
120 Hz = 25–33 ms with zero tick cost — the measured phone signature.
|
||
Verified on emulator: nice holds at 0 while foreground; Android
|
||
re-nices backgrounded processes (expected, irrelevant).
|
||
- Verification on phone: check the two log lines, then
|
||
`for t in /proc/$(pidof wow.gmp.gallery)/task/*; do ...` and confirm
|
||
thumb-* at 2, slint vsync at 0, before/while the user scrolls.
|
||
5. Candidate code changes (after the A/B): GRID_TARGET 500→400 (one-time
|
||
cache invalidation — don't do it before knowing the culprit), 1 painted
|
||
cell/tick instead of 2.
|
||
|
||
## 9. Milestones
|
||
|
||
- **M0 — scaffold (DONE)**: builds (x86_64 + aarch64 toolchain), installs,
|
||
runs on emulator; permission-aware init + Resume hook; build script;
|
||
manifest with permissions; this document.
|
||
- **M1 — metadata (DONE, 2026-10-27)**: `db.rs` + `scan.rs`; background scan
|
||
thread → sqlite → `VecModel` in a `ListView` (paths as text). Acceptance
|
||
verified on emulator: **5002 files in 153 ms**; relaunch `0+ 0~ 0-` in 2 ms;
|
||
mtime-touch → `1~`, delete → `1-`. Status line shows `N files (a+ u~ d–) in X ms`.
|
||
List paint is instant on relaunch: the DB list is shown first
|
||
("N files (cached)"), the background walk then runs and refreshes
|
||
("Updating… N"). First launch streams results into the list in 512-file
|
||
batches while scanning — the UI never waits for the full DB build
|
||
(`walk_stream`). Hidden dirs (`.thumbnails`, …) are skipped.
|
||
Real-device numbers (Pixel-class, 8.5k photos): cold scan ~4.2–5.9 s
|
||
(filesystem metadata cost — the walk is the bottleneck, not sqlite),
|
||
relaunch paint instant, rescan ~4.2 s. Walk parallelism is an M2/M7
|
||
optimization candidate.
|
||
Gotchas hit (also see §6/§12):
|
||
- `slint::Result` / `slint::WindowHandle` **do not exist in 1.18.1** as
|
||
published on crates.io — desktop bin is `fn main() -> Result<(),
|
||
slint::PlatformError>` + `use slint::ComponentHandle;`. (The M0 desktop
|
||
bin never actually compiled; only the Android lib was built. The eval
|
||
sample `slint_app` is broken the same way.)
|
||
- `.slint` modulo is `Math.mod(x, y)`, not `%`.
|
||
- ListView rows: `for path[index] in root.items: …` (`index` is not in
|
||
scope unless named; `parent` = the ListView).
|
||
- `rusqlite(bundled)` needs `CC_<triple>` pointing at NDK clang for
|
||
Android — `scripts/build.sh` now exports them.
|
||
- Android DB path: `EXTERNAL_DATA_DIR` is **not set** by this
|
||
NativeActivity setup and `temp_dir()` = `/data/user/0` (unwritable).
|
||
Use `/storage/emulated/0/Android/data/<pkg>/files`, internal files dir
|
||
fallback (see `db::android_data_dir`).
|
||
- Emulator: grant all-files access once via
|
||
`adb shell appops set wow.gmp.gallery MANAGE_EXTERNAL_STORAGE allow`; it
|
||
survives `adb install -r`.
|
||
- Files pushed with `adb push` are owned by `shell` mode 640 — `chmod 644`
|
||
or the app can't read them.
|
||
- **M2 — thumbnail engine (DONE, 2026-10-27)**: `images.rs` (turbojpeg
|
||
DCT-level shrink decode via 1/2/1/4/1/8 `ScalingFactor`, all 8 EXIF
|
||
orientations as uniform `(x,y)→(dx,dy)` pixel maps, `image::imageops::resize`
|
||
final fit, re-encode to disk cache at q85; PNG via `image`) + `thumb.rs`
|
||
(dispatcher thread round-robining to N=cores/2 workers clamped 2..8,
|
||
unbounded job queue, disk cache `thumbs/{sha1(path|mtime|size|target)}.jpg`
|
||
— target in the key so list-64px and grid-500px coexist; atomic
|
||
`.tmp`→rename; generation counter for cancellation; per-worker sqlite
|
||
connection writes `set_thumb` w/h/orientation/thumb_path) + `lib.rs`
|
||
(process-wide `AppState` in `OnceLock`, 10 Hz drain timer — `Timer` is
|
||
`!Send`, leaked with `std::mem::forget`, scroll-prefetch with 50-row
|
||
margin, `do_prefetch` flag so streaming scan batches don't re-prefetch) +
|
||
`main.slint` (rows carry `image` now).
|
||
*Accept, verified:* 12 unit tests + 5k synthetic engine test (cold fill,
|
||
100% cache warm refill, generation cancel). Release aarch64 APK 9.8 MB.
|
||
**Real device (Pixel 9 Pro, 8577 photos):** cold scan ~5 s with list
|
||
streaming; warm relaunch paints the cached list instantly; thumbnails
|
||
render in every visible row (screenshot-verified); relaunch cache hits
|
||
~100% (255–258 of first 256 jobs); 2 WebP files misnamed `.jpg` fail
|
||
gracefully (logged, row stays empty). Worker log cadence: every 256 jobs.
|
||
*Known transient (unreproducible): the very first aarch64-release install
|
||
rendered a pure-black screen (no drain-timer "painted" logs) until the
|
||
next launch; it did **not** reproduce on the emulator (screen-off launch +
|
||
wake) nor across repeated device launches. Suspected screen-lock / display
|
||
state race on first GL frame. If it recurs: force a window refresh on
|
||
Resume (see §10.7).*
|
||
*Build gotchas hit (also in build.sh comments):*
|
||
- `ANDROID_PLATFORM=android-26` (NDK CMake toolchain) **conflicts** with
|
||
the `android-build` crate (Slint's Java compile) unless `ANDROID_JAR`
|
||
is set explicitly to `platforms/android-35/android.jar`.
|
||
- x86_64 host builds of turbojpeg-sys need `nasm` (`~/.local/bin/nasm`;
|
||
aarch64 does not).
|
||
- `ADB_SERIAL=…` env var picks the device in build.sh (phone + emulator
|
||
are often both attached).
|
||
- build.sh keywords: `debug` (x86_64), `debug-aarch64`, `x86_64`,
|
||
`aarch64`; second arg `no-install` to skip install.
|
||
- **M3 — grid UI (DONE 2026-10-27)**: 3-column `ListView` grid (`GridRow` =
|
||
one list item holding up to 3 square `CellView`s), shimmer placeholder via
|
||
a time-driven `animation-tick()` expression, scroll-driven prefetch
|
||
(`scrolled → on_scroll`), memory-bound retention (cells drop to shimmer
|
||
when they leave the active range; disk cache repaints them in ~10 ms).
|
||
Key fixes along the way (see §6.11–13): `in property` must be assigned
|
||
explicitly in a `for`; cell updates must replace the whole outer row
|
||
(`apply_row_batch`) and are capped at 16 rows/tick to avoid ANR on fast
|
||
scroll; `on_scroll` only re-anchors the active range + submits prefetch
|
||
(no model mutation), all mutation is in the 10 Hz drain timer.
|
||
|
||
**Thumbnail loading: viewport reconciliation + worker-level priority
|
||
(rewritten 2026-10-27 after two field-reported bugs).** Design doc is the
|
||
header comment of `src/lib.rs`; engine in `src/thumb.rs`. The invariants:
|
||
|
||
1. *The viewport is the only source of truth.* `on_scroll` only records the
|
||
viewport range (zero work). The 10 Hz tick is the sole authority AND the
|
||
sole submitter: CONSUME finished results (paint viewport ones, capped),
|
||
EVICT far pixels (hard cap `MAX_PAINTED=120`), FILL the viewport
|
||
(center-first), PRUNE queued jobs outside the retention band. After each
|
||
tick: every viewport cell is PAINTED, IN-FLIGHT, or FAILED — restored,
|
||
not maintained, so no scroll pattern can leave a visible cell forgotten
|
||
(the original bug: re-submission only happened in `on_scroll`, so when
|
||
scrolling STOPPED, forgotten cells stayed empty until the next small
|
||
move). `debug_assert`ed every tick.
|
||
2. *The viewport is first at EVERY queue level.* The central queue
|
||
(`Mutex<VecDeque>` + `Condvar`, prepend-on-submit) is drained as
|
||
priority-ordered bursts and PREPENDED to the per-worker queues (also
|
||
`Mutex<VecDeque>` + `Condvar`, NOT mpsc — mpsc is FIFO-only). Each
|
||
worker queue is capped (96 ≈ two screens): overflow is dropped from the
|
||
TAIL (oldest transit work) with a `skipped: true` result; the UI clears
|
||
the in-flight marker, the cell is NOT failed, re-entry re-submits it
|
||
(disk-cache hit). A burst ≤ viewport (~45) < cap, so fresh viewport work
|
||
is never dropped. `in_flight` accounting: +1 at submit, −1 at finish OR
|
||
at cap-drop; the UI's `inflight` map (path→time) is the dedupe key and
|
||
is cleared on every result (incl. `skipped`) — a dropped job whose
|
||
path has a newer queued job may be briefly re-submitted (one wasted
|
||
decode, self-cleans).
|
||
|
||
Regression tests: `thumb::tests::scroll_back_cache_hit_not_starved`
|
||
(central-level: 6.3 s FIFO starvation → 130 ms) and
|
||
`thumb::tests::fresh_batch_preempts_stale_worker_queue` (worker-level: a
|
||
fresh job submitted while a slow batch is queued must finish within the
|
||
first two results, never behind the whole old queue).
|
||
|
||
*Stress results (emulator x86_64 DEBUG build — slowest plausible hardware —
|
||
then release aarch64 on the Pixel 9 Pro):* fast up/down ×5 (the reported
|
||
repro): viewport fully covered 0–119 ms after stop (4/5 already covered at
|
||
stop); cold landings: 1.0–1.2 s = pure decode throughput (p rises
|
||
monotonically while stale jobs wait BEHIND, oldest growing 700→1900 ms
|
||
harmlessly); chaos ×6 + 25 s idle soak: covered before every stop, max
|
||
oldest in-flight job 217 ms (pre-fix: 8,055 ms), max queue depth 45 (one
|
||
viewport), 0 invariant violations in ~1,450 ticks, 0 panics, idle
|
||
converges to 0 queued. Phone smoke: 99.98% cache hits during 8k-cell
|
||
stress scroll, 0 panics/ANRs, one 541 ms transient stall alarm (cold
|
||
decode burst).
|
||
|
||
*Verified (emulator, 150 real JPEGs + 5k stubs): first paint 0.6 s cold;
|
||
warm relaunch from disk cache; 0 ANRs through 16 fast swipes; backlog
|
||
drains to 0 when idle. Verified on Pixel 9 Pro (release aarch64, 8,577
|
||
photos): first screen ~0.7 s cold at 500 px; scroll = sustained 60 fps
|
||
with 0 display hitches (cold and warm cache) — see §8 for the
|
||
measurement method (gfxinfo does NOT track Slint's window).*
|
||
- **M4 — viewer**: tap cell → full decode + overlay; pinch/pan (manual
|
||
math), double-tap zoom; back gesture/button; EXIF orientation correct.
|
||
- **M5 — permissions**: implement `perm.rs` per §7; launch flow when
|
||
un-granted (settings card → open Settings → recheck on Resume); test the
|
||
exact passgo-style flow on a fresh install.
|
||
- **M6 — polish**: dark theme, animations (slide/fade), search/filter
|
||
(name/date), sort; double-tap zoom; battery-friendly (pool idles, no
|
||
busy repaint — Slint is retained, verify with gfxinfo "total frames").
|
||
- **M7 — perf & release**: real-device (aarch64) pass; Perfetto; memory
|
||
bounds; release APK size check (expect ~8–9 MB like the sample);
|
||
consider the 1.19 upgrade (verify pointer-scroll fix first).
|
||
|
||
## 10. Open questions & risks
|
||
|
||
1. **HEIC (RESOLVED for now)**: the real device has 0 HEIC files (9109
|
||
JPEG, 27 PNG), so HEIC is deferred. When needed: add a decoder to
|
||
`images.rs` dispatch (libheif-rs / `image` heic feature / libvips);
|
||
the engine API is decoder-agnostic.
|
||
2. **Slint pointer-scroll `todo!()` panic** (open issue in the Android
|
||
backend at eval time): could fire on trackpad/touch scroll. Test scrolling
|
||
on M3; if it fires, patch locally (the fix is small) or upgrade to 1.19
|
||
when available.
|
||
3. **Max texture size**: full-screen decode of 12 MP+ images (e.g.
|
||
8192 px) may exceed GPU max texture size on some devices — verify on
|
||
M4; fall back to a 2×-max downscale for display.
|
||
4. **Slint license**: royalty-free tier covers proprietary mobile/desktop/
|
||
web; embedded devices excluded — irrelevant for us, but the repo README
|
||
should state the chosen tier (royalty-free) once the app ships.
|
||
5. **Docs drift**: trust the pinned source (`~/prog/android/ui/slint-src`
|
||
@ v1.18.1) over online docs; bump pin + re-verify after any Slint upgrade.
|
||
6. **rusqlite `bundled`** compiles SQLite from source in every target
|
||
(adds ~30–60 s to clean builds) — acceptable; revisit if it hurts.
|
||
7. **Transient black screen (M2 first launch, unreproducible)**: one
|
||
aarch64-release launch rendered pure black with a dead drain timer until
|
||
relaunch; not reproduced on emulator or device since. Candidate fix if it
|
||
recurs: in the `init_with_event_listener` Resume handler, call
|
||
`window.request_redraw()` / check `window.size()` (a 0×0 or stale surface
|
||
after a screen-off start would match the symptoms). Don't add it
|
||
speculatively.
|
||
|
||
## 11. Pointers
|
||
|
||
- Framework eval (full report + raw notes + both sample apps + screenshots):
|
||
`~/prog/android/ui/` — `REPORT.md`, `NOTES.md`, `slint_app/`, `eframe_app/`,
|
||
`slint-src/` (v1.18.1), `egui-src/` (0.37.0).
|
||
- Slint docs (version-sensitive!): https://slint.dev/docs — Android guide:
|
||
`docs/guide/platforms/mobile/android.mdx` in the source tree.
|
||
- `examples/native-gestures`, `demos/` in `slint-src` — reference code for
|
||
gestures, lists, images.
|
||
- passgo (existing Go app using the same all-files-access pattern):
|
||
`~/prog/go/passgo` — its `scripts/build.sh` and manifest handling are a
|
||
proven reference for the permission UX flow.
|
||
- Emulator AVD: `pad_avd` (google_apis x86_64, API 35); start command in §3.
|
||
- Gitea: push target `git+ssh://gitea@git.wow.st/gmp/gallery.git`
|
||
(repo must exist on the server first, like the other gmp/* repos).
|