Skip to content
sysout.dev

In My Head - Implementation Plan

Current checkpoint

Android has a working local-first record → transcribe app. It is not a production-quality voice-note app yet; this document records what is actually implemented and the gates before building the Gemma cleanup experience or broader product UI.

  • The app records PCM audio in an Android foreground service and preserves the resulting WAV files in app-private storage.

  • It downloads and uses Whisper small.en locally. The active model is about 488 MB; medium.en is catalogued but is not selected or benchmarked.

  • It transcribes only after Stop, stores the raw transcript locally, keeps audio when transcription fails, and lets the user retry without rerecording.

  • The transcript list and detail screens show a failure explanation and the recorded transcription duration.

  • The optional LiteRT-LM/Gemma integration remains intentionally behind the Whisper quality work; it is not the current product flow to validate.

Completed foundation

Phase 1 - Theme and app shell

Completed:

  • Material 3 light/dark theme from the visual design document.

  • Record, transcript list, and transcript detail screens.

  • Shared domain models and ViewModels.

Phase 2 - Local persistence

Completed:

  • SQLDelight recordings and transcripts storage.

  • A versioned database migration for transcription failure text and processing duration.

  • One Android repository provider shared by the Activity and foreground service, so status changes reach the UI immediately.

Phase 3 - Android capture pipeline

Completed first pass:

  • AudioRecord capture in a microphone foreground service.

  • Automatic 25-second WAV chunks plus a valid final partial chunk after Stop.

  • Recording files remain available after a failed transcription for retry.

  • Capture stays available while a durable first-in, first-out queue processes one earlier recording at a time.

Still to prove:

  • Background, screen-off, Bluetooth, and OEM battery-management behaviour on a representative set of physical devices.

  • Long-session reliability (15-20 minutes or more).

Phase 4 - whisper.cpp integration and performance baseline

Completed first pass:

  • Standalone :whisperNative Android library with CMake/JNI bridge.

  • First-run atomic model download into app-private storage.

  • small.en selected as the current English-only model; the app does not treat the previous base.en download as the active model.

  • Native debug builds now compile Whisper and its GGML compute libraries with -O3, so device testing does not benchmark unoptimized native code.

Verified on the current Motorola Edge 50 Fusion test device:

  • A 6.0-second recording transcribed with small.en in 23.7 seconds, roughly 4x real time. This is a useful baseline, not a quality or thermal benchmark.

Current native decision:

  • Debug and release use GGML’s existing multithreaded worker pool, with GGML_OPENMP=OFF. No app-owned OpenMP module or runtime selector remains.

  • On 2026-11-03, representative saved-recording comparisons favoured the worker pool. OpenMP warm passes took 10.2% and 14.2% more time at four threads in opposite backend orders, and 54.3% more at six threads. Outputs matched. Uncontrolled device conditions limit generalization; they do not establish an optimal production thread count.

  • The full experiment is preserved on local Git branch archive/openmp-experiment-20261103. See OpenMP experiment for the measured results, limitations, native integration, and restoration context.

2026-09-11 short-fixture A/B

Historical experiment, superseded by the 2026-11-03 decision above. The archived branch retains the implementation; these results are not the current build.

The first gate used one 3.2-second saved WAV and deliberately stopped short of a thermal or battery workload. Results are wall-clock transcription time:

Threads Worker pool cold / warm OpenMP cold / warm Result

2

35.894 s / 35.355 s

35.924 s / 35.350 s

Equivalent

4, three-run mean

22.357 s / 21.585 s

22.266 s / 21.270 s

OpenMP was 0.4% faster cold and 1.46% faster warm; too small to choose a backend

6

18.745 s / 18.294 s

18.909 s / 18.111 s

Equivalent and fastest overall

8

36.869 s / 39.731 s

28.073 s / 29.186 s

OpenMP was 24-27% faster, but both were slower than 6 threads

8, reverse-order repeat

38.778 s / 38.415 s

28.553 s / 32.387 s

Confirmed the OpenMP scheduling advantage under oversubscription

These figures do not contradict the approximately 1.5x-real-time behaviour seen on normal recordings. whisper.cpp defines a 30-second inference chunk and its mel-spectrogram path appends 30 seconds of zero padding. A 3.2-second clip therefore pays almost the same fixed inference-window cost as a much longer clip; dividing that cost by 3.2 seconds produces a misleading real-time factor. Use recordings of at least 30 seconds for representative throughput ratios.

Debug builds expose Settings > Debug transcription. It selects any finalized saved recording and a 2/4/6/8 thread count, then runs a fresh-context cold pass and a same-context warm pass with the worker pool: two passes total. It never changes the stored transcript or version history. The context is released when the benchmark ends. The screen reports both timings, real-time factors (time divided by audio duration), the transcript hash, and whether both outputs match. The full result is logged as IMH_DEBUG_TRANSCRIPTION. Cold means a fresh model context, not an empty OS file cache. Existing paired-comparison preferences remain readable as the worker-pool result; obsolete comparison keys are ignored.

The debug rerun is owned by the existing RecordingService, using its data-sync foreground notification and transcription-queue exclusion. It no longer relies on an Activity-owned coroutine that an OEM can freeze after the app is hidden. A partial wake lock is held only during the test (with a two-hour timeout), and released on completion or failure. Title/polish model work is serialized with the benchmark as well, avoiding competing inference during a timing run.

Settings and the notification show the worker pool, cold/warm pass, and completed chunks. Each pass completion is logged separately to logcat and the app-private files/debug-transcription.log, without transcript content. The last completed result is retained in debug preferences so it remains available when returning after the service has finished. Saved audio, recording status, transcript text, and version history are not rewritten by a debug rerun.

During the historical short-fixture test the phone rose from 32 C to 36 C and fell from 9% to 8% while USB charging. Those readings are context only, not thermal or energy evidence. Future throughput work should use representative saved audio and controlled conditions with the worker pool; no further OpenMP comparison is required for the current decision.

Implemented on 2026-11-04: transcription checkpoints and recovery: durable per-chunk results, stage-aware resume, and atomic transcript/session finalization. The additive migration upgrades database version 10 to 11 while preserving existing recordings and versions. The worker pool and CPU-based thread selection are retained. See that page for validation and device acceptance.

Extended on 2026-11-04 to queue each closed WAV chunk while microphone capture continues. A growing checkpoint manifest is sealed at Stop; publication and automatic polish wait for the complete range. Schema version 12 adds durable capture/job end indices and preserves version-11 jobs/results. Live transcription failure leaves capture running and exposes Retry after Stop. The extension’s design, failure cases, and current validation are on the same recovery page.

Next: Whisper benchmark and quality gate

Before adding Gemma cleanup or redesigning screens, create a repeatable device benchmark record for small.en:

  • Measure 30-second, 1-minute, and multi-chunk recordings; record audio length, transcription duration, device temperature/charging state, and battery impact.

  • Compare cold-model and warm-model runs.

  • Test English with Indian and US accents, normal room noise, and sequential speakers in a meeting-room setting. Speaker labels/diarization are not part of this pass; the output is one ordered transcript.

  • Confirm the transcript preserves chunk boundaries naturally enough. There is no overlap or boundary de-duplication today.

  • Compare worker-pool thread counts using the saved-WAV benchmark under controlled conditions; its log records GGML system information.

  • Decide whether small.en is sufficient. Do not move to medium.en without measuring its quality gain, processing time, storage, memory, and thermal cost on the same device.

The exit criterion is not a particular benchmark number alone: Whisper must be fast enough and accurate enough that a user will wait for the result, while recording failure never loses their audio.

Follow-on product work

UI/UX pass

The 0.1 decisions are recorded in product/0.1-release.adoc. Implement them as focused, separately validated changes:

  • Separate model download/retry from recording. The record action is disabled until Whisper is ready, and downloading does not request microphone access.

  • Add a static Settings destination with on-device Whisper/Gemma status and private-storage information; it must not imply a missing Gemma download feature.

  • Expose existing recording/transcription diagnostics, copy/selectable text, generated recording titles, and sort/grouping without adding sessions, streaming transcription, or an audio player.

  • Apply Material 3 semantic colour roles, typography, standard shapes, and accessible labels. The selective record action remains primary/red; supporting actions and errors use their semantic roles.

Gemma cleanup pass

Deferred until the Whisper quality gate passes:

  • Define what cleanup should produce, rather than treating it as generic note formatting.

  • Choose the Gemma prompt, model bundle, download/storage behaviour, and failure semantics.

  • Preserve and show raw Whisper text whenever cleanup is unavailable or fails.

  • Benchmark cleanup separately from transcription.

Output and retention

Pending after the core record/transcribe/cleanup flow:

  • Plain-text export.

  • User-controlled deletion and storage management for recordings and models.

  • Cross-app model sharing, if a second Sysout app has a concrete requirement.

Validation and commit workflow

Each focused change follows this loop:

  1. Implement one bounded change.

  2. Compile the relevant shared/native/app targets and run applicable tests.

  3. Install on the physical Android device when behaviour or native code changes.

  4. User tests the behaviour.

  5. Commit only the intended source and documentation files after that test.