An on-device drilldown proved Gmail server-side throttles LibreMail's
connect-per-operation IMAP: every op was a fresh CONNECT+TLS+LOGIN, and
full-history backfill's body+attachment prefetch generated ~601 connections in
~22 min, tripping (and sustaining) Gmail's per-account rate/bandwidth clamp
(body download collapsed to ~4 KB/s). The `live` gauge peaked at only 5 (Gmail
allows ~15), so it is connection *volume*, not count. Outlook IMAP on the same
device opened in 2-3 s. Reusing one warm socket per account (~601 -> ~1) removes
the throttle's trigger. This wires the reuse path the #125 spike built and left
OFF (issue #357 Part 2 — connection reuse only; prefetch is a separate PR).
How it is enabled (with a safety switch):
- New `BuildConfig.IMAP_CONNECTION_REUSE` (default true) drives the production
`ImapClient` no-arg `@Inject` constructor. To disable if a server misbehaves,
flip it to "false" in app/build.gradle.kts — a build-config change, no Kotlin
edit. The internal `ImapClient(reuseConnections, reuseIdleTimeoutMillis)`
constructor stays the test/harness seam.
- Universal: applies to all providers (incl. Outlook). No per-provider caps or
throttling here — that is a separate effort (#356/#360-#364).
Hardening `ImapConnectionCache` for production (was a spike):
- Transparent stale recovery: broadened drop detection to Angus's own
`iap.ConnectionException` (and a MessagingException caused by one) — the real
signal `folder.open()` throws on a server-dropped idle socket, which the
IOException-only check missed, so the reconnect now actually fires. A dropped
reused socket is rebuilt once and the op retried, so callers see no spurious
error; a genuine app error (e.g. message-not-found) is never retried.
- Idle eviction: `evictIdle()` closes a connection unused past the reuse idle
timeout (default 5 min), swept every 2 min by `IdleService`; skips any
in-use connection.
- Teardown: `IdleService` also tears down reused connections on the low-battery
push-teardown path (#88/#89/#90), mirroring the IDLE connection teardown.
- Concurrency: one connection per account behind a per-account mutex; the
eviction sweep takes the lock non-blockingly so it never stalls or interrupts
an in-flight op. Coexists with IMAP IDLE (its own separate connection).
- PII-free AppLog on the lifecycle (open / reuse-hit / reconnect-stale / evict /
teardown) keyed by an opaque per-cache ordinal, plus the #358 ImapPerf
breadcrumb (connect~=0ms on a reuse hit).
Tests (all via the fast gate, no emulator):
- ImapConnectionCacheTest: reuse, retry-once stale recovery, narrow drop
detection, deterministic idle eviction (injected clock), teardown.
- ImapFolderOpenLatencyTest (GreenMail + counting proxy): N ops share one
connection/LOGIN; a force-dropped socket is transparently reconnected; an app
error does not reconnect; idle eviction LOGS-OUT and the next op reconnects.
- Correctness suites (ImapClientTest/ImapClientBackfillTest/MailBackfillerTest)
pinned to reuse-off to keep their connect-per-op assertions unchanged.
Fast gate green: assembleDebug, testDebugUnitTest, compileDebugAndroidTestKotlin,
lintDebug, ktlintCheck, detekt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Prototype the per-account connection reuse the #125 investigation recommended
and deferred, behind an OFF-by-default flag so it cannot destabilize `main`.
- ImapConnectionCache: keeps one authenticated Store alive per account, guarded
by a per-account mutex, keyed by connection identity (not the rotating
secret), with lazy catch-and-retry-once stale handling. No eviction policy
yet beyond an explicit closeReusedConnections() hook.
- ImapClient gains a `reuseConnections` flag (default false via the @Inject
no-arg constructor). With it off, withStore is byte-for-byte the previous
connect + LOGOUT-per-call; with it on, calls borrow the kept-alive Store.
- ImapFolderOpenLatencyTest flips the flag on: the same real-IMAP operations
that cost N connections / N LOGINs collapse to 1 connection / 1 LOGIN, with
the necessary per-open EXAMINE unchanged (proven via CountingImapProxy +
GreenMail; localhost is ~0 RTT so this proves structure, not wall-clock).
- docs/perf/issue-125-connection-reuse-spike.md: prototype design, the
flag-off-vs-on proof, per-decision trade-offs, and the refined real-device
validation plan. References #125; does not close it (needs device validation).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unified inbox query (WHERE folder = ?, no accountId) has no folder-leading
index, so it scans in timestamp order and materializes the whole unified inbox
(~4k rows at a 20k cache) into memory on every emission. Apply Paging 3 to the
unified browse path so query, mapping, and recomposition cost scale with the
visible window, not the total cache.
- MessageDao.pagingUnifiedFolderSummaries: a PagingSource over the folder's
synced rows (inInbox = 1); unified search keeps the whole-folder query so it
can still surface transient server-search hits.
- MailRepository.pagedUnifiedFolderMessages: a Pager (pageSize 40, initialLoad
120, no placeholders) mapping summaries to domain.
- MailboxViewModel.pagedMessages: paged while browsing the unified inbox, else
empty; the messages list flow stays empty in that state so the whole cache is
never materialized. Selection captures each row's accountId at tap time, so
"Move" still resolves the selection's account without an in-memory list.
- MailboxScreen renders the unified browse list via collectAsLazyPagingItems;
per-account and search views render the flat list unchanged (issue #86 stays
flat).
Profiling (docs/perf/issue-124-unified-inbox-paging.md) on an api29 emulator:
current whole-inbox first-emit ~24.6 ms at a 20k cache vs. the paged first page
~6.8 ms and flat regardless of cache size (~3.6x). EXPLAIN QUERY PLAN shows the
paged query still stops early on the existing timestamp index, so no
(folder, ...) index and no schema migration are added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Investigate IMAP folder-open latency (follow-up to #86). Localhost GreenMail
has ~0 RTT, so real wall-clock latency can't be measured here; instead this
pins the folder-open round-trip STRUCTURE deterministically.
Finding: ImapClient.withStore wraps every operation in its own short-lived
Store, so each folder-open pays a full CONNECT + TLS + LOGIN + EXAMINE +
FETCH + LOGOUT. Only EXAMINE + FETCH is intrinsic to opening a folder; the
whole connection-setup group is avoidable on the 2nd+ operation if a
connection were reused. Optimistic render-from-cache already exists
(selectFolder renders cached rows; the network sync is a background refresh).
Adds:
- CountingImapProxy: a localhost TCP proxy that forwards a cleartext IMAP
session to GreenMail while counting TCP connections and parsing IMAP
command words.
- ImapFolderOpenLatencyTest: asserts the current no-reuse behaviour (N opens
=> N connections and N LOGINs; list+read => 2 connections) against a real
in-process IMAP server. Doubles as the harness to validate a future
connection-reuse fix (flip the counts to assert reuse).
- docs/perf/issue-125-imap-folder-open.md: the per-open round-trip sequence,
avoidable vs. necessary round-trips, and the recommended per-account
connection-reuse/keep-alive mitigation with its IDLE / thread-safety /
battery / stale-connection constraints.
Analysis + harness only; the connection-reuse fix is deferred pending
real-network + real-device measurement (see the doc's measurement plan), so
this references #125 without closing it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mailbox list observed the entire `messages` table (observeSummaries, no
WHERE/LIMIT), mapped every cached row to a domain Message, and filtered down to
the visible account+folder in MailboxViewModel — so its cost scaled with the
whole cache and re-ran on every write to `messages` (IDLE delivery, a flag
toggle, a backfill page, any folder sync). On a 20k-row cache that is ~125 ms of
work per unrelated write.
Push the account/folder filter into SQL (observeFolderSummaries /
observeUnifiedFolderSummaries, exposed via observeFolderMessages /
observeUnifiedFolderMessages) and flatMapLatest the ViewModel over the selected
account+folder. The only remaining client-side pass separates the normal list
from an active search over the small folder-scoped set.
Validated on an emulator against 1k/5k/20k-row caches (docs/perf/issue-86-
profiling.md): the account-scoped query is ~1.5 ms flat (~80x faster at 20k) and
is already served by the existing (accountId, folder, uid) index — so NO
composite index and NO schema migration are added. The ticket's proposed
(accountId, folder, inInbox, timestampMillis) index changes timing only within
noise and isn't even preferred by SQLite's planner. The unified "All inboxes"
view stays an O(N) folder scan (still 5.6x better) and is a follow-up for paging;
IMAP latency on folder open is a separate, unmeasured concern.
Closes#86
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JMR-devandClaude Opus 4.8