perf(gmail): respect Gmail IMAP connection & bandwidth limits in sync/backfill #469

Merged
JMR-dev merged 1 commits from feat-361-gmail-imap-limits into main 2026-07-09 01:15:22 +00:00
JMR-dev commented 2026-07-09 00:24:19 +00:00 (Migrated from github.com)

Closes #361

What

Gmail-specific IMAP connection and bandwidth caps, applied as provider-scoped config/policy that
feeds the existing #360 (AccountThrottleGate, reactive backoff) and #356 (BackfillPacer,
proactive inter-slice cooldown) machinery — neither is modified.

Gmail caps applied (GmailSyncLimits, pure constants + appliesTo(account))

  • MAX_IMAP_CONNECTIONS = 15, INTERACTIVE_RESERVED_CONNECTIONS = 1 ->
    MAX_BACKGROUND_IMAP_CONNECTIONS = 14.
  • DAILY_DOWNLOAD_BUDGET_BYTES = 2,500 MB, DAILY_UPLOAD_BUDGET_BYTES = 500 MB.
  • MAX_MESSAGES_PER_LABEL = 10,000, MAX_LABELS = 10,000.

Connection cap: already satisfied by the existing architecture. ImapConnectionCache
(#125/#357) keeps at most one reused connection per account plus one dedicated IMAP-IDLE
connection — 2 total, well under 14. GmailSyncLimitsTest pins that invariant against the
documented ceiling so a future change that grows per-account concurrency (e.g. a real connection
pool) trips a test before it could approach Gmail's real limit. No enforcement code was needed (or
added) beyond that assertion — building a pool here risked colliding with the sibling Yahoo/iCloud
tickets' own connection-cap work on the same ImapConnectionCache/ImapClient.

Bandwidth-aware pacing: the genuinely new mechanism. GmailBandwidthTracker (new,
@Singleton, mirrors AccountThrottleGate's shape — ConcurrentHashMap state, injectable clock,
PII-free once-per-crossing AppLog breadcrumb) tracks per-account, per-day download bytes.
MailRepositoryImpl.prefetchMessage — the single funnel both MailBackfiller and MailSyncer's
background prefetch already share — records bytes actually pulled over the network (body chars +
actually-downloaded attachment bytes, not cache hits) for Gmail accounts only.
MailBackfiller/MailSyncer's prefetchIfEnabled each consult isOverDailyBudget once per
account before starting a prefetch batch and defer for the rest of the day once Gmail's budget is
reached. Header paging/sync is never gated, and interactive fetches (message open, attachment tap,
inline images) are never gated either — same interactive-priority principle #355/#360 already
apply elsewhere.

10k-messages-per-label: captured as a documented constant only, deliberately NOT wired into a
backfill stop condition — issue #12's full-history backfill is intentional and a real large mailbox
can exceed 10k messages, so treating this as a hard ceiling would silently truncate history for
exactly the users the feature is for.

How this composes with #360 / #356

  • AccountThrottleGate (#360) is unmodified — it still handles the reactive case (a real
    provider throttle/lockout response); ThrottleClassifier's existing generic patterns already
    match Gmail's real-world throttle text (e.g. "too many simultaneous connections"), so no
    Gmail-specific classifier changes were needed either.
  • BackfillPacer (#356) is unmodified — it still paces slice cadence, not bytes.
  • GmailBandwidthTracker is a new, orthogonal, proactive mechanism — the same relationship
    InteractiveImapGate documents having to AccountThrottleGate: composes with the other two,
    doesn't duplicate or replace either.

Shared files touched (for conflict-awareness with #362/#363/#364)

Kept additive and Gmail-scoped throughout — no changes to MailProvider.kt, AccountThrottleGate.kt,
BackfillPacer.kt, ThrottleSignal.kt/ThrottleClassifier.kt/ThrottleBackoff.kt,
InteractiveImapGate.kt, or ImapClient.kt/ImapConnectionCache.kt. Files touched that a sibling
provider ticket could also want to touch:

  • MailBackfiller.kt / MailSyncer.kt — small additive change to prefetchIfEnabled (new
    Gmail-only budget check + a new GmailBandwidthTracker constructor param). Yahoo/iCloud connection
    caps are more likely to land in ImapClient/ImapConnectionCache instead (untouched here), but
    flagging in case #362/#363 also touch these two files.
    • MailBackfiller.prefetchIfEnabled also gained an account: Account parameter (previously just
      ids: List<String>) since the Gmail check needs the account to detect the provider.
  • MailRepositoryImpl.kt — new bandwidthTracker: GmailBandwidthTracker constructor param;
    prefetchMessage now records downloaded bytes; ensureAttachmentFile (private) now returns a
    small AttachmentFetch(file, downloadedBytes) instead of a bare File (its 3 call sites updated
    accordingly). MailRepository's public interface is unchanged. Unlikely to overlap with
    #362/#363 (no byte-budget in their issues); #364 (Outlook/Graph) would likely land in a separate
    Graph REST client rather than this IMAP path, but flagging since Outlook accounts currently still
    flow through ImapClient/this same repository.
  • config/detekt/detekt.yml — one appended line, excluding the new
    GmailBandwidthTrackerTest.kt from the android.util.Log forbidden-import rule (it
    mockkStatic(Log::class) the same way AccountThrottleGateTest/BackfillPacerTest already do).

Tests

  • GmailSyncLimitsTest — documented constants, the connection-cap architecture invariant,
    appliesTo for Gmail (incl. the legacy imap.googlemail.com host) vs. Yahoo/iCloud/AOL/Outlook.
  • GmailBandwidthTrackerTest — accumulation, per-account isolation, day-rollover reset,
    threshold-crossing, no-op on non-positive bytes, PII-free once-per-crossing logging.
  • GmailBandwidthTrackerInstrumentedTest (new, mock-free, mirrors BackfillPacerInstrumentedTest's
    idiom) — concurrent-update correctness and account isolation on the real dispatcher/JVM, and
    appliesTo resolving the real MailProvider presets on-device.
  • New/updated cases in MailBackfillerTest, MailSyncerTest, MailRepositoryImplTest composing
    the new Gmail gating with the real backfill/sync/prefetch paths (incl. proving a non-Gmail account
    is unaffected by an over-budget tracker entry for the same account id, and that header
    paging/sync keeps running while prefetch is deferred).

Verification

Fast gate green locally (JDK 21): assembleDebug, testDebugUnitTest,
jacocoTestCoverageVerification, compileDebugAndroidTestKotlin, lintDebug, ktlintCheck,
detekt. Per the dispatch instructions, local emulator E2E was skipped as flaky/non-authoritative —
CI's full matrix is authoritative here.

Not armed for auto-merge; please review.

Closes #361 ## What Gmail-specific IMAP connection and bandwidth caps, applied as provider-scoped config/policy that feeds the existing #360 (`AccountThrottleGate`, reactive backoff) and #356 (`BackfillPacer`, proactive inter-slice cooldown) machinery — neither is modified. ## Gmail caps applied (`GmailSyncLimits`, pure constants + `appliesTo(account)`) - `MAX_IMAP_CONNECTIONS = 15`, `INTERACTIVE_RESERVED_CONNECTIONS = 1` -> `MAX_BACKGROUND_IMAP_CONNECTIONS = 14`. - `DAILY_DOWNLOAD_BUDGET_BYTES = 2,500 MB`, `DAILY_UPLOAD_BUDGET_BYTES = 500 MB`. - `MAX_MESSAGES_PER_LABEL = 10,000`, `MAX_LABELS = 10,000`. **Connection cap:** already satisfied by the existing architecture. `ImapConnectionCache` (#125/#357) keeps at most one reused connection per account plus one dedicated IMAP-IDLE connection — 2 total, well under 14. `GmailSyncLimitsTest` pins that invariant against the documented ceiling so a future change that grows per-account concurrency (e.g. a real connection pool) trips a test before it could approach Gmail's real limit. No enforcement code was needed (or added) beyond that assertion — building a pool here risked colliding with the sibling Yahoo/iCloud tickets' own connection-cap work on the same `ImapConnectionCache`/`ImapClient`. **Bandwidth-aware pacing:** the genuinely new mechanism. `GmailBandwidthTracker` (new, `@Singleton`, mirrors `AccountThrottleGate`'s shape — `ConcurrentHashMap` state, injectable clock, PII-free once-per-crossing `AppLog` breadcrumb) tracks per-account, per-day download bytes. `MailRepositoryImpl.prefetchMessage` — the single funnel both `MailBackfiller` and `MailSyncer`'s background prefetch already share — records bytes actually pulled over the network (body chars + actually-downloaded attachment bytes, not cache hits) for Gmail accounts only. `MailBackfiller`/`MailSyncer`'s `prefetchIfEnabled` each consult `isOverDailyBudget` once per account before starting a prefetch batch and defer for the rest of the day once Gmail's budget is reached. Header paging/sync is never gated, and interactive fetches (message open, attachment tap, inline images) are never gated either — same interactive-priority principle #355/#360 already apply elsewhere. **10k-messages-per-label:** captured as a documented constant only, deliberately NOT wired into a backfill stop condition — issue #12's full-history backfill is intentional and a real large mailbox can exceed 10k messages, so treating this as a hard ceiling would silently truncate history for exactly the users the feature is for. ## How this composes with #360 / #356 - `AccountThrottleGate` (#360) is unmodified — it still handles the *reactive* case (a real provider throttle/lockout response); `ThrottleClassifier`'s existing generic patterns already match Gmail's real-world throttle text (e.g. "too many simultaneous connections"), so no Gmail-specific classifier changes were needed either. - `BackfillPacer` (#356) is unmodified — it still paces slice *cadence*, not bytes. - `GmailBandwidthTracker` is a new, orthogonal, *proactive* mechanism — the same relationship `InteractiveImapGate` documents having to `AccountThrottleGate`: composes with the other two, doesn't duplicate or replace either. ## Shared files touched (for conflict-awareness with #362/#363/#364) Kept additive and Gmail-scoped throughout — no changes to `MailProvider.kt`, `AccountThrottleGate.kt`, `BackfillPacer.kt`, `ThrottleSignal.kt`/`ThrottleClassifier.kt`/`ThrottleBackoff.kt`, `InteractiveImapGate.kt`, or `ImapClient.kt`/`ImapConnectionCache.kt`. Files touched that a sibling provider ticket *could* also want to touch: - `MailBackfiller.kt` / `MailSyncer.kt` — small additive change to `prefetchIfEnabled` (new Gmail-only budget check + a new `GmailBandwidthTracker` constructor param). Yahoo/iCloud connection caps are more likely to land in `ImapClient`/`ImapConnectionCache` instead (untouched here), but flagging in case #362/#363 also touch these two files. - `MailBackfiller.prefetchIfEnabled` also gained an `account: Account` parameter (previously just `ids: List<String>`) since the Gmail check needs the account to detect the provider. - `MailRepositoryImpl.kt` — new `bandwidthTracker: GmailBandwidthTracker` constructor param; `prefetchMessage` now records downloaded bytes; `ensureAttachmentFile` (private) now returns a small `AttachmentFetch(file, downloadedBytes)` instead of a bare `File` (its 3 call sites updated accordingly). `MailRepository`'s public interface is unchanged. Unlikely to overlap with #362/#363 (no byte-budget in their issues); #364 (Outlook/Graph) would likely land in a separate Graph REST client rather than this IMAP path, but flagging since Outlook accounts currently still flow through `ImapClient`/this same repository. - `config/detekt/detekt.yml` — one appended line, excluding the new `GmailBandwidthTrackerTest.kt` from the `android.util.Log` forbidden-import rule (it `mockkStatic(Log::class)` the same way `AccountThrottleGateTest`/`BackfillPacerTest` already do). ## Tests - `GmailSyncLimitsTest` — documented constants, the connection-cap architecture invariant, `appliesTo` for Gmail (incl. the legacy `imap.googlemail.com` host) vs. Yahoo/iCloud/AOL/Outlook. - `GmailBandwidthTrackerTest` — accumulation, per-account isolation, day-rollover reset, threshold-crossing, no-op on non-positive bytes, PII-free once-per-crossing logging. - `GmailBandwidthTrackerInstrumentedTest` (new, mock-free, mirrors `BackfillPacerInstrumentedTest`'s idiom) — concurrent-update correctness and account isolation on the real dispatcher/JVM, and `appliesTo` resolving the real `MailProvider` presets on-device. - New/updated cases in `MailBackfillerTest`, `MailSyncerTest`, `MailRepositoryImplTest` composing the new Gmail gating with the real backfill/sync/prefetch paths (incl. proving a non-Gmail account is unaffected by an over-budget tracker entry for the same account id, and that header paging/sync keeps running while prefetch is deferred). ## Verification Fast gate green locally (JDK 21): `assembleDebug`, `testDebugUnitTest`, `jacocoTestCoverageVerification`, `compileDebugAndroidTestKotlin`, `lintDebug`, `ktlintCheck`, `detekt`. Per the dispatch instructions, local emulator E2E was skipped as flaky/non-authoritative — CI's full matrix is authoritative here. Not armed for auto-merge; please review.
mergify[bot] commented 2026-07-09 00:45:41 +00:00 (Migrated from github.com)

Merge Queue Status

This pull request spent 29 minutes 44 seconds in the queue, including 26 minutes 46 seconds running CI.

Required conditions to merge
<!--- DO NOT EDIT -*- Mergify Payload -*- {"version": 1, "state": "merged", "queue_rule_name": "default", "queued_at": "2026-07-09T00:45:40.203182+00:00", "estimated_time_of_merge": null, "speculative_check_pr": null, "required_conditions": []} -*- Mergify Payload End -*- --> # Merge Queue Status - ✅ **Entered queue** — `2026-07-09 00:45 UTC` · Rule: `default` · triggered by merge protections - ✅ **Checks passed** · on draft #473 - ✅ **Merged** — `2026-07-09 01:15 UTC` · at `b1a7931dfac27eec4cb5cc195d3e02af66b8d5e0` · merge This pull request spent **29 minutes 44 seconds** in the queue, including **26 minutes 46 seconds** running CI. <details> <summary>Required conditions to merge</summary> - `-conflict` - [X] #467 - [X] #469 - `-draft` - [X] #467 - [X] #469 - [X] `base = main` - [X] `check-success = CI passed` - `github-review-approved` [🛡 GitHub repository ruleset rule `main`] - [X] #467 - [X] #469 - `label != broken` - [X] #467 - [X] #469 - [X] any of [🛡 GitHub branch protection]: - [X] `check-success = Debug build` - [ ] `check-neutral = Debug build` - [ ] `check-skipped = Debug build` - [X] any of [🛡 GitHub branch protection]: - [X] `check-success = Unit tests` - [ ] `check-neutral = Unit tests` - [ ] `check-skipped = Unit tests` - [X] any of [🛡 GitHub branch protection]: - [X] `check-success = CI passed` - [ ] `check-neutral = CI passed` - [ ] `check-skipped = CI passed` - [X] any of [🛡 GitHub repository ruleset rule `main`]: - [X] `check-success = @github-actions/CI passed` - [ ] `check-neutral = @github-actions/CI passed` - [ ] `check-skipped = @github-actions/CI passed` </details>
JMR-dev commented 2026-07-09 01:16:05 +00:00 (Migrated from github.com)

@Mergifyio refresh

@Mergifyio refresh
mergify[bot] commented 2026-07-09 01:16:12 +00:00 (Migrated from github.com)

refresh

✅ Pull request refreshed

> refresh #### ✅ Pull request refreshed
Sign in to join this conversation.