feat(sync): graceful degradation + exponential backoff on provider throttling #436

Merged
JMR-dev merged 2 commits from feat-360-throttling-backoff into main 2026-07-08 19:45:47 +00:00
JMR-dev commented 2026-07-08 12:43:12 +00:00 (Migrated from github.com)

What & why

Closes #360. Adds the reactive throttle layer: when a provider rate-limits or locks us, the app backs off exponentially (with jitter) and degrades gracefully — pausing the offending activity (esp. background backfill) instead of hammering the server. The on-device perf drilldown (docs/perf/issue-125-*) proved that hammering a throttling server makes throttling worse and can trip a lasting lockout (e.g. Yahoo's ~1h lock); connection reuse (#357) is the proactive fix, this is the reactive one.

Approach

Classification signals (ThrottleClassifier, message-text + HTTP-status):

  • Rate limit: IMAP [THROTTLED] / "too many requests" / "too many simultaneous connections", RFC 5530 [LIMIT]/[UNAVAILABLE], SMTP "too many messages", HTTP 429/503.
  • Lockout: "account (temporarily) locked/suspended", "too many login attempts".
  • Deliberately conservative: a wrong password or the #390 "IMAP disabled" state is never misclassified (a false positive would silently stall an account for hours).

Backoff schedule (ThrottleBackoff, pure/clock-free):

  • Exponential base * 2^(attempt-1), equal jitter → [capped/2, capped], bounded max cap.
  • Rate limit: 30s base, 15m cap. Lockout: 1h base (sized to Yahoo's lock), 4h cap.
  • Honors a provider Retry-After as a lower bound.

Degradation + per-account isolation (AccountThrottleGate, @Singleton):

  • Per-account backoff window keyed by account id — one throttled account never stalls the others.
  • MailBackfiller skips an account inside its window, and stops paging one that throttles mid-slice (records the backoff, does not set moreWork, so BackfillWorker's slice-chaining loop stops rather than tight-looping against the server). Resumes on a later scheduled slice once the window elapses; resets on the next successful page.
  • MailSyncer (interactive/foreground sync) feeds the gate on a throttle failure but is never blocked by it — interactive priority over backfill.

Integration, not duplication: builds on the existing WorkManager retry (#403) and connection reuse (#357). The classifier also exposes classifyHttpStatus(...) + Retry-After honoring so the Outlook/Graph send path can wire in via the per-provider follow-up issues; this PR wires the IMAP backfill + foreground-sync paths.

Tests

  • ThrottleClassifierTest — positive + negative classification across IMAP/SMTP/HTTP, cause-chain + cyclic-chain, the #390 exclusion.
  • ThrottleBackoffTest — exponential growth, jitter bounds, cap, lockout window, Retry-After floor.
  • AccountThrottleGateTest — escalation, per-account isolation, reset-on-success, and the window clearing exactly when the backoff elapses (coroutines-test virtual time).
  • MailBackfillerTest / MailSyncerTest — throttle pauses/skips the account (no tight loop) and foreground sync records-without-blocking.
  • New source files carry SPDX headers; all logging is PII-free AppLog (accountLogRef + durations/counts).

Gate

Local fast gate green: assembleDebug + testDebugUnitTest + jacocoTestCoverageVerification (0.84 floor held) + compileDebugAndroidTestKotlin + lintDebug + ktlintCheck + detekt. No instrumented test touched (background logic; unit-tested); E2E via the CI matrix.

Notes / assumptions

  • Gate state is in-process (@Singleton); a process restart resets it — fine, WorkManager job backoff covers the cross-process case and a fresh process re-probes.
  • This is the umbrella #360's reactive layer. The per-provider concurrency caps (Gmail 15 / Yahoo 5 / iCloud 5 / Graph 4) and the Graph Retry-After send-path wiring remain for the per-provider sub-issues; the shared classifier/backoff/gate are ready for them.
## What & why Closes #360. Adds the **reactive throttle layer**: when a provider rate-limits or locks us, the app backs off exponentially (with jitter) and **degrades gracefully** — pausing the offending activity (esp. background backfill) instead of hammering the server. The on-device perf drilldown (`docs/perf/issue-125-*`) proved that hammering a throttling server makes throttling worse and can trip a lasting lockout (e.g. Yahoo's ~1h lock); connection reuse (#357) is the *proactive* fix, this is the *reactive* one. ## Approach **Classification signals** (`ThrottleClassifier`, message-text + HTTP-status): - Rate limit: IMAP `[THROTTLED]` / "too many requests" / "too many simultaneous connections", RFC 5530 `[LIMIT]`/`[UNAVAILABLE]`, SMTP "too many messages", HTTP 429/503. - Lockout: "account (temporarily) locked/suspended", "too many login attempts". - Deliberately conservative: a wrong password or the #390 "IMAP disabled" state is **never** misclassified (a false positive would silently stall an account for hours). **Backoff schedule** (`ThrottleBackoff`, pure/clock-free): - Exponential `base * 2^(attempt-1)`, **equal jitter** → `[capped/2, capped]`, bounded max cap. - Rate limit: 30s base, 15m cap. Lockout: 1h base (sized to Yahoo's lock), 4h cap. - Honors a provider `Retry-After` as a lower bound. **Degradation + per-account isolation** (`AccountThrottleGate`, `@Singleton`): - Per-account backoff window keyed by account id — one throttled account never stalls the others. - `MailBackfiller` skips an account inside its window, and stops paging one that throttles mid-slice (records the backoff, does **not** set `moreWork`, so `BackfillWorker`'s slice-chaining loop stops rather than tight-looping against the server). Resumes on a later scheduled slice once the window elapses; resets on the next successful page. - `MailSyncer` (interactive/foreground sync) **feeds** the gate on a throttle failure but is **never blocked** by it — interactive priority over backfill. **Integration, not duplication:** builds on the existing WorkManager retry (#403) and connection reuse (#357). The classifier also exposes `classifyHttpStatus(...)` + `Retry-After` honoring so the Outlook/Graph send path can wire in via the per-provider follow-up issues; this PR wires the IMAP backfill + foreground-sync paths. ## Tests - `ThrottleClassifierTest` — positive + negative classification across IMAP/SMTP/HTTP, cause-chain + cyclic-chain, the #390 exclusion. - `ThrottleBackoffTest` — exponential growth, jitter bounds, cap, lockout window, Retry-After floor. - `AccountThrottleGateTest` — escalation, per-account isolation, reset-on-success, and the window clearing exactly when the backoff elapses (**coroutines-test virtual time**). - `MailBackfillerTest` / `MailSyncerTest` — throttle pauses/skips the account (no tight loop) and foreground sync records-without-blocking. - New source files carry SPDX headers; all logging is PII-free `AppLog` (accountLogRef + durations/counts). ## Gate Local fast gate green: `assembleDebug` + `testDebugUnitTest` + `jacocoTestCoverageVerification` (0.84 floor held) + `compileDebugAndroidTestKotlin` + `lintDebug` + `ktlintCheck` + `detekt`. No instrumented test touched (background logic; unit-tested); E2E via the CI matrix. ## Notes / assumptions - Gate state is in-process (`@Singleton`); a process restart resets it — fine, WorkManager job backoff covers the cross-process case and a fresh process re-probes. - This is the umbrella #360's reactive layer. The per-provider concurrency caps (Gmail 15 / Yahoo 5 / iCloud 5 / Graph 4) and the Graph `Retry-After` send-path wiring remain for the per-provider sub-issues; the shared classifier/backoff/gate are ready for them.
JMR-dev commented 2026-07-08 18:50:14 +00:00 (Migrated from github.com)

@Mergifyio refresh

@Mergifyio refresh
mergify[bot] commented 2026-07-08 18:50:28 +00:00 (Migrated from github.com)

refresh

✅ Pull request refreshed

> refresh #### ✅ Pull request refreshed
mergify[bot] commented 2026-07-08 19:18:12 +00:00 (Migrated from github.com)

Merge Queue Status

This pull request spent 27 minutes 41 seconds in the queue, including 27 minutes 14 seconds running CI.

Required conditions to merge
<!--- DO NOT EDIT -*- Mergify Payload -*- {"version": 1, "state": "merged", "queue_rule_name": "default", "queued_at": "2026-07-08T19:18:06.523724+00:00", "estimated_time_of_merge": null, "speculative_check_pr": null, "required_conditions": []} -*- Mergify Payload End -*- --> # Merge Queue Status - ✅ **Entered queue** — `2026-07-08 19:18 UTC` · Rule: `default` · triggered by merge protections - ✅ **Checks passed** · on draft #456 - ✅ **Merged** — `2026-07-08 19:45 UTC` · at `b63354b89a03d9358b5e05f87c69b51f60cf5eb4` · merge This pull request spent **27 minutes 41 seconds** in the queue, including **27 minutes 14 seconds** running CI. <details> <summary>Required conditions to merge</summary> - `-conflict` - [X] #434 - [X] #436 - `-draft` - [X] #434 - [X] #436 - [X] `base = main` - [X] `check-success = CI passed` - `github-review-approved` [🛡 GitHub repository ruleset rule `main`] - [X] #434 - [X] #436 - `label != broken` - [X] #434 - [X] #436 - [X] any of [🛡 GitHub branch protection]: - [X] `check-success = Debug build` - [ ] `check-neutral = Debug build` - [ ] `check-skipped = Debug build` - [X] any of [🛡 GitHub branch protection]: - [X] `check-success = Unit tests` - [ ] `check-neutral = Unit tests` - [ ] `check-skipped = Unit tests` - [X] any of [🛡 GitHub branch protection]: - [X] `check-success = CI passed` - [ ] `check-neutral = CI passed` - [ ] `check-skipped = CI passed` - [X] any of [🛡 GitHub repository ruleset rule `main`]: - [X] `check-success = @github-actions/CI passed` - [ ] `check-neutral = @github-actions/CI passed` - [ ] `check-skipped = @github-actions/CI passed` </details>
Sign in to join this conversation.