fix(sync): a burst of more than 50 new messages between syncs leaves a permanent message gap #487

Open
opened 2026-07-10 19:14:33 +00:00 by JMR-dev · 0 comments
JMR-dev commented 2026-07-10 19:14:33 +00:00 (Migrated from github.com)

Verified finding(s) from the 2026-07-09 whole-repo multi-agent review (independent finder, then adversarial verifier; verdict CONFIRMED).

Triage: above the cut — fix dispatched immediately; this issue tracks the fix to Done.

app/src/main/kotlin/org/libremail/data/sync/MailBackfiller.kt:155 — high

Once a folder's backfill progress is marked complete, backfillFolder returns immediately forever, but foreground sync only fetches the newest 50 headers — so any burst of more than FETCH_LIMIT new messages between syncs leaves a permanent gap of messages that are never fetched and never shown in the offline-first UI.

Failure scenario: A folder finishes backfill (progress.complete=true — the normal steady state). The device is off/airplane-mode for a weekend and 200 new messages arrive. The next MailSyncer sync inserts only the newest 50 (FETCH_LIMIT); messages ranked 51-200 sit between the old backfill boundary and the new recent window. Backfill is the only caller of fetchOlderThan, and it early-returns on complete (progress is only reset by a retention-settings change), so those 150 messages are never cached and never appear in the message list, silently and permanently.

Verifier justification (CONFIRMED): MailBackfiller.kt:155 returns immediately once progress.complete==true, and even without that flag backfill only pages downward from the persisted low-water mark via fetchOlderThan — it can never fetch UIDs above its boundary. Foreground sync (MailSyncer.kt:102-103, FETCH_LIMIT=50; ImapClient.fetchRecent fetches total-limit+1..total) only inserts the newest 50. The only BackfillProgress resets are retention-settings changes (SettingsViewModel/AccountSettingsViewModel -> resetBackfillProgress); no code detects a recent window whose lowest UID exceeds the highest cached UID. So if >50 messages arrive in a folder between syncs after backfill completes (e.g. device off for a weekend), messages 51..N fall between the old cached region and the new window and are never fetched by any path — a silent, permanent gap in the offline-first message list. This is not the known-intentional marks-complete-at-floor behavior, which concerns the retention floor, not new-mail bursts above the boundary.

Defective line: if (progress?.complete == true) { return FolderResult(batches = 0, moreWork = false) }

Fix hint: In MailSyncer.syncFolderHeaders, after a FULL recent window lands, detect a gap (fetched window's minimum positive UID greater than the previously highest cached synced UID) and upsert the folder's BackfillProgressEntity with complete=false and nextBeforeUid=minWindowUid so the backfiller pages down through the gap (insertNew IGNORE makes any overlap with existing rows harmless).

app/src/main/kotlin/org/libremail/data/sync/MailSyncer.kt:103 — high

When more than FETCH_LIMIT (50) messages arrive in a folder between two syncs, the messages below the new 50-message recent window are never fetched by any path: foreground sync only ever fetches the newest 50, and MailBackfiller only pages monotonically downward from a boundary already below the gap — so those messages permanently never appear in the app.

Failure scenario: Device is in polling mode (low battery drops IDLE per SyncResourcePolicy, 15-min periodic sync) or offline overnight; a busy inbox receives 60 messages before the next sync. fetchRecent(params, folder, 50) returns only the newest 50 (ImapClient.kt:179 takes messages total-49..total); the 10 oldest of the burst are never inserted. Backfill cannot heal it: backfillFolder resumes from nextBeforeUid/lowestSyncedUid which is below the gap and 'never re-descends' (MailBackfiller.kt:159-168), and once progress.complete=true (MailBackfiller.kt:155) the folder is never paged again. Since the UI is offline-first and reads only Room, those 10 messages are invisible in LibreMail forever while visible in every other client — silently missed mail.

Verifier justification (CONFIRMED): Verified all three legs of the failure. (1) Every sync path fetches only the newest window: MailSyncer.kt:103 imapClient.fetchRecent(params, folder, window) with window <= FETCH_LIMIT=50, and ImapClient.kt:179 mailbox.getMessages(maxOf(1, total - limit + 1), total) materializes only the newest 50. (2) The deletion reconcile is bounded below by minWindowUid (MailSyncer.kt:144-147), so it neither sees nor heals anything between the previously-cached max UID and the new window's min UID. (3) MailBackfiller.backfillFolder returns early when progress.complete==true (line 155) and otherwise resumes from nextBeforeUid/lowestSyncedUid (lines 166-168), a boundary already below the gap, with fetchOlderThan paging strictly downward — the KDoc itself says paging "never re-descends". Repo-wide grep shows no UIDNEXT tracking or gap detection anywhere; resetBackfillProgress is only invoked on user retention-settings changes. Trigger: 51+ messages arrive in a folder between two syncs (offline overnight, 15-min polling mode, or a non-inbox folder only synced on open); the oldest N-50 of the burst land in a UID range no code path will ever fetch, so they are permanently invisible in the Room-backed offline-first UI. Not covered by any known-intentional design (marks-complete-at-floor concerns retention floors, not arrival bursts above the backfill boundary).

Defective line: val fetched = imapClient.fetchRecent(params, folder, window) // cancellable network I/O

Fix hint: In syncFolderHeaders, after a non-empty fetch compare minWindowUid against the highest previously-cached synced UID for the folder; if minWindowUid > cachedMaxUid + 1, the burst overflowed the window — either loop fetchOlderThan(params, folder, minWindowUid, ...) until the page overlaps cachedMaxUid, or persist a bounded per-folder gap (loUid=cachedMaxUid, hiUid=minWindowUid) that MailBackfiller drains even when progress.complete==true.

Verified finding(s) from the 2026-07-09 whole-repo multi-agent review (independent finder, then adversarial verifier; verdict **CONFIRMED**). **Triage: above the cut — fix dispatched immediately; this issue tracks the fix to Done.** ## `app/src/main/kotlin/org/libremail/data/sync/MailBackfiller.kt:155` — high Once a folder's backfill progress is marked complete, backfillFolder returns immediately forever, but foreground sync only fetches the newest 50 headers — so any burst of more than FETCH_LIMIT new messages between syncs leaves a permanent gap of messages that are never fetched and never shown in the offline-first UI. **Failure scenario:** A folder finishes backfill (progress.complete=true — the normal steady state). The device is off/airplane-mode for a weekend and 200 new messages arrive. The next MailSyncer sync inserts only the newest 50 (FETCH_LIMIT); messages ranked 51-200 sit between the old backfill boundary and the new recent window. Backfill is the only caller of fetchOlderThan, and it early-returns on complete (progress is only reset by a retention-settings change), so those 150 messages are never cached and never appear in the message list, silently and permanently. **Verifier justification (CONFIRMED):** MailBackfiller.kt:155 returns immediately once progress.complete==true, and even without that flag backfill only pages downward from the persisted low-water mark via fetchOlderThan — it can never fetch UIDs above its boundary. Foreground sync (MailSyncer.kt:102-103, FETCH_LIMIT=50; ImapClient.fetchRecent fetches total-limit+1..total) only inserts the newest 50. The only BackfillProgress resets are retention-settings changes (SettingsViewModel/AccountSettingsViewModel -> resetBackfillProgress); no code detects a recent window whose lowest UID exceeds the highest cached UID. So if >50 messages arrive in a folder between syncs after backfill completes (e.g. device off for a weekend), messages 51..N fall between the old cached region and the new window and are never fetched by any path — a silent, permanent gap in the offline-first message list. This is not the known-intentional marks-complete-at-floor behavior, which concerns the retention floor, not new-mail bursts above the boundary. **Defective line:** `if (progress?.complete == true) { return FolderResult(batches = 0, moreWork = false) }` **Fix hint:** In MailSyncer.syncFolderHeaders, after a FULL recent window lands, detect a gap (fetched window's minimum positive UID greater than the previously highest cached synced UID) and upsert the folder's BackfillProgressEntity with complete=false and nextBeforeUid=minWindowUid so the backfiller pages down through the gap (insertNew IGNORE makes any overlap with existing rows harmless). ## `app/src/main/kotlin/org/libremail/data/sync/MailSyncer.kt:103` — high When more than FETCH_LIMIT (50) messages arrive in a folder between two syncs, the messages below the new 50-message recent window are never fetched by any path: foreground sync only ever fetches the newest 50, and MailBackfiller only pages monotonically downward from a boundary already below the gap — so those messages permanently never appear in the app. **Failure scenario:** Device is in polling mode (low battery drops IDLE per SyncResourcePolicy, 15-min periodic sync) or offline overnight; a busy inbox receives 60 messages before the next sync. fetchRecent(params, folder, 50) returns only the newest 50 (ImapClient.kt:179 takes messages total-49..total); the 10 oldest of the burst are never inserted. Backfill cannot heal it: backfillFolder resumes from nextBeforeUid/lowestSyncedUid which is below the gap and 'never re-descends' (MailBackfiller.kt:159-168), and once progress.complete=true (MailBackfiller.kt:155) the folder is never paged again. Since the UI is offline-first and reads only Room, those 10 messages are invisible in LibreMail forever while visible in every other client — silently missed mail. **Verifier justification (CONFIRMED):** Verified all three legs of the failure. (1) Every sync path fetches only the newest window: MailSyncer.kt:103 `imapClient.fetchRecent(params, folder, window)` with window <= FETCH_LIMIT=50, and ImapClient.kt:179 `mailbox.getMessages(maxOf(1, total - limit + 1), total)` materializes only the newest 50. (2) The deletion reconcile is bounded below by minWindowUid (MailSyncer.kt:144-147), so it neither sees nor heals anything between the previously-cached max UID and the new window's min UID. (3) MailBackfiller.backfillFolder returns early when progress.complete==true (line 155) and otherwise resumes from nextBeforeUid/lowestSyncedUid (lines 166-168), a boundary already below the gap, with fetchOlderThan paging strictly downward — the KDoc itself says paging "never re-descends". Repo-wide grep shows no UIDNEXT tracking or gap detection anywhere; resetBackfillProgress is only invoked on user retention-settings changes. Trigger: 51+ messages arrive in a folder between two syncs (offline overnight, 15-min polling mode, or a non-inbox folder only synced on open); the oldest N-50 of the burst land in a UID range no code path will ever fetch, so they are permanently invisible in the Room-backed offline-first UI. Not covered by any known-intentional design (marks-complete-at-floor concerns retention floors, not arrival bursts above the backfill boundary). **Defective line:** `val fetched = imapClient.fetchRecent(params, folder, window) // cancellable network I/O` **Fix hint:** In syncFolderHeaders, after a non-empty fetch compare minWindowUid against the highest previously-cached synced UID for the folder; if minWindowUid > cachedMaxUid + 1, the burst overflowed the window — either loop fetchOlderThan(params, folder, minWindowUid, ...) until the page overlaps cachedMaxUid, or persist a bounded per-folder gap (loUid=cachedMaxUid, hiUid=minWindowUid) that MailBackfiller drains even when progress.complete==true.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: JMR-dev/LibreMail#487