sync: age-based retention re-downloads pruned messages on low-traffic mailboxes (window not bounded by age cutoff) #193

Closed
opened 2026-07-03 02:06:38 +00:00 by JMR-dev · 0 comments
JMR-dev commented 2026-07-03 02:06:38 +00:00 (Migrated from github.com)

Flagged during #53's concurrency analysis — out of scope there: this is a re-download efficiency issue, not a correctness bug (no rows are lost/duplicated/wrongly-deleted), and it happens sequentially, not only under concurrency.

Problem

Retention has two modes (data/local/entity/AccountSettingsEntity.kt): count-based (retentionCount, newest-N per folder) and age-based (retentionMonths). The sync window and the pruner are only aligned in COUNT mode:

  • MailSyncer.recentWindowFor fetches the newest-by-UID window and caps it to the retention count, so in count mode sync + prune are disjoint (sync only reconciles uid >= minWindowUid via MessageDao.deleteSyncedInWindowNotIn; prune deletes below that floor).
  • In AGE mode, MailPruner deletes messages older than a date cutoff (policy.ageCutoffMillis(now) → MessageDao.syncedIdsOlderThan(accountId, cutoff)), but the sync window is still count/UID-based and is not bounded by that date cutoff.

On a low-traffic mailbox where fewer than the window-count of messages fall within the age window, the newest-N UIDs extend past the age cutoff — so a sync re-fetches and re-inserts messages the age-pruner just deleted, and the next prune deletes them again. Result: a churn loop of wasted IMAP fetches + DB writes + prune deletes every cycle (bandwidth, battery, IO) for messages the user chose not to retain.

MailPruner's KDoc currently asserts the sync/prune working sets are "disjoint" — true for count mode, not age mode; doc + behavior should be reconciled.

Repro

  • Age-based retention (retentionMonths) active for a folder.
  • The folder's newest-N (sync-window count) messages span dates OLDER than the age cutoff (low message volume vs. window size).
  • After an age-prune, the next sync re-downloads the just-pruned old messages; they're pruned again next cycle.

Fix directions (pick simplest; keep count-mode behavior unchanged)

  • Bound the sync fetch/reconcile window by the age cutoff when age mode is active (intersect the recent-UID window with the age boundary so it never re-inserts below what prune deletes), OR
  • Have sync + prune share one "oldest-to-keep" boundary correct for both modes, OR
  • Track the pruned boundary and have sync skip re-inserting below it.

Acceptance

  • Low-traffic mailbox + age retention: a sync after an age-prune does NOT re-fetch/re-insert the pruned messages (no re-download loop).
  • Count-mode behavior unchanged.
  • MailPruner's "disjoint working sets" KDoc accurate for both modes.
  • A test covers the age-mode low-traffic case (sync after age-prune doesn't resurrect pruned rows).

Relevant code

  • data/sync/MailPruner.kt (age cutoff -> syncedIdsOlderThan)
  • MailSyncer.recentWindowFor (count/UID window; caps to retentionCount only)
  • data/local/dao/MessageDao.kt (deleteSyncedInWindowNotIn uid >= minWindowUid; syncedIdsOlderThan)
  • data/local/entity/AccountSettingsEntity.kt (retentionCount vs retentionMonths)

Not a data-integrity bug — purely re-download efficiency/churn; priority accordingly.

Flagged during #53's concurrency analysis — **out of scope there**: this is a re-download **efficiency** issue, not a correctness bug (no rows are lost/duplicated/wrongly-deleted), and it happens sequentially, not only under concurrency. ## Problem Retention has two modes (`data/local/entity/AccountSettingsEntity.kt`): count-based (`retentionCount`, newest-N per folder) and age-based (`retentionMonths`). The sync window and the pruner are only aligned in COUNT mode: - `MailSyncer.recentWindowFor` fetches the newest-by-UID window and caps it to the retention **count**, so in count mode sync + prune are disjoint (sync only reconciles `uid >= minWindowUid` via `MessageDao.deleteSyncedInWindowNotIn`; prune deletes below that floor). - In AGE mode, `MailPruner` deletes messages older than a **date** cutoff (`policy.ageCutoffMillis(now)` → `MessageDao.syncedIdsOlderThan(accountId, cutoff)`), but the sync window is still count/UID-based and is **not** bounded by that date cutoff. On a **low-traffic mailbox** where fewer than the window-count of messages fall within the age window, the newest-N UIDs extend past the age cutoff — so a sync re-fetches and re-inserts messages the age-pruner just deleted, and the next prune deletes them again. Result: a churn loop of wasted IMAP fetches + DB writes + prune deletes every cycle (bandwidth, battery, IO) for messages the user chose not to retain. `MailPruner`'s KDoc currently asserts the sync/prune working sets are "disjoint" — true for count mode, not age mode; doc + behavior should be reconciled. ## Repro - Age-based retention (`retentionMonths`) active for a folder. - The folder's newest-N (sync-window count) messages span dates OLDER than the age cutoff (low message volume vs. window size). - After an age-prune, the next sync re-downloads the just-pruned old messages; they're pruned again next cycle. ## Fix directions (pick simplest; keep count-mode behavior unchanged) - Bound the sync fetch/reconcile window by the age cutoff when age mode is active (intersect the recent-UID window with the age boundary so it never re-inserts below what prune deletes), OR - Have sync + prune share one "oldest-to-keep" boundary correct for both modes, OR - Track the pruned boundary and have sync skip re-inserting below it. ## Acceptance - Low-traffic mailbox + age retention: a sync after an age-prune does NOT re-fetch/re-insert the pruned messages (no re-download loop). - Count-mode behavior unchanged. - `MailPruner`'s "disjoint working sets" KDoc accurate for both modes. - A test covers the age-mode low-traffic case (sync after age-prune doesn't resurrect pruned rows). ## Relevant code - `data/sync/MailPruner.kt` (age cutoff -> `syncedIdsOlderThan`) - `MailSyncer.recentWindowFor` (count/UID window; caps to `retentionCount` only) - `data/local/dao/MessageDao.kt` (`deleteSyncedInWindowNotIn` `uid >= minWindowUid`; `syncedIdsOlderThan`) - `data/local/entity/AccountSettingsEntity.kt` (`retentionCount` vs `retentionMonths`) Not a data-integrity bug — purely re-download efficiency/churn; priority accordingly.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: JMR-dev/LibreMail#193