perf(search): Unicode-aware case-insensitive search via a casefold column (approach A) #232

Closed
opened 2026-07-03 16:21:46 +00:00 by JMR-dev · 1 comment
JMR-dev commented 2026-07-03 16:21:46 +00:00 (Migrated from github.com)

Approach A from the #227 examination (recommended). Restores Unicode-aware case-insensitive substring search — identical semantics to pre-#223, minus the ASCII-only LIKE limitation.

Implementation

  • Schema v18→v19: add a searchText column to messages (MessageEntity), populated at write time in the entity mapper as lowercase() of the searchable fields concatenated (sender + senderEmail + subject + snippet). Kotlin String.lowercase() is Unicode-aware and locale-independent, so it casefolds beyond ASCII.
  • Migration: add the column + backfill existing rows. SQL lower() is ASCII-only, so either (a) backfill via SQL lower() and let non-ASCII rows self-heal on their next sync/rewrite, or (b) backfill in Kotlin for immediate correctness. Export the schema and add a MigrationTest v18→v19 case.
  • Queries: replace the four-way LIKE in MessageDao.pagingUnifiedFolderSearchSummaries / pagingFolderSearchSummaries with a single WHERE searchText LIKE :pattern; the repo builds pattern from the lowercased query via the existing likePattern (keep \ % _ escaping + ESCAPE '\').
  • Keep inInbox unfiltered for search (still surfaces transient server-search hits), and keep the paged window loading + load-state-gated empty state from #223.

Tests

  • MailRepositoryImplTest: a Unicode case-insensitive match (e.g. query "ÄPFEL" matches stored "äpfel"), and the existing LIKE-escape assertion still holds against the lowercased pattern.
  • MigrationTest: v18→v19 adds the column and preserves rows.

Notes

  • Perf profile is unchanged (leading-% scan, folder-scoped + paged) — a correctness fix, not a perf regression; one comparison instead of four.
  • Rejected alternatives (ICU not compiled into the SQLCipher build; FTS changes substring→token semantics) are documented in #227.
**Approach A** from the #227 examination (recommended). Restores Unicode-aware case-insensitive **substring** search — identical semantics to pre-#223, minus the ASCII-only `LIKE` limitation. ## Implementation - **Schema v18→v19:** add a `searchText` column to `messages` (`MessageEntity`), populated at write time in the entity mapper as `lowercase()` of the searchable fields concatenated (`sender` + `senderEmail` + `subject` + `snippet`). Kotlin `String.lowercase()` is Unicode-aware and locale-independent, so it casefolds beyond ASCII. - **Migration:** add the column + backfill existing rows. SQL `lower()` is ASCII-only, so either (a) backfill via SQL `lower()` and let non-ASCII rows self-heal on their next sync/rewrite, or (b) backfill in Kotlin for immediate correctness. Export the schema and add a `MigrationTest` v18→v19 case. - **Queries:** replace the four-way `LIKE` in `MessageDao.pagingUnifiedFolderSearchSummaries` / `pagingFolderSearchSummaries` with a single `WHERE searchText LIKE :pattern`; the repo builds `pattern` from the **lowercased** query via the existing `likePattern` (keep `\ % _` escaping + `ESCAPE '\'`). - Keep `inInbox` unfiltered for search (still surfaces transient server-search hits), and keep the paged window loading + load-state-gated empty state from #223. ## Tests - `MailRepositoryImplTest`: a Unicode case-insensitive match (e.g. query `"ÄPFEL"` matches stored `"äpfel"`), and the existing LIKE-escape assertion still holds against the lowercased pattern. - `MigrationTest`: v18→v19 adds the column and preserves rows. ## Notes - Perf profile is unchanged (leading-`%` scan, folder-scoped + paged) — a correctness fix, not a perf regression; one comparison instead of four. - Rejected alternatives (ICU not compiled into the SQLCipher build; FTS changes substring→token semantics) are documented in #227.
JMR-dev commented 2026-07-03 16:27:34 +00:00 (Migrated from github.com)

Implementation finding — single searchText won't stay in sync; use per-field casefold columns

The searchable fields aren't written only through the entity mapper — they're maintained by partial UPDATEs that don't carry all four fields:

  • MessageDao.insertNew (via toEntity) sets sender/senderEmail/subject, but snippet is "" at insert (Mappers.kt:142).
  • MessageDao.updateBody (MessageDao.kt:179) sets the snippet later, when the body is fetched — so the snippet only exists post-insert.
  • MessageDao.updateHeaderContent (MessageDao.kt:162) refreshes sender/senderEmail/subject (headers rarely change, but for correctness).

A single concatenated searchText populated only in toEntity would therefore never include the snippet and would go stale on a header refresh — breaking snippet search. Recomputing a concatenated column inside a partial UPDATE would need the other fields, which those methods don't have.

Corrected design (still approach A — casefold columns)

Per-field fold columns, because each depends only on its own source field, so each write site can maintain its own:

  • Add senderFold, senderEmailFold, subjectFold, snippetFold (TEXT NOT NULL DEFAULT ''), each = source.lowercase() (Kotlin, Unicode-aware).
  • toEntity sets all four (snippetFold = ""); updateBody also sets snippetFold; updateHeaderContent also sets the three header folds.
  • Search: WHERE senderFold LIKE :p OR senderEmailFold LIKE :p OR subjectFold LIKE :p OR snippetFold LIKE :p, :p built from the lowercased query (keep \ % _ escaping + ESCAPE '\').
  • Migration v18→v19: add the columns + ASCII lower() backfill (non-ASCII rows self-heal on their next write). Export schema + MigrationTest.

Scope options

  • Full (recommended): all four fold columns — matches the pre-#223 behaviour, which searched the snippet too. Touches toEntity + updateBody + updateHeaderContent.
  • Headers-only: senderFold/senderEmailFold/subjectFold (3 columns), snippet not searchable — drops the updateBody change and one column, but you lose "find by body-preview text."
## Implementation finding — single `searchText` won't stay in sync; use per-field casefold columns The searchable fields aren't written only through the entity mapper — they're maintained by **partial `UPDATE`s** that don't carry all four fields: - `MessageDao.insertNew` (via `toEntity`) sets sender/senderEmail/subject, but **snippet is `""` at insert** (`Mappers.kt:142`). - `MessageDao.updateBody` (`MessageDao.kt:179`) sets the **snippet** later, when the body is fetched — so the snippet only exists *post-insert*. - `MessageDao.updateHeaderContent` (`MessageDao.kt:162`) refreshes sender/senderEmail/subject (headers rarely change, but for correctness). A single concatenated `searchText` populated only in `toEntity` would therefore **never include the snippet** and would go stale on a header refresh — breaking snippet search. Recomputing a concatenated column inside a partial `UPDATE` would need the *other* fields, which those methods don't have. ### Corrected design (still approach A — casefold columns) **Per-field fold columns**, because each depends only on its own source field, so each write site can maintain its own: - Add `senderFold`, `senderEmailFold`, `subjectFold`, `snippetFold` (`TEXT NOT NULL DEFAULT ''`), each `= source.lowercase()` (Kotlin, Unicode-aware). - `toEntity` sets all four (`snippetFold = ""`); `updateBody` also sets `snippetFold`; `updateHeaderContent` also sets the three header folds. - Search: `WHERE senderFold LIKE :p OR senderEmailFold LIKE :p OR subjectFold LIKE :p OR snippetFold LIKE :p`, `:p` built from the **lowercased** query (keep `\ % _` escaping + `ESCAPE '\'`). - Migration v18→v19: add the columns + ASCII `lower()` backfill (non-ASCII rows self-heal on their next write). Export schema + `MigrationTest`. ### Scope options - **Full (recommended):** all four fold columns — matches the pre-#223 behaviour, which searched the snippet too. Touches `toEntity` + `updateBody` + `updateHeaderContent`. - **Headers-only:** `senderFold`/`senderEmailFold`/`subjectFold` (3 columns), snippet **not** searchable — drops the `updateBody` change and one column, but you lose "find by body-preview text."
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: JMR-dev/LibreMail#232