Merge pull request #408 from JMR-dev/ci-407-mergify-spec

docs(ci): propose Mergify merge-queue integration spec (#407)
This commit was merged in pull request #408.
This commit is contained in:
Jason Ross
2026-07-06 22:08:34 -05:00
committed by GitHub
2 changed files with 457 additions and 0 deletions
+341
View File
@@ -0,0 +1,341 @@
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
# Mergify integration spec (proposal for issue #407)
> **Status: PROPOSAL — NOTHING IS ACTIVE.** This document and its companion
> `docs/ci/mergify.yml.proposed` are a review artifact. No Mergify app is installed and
> **no live `.mergify.yml` exists** in the repo, so Mergify does nothing until a maintainer
> explicitly adopts it. This spec exists to be reviewed *before* adoption. It must not weaken
> the `require branches to be up to date` invariant — see [The require-up-to-date trilemma](#the-require-up-to-date-trilemma),
> which is the central decision this proposal surfaces.
## TL;DR / recommendation
- **Mergify is free for open source** (public repos, unlimited contributors) and the free
"Open Source" plan includes the full **Merge Queue + batching + speculative (parallel) checks**.
- Adopting it lets us **retire the entire hand-rolled traffic-controller** (`autoupdate.yml`
+ `ci-trigger.yml` + the mothballed `traffic-control.yml`) and the manual serial-bump grind,
replacing a "poor-man's merge queue" with a real one.
- **Recommendation: adopt Mergify, in two phases.**
- **Phase 1 (adopt now, zero impact on the hard rule):** a **serial** queue
(`batch_size: 1`, `merge_method: merge`). Mergify respects branch protection, keeps
`require branches up to date` **literally ON**, does the up-to-date-ing + merge itself.
This alone kills the manual grind. No throughput multiplier yet.
- **Phase 2 (the batching throughput win — needs one conscious maintainer decision):**
batch N test-only PRs into **one** CI run. Batching is **incompatible with GitHub's
literal `require branches up to date` checkbox**; to get it, the guarantee must move
*into* Mergify's queue (checkbox off, invariant preserved/strengthened by speculative
testing) — or history must go linear via `fast-forward` (which breaks the merge-commit
policy). **This is a trilemma: `require-up-to-date` literal × merge-commits × batching —
you can keep at most two.** Phase 2 should not ship until the maintainer picks which
constraint to relax.
- **Do I recommend batching?** Yes, *if* the maintainer accepts moving the up-to-date
guarantee from GitHub's checkbox into Mergify (Phase 2, `merge-batch`). That preserves the
merge-commit policy **and** the actual invariant (nothing merges without being tested on top
of the latest `main`), and it is the only path to the #373-style throughput win now that the
native merge queue is off the table. If the literal checkbox is non-negotiable *and* linear
history is unacceptable, batching is unreachable and we stop at Phase 1.
---
## 1. Problem recap
The merge cascade (see the `ci-merge-cascade` note + issue #349) comes from three things
multiplying together:
1. **Branch protection `require branches to be up to date before merging`** — a **hard
maintainer rule that stays**. Every PR must be tested against the current tip of `main`
before it merges.
2. **Serial merges** — merging PR *A* advances `main`, which makes every other open PR
out-of-date, so each must be re-bumped and re-run before it can merge.
3. **A slow, wedge-prone E2E matrix** — each PR fans out to API 29–36 + the API 37 preview
(~15 min, occasionally wedging on an emulator boot race).
Draining a multi-PR batch is therefore painful: the #373 Robolectric epic was ~6 PRs, each a
full E2E run, hand-bumped one at a time. GitHub's **native merge queue** solves exactly this,
but it is **org/enterprise-only** and **JMR-dev is a user account**, so it is unavailable.
The repo's current mitigation is a hand-built "poor-man's merge queue":
| Piece | File | Role | Status |
|---|---|---|---|
| Auto-update | `.github/workflows/autoupdate.yml` | rebases behind PRs with `GITHUB_TOKEN` (no CI re-trigger, by anti-recursion) | active, being mothballed |
| CI trigger / scheduler | `.github/workflows/ci-trigger.yml` | re-triggers CI for the top-priority PRs, `MAX_INFLIGHT_RUNS: 2` | active, being mothballed |
| Runner priority | `.github/workflows/traffic-control.yml` | in-run P0-preempt / P1–P9 hold-back ordering | **mothballed/disabled** |
| Decision core | `.github/scripts/traffic_control.py` | pure, unit-tested priority logic | kept (still unit-tested by `traffic-control-tests`) |
**Mergify replaces the first three outright.** The single required gate — the `ci-passed`
job, status name **`CI passed`** — stays exactly as is.
---
## 2. What Mergify is + free-for-OSS eligibility
Mergify is a third-party GitHub App providing a real **merge queue** with **batching**,
**speculative/parallel checks**, auto-update/rebase, and label-driven **priority**.
- **Free for open source.** The **"Open Source" plan is $0** with **unlimited users** for
public repositories (private teams are free up to 5 active contributors). LibreMail is a
public GPL-3.0 repo, so it qualifies. ([pricing](https://mergify.com/pricing),
[marketplace](https://github.com/marketplace/mergify))
- **Full product on the free tier.** "Every plan includes … Merge Queue, Merge Protections …"
— batching and speculative checks are **not** gated behind a paid tier.
([pricing](https://mergify.com/pricing))
---
## 3. How Mergify's merge queue works
1. A PR is **queued** (by a `pull_request_rules` `queue` action when it is green + approved,
or manually via a `@mergifyio queue` comment).
2. Mergify builds a **temporary branch** = *latest `main`* + the queued PR(s), and runs your
CI on that temporary branch — i.e. it tests **against an up-to-date base speculatively**,
without touching the PR's own branch. ([merge queue](https://docs.mergify.com/merge-queue/),
[parallel checks](https://docs.mergify.com/merge-queue/parallel-checks/))
3. When the temporary branch's checks pass, Mergify **merges** the original PR(s).
4. **Speculative / parallel checks:** Mergify validates several positions of the queue at once
— `(A)`, `(A+B)`, `(A+B+C)` — so it does not pay one-CI-run-per-PR serially.
`merge_queue.max_parallel_checks` bounds the concurrency (our analogue of the old
`MAX_INFLIGHT_RUNS: 2`). ([parallel checks](https://docs.mergify.com/merge-queue/parallel-checks/))
5. **Batching** (`batch_size > 1`): Mergify combines the next *N* PRs into **one** temporary
batch branch and validates the whole batch with **a single CI run** — the key win for
test-only batches. It groups by priority + similarity (touching-similar-files) so batches
are cohesive. ([batches](https://docs.mergify.com/merge-queue/batches/))
6. **Batch failure → automatic bisection:** a red batch is **not** wholesale dequeued; Mergify
splits it (down toward single PRs) to isolate the culprit, dequeues only that PR, and lets
the rest proceed — bounded by `batch_max_failure_resolution_attempts`.
([batches](https://docs.mergify.com/merge-queue/batches/))
---
## The require-up-to-date trilemma
**This is the crux of the whole proposal.** Mergify's own docs are explicit:
> "Batches require the branch protection setting *Require branches to be up to date before
> merging* to be **disabled**." — [Mergify: Merge Queue Batches](https://docs.mergify.com/merge-queue/batches/)
The reason: with batching, Mergify tests a *temporary batch branch* that is up-to-date, then
merges the **original** PRs — whose own branches are **not** up-to-date per GitHub — so the
literal checkbox would block the merge. The documented ways out each cost something:
| You want to keep… | …and also… | …then batching is | Cost |
|---|---|---|---|
| `require-up-to-date` checkbox **literally on** | merge commits (`merge_method: merge`) | **serial only** (no batching) | no throughput win |
| `require-up-to-date` checkbox **literally on** | **batching** | via `queue_branch_merge_method: fast-forward` | **linear history — breaks the merge-commit policy** ([discussion #5138](https://github.com/Mergifyio/mergify/discussions/5138)) |
| merge commits **and** batching | (the throughput win) | yes, `merge_method: merge-batch` | **checkbox must be off**; guarantee moves into Mergify |
So `{ require-up-to-date literal · merge-commits · batching }` — **pick two.**
**The key reframing:** the maintainer rule is really about the **invariant** — *never merge
code that wasn't tested against the latest `main`.* Mergify's speculative queue **enforces
that invariant by construction** (it literally builds `latest-main + PRs` and tests *that*
before merging). Whether the *invariant* is enforced by GitHub's checkbox or by Mergify's
queue, it holds either way. Turning the checkbox off in Phase 2 does **not** weaken the
invariant — arguably it strengthens it (GitHub's checkbox only guarantees the branch *was*
up-to-date at some point and then requires a fresh run; Mergify guarantees the *exact merged
commit set* passed on top of the exact tip). But it **does** flip a literal setting the rule
names, so it is a conscious call for the maintainer, not something this proposal does silently.
**Recommended resolution:** Phase 1 keeps the checkbox literally on (serial). Phase 2 uses
**`merge_method: merge-batch`** (one merge commit per batch — preserves the merge-commit
policy), turns the **checkbox off**, and closes the bypass with "block any merge outside
Mergify" (below). The `fast-forward` row is documented for completeness but **not
recommended** — it would silently drop merge commits.
> **Open item to verify in a trial** (do not take on faith from docs): confirm on a throwaway
> branch that `merge-batch` lands a real merge commit and that, with the checkbox off + "merge
> only via Mergify" set, no path exists to merge a stale PR outside the queue.
---
## 4. Coexistence with the single `CI passed` gate
- Branch protection keeps requiring the one context **`CI passed`**. No change to
`ci.yml`'s `ci-passed` fan-in.
- Mergify **injects the repo's required checks as queue conditions automatically**
(`branch_protection_injection_mode`, default `queue`), and the proposed config *also* names
`check-success = CI passed` in `merge_conditions` explicitly, so a PR merges **only** when
the same gate branch protection requires is green. ([queue action](https://docs.mergify.com/workflow/actions/queue/))
- During Phase-1 serial operation Mergify "**respects your GitHub branch protections and
rulesets**" out of the box — including `require-up-to-date`, which it satisfies by updating
the branch itself before merging. ([setup](https://docs.mergify.com/merge-queue/setup/))
> **Gotcha:** the condition string must match the check name **exactly** — `CI passed` (the
> `name:` of the `ci-passed` job), *not* `ci-passed`. A wrong name means PRs queue but never
> merge (or never queue).
---
## 5. Priority: mapping the P0–P9 labels
Mergify `priority_rules` assign a priority from PR conditions; higher merges first (keywords
`low`=1000 / `medium`=2000 / `high`=3000, or numeric 1–10000; unmatched → `medium`).
([priority](https://docs.mergify.com/merge-queue/priority/)) The proposed config maps
**P0 → 10000 … P9 → 1000** linearly, so:
- **P0** (emergency-only) outranks everything and is picked first for the next batch.
- Unlabelled PRs fall to Mergify's default `medium` (2000) — the same "no label ⇒ P5-ish
default" behaviour `traffic_control.py` uses today.
- `broken`/`draft` PRs are **excluded from queueing** (they simply shouldn't merge), rather
than run at the bottom as the old traffic-controller did.
`allow_checks_interruption` (default true) reproduces the old **P0-preempts** behaviour: a
higher-priority arrival can interrupt in-flight speculative checks of lower-priority PRs.
---
## 6. What it replaces
| Old piece | Fate under Mergify |
|---|---|
| `autoupdate.yml` (branch bumping) | **Disable** — the queue updates branches itself. |
| `ci-trigger.yml` (priority scheduler, `MAX_INFLIGHT_RUNS`) | **Disable** — replaced by the queue + `max_parallel_checks`. |
| `traffic-control.yml` (already mothballed) | **Delete** (or leave disabled). |
| `traffic_control.py` + `traffic-control-tests` job | **Keep** — harmless, still unit-tested; can be removed later. Note: `ci-passed` currently lists `traffic-control-tests` in its `needs`; leave it or drop it, either is fine. |
| `AUTOUPDATE_TOKEN` secret / PAT plumbing | Becomes unnecessary — Mergify acts via its App installation, not a PAT. |
Net: a large amount of bespoke CI-orchestration YAML + Python is superseded by one
`.mergify.yml` and the App.
---
## 7. Interaction with the other CI workstreams
- **E2E path-filter (#399 / #402):** synergistic. Mergify runs CI on the *batch* branch, so
the path filter evaluates the batch's *combined* diff: a docs-only batch still skips E2E and
merges fast; a test-only batch still triggers the E2E it needs. The one requirement — that a
path-skipped run still reports a green **`CI passed`** — is already how branch protection is
satisfied, and Mergify keys on the same context, so no extra work. (Verify the skip path
produces `CI passed`, not an absent check, or the queue will wait forever.)
- **E2E sharding (#372, and the API 37 N=2 shards):** orthogonal and compounding — faster CI
⇒ faster batch validation ⇒ shorter queue latency. No config interaction.
- **Wedge diagnostics (#404 / #406):** still run inside the CI Mergify triggers. Better,
`checks_timeout` on the queue bounds a wedged leg: instead of a hung run blocking the queue,
the batch times out and Mergify bisects/re-runs — turning "hand-babysit a wedged emulator"
into an automatic dequeue of the offending PR.
---
## 8. Proposed `.mergify.yml`
The full proposed config is in **`docs/ci/mergify.yml.proposed`** (kept as `*.proposed`, **not**
a live `.mergify.yml`, so nothing activates). Outline below.
```yaml
# NOT ACTIVE — see docs/ci/mergify.yml.proposed for the annotated version.
queue_rules:
- name: default
merge_conditions:
- check-success = CI passed # the single required gate
- "#approved-reviews-by >= 1"
- -draft
- label != broken
# Phase 1 (keeps require-up-to-date literally ON): batch_size: 1 + merge_method: merge
# Phase 2 (batching — checkbox off, guarantee moved into the queue):
batch_size: { min: 1, max: 5 } # dynamic; one CI run per batch
batch_max_wait_time: 5 min
merge_method: merge-batch # one MERGE COMMIT per batch (keeps merge policy)
checks_timeout: 45 min # slow/wedge-prone E2E: time out -> bisect
merge_queue:
max_parallel_checks: 2 # == old MAX_INFLIGHT_RUNS
priority_rules: # P0 -> 10000 ... P9 -> 1000 (higher merges first)
- { name: p0-emergency, conditions: [label = P0], priority: 10000 }
# … P1..P9 …
pull_request_rules:
- name: Queue green, approved, non-draft PRs targeting main
conditions:
- base = main
- -draft
- label != broken
- "#approved-reviews-by >= 1"
- check-success = CI passed
actions:
queue: { name: default }
```
---
## 9. Setup steps (free-OSS)
1. **Install the Mergify GitHub App** on `JMR-dev/LibreMail` from the
[marketplace](https://github.com/marketplace/mergify) (Open Source plan, $0). Requires
GitHub admin on the repo. The App requests read/write on pull requests, checks, and
contents (it must be able to create temporary branches and merge).
2. **Commit `.mergify.yml`** at the repo root — start from `docs/ci/mergify.yml.proposed`,
**Phase-1 variant** (`batch_size: 1`, `merge_method: merge`). Mergify validates the config
on push.
3. **Keep** branch protection `require branches up to date` + the required `CI passed` context
unchanged (Phase 1). Confirm PRs queue and merge.
4. **Phase 2 (only after the maintainer signs off on the trilemma):** switch to
`merge_method: merge-batch` + `batch_size`, **turn off** GitHub's literal
`require branches up to date` checkbox, and **block merges outside Mergify** (restrict who
can push/merge to `main` so the queue is the only merge path — Mergify's recommended
companion to disabling the checkbox
([discussion #5138](https://github.com/Mergifyio/mergify/discussions/5138))).
5. **Disable** `autoupdate.yml` and `ci-trigger.yml`; delete/retire `traffic-control.yml`.
---
## 10. Risks & tradeoffs
- **Third-party in the merge path.** Mergify gets write/merge authority on `main`. It is an
OSS-standard, widely used app, but it is a new external dependency and a supply-chain /
availability surface. If Mergify is down, merges pause (fail-safe, not fail-open).
- **The require-up-to-date decision (Phase 2).** Turning off the literal checkbox is the
price of batching. Mitigated by moving the guarantee into the queue **and** blocking
out-of-queue merges — but it is a real, conscious change to a rule the maintainer has
pinned, and must be signed off, not assumed.
- **Speculative-check CI cost.** Parallel checks and batch bisection **run CI more**: a failed
batch can re-run the ~15-min E2E matrix several times while splitting to find the culprit.
On a slow/wedge-prone matrix that is a genuine runner-minute cost. Mitigate with a small
`batch_size` cap, `batch_max_wait_time`, `checks_timeout`, and modest `max_parallel_checks`.
- **Batching hides which PR broke it (until bisection finishes).** A red batch doesn't
immediately name the culprit; Mergify bisects automatically, but that costs runs + latency.
Best for **test-only / low-risk** batches (the #373 case); route risky feature PRs through a
serial path (batch_size 1) or lower priority so they validate alone.
- **Config complexity + a new failure mode.** `.mergify.yml` is another surface. The classic
footgun is a condition that doesn't match the real check name (`CI passed`) — PRs then queue
but never merge. Needs a short trial to shake out.
- **Loss of bespoke control.** The traffic-controller's exact P0-preempt / hold-back semantics
are approximated (not byte-identical) by Mergify priorities + `allow_checks_interruption`.
Close enough in practice, but not the same code.
---
## 11. Recommendation
**Adopt Mergify, phased.**
- **Phase 1 now:** serial queue. Immediate elimination of the manual serial-bump grind and the
whole traffic-controller apparatus, with **zero** change to the require-up-to-date rule or
the merge-commit policy. Low risk, high daily-quality-of-life win.
- **Phase 2 when the maintainer signs off:** batching via `merge-batch`, moving the
up-to-date **guarantee** into the queue (checkbox off, invariant preserved, merges locked to
Mergify). This is the only remaining path to the #373-scale throughput win (native queue is
unavailable) and it keeps the merge-commit policy. Gate it on the explicit trilemma decision
and a throwaway-branch trial that verifies (a) `merge-batch` really lands merge commits and
(b) no stale PR can merge outside the queue.
If the literal checkbox is truly immovable *and* linear history is unacceptable, **stop at
Phase 1** — that is still a large win over today.
---
## Sources
- [Mergify — Merge Queue](https://docs.mergify.com/merge-queue/)
- [Mergify — Merge Queue Batches](https://docs.mergify.com/merge-queue/batches/)
- [Mergify — Parallel Checks](https://docs.mergify.com/merge-queue/parallel-checks/)
- [Mergify — Using Queue Rules](https://docs.mergify.com/merge-queue/rules/)
- [Mergify — Using Priorities](https://docs.mergify.com/merge-queue/priority/)
- [Mergify — Merge Strategies](https://docs.mergify.com/merge-queue/merge-strategies/)
- [Mergify — Configuration File Format](https://docs.mergify.com/configuration/file-format/)
- [Mergify — queue action](https://docs.mergify.com/workflow/actions/queue/)
- [Mergify — Merge Queue setup](https://docs.mergify.com/merge-queue/setup/)
- [Mergify changelog — new `merge-batch` merge method](https://docs.mergify.com/changelog/2026-04-07-new-merge-batch-merge-method-for-merge-queue/)
- [Mergify discussion #5138 — Merge Queue and branch up-to-date protection](https://github.com/Mergifyio/mergify/discussions/5138)
- [Mergify pricing](https://mergify.com/pricing) · [Mergify on GitHub Marketplace](https://github.com/marketplace/mergify)
- Repo: `.github/workflows/ci.yml`, `autoupdate.yml`, `ci-trigger.yml`, `traffic-control.yml`, `.github/scripts/traffic_control.py`; issue #349 + the `ci-merge-cascade` note.
+116
View File
@@ -0,0 +1,116 @@
# SPDX-License-Identifier: GPL-3.0-or-later
#
# ============================================================================
# PROPOSED — NOT ACTIVE. This is docs/ci/mergify.yml.proposed, a review
# artifact for issue #407. It is deliberately NOT committed as a live
# `.mergify.yml` at the repo root, so Mergify does NOTHING until a maintainer
# (a) installs the Mergify GitHub App and (b) moves/renames this file to
# `.mergify.yml`. Read docs/ci/mergify-integration-spec.md first — especially
# the require-up-to-date trilemma — before adopting.
# ============================================================================
#
# Schema: https://docs.mergify.com/configuration/file-format/ (validated against
# https://docs.mergify.com/mergify-configuration-schema.json on push once live).
# ---------------------------------------------------------------------------
# queue_rules — how a queued PR is validated and merged.
# ---------------------------------------------------------------------------
queue_rules:
- name: default
# ---- Final merge gate --------------------------------------------------
# Merge only when the SINGLE aggregating required context is green — the exact
# same check branch protection already requires — plus review + not-draft.
# "CI passed" is the name: field of the `ci-passed` job in .github/workflows/ci.yml.
merge_conditions:
- check-success = CI passed
- "#approved-reviews-by >= 1"
- -draft
- label != broken
# ---- Speculative validation on top of the latest main ------------------
# Mergify builds a temporary batch branch = (latest main + the next PRs) and runs
# CI on THAT, so what merges was tested against an up-to-date base. This is the
# up-to-date GUARANTEE, enforced by the queue instead of by a stale-branch re-push.
#
# PHASE 1 (safe, keeps GitHub "require branches up to date" literally ON):
# set batch_size: 1 and merge_method: merge below, and comment out the Phase-2
# block. Serial queue = automates the manual bumping, NO batching throughput win.
#
# PHASE 2 (the throughput win — REQUIRES the maintainer decision in the spec,
# §"The require-up-to-date trilemma"): batch N PRs into ONE CI run. Batching is
# incompatible with GitHub's literal "require branches up to date" checkbox
# (Mergify docs), so that checkbox must be turned OFF and its guarantee moved
# into this queue — OR use queue_branch_merge_method: fast-forward to keep the
# checkbox at the cost of linear history (breaks the merge-commit policy).
batch_size: # dynamic: stays at min when runners are free, grows toward max on a backlog
min: 1
max: 5 # cap for the test-only batches (e.g. the #373 Robolectric epic: ~6 PRs -> ~1 run)
batch_max_wait_time: 5 min # how long to accumulate a partial batch before validating it
# One MERGE COMMIT per batch via the GitHub merge API — preserves the repo's
# merge-commit policy (never squash/rebase) at the batch granularity.
merge_method: merge-batch
# Slow + wedge-prone E2E matrix: bound the wait so a wedged emulator leg times the
# batch out (then Mergify bisects/splits it) instead of blocking the queue forever.
checks_timeout: 45 min
# How many speculative batches validate in parallel. This is the Mergify equivalent
# of the traffic-controller's MAX_INFLIGHT_RUNS (was 2). Each parallel check fans out
# to the full E2E matrix, so keep it modest against the runner budget.
# (Set under `merge_queue:` — see below — in current schema; kept here for context.)
# max_parallel_checks lives under the top-level merge_queue options.
merge_queue:
max_parallel_checks: 2 # == old MAX_INFLIGHT_RUNS; raise for throughput vs runner cost
# ---------------------------------------------------------------------------
# priority_rules — map the repo's P0–P9 labels onto queue priority.
# Higher number merges first (Mergify: low=1000, medium=2000, high=3000; range 1–10000).
# Unlabelled PRs get Mergify's default `medium` (2000) — equivalent to the old P5 default
# in traffic_control.py. P0 is emergency-only and outranks everything.
# ---------------------------------------------------------------------------
priority_rules:
- name: p0-emergency
conditions: [label = P0]
priority: 10000
- name: p1
conditions: [label = P1]
priority: 9000
- name: p2
conditions: [label = P2]
priority: 8000
- name: p3
conditions: [label = P3]
priority: 7000
- name: p4
conditions: [label = P4]
priority: 6000
- name: p5
conditions: [label = P5]
priority: 5000
- name: p6
conditions: [label = P6]
priority: 4000
- name: p7
conditions: [label = P7]
priority: 3000
- name: p8
conditions: [label = P8]
priority: 2000
- name: p9
conditions: [label = P9]
priority: 1000
# ---------------------------------------------------------------------------
# pull_request_rules — WHICH PRs enter the queue (the `queue` action).
# A maintainer opts a PR in by getting it green + approved (or by comment
# `@mergifyio queue`). `broken`/`draft` PRs are never queued.
# ---------------------------------------------------------------------------
pull_request_rules:
- name: Queue green, approved, non-draft PRs targeting main
conditions:
- base = main
- -draft
- label != broken
- "#approved-reviews-by >= 1"
- check-success = CI passed
actions:
queue:
name: default