Media3 + FFmpeg conversion pipeline #1

Merged
JMR-dev merged 26 commits from feat/media3-conversion-pipeline into main 2026-08-22 02:44:12 +00:00
JMR-dev commented 2026-08-20 18:25:03 +00:00 (Migrated from github.com)

Why

main is a bare Android Studio scaffold: one commit, no application code, no activity in the manifest, and a com.example package. This turns it into a working media converter.

The shape of the app is driven by one constraint that isn't obvious up front: Android has no GPU video codec path. There is fixed-function codec silicon reached through MediaCodec, GPU shader cores usable only for filters, and CPU. FFmpeg's -hwaccel is meaningful on Android only as mediacodec, and Vulkan Video — real in FFmpeg 8.0+ — is exposed by no shipping Android driver.

So "hardware accelerated" has to mean MediaCodec, and that is what Media3 Transformer packages. But Media3 cannot mux Matroska, cannot encode MP3 (Android has no MP3 encoder at any API level — a platform gap, not a Media3 one), and has no GIF muxer. Hence two engines and an explicit router rather than a wrapper around either one alone.

A second constraint shaped the supply chain: arthenica/ffmpeg-kit is archived and its binaries were deleted from Maven Central. Every com.arthenica:ffmpeg-kit-* coordinate 404s, and its successor is source-only. Building FFmpeg ourselves is the only remaining option, not a preference.

What

Two engines behind a router. Media3 Transformer takes the hardware path; FFmpeg takes everything it structurally cannot do — MKV, MP3, GIF, PNG frames, inputs with no platform decoder, and the CRF quality tier. The routing reason is shown in the UI rather than hidden, so a slow job explains itself, and there is a per-job engine override.

Quality tiers, because the licence choice and the engine choice are the same decision. Fast is hardware and bitrate-targeted; Best is x264/x265 with CRF. That tier is the entire reason the shipped binary is GPL-3.0 — no Android hardware encoder exposes CRF or two-pass.

Durable background work. WorkManager with a foreground service, covering three regimes across the supported range: no service type below API 34, dataSync at 34, mediaProcessing from 35. A budget timeout is modelled as a pause, not an error — the work is still valid, there is simply no budget.

Joining picks stream-copy or re-encode by inspecting inputs, because the concat demuxer does not reliably reject mismatched files: it can emit output whose later segments are garbled.

FFmpeg build as a container recipe, producing a 16 KB-aligned GPL archive. The archive is committed under bin/ so test runs do not depend on a 40-minute cross-compile succeeding.

Licensing is documented rather than assumed: source is MIT, the distributed APK is GPL-3.0 because of x264/x265. Notably libass is ISC, so subtitle burn-in is not what forces GPL.

How it was verified

66 unit tests covering the routing matrix, the FFmpeg argument builder, the concat planner and the retry rule — run against fabricated device profiles so branches like "this device cannot encode HEVC" are reachable regardless of the test machine.

40 instrumented tests, passing on:

Target Result
Pixel 10 Pro XL (API 37, real hardware) 40 pass
Emulators API 33, 34, 35, 36 40 pass each

The 2 skips on every run are an opt-in benchmark that needs uncommitted media.

Every failure branch in the conversion and join paths is forced by a test. That needed a seam — the worker previously constructed its engines directly, so nothing could make them fail. The foreground-service timeout is the exception: its trigger is a six-hour budget no test can reach, so the rule was extracted and verified across every WorkInfo stop reason instead.

Real footage found two bugs that synthetic fixtures could not. A 4:4:4 H.264 file (avc1.F4001F) cannot be decoded by Media3 on any device — hardware AVC decoders implement High 4:2:0 and Android's software decoder does not cover 4:4:4 either. Following it through exposed a missing -pix_fmt on most encode paths, and that FFmpeg's *_mediacodec wrappers fail on real input. Those wrappers are no longer used at all: Media3 already does hardware encode properly, and a job only reaches FFmpeg because Media3 could not handle it. Both findings are now regression tests, one of which ships its own 4:4:4 fixture.

Measured on the Pixel: AV1 1080p through the hardware path runs at 7.9× realtime, confirming the figure the two-engine design was based on. Software x264 CRF at 720p runs at 4.4×.

Release build verified, not just debug: R8 enabled, all 22 native libraries surviving minification and still 16 KB aligned in the APK. That matters — R8 caught a latent crash that debug builds tolerated through lazy class loading.

Known gaps

  • The foreground-service timeout path is reasoned, not exercised.
  • status_check.yml has not run yet; CI behaviour is only truly known once this PR triggers it.
  • F-Droid submission would need a scandelete entry for bin/, since its scanner flags checked-in native libraries. The from-source recipe in tools/ffmpeg/ remains the authority.
  • The repository grows from ~1 MB to ~35 MB, and future rebuilds of the archive add further permanent blobs.

🤖 Generated with Claude Code

## Why `main` is a bare Android Studio scaffold: one commit, no application code, no activity in the manifest, and a `com.example` package. This turns it into a working media converter. The shape of the app is driven by one constraint that isn't obvious up front: **Android has no GPU video codec path.** There is fixed-function codec silicon reached through `MediaCodec`, GPU shader cores usable only for filters, and CPU. FFmpeg's `-hwaccel` is meaningful on Android only as `mediacodec`, and Vulkan Video — real in FFmpeg 8.0+ — is exposed by no shipping Android driver. So "hardware accelerated" has to mean MediaCodec, and that is what Media3 Transformer packages. But Media3 cannot mux Matroska, cannot encode MP3 (Android has **no** MP3 encoder at any API level — a platform gap, not a Media3 one), and has no GIF muxer. Hence two engines and an explicit router rather than a wrapper around either one alone. A second constraint shaped the supply chain: **`arthenica/ffmpeg-kit` is archived and its binaries were deleted from Maven Central.** Every `com.arthenica:ffmpeg-kit-*` coordinate 404s, and its successor is source-only. Building FFmpeg ourselves is the only remaining option, not a preference. ## What **Two engines behind a router.** Media3 Transformer takes the hardware path; FFmpeg takes everything it structurally cannot do — MKV, MP3, GIF, PNG frames, inputs with no platform decoder, and the CRF quality tier. The routing reason is shown in the UI rather than hidden, so a slow job explains itself, and there is a per-job engine override. **Quality tiers**, because the licence choice and the engine choice are the same decision. *Fast* is hardware and bitrate-targeted; *Best* is x264/x265 with CRF. That tier is the entire reason the shipped binary is GPL-3.0 — no Android hardware encoder exposes CRF or two-pass. **Durable background work.** WorkManager with a foreground service, covering three regimes across the supported range: no service type below API 34, `dataSync` at 34, `mediaProcessing` from 35. A budget timeout is modelled as a pause, not an error — the work is still valid, there is simply no budget. **Joining** picks stream-copy or re-encode by inspecting inputs, because the `concat` demuxer does not reliably reject mismatched files: it can emit output whose later segments are garbled. **FFmpeg build** as a container recipe, producing a 16 KB-aligned GPL archive. The archive is committed under `bin/` so test runs do not depend on a 40-minute cross-compile succeeding. **Licensing** is documented rather than assumed: source is MIT, the distributed APK is GPL-3.0 because of x264/x265. Notably `libass` is ISC, so subtitle burn-in is *not* what forces GPL. ## How it was verified **66 unit tests** covering the routing matrix, the FFmpeg argument builder, the concat planner and the retry rule — run against fabricated device profiles so branches like "this device cannot encode HEVC" are reachable regardless of the test machine. **40 instrumented tests**, passing on: | Target | Result | |---|---| | Pixel 10 Pro XL (API 37, real hardware) | 40 pass | | Emulators API 33, 34, 35, 36 | 40 pass each | The 2 skips on every run are an opt-in benchmark that needs uncommitted media. **Every failure branch in the conversion and join paths is forced by a test.** That needed a seam — the worker previously constructed its engines directly, so nothing could make them fail. The foreground-service timeout is the exception: its trigger is a six-hour budget no test can reach, so the *rule* was extracted and verified across every `WorkInfo` stop reason instead. **Real footage found two bugs that synthetic fixtures could not.** A 4:4:4 H.264 file (`avc1.F4001F`) cannot be decoded by Media3 on any device — hardware AVC decoders implement High 4:2:0 and Android's software decoder does not cover 4:4:4 either. Following it through exposed a missing `-pix_fmt` on most encode paths, and that FFmpeg's `*_mediacodec` wrappers fail on real input. Those wrappers are no longer used at all: Media3 already does hardware encode properly, and a job only reaches FFmpeg because Media3 could not handle it. Both findings are now regression tests, one of which ships its own 4:4:4 fixture. **Measured on the Pixel:** AV1 1080p through the hardware path runs at **7.9× realtime**, confirming the figure the two-engine design was based on. Software x264 CRF at 720p runs at 4.4×. **Release build verified**, not just debug: R8 enabled, all 22 native libraries surviving minification and still 16 KB aligned in the APK. That matters — R8 caught a latent crash that debug builds tolerated through lazy class loading. ### Known gaps - The foreground-service timeout path is reasoned, not exercised. - `status_check.yml` has not run yet; CI behaviour is only truly known once this PR triggers it. - F-Droid submission would need a `scandelete` entry for `bin/`, since its scanner flags checked-in native libraries. The from-source recipe in `tools/ffmpeg/` remains the authority. - The repository grows from ~1 MB to ~35 MB, and future rebuilds of the archive add further permanent blobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.