docs: add a README covering resume, retention and hardware routing
Explains the parts that are non-obvious from the code: why artifacts rather than a stored stage marker are the source of truth, why a .part without its aria2 control file is discarded, which aria2c flags actually change behaviour versus which are already yt-dlp defaults, and why NVIDIA and AMD ship as interfaces rather than implementations. README-intel.md keeps the hardware setup; this covers the pipeline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,151 @@
|
||||
# ccn-transcribe
|
||||
|
||||
Download a video, extract its audio losslessly as FLAC, and transcribe it with
|
||||
Whisper on whatever GPU the machine actually has.
|
||||
|
||||
Resumable from any completed stage, with errors that tell you what to do next.
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
uv run ccn-transcribe doctor # check the environment first
|
||||
uv run ccn-transcribe run 'https://youtu.be/...'
|
||||
uv run ccn-transcribe run -i urls.txt --formats txt,srt
|
||||
```
|
||||
|
||||
Hardware setup for the Intel path is documented separately in
|
||||
[README-intel.md](README-intel.md).
|
||||
|
||||
## What it does
|
||||
|
||||
```
|
||||
yt-dlp (aria2c) -> ffmpeg -> Whisper -> txt / srt / vtt / json
|
||||
video.mkv audio.flac OpenVINO GenAI
|
||||
```
|
||||
|
||||
Each video becomes one job directory under `--workdir` (default `transcripts/`):
|
||||
|
||||
```
|
||||
transcripts/jobs/youtube-<id>/
|
||||
state.json media/video.mkv out/transcript.{txt,srt,vtt,json}
|
||||
events.jsonl media/audio.flac logs/{job,aria2}.log
|
||||
```
|
||||
|
||||
## Resume
|
||||
|
||||
**Artifacts on disk are the source of truth.** `state.json` records only what the
|
||||
filesystem cannot answer — chiefly *why* a file is absent. There is deliberately
|
||||
no stored "current stage": persisting one is how "marked done but the file is
|
||||
gone" bugs happen. Every run verifies what is actually present (`ffprobe`
|
||||
duration > 0, non-empty, audio length matching the video) and enters at the first
|
||||
stage whose output does not check out.
|
||||
|
||||
| Transcript | FLAC | Video | Runs |
|
||||
|---|---|---|---|
|
||||
| missing | ok | anything | transcribe, outputs |
|
||||
| missing | missing | ok | extract, transcribe, outputs |
|
||||
| missing | missing | missing | everything |
|
||||
| ok | — | — | nothing |
|
||||
|
||||
Killing a run mid-download resumes rather than restarting, because yt-dlp keeps
|
||||
its `.part` and aria2 keeps its control file. A `.part` **without** that control
|
||||
file is treated as unusable and restarted: aria2 writes segments out of order, so
|
||||
such a file is sparse with holes rather than a valid prefix.
|
||||
|
||||
Deleting one subtitle file re-renders it from the segments saved in
|
||||
`transcript.json` instead of re-transcribing the whole recording.
|
||||
|
||||
`--dry-run` prints the plan for every job without touching anything.
|
||||
|
||||
## Keeping or discarding media
|
||||
|
||||
Video and audio are kept by default.
|
||||
|
||||
| Flag | Effect |
|
||||
|---|---|
|
||||
| `--no-retain-video` | delete the video once the FLAC is extracted and verified |
|
||||
| `--no-retain-audio` | delete the FLAC once every output is written |
|
||||
| `--no-retain` | both |
|
||||
|
||||
Deletion always happens **after** the next stage is committed, so interrupting a
|
||||
run is still safe and resume still works. What you actually give up is re-run
|
||||
flexibility: re-transcribing later with a different `--language`, `--model` or
|
||||
`--task` will need another download.
|
||||
|
||||
These flags prompt once per run. On a non-TTY they refuse outright rather than
|
||||
proceeding — a piped or cron invocation should never delete gigabytes with nobody
|
||||
seeing the warning. Pass `--yes` to confirm non-interactively.
|
||||
|
||||
## Hardware routing
|
||||
|
||||
Detection runs in two phases: a cheap hardware probe that imports nothing, then a
|
||||
toolchain probe that may import. The toolchain probe is only reached once the
|
||||
hardware is confirmed, which is what keeps torch off a machine with no NVIDIA or
|
||||
AMD GPU.
|
||||
|
||||
Order is **NVIDIA → AMD → Intel GPU → CPU**, and `--device` overrides it.
|
||||
|
||||
Fallback happens at runtime, not at detection: constructing the backend and its
|
||||
first `generate()` are both guarded, because a `/dev/dri` permission failure and
|
||||
an OpenCL JIT failure surface there rather than at device enumeration. A backend
|
||||
that fails is demoted for the rest of the process, so a long batch does not retry
|
||||
it once per job. An explicit `--device` never falls back silently — getting the
|
||||
CPU when you asked for the GPU would make any measurement a lie.
|
||||
|
||||
**NVIDIA and AMD are interface-only in this build.** Detection and routing are
|
||||
real and the error says what to implement, but there is no inference code: the
|
||||
cached model is OpenVINO IR, which cannot load on CUDA or ROCm, and neither path
|
||||
could be tested here. CTranslate2 has no ROCm support, so they are genuinely two
|
||||
implementations rather than one.
|
||||
|
||||
## Download speed
|
||||
|
||||
yt-dlp already passes aria2c `-x16 -j16 -s16 --min-split-size 1M`, and `-x16` is
|
||||
aria2's per-server cap, so re-sending connection counts changes nothing. What
|
||||
this adds is the flags that do:
|
||||
|
||||
| Flag | aria2 default | Why |
|
||||
|---|---|---|
|
||||
| `--retry-wait=3` | 0 | retrying with no delay burns all five tries in under a second |
|
||||
| `--timeout=30` / `--connect-timeout=15` | 60 | a dead connection otherwise holds a slot for a full minute |
|
||||
| `--lowest-speed-limit=50K` | 0 (off) | without it a wedged connection hangs forever |
|
||||
| `--disk-cache=64M` | 16M | sixteen writers at out-of-order offsets thrash a small cache |
|
||||
| `--file-allocation=falloc` | none | instant on btrfs/ext4/xfs; avoids fragmenting multi-GB files |
|
||||
| `--log=logs/aria2.log` | none | the console level hides everything, leaving no forensics |
|
||||
|
||||
`--continue` is deliberately never passed — see the `.part` note above.
|
||||
|
||||
aria2c only handles `http/https/ftp/ftps`, so HLS and DASH fall through to
|
||||
yt-dlp's native downloader; `--concurrent-fragments` is the knob there. A missing
|
||||
aria2c downgrades with a warning rather than failing.
|
||||
|
||||
Because yt-dlp calls an external downloader exactly once and does not retry it,
|
||||
the retry loop lives here instead.
|
||||
|
||||
## Options worth knowing
|
||||
|
||||
`--formats txt,srt,vtt,json` · `--language en` · `--task translate` ·
|
||||
`--playlist` (a URL carrying `&list=` stays one video unless you ask) ·
|
||||
`--audio-profile whisper` (16 kHz mono FLAC instead of source quality) ·
|
||||
`--force` / `--force-stage` · `--cookies-from-browser firefox` ·
|
||||
`--js-runtime` · `-v`
|
||||
|
||||
## Requirements
|
||||
|
||||
`ffmpeg`, `ffprobe`, and optionally `aria2c`. Everything else comes from
|
||||
`uv sync`, including the deno binary YouTube needs for its signature challenges —
|
||||
yt-dlp defaults to deno only, and installing it as a dependency means it works
|
||||
under cron without any PATH setup.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
pre-commit install --hook-type pre-commit --hook-type pre-push --hook-type commit-msg
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
Commits run ruff, pyright and the full suite at 100% coverage; pushes run pylint
|
||||
and mypy. No unit test touches the GPU, the network or a real model, which is
|
||||
what keeps the commit gate fast. Live checks sit behind `CCN_LIVE=1`.
|
||||
|
||||
Conventional commits are enforced.
|
||||
Reference in New Issue
Block a user