mudler/moss-transcribe.cpp-gguf

MOSS-Transcribe-Diarize GGUF (for moss-transcribe.cpp)

GGUF weights for moss-transcribe.cpp, a from-scratch C++/ggml inference port of OpenMOSS MOSS-Transcribe-Diarize. Joint long-form transcription, speaker diarization, and timestamps in one pass, on CPU (and on GPU through ggml backends as those land), with no Python, PyTorch, or CUDA toolkit at inference. Each file is fully self-contained (weights, tokenizer, mel filterbank, and all config live inside the GGUF).

Brought to you by the LocalAI team.

Variants

Every file was verified end-to-end against the reference on the JFK sample (CPU, greedy, 8 threads). "Transcript" is versus the original PyTorch model; "speed" is total wall time on an 11 s clip on a 20-core x86 CPU at 8 threads (the whole model, load included; the autoregressive decode is memory-bandwidth bound, so smaller weights run faster).

file	size	vs f32	wall (11 s)	speed vs f32	transcript vs reference
moss-transcribe-f16.gguf	1.8 GB	50%	4.96 s	1.6x	byte-identical
moss-transcribe-q8_0.gguf	942 MB	27%	3.97 s	2.0x	byte-identical
moss-transcribe-q6_k.gguf	733 MB	21%	4.16 s	1.9x	byte-identical
moss-transcribe-q5_k.gguf	619 MB	18%	4.47 s	1.8x	byte-identical
moss-transcribe-q5_0.gguf	619 MB	18%	3.81 s	2.1x	byte-identical
moss-transcribe-q4_k.gguf	511 MB	15%	3.81 s	2.1x	word-identical (one timestamp off 0.02 s)
moss-transcribe-q4_0.gguf	511 MB	15%	3.57 s	2.2x	word-identical (one timestamp off 0.07 s)

Which to pick: q5_k or q5_0 for the best size and accuracy (byte-identical to the reference at about one sixth the size); q4_k/q4_0 for the smallest and fastest (word-identical); q8_0 for the largest fidelity margin; f16 if you want the near-lossless full-precision equivalent. The F32 GGUF (3.4 GB, the parity reference) is not published here; produce it with the converter if you need it.

Only the large ggml_mul_mat-fed weights (Qwen3 and Whisper attention/FFN projections, the adaptor linears, and the token embedding, 343 tensors) are quantized; norms, biases, the conv stem, positional embeddings, and the mel filterbank stay F32.

Benchmarks

Same audio, same F32 weights, same threads, byte-identical transcript. moss-transcribe.cpp (ggml, CPU) stays under real time where PyTorch does not, and the gap holds as clips get longer:

Quantization makes the model both smaller and faster (the decode is memory-bandwidth bound), with the transcript byte-identical through q5:

Full methodology and the reproducible harness are in the benchmarks.

Usage

git clone --recursive https://github.com/mudler/moss-transcribe.cpp
cd moss-transcribe.cpp
cmake -B build && cmake --build build -j

hf download mudler/moss-transcribe.cpp-gguf moss-transcribe-q5_k.gguf --local-dir .
./build/moss-transcribe transcribe moss-transcribe-q5_k.gguf audio.wav

Output is the compact [start][Sxx]text[end] transcript with inline speaker tags and timestamps, for example:

[0.28][S01] And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.[10.59]

Set MTD_THREADS to tune CPU threads (8 is a good default on a 20-core box; the decode is bandwidth bound, so fewer busy threads often beat more).

For production serving

Use LocalAI for an OpenAI-compatible /v1/audio/transcriptions endpoint, a model gallery, concurrency, auth, and metrics.

Model

The MOSS-Transcribe-Diarize model is by the OpenMOSS / MOSI.AI team (arXiv:2601.01554), released under Apache-2.0. moss-transcribe.cpp is MIT-licensed; these GGUF weights keep the model's Apache-2.0 license.

Citation

@software{moss_transcribe_cpp,
  title  = {moss-transcribe.cpp: a C++/ggml inference engine for MOSS-Transcribe-Diarize},
  author = {Di Giacinto, Ettore},
  url    = {https://github.com/mudler/moss-transcribe.cpp},
  year   = {2026}
}

---

Built by the LocalAI team. If you want to run speech transcription and diarization (and LLMs, vision, voice, image, and video models) locally on any hardware with an OpenAI-compatible API, give LocalAI a star.