Audio8-TTS: High-Fidelity Zero-Shot Speech Synthesis

Two Clocks for One Voice

Generating a single sentence runs on two clocks at once. The slow clock follows the text: it decides where one frame ends and the next begins, and keeps the sentence coherent across its full semantic arc. The fast clock fills each frame with timbre, articulation and spectral detail — precisely the details that make a voice unmistakably belong to a particular speaker. Running one sequence model at two rates simultaneously is certainly possible, but the sequence becomes needlessly long, and structure and fidelity are forced to compete for the same representation.

Audio8 TTS decouples these two jobs with a DualAR architecture inspired by Fish Audio S2 Pro. Slow AR advances one semantic token per codec frame; Fast AR reads the Slow AR hidden states plus the codebooks already generated within the current frame, then renders that frame's acoustic codes. The neural audio codec shipped with the model encodes the optional reference recording and, at the end of the pipeline, decodes the generated codes into a 44.1 kHz waveform.

Multi-Layer RVQ: More Than One Token per Frame

The codec produces every frame through 10 residual codebooks, each a table of 4,096 entries. The first few codebooks lay down the rough skeleton of the frame; later codebooks add residuals layer by layer, sharpening articulation, timbre and finer acoustic strata. Fast AR predicts this stack of codebooks along the depth axis, so the model can spend capacity on detail instead of making the semantic stream carry the entire waveform alone.

This also explains why the two autoregressive branches fit together so naturally. Slow AR advances at roughly 21.5 frames per second, unrolling a stable sequence that long-range context can depend on; Fast AR then completes a much shorter within-frame decode against that backdrop. The whole pipeline remains end-to-end autoregressive, but the token budget is finally allocated along the structure of speech, instead of flattening every layer of detail into a single sequence.

Audio8-TTS-0.6B (Preview)

The main model contains 601,159,424 parameters, not counting the neural audio codec. Both branches share a width of 896, but the depth is deliberately asymmetric: a 24-layer Slow AR has room to carry speech content across time, while a 4-layer Fast AR focuses on the ten codebooks inside each frame.

At 44.1 kHz, 2,048 samples correspond to roughly 46.4 ms. Per second of audio, Slow AR makes about 21.5 semantic decisions, and Fast AR fills each decision with ten layers of residual information. A static KV cache keeps earlier positions available at all times, so the model never recomputes the whole prefix at every audio frame.

The codec ships with the model and does double duty: the same component encodes the reference timbre and decodes the generated codes. The entire inference path therefore manages just this one component, with no separate audio checkpoint to maintain alongside the pipeline.

The 170M Branch

Audio8-TTS-0.1B (Preview) keeps the DualAR contract but replaces the Slow AR backbone. Instead of a pure Transformer it adopts Falcon-H1, combining attention with Mamba 2 state-space layers. This single architectural change is what lets the main model shrink to 170M parameters while preserving the semantic-to-acoustic handoff and continuing to emit the same ten codebooks.

170M describes only the main generative model. The audio decoder is a separate component of roughly 120M parameters; generating an actual waveform still counts it against storage and memory budgets. Keeping the two numbers separate keeps scale comparisons honest, rather than pretending the codec costs nothing.

Falcon-H1 also reshapes the runtime state. Each request carries its own FalconHybridMambaAttentionDynamicCache for Slow AR; semantic sampling, Fast AR and audio decoding remain shared with the 0.6B path. In GPU serving, Mamba state is passed along the eager path instead of the usual SGLang KV pages.

Letting the Training Objective Mirror the Architecture

The split lives not only in the inference graph but throughout training. Audio is first encoded into a [10, T] array of codebook indices; supervised fine-tuning then optimizes the Slow AR semantic/EOS objective and the Fast AR codebook objective jointly under teacher forcing. When only one side of the generation pipeline needs to change, the other can stay frozen.

The data interface stays deliberately small. Target audio is required; reference_audio and reference_text may be omitted, but whenever they are used they must be provided as a pair. At inference time, text and reference audio share one packed 2,048-position context. The repository recommends keeping a single input to roughly 150 characters and splitting longer content at full semantic boundaries.

Zero-Shot Timbre Cloning: No Speaker-Specific Fine-Tuning

Timbre cloning is only useful if the model can tell apart “who is speaking” from “what is said”. Audio8 TTS never fine-tunes per new speaker: it encodes a short recording together with its exact transcript, then uses that pair as the condition for new text.

inputs = processor(
    text=["Welcome to Audio8 TTS."],
    reference_audio=["reference.wav"],
    reference_text=["The exact transcript of the reference recording."],
    return_tensors="pt",
)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.8,
        top_p=0.95,
        top_k=50,
        do_sample=True,
        return_dict_in_generate=True,
    )
    waveforms, lengths = model.decode_audio(output.codes)

The reference transcript is part of the condition, not a note attached to the recording after the fact. It aligns what was said with the reference waveform, leaving the model a cleaner speaker-identity signal. Clean recordings and accurate transcripts usually produce the most stable results; noise, over-long clips and wrong transcriptions all degrade stability and timbre similarity.

The reference condition is optional. Omit both reference fields and the same checkpoint becomes an ordinary text-to-speech model.

Language Changes, Voice Doesn't

Audio8-TTS-0.6B (Preview) covers Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, Cantonese and Spanish, with more languages continuing to be added. The harder test is cross-lingual synthesis: a Chinese reference recording can guide an English utterance, and an English reference can guide Chinese. The language of the text changes; the voice anchor never moves.

The 0.1B Preview draws a narrower boundary. Chinese and English are the primary languages; German, Spanish, French, Italian, Japanese and Korean are experimentally supported. Preview is a boundary marker, not decoration: a language appearing in the tokenizer or an evaluation table does not guarantee consistent quality across accents, dialects and texts.

SOTA-Class Performance at a Compact Scale

In the comparisons below, Audio8-TTS-0.6B (Preview) is the smallest main model in the table. On Seed-TTS it achieves the lowest English WER listed and keeps Chinese CER competitive; on CV3 it remains in the top tier of public multilingual evaluations. We describe this position as SOTA-class without claiming universal leadership on every metric or in every language.

Seed-TTS

Lower is better for WER/CER; higher is better for SIM. SIM is reported as a percentage.

CV3 Multilingual Error Rates

Lower is better for all metrics below.

The 170M model is not a same-quality substitute for the 0.6B. Its public Seed-TTS results are 1.662 WER / 56.7 SIM for English and 1.13 CER / 68.2 SIM for Chinese; on CV3 it scores 3.619 in Chinese and 3.307 in English, with wider gaps in some experimental languages. Its value lies elsewhere: keeping zero-shot conditioning usable under a dramatically smaller deployment budget.

All cross-project numbers come from public model cards and technical reports. Normalization, evaluators and test sets differ, so these figures should be read as reference comparisons, not as a strictly controlled unified ranking.

From Checkpoint to a Runnable Serving Stack

Parameter counts only mean something when a runtime can carry them. Audio8 TTS therefore ships two deployment paths with different emphases: the 0.6B pursues GPU throughput, while the 170M comes with a self-contained CPU runtime.

GPU Serving for the 0.6B

The Audio8 repository provides an SGLang Omni adapter with paged attention, dynamic batching, reference-audio encoding, waveform decoding, and an OpenAI-compatible /v1/audio/speech endpoint.

The adapter is validated against SGLang Omni 0.1.0 (commit 68a572348837f7b004857b4b07993c20ade4c017), SGLang 0.5.8, PyTorch 2.9.1+cu128, Transformers 4.57.1 and BF16 weights. In warm single-stream tests on a single NVIDIA H20, generating 128 frames takes a p50 latency of 0.691 s at an RTF of 0.116. The tests exclude cold start and compile time; these numbers describe the test configuration only, not a universal latency guarantee.

ONNX INT8: A 170M Path Built for Edge Systems

The separately released Audio8-TTS-0.1B ONNX INT8 is more than a quantized checkpoint. It packages the autoregressive loop, audio codec, timbre registration and serving interfaces around ONNX Runtime — once the model is downloaded, the runtime needs no CUDA, PyTorch or Transformers.

The Slow AR graph passes attention and Mamba state explicitly between adjacent tokens. Valid-prefix attention confines compute to the valid prefix. Fast AR follows the same per-token boundary, emitting the codebooks inside each frame. Exposing the loop state explicitly is easier to schedule, inspect and memory-bound on CPU than wrapping the whole loop into one monolithic static graph.

In reported tests, regular inference occupies roughly 0.4 GiB after loading. Test platform: Linux x86_64. Memory footprint varies with ONNX Runtime build, allocator, thread count and hardware. On an 8-thread host, Slow AR takes about 19 ms per token, Fast AR about 8 ms per frame, and the one-time prefill costs roughly 19 ms per prompt token. These numbers are useful reference points; production deployments should re-measure on the target CPU.

Moving Timbre Registration off the Hot Path

Ordinary synthesis needs only Slow AR, Fast AR and the audio decoder. The audio encoder is loaded only when registering a new reference timbre, converting the recording and its transcript into reusable timbre codes; once done, the encoder can be unloaded and synthesis resumes. Timbre registration therefore becomes a one-off lifecycle operation rather than a standing cost paid on every request.

The standalone onnx_runtime_0_1b_int8 implementation includes command-line inference, a local HTTP server, streaming PCM, timbre registration and an OpenAI-compatible API. From model files to serving boundary, the deployment chain is fully covered — not just a set of exported compute graphs.

Preview Boundaries and Responsible Use

Both versions are currently in Preview. Language coverage has explicit boundaries, and experimental languages and dialects should be tested on the data and hardware the target application actually uses. INT8 quantization can change the sampled token sequence, so the ONNX model should also be evaluated separately from the original PyTorch checkpoint.

Zero-shot cloning works best with clean reference recordings and accurate transcripts, and requires the speaker's consent. Synthesized speech can be used for impersonation or misleading content; disclose generated audio where appropriate, and evaluate accuracy, safety and legal compliance before production use.