Fish Audio and the Case for Open Voice Models

For a long time the weakest link in a voice agent was the voice. Text to speech was flat, oddly paced, and gave the game away in the first sentence. Callers heard a machine and behaved accordingly.

That has stopped being true. Fish Audio's S1 model currently sits at the top of TTS-Arena2, the leaderboard most people cite for speech quality, and its successor S2 leads on word error rate in both English and Chinese. S1 was trained on more than 700,000 hours of multilingual audio, which shows up as prosody: it puts stress and pauses roughly where a person would, which is the thing that actually sells a synthetic voice.

Voice cloning has come down to about 15 seconds of reference audio, and a cloned voice carries across 30-plus languages. You can have a voice speak a language its owner does not.

The more interesting development is not quality, it is that Fish Audio publishes an open model. S1-mini can be self-hosted with no per-character API cost, and the flagship models are distributed through Hugging Face.

Voice is unusually sensitive to this, for two reasons. The first is unit economics: a voice agent that handles thousands of minutes a month pays for every one of them, and TTS is billed per character or per second. At low volume the API is obviously right. At high volume, a self-hosted model changes the shape of the cost from variable to fixed.

The second is latency. In conversation people expect a reply within roughly half a second, and every network round trip spends from that budget. A model running close to your own infrastructure removes a hop you cannot otherwise optimise away.

Self-hosting is not free, it just moves the cost. You take on GPU capacity, deployment, monitoring, and the work of keeping up with model releases. An API vendor does all of that and charges you for it, which is often a good deal.

So the decision is volume and control, not quality. Below a few thousand minutes a month, a hosted API is almost certainly cheaper than the engineering time to run your own. Above that, the arithmetic starts to flip, and if you have data-residency constraints it may flip regardless of volume.

Cloning raises a separate question that is not technical. A 15-second sample is a low bar, and using someone's voice needs their consent. We treat that as a hard requirement on client work rather than a preference.

Most voice agents we ship start on a hosted API, because the volume does not justify anything else and getting the conversation design right matters far more than the last few milliseconds. The voice is rarely what makes an agent fail.

When a system grows into real call volume, moving TTS to a self-hosted open model is one of the cleaner cost reductions available, precisely because it changes nothing the caller hears. That is a better lever than most, and it only exists because open models caught up.