---
title: Only 2 of 15 speech-to-text models caught every repeat | Plainlamp
description: Our AI narrator once repeated 9 seconds of its script, and our pace check passed it. So we tested speech-to-text models on one job: hearing exactly what was said. Only 2 of 15 caught every repeat we planted. What we found, and what we chose.
url: https://plainlamp.com/blog/speech-to-text-repeats
---

31 October 2026, 12 min read

# Only 2 of 15 speech-to-text models caught every repeat

Plainlamp turns a hard PDF, such as a paper, a policy or a standard, into a short narrated video lesson and a guide. The narration is an AI voice, recorded in takes of about half a minute, one for each part of the script.

In October one take went wrong. The voice jumped back and said about 9 seconds of the script a second time. The voice service's own word times did not show it, so our pace check passed the take, and a later step failed twice on a gap no picture could fill.

Since then, a speech recognizer listens to every take, and we compare what it heard with the script, word for word. Extra words mean a repeat. Missing words mean a cut. Until now that recognizer was a Whisper model that we ran ourselves. It found that repeat, but it is slow, and as the results below show, it is not the most accurate. We want every step of Plainlamp to run in the cloud. So we tested which speech-to-text service should do the job.

## What the check needs

Many speech-to-text models are made to write down what a speaker meant. A transcript of a meeting is better without the "um", the false start and the word said twice.

We need the opposite: exactly what was said, slips included. If a model leaves a repeat out, our check cannot see it.

## The test

![Flow: 71 real takes from three approved lessons and one with a real repeat; 52 faults planted in the audio (3 to 10 words said again or cut out, 5 words from elsewhere put in); 12 harmless versions (one word said twice, a pause, quiet added at the end); 15 recognizers listen (12 cloud services and 3 local Whisper models as a yardstick); words compared with the script: caught, let through or a false alarm. Four ran twice with the same verdicts.](https://plainlamp.com/blog/speech-to-text-repeats/stt-4-test-1080.webp)

- **Real takes:** 71 takes from three lessons we had already watched and approved. None of them should raise an alarm. We added the take with the real 9-second repeat.
- **52 faults to catch:** we edited the audio of 8 clean takes. A run of 3 to 10 words is said a second time (a jump back, like the real fault) or cut out, or 5 words from another take are put in.
- **12 harmless versions:** 4 of those takes, each with one word said twice, a pause of about 3 seconds, or about 4 seconds of quiet added at the end. Our check lets these through, so a model must too.
- **15 recognizers:** 12 cloud services, and 3 Whisper models run locally as a yardstick. Eight services ran through OpenRouter, seven of them on endpoints that keep no data (for Gemini, see below). Inworld and Mistral we called directly, under their own terms. We tested one more service, which is not shown: its terms do not allow publishing benchmark results.
- **Scoring:** the words each model heard went through our word check. For this test we made it lenient on spelling: a number written as digits or as words counted as the same, and so did any word of five or more letters spelled within two letters of the script. Our check as it runs today is stricter. With it, Gemini and Inworld score the same, but Nemotron, Parakeet and Scribe v2 raise 1 or 2 false alarms on the approved takes and let fewer harmless versions through.
- **Twice:** four services heard the first 24 planted takes a second time. Each gave the same verdict on every take both times.
- **How the faults were made:** we cut and joined the audio at gaps between words. A repeat jumps back and says the words again; a cut leaves them out. All runs were on 30 October 2026, with the model versions offered that day.

The comparison cost about US$3.70.

## The results

| Recognizer | Faults caught (of 52) | Harmless versions let through (of 12) | False alarms (of 71) | Cost a lesson | Time a lesson |
| --- | --- | --- | --- | --- | --- |
| Gemini 3.5 Transcribe (Google) | 52 | 12 | 0 | US$0.11 | 3.2 min |
| Inworld STT 1 | 52 | 10 | 0 | US$0.05 | 0.7 min |
| Parakeet TDT 0.6B v3 (NVIDIA) | 51 | 12 | 1 | US$0.03 | 1.1 min |
| Scribe v2 (ElevenLabs) | 47 | 12 | 0 | US$0.07 | 1.8 min |
| MAI-Transcribe-2 (Microsoft) | 43 | 12 | 0 | US$0.03 | 1.0 min |
| Voxtral Mini Transcribe 2 (Mistral) | 41 | 12 | 0 | US$0.06 | 2.1 min |
| Whisper large-v3 turbo (cloud) | 40 | 12 | 3 | under US$0.01 | 2.0 min |
| Nemotron 3.5 Transcribe (NVIDIA) | 38 | 12 | 0 | US$0.04 | 1.7 min |
| Whisper large-v3 (cloud) | 36 | 10 | 4 | US$0.01 | 2.5 min |
| Voxtral Mini Transcribe 3 (Mistral) | 31 | 12 | 0 | US$0.07 | 1.2 min |
| Voxtral Small Transcribe 3 (Mistral) | 31 | 12 | 0 | US$0.08 | 2.0 min |
| MAI-Transcribe-1.5 (Microsoft) | 26 | 12 | 0 | US$0.12 | 1.8 min |
| Whisper base.en (local) | 47 | 12 | 1 | – | 1.3 min |
| Whisper small.en (local) | 45 | 12 | 1 | – | 3.2 min |
| Whisper medium.en (local) | 41 | 10 | 3 | – | 8.8 min |

A lesson is about 20 minutes of takes, retakes included. Time is with one request at a time; the local times are on one desktop computer. Costs are what each service reported for this test; for Inworld and Mistral, the billed audio length at their list price.

![Table of the six best of 12 services. Faults caught of 52, harmless versions let through of 12, false alarms on 71 approved takes, cost a lesson: Gemini 3.5 Transcribe 52, 12, 0, US$0.11; Inworld STT 1 (chosen) 52, 10, 0, US$0.05; Parakeet TDT 0.6B v3 51, 12, 1, US$0.03; Scribe v2 47, 12, 0, US$0.07; MAI-Transcribe-2 43, 12, 0, US$0.03; Voxtral Mini Transcribe 2 41, 12, 0, US$0.06.](https://plainlamp.com/blog/speech-to-text-repeats/stt-2-scores-1080.webp)

## Cuts are easy to hear. Repeats are not.

![Two bars for each of 12 speech-to-text services: cuts caught of 24 and repeats caught of 24. Gemini 3.5 Transcribe 24 and 24, Inworld STT 1 24 and 24, Parakeet TDT 0.6B v3 24 and 23, Scribe v2 23 and 21, MAI-Transcribe-2 23 and 17, Whisper large-v3 turbo 22 and 14, Voxtral Mini Transcribe 2 24 and 13, Nemotron 3.5 Transcribe 24 and 10, Whisper large-v3 22 and 10, Voxtral Small Transcribe 3 22 and 8, Voxtral Mini Transcribe 3 24 and 6, MAI-Transcribe-1.5 20 and 4.](https://plainlamp.com/blog/speech-to-text-repeats/stt-1-repeats-1080.webp)

Every model heard nearly every cut: 20 to 24 of 24. Repeats split them. Gemini and Inworld caught all 24, Parakeet 23. Some well-known models caught fewer than half: Whisper large-v3 10, the Voxtral Transcribe 3 models 6 and 8, MAI-Transcribe-1.5 4.

Could that order be chance? Inworld and Gemini caught every repeat. Parakeet missed 1 and Scribe v2 missed 3, few enough to be chance: on repeats, the test cannot tell the top four apart. The other eight each missed 7 to 20 that Inworld caught, too many for chance, though for MAI-Transcribe-2, the closest, that is less sure (see below). But that does not make two clean groups: the test cannot tell Parakeet and Scribe v2 apart from MAI-Transcribe-2, the closest of the eight. More, and where these gaps are less sure, under "How sure we can be", below.

All 15 caught the real repeat of 9 seconds. Short repeats of 3 to 5 words were missed most often, nearly half the time across all the models. But even 10-word repeats were missed about 1 time in 4.

Our guess is that many models drop what sounds like a slip, as a clean transcript would. The test shows which models miss repeats, not why.

Two more things surprised us:

- **Bigger did not help Whisper.** Run locally, the smallest model we tried (base.en) caught 47 faults, and the largest (medium.en) 41.
- **A higher version number did not help.** Mistral's Voxtral Transcribe 3 models caught fewer repeats than Voxtral Mini Transcribe 2, and they give times for whole segments, not for each word.

## What the models heard in the quiet

![Words the models wrote that nobody said. Inworld STT 1, in the quiet at the end of a take (2 of 4 planted takes, and once on a real take): "I'm not sure what you're talking about". Whisper medium.en, run locally, at the end of an approved take: "Thanks for watching and have a great day!". Whisper small.en, on another take: "Thanks for watching."](https://plainlamp.com/blog/speech-to-text-repeats/stt-3-silence-1080.webp)

Some models write words nobody said, most of all in a quiet stretch. At the end of two takes we had approved, our local Whisper models wrote "Thanks for watching and have a great day!" and "Thanks for watching." In our check, that is a false alarm.

Inworld STT 1 did it too. In about 4 seconds of quiet added at the end of a take, it heard "I'm not sure what you're talking about", in 2 of the 4 takes we tested this way. Those are its only misses in the table: the 2 harmless versions it did not let through. It heard the same words once more, at the end of a real take with less than a second of quiet.

## How sure we can be

Every model heard the same planted takes. So to compare two models, we count the takes that only one of them got right. If the two were equally good, those would split about evenly between them. We call a gap too big for chance when chance alone gives a split at least as uneven less than 1 time in 20 (an exact sign test). A 95% confidence interval is a range worked out so that, for tests like this, it holds the true share 19 times in 20.

- **Reads hardly vary.** Four services (Inworld, MAI-Transcribe-2, Gemini and Scribe v2) heard 24 planted takes twice and gave the same verdict on every one. We did not run the others twice and take it that they vary as little. So the chance is mostly in which faults we happened to plant.
- **Repeats, the top four:** Inworld and Gemini caught all 24. Parakeet missed 1, which could be chance (one take alone cannot tell two models apart). Scribe v2 missed 3, which could be chance (about 1 time in 4).
- **Repeats, the other eight:** each missed 7 to 20 repeats that Inworld caught: too big for chance (about 1 time in 64 for MAI-Transcribe-2, the closest). Parakeet caught 7 repeats that MAI-Transcribe-2 missed, and MAI-Transcribe-2 1 that Parakeet missed, which could be chance (about 1 time in 14). Scribe v2 caught 6 repeats that MAI-Transcribe-2 missed, and MAI-Transcribe-2 2 that Scribe v2 missed, which could be chance (about 1 time in 3).
- **All 52 faults:** Inworld and Gemini caught all 52. Parakeet missed 1, which could be chance (one take alone cannot tell two models apart). Scribe v2 missed 5, which could be chance (about 1 time in 16). The other eight each missed 9 to 26: too big for chance.
- **How sure one count is:** catching 24 of 24 does not prove a model catches every repeat. With 95% confidence, the share of repeats each one catches on takes like ours is Gemini and Inworld 86 to 100%, Parakeet 80 to 99%, Scribe v2 69 to 96%, MAI-Transcribe-2 51 to 85%, Whisper large-v3 turbo 39 to 76%, Voxtral Mini Transcribe 2 35 to 72%, Nemotron 3.5 Transcribe and Whisper large-v3 24 to 61%, Voxtral Small Transcribe 3 18 to 53%, Voxtral Mini Transcribe 3 12 to 45% and MAI-Transcribe-1.5 7 to 36% (Wilson intervals). These ranges treat each repeat as a separate test. The repeats were planted in 8 takes, so the true ranges are wider: counted by take, a model that caught every repeat in all 8, like Inworld, gets 68 to 100%.
- **Our two surprises:** Whisper base.en caught 7 faults that medium.en missed, and medium.en 1 that base.en missed, which could be chance (about 1 time in 14). Voxtral Mini Transcribe 2 caught 7 repeats that Voxtral Mini Transcribe 3 missed, and none the other way: too big for chance (about 1 time in 64). Against Voxtral Small Transcribe 3 it was 5 to 0, which could be chance (about 1 time in 16). We picked these pairs after seeing the results, which makes a gap look surer than it is.
- **Planted in 8 takes:** the 52 faults were planted in 8 recorded takes, and a model's misses may bunch in one of them. Counted take by take (a take counts for the model that caught more of its faults), Inworld's gap to MAI-Transcribe-2 on repeats could be chance (5 takes to 0, about 1 time in 16). Every other gap called too big for chance above still holds.
- **Many comparisons:** this section makes 27 comparisons, and among that many, one gap can pass the 1-in-20 line by chance alone. Allowing for that (Holm's method), Inworld's gap to MAI-Transcribe-2 on repeats, Inworld's gap to MAI-Transcribe-2 on all faults and Voxtral Mini Transcribe 2's gap to Voxtral Mini Transcribe 3 could be chance too; the other gaps called too big for chance still hold.

## What we chose

**Inworld STT 1.** It caught all 52 faults and raised no false alarm on the 71 approved takes. On faults, the test cannot tell it apart from Gemini, Parakeet or Scribe v2, nor, once the many comparisons are allowed for, from MAI-Transcribe-2 (see above). Of those five it is the fastest, at under a minute a lesson. At about 5 cents it costs less than Gemini or Scribe v2, though Parakeet and MAI-Transcribe-2 cost less still. And Inworld already makes our voices, so no new company hears the lessons.

For the invented words, our check now cuts the quiet end of a take before it is sent. With that cut, Inworld STT 1 heard all the takes again: every fault caught, all 12 harmless versions let through, no false alarm, and the real repeat caught. That run cost US$0.22. The table above shows Inworld without the cut, like every other model.

**Why not Gemini 3.5 Transcribe?** On our scored takes it was the only perfect one: every fault caught, every harmless version let through, no false alarm. But on the route we tested, Google's terms let it keep the audio. We allowed that for this test only, on our own narration. It also costs about twice as much and takes more than four times as long.

**Not tested:** GPT-4o Transcribe and GPT-4o mini Transcribe. OpenRouter had no endpoint for them that keeps no data, and we chose not to make an exception as we did for Gemini. OpenRouter refused the requests, so nothing reached OpenAI.

## What this test does not tell you

- All the takes are AI narration in English, from one voice service, not people talking. Human speech, other languages and noisy rooms may give a different order.
- That voice service is Inworld, which also made the recognizer we chose. It may hear its own voices better than others do.
- We planted the faults ourselves, cut at gaps between words. A real slip may sound different.
- "Approved" takes are ones we had passed by ear. A fault we missed by ear would count as a false alarm here.
- 52 faults from 8 takes is a small set. One fault more or less moves a model one or two places.
- We also ran 40 earlier takes whose right answer we don't know. Gemini and Inworld each flagged one of them, which looks like a mistake.
- Prices and models change. These are the ones of 30 October 2026.

If your job is a clean transcript, most of these models will do it well. If you need to know exactly what was said, test for repeats first.

**About Plainlamp.** Plainlamp turns a difficult PDF into a short narrated video and a plain-English guide. Every point shows the page it comes from.

[See what a lesson looks like](https://plainlamp.com/#lesson)

[All posts](https://plainlamp.com/blog)