Why the karaoke highlight is the hard part of a language app

Tori is a language app built on a simple loop: you read a story you actually want to read. Each sentence is shown in the language you are learning, with its translation directly below. Press play and the narrator reads the story sentence by sentence, alternating between the two languages, while the words being spoken light up in the text.

That last part — the highlight — is the feature everything else leans on. If it is right, the text and the voice feel like one thing and you stop noticing either. If it is late by a quarter of a second, the reading feels broken in a way people cannot quite name, and the method stops working.

The target is a number, not a feeling

We set the accuracy target at ±60 milliseconds per word. That is not an arbitrary precision contest. It is roughly the point where a highlight still reads as "the word being spoken" rather than "the word that was just spoken."

The data contract follows from that. Each sentence's audio lives in object storage with a companion JSON file. Every word carries a start and end in milliseconds within that sentence, plus where the timing came from — the synthesis engine itself, a forced aligner, or an estimate — and a confidence value. The invariants are checked, not assumed: a word cannot start before the sentence or end after it, and a gap of more than about 400 ms between consecutive words is flagged as a possible misalignment.

The reader does one job with this: given the audio position, find the last sentence whose start is at or before it, then the word inside it. That is a reverse scan over small arrays, cheap enough to run on every animation frame.

Where timing actually comes from

There are three ways to get word boundaries, and they are not equally good.

The engine can tell you. Some text-to-speech systems emit word boundaries as they synthesize, or expose a duration predictor that gives the length of every token. A VITS-style model, for example, decides how long each phoneme lasts before it renders the audio. Hooking that predictor gives exact boundaries for free: one Spanish sentence came back as 91 tokens spanning 0 to 1750 ms, with the boundaries falling on the hop grid. The important part is not the precision — it is that audio and timing are produced in the same pass, so they cannot drift apart.

A forced aligner can infer them. Given the audio and the known transcript, an aligner can place each word. This is the fallback for engines that emit no boundaries. It is genuinely useful, but it has failure modes: it is slower, it is less accurate on short or noisy clips, and if you ask a speech recogniser to transcribe freely it can hallucinate words that are not in the text. The rule we settled on is to never re-transcribe — always align to the known transcript, which keeps the text and the timing from disagreeing.

You can estimate. If neither works for a sentence, the highlight still has to move. The naive version spreads the words evenly across the clip, which looks wrong whenever the speaker pauses. A better estimate detects the actual speech spans — 20 ms RMS frames against a threshold relative to the clip's peak, bridging short gaps — and places words only over the regions where someone is speaking. Even the fallback should follow the audio.

The first reader prototype used the browser's own speech synthesis and its word-boundary events. That was enough to prove the loop, and not enough to ship: the timing quality belonged to the browser, not to us.

The decision that saved a pass

The planning assumption for a large library was that audio and word timing were two separate jobs: synthesize everything, then run forced alignment over the whole corpus. At the scale we were considering — a hundred books across two dozen languages — the alignment pass was going to cost more than the audio.

The way out was to choose an engine whose architecture already knows how long each token lasts. That collapsed two passes into one: the same synthesis run that writes the audio also writes the karaoke contract. The consequence is not just cost. A single pass cannot produce audio and timing from different runs of the model, which is exactly the class of bug that makes a highlight drift on one sentence and not the next.

Not every language has that option. Some will rely on alignment, and tonal languages need listening QA that no timestamp check replaces. The contract is deliberately engine-agnostic so a language can change engines without changing the reader.

What the reader gets

The payoff is small and specific: you tap a word and its meaning appears in place, without leaving the sentence; you press play and the narration follows the passage as you scroll; the translation stays beside the original instead of replacing it. None of that is impressive in a demo. All of it depends on the highlight landing on the right word at the right time.

A language app can have a beautiful library and a clever review system, and still fail at the moment a learner looks away from the text because the voice and the highlight have stopped agreeing. Sixty milliseconds is where that agreement is won.