Speech: ASR and Text-to-Speech

Speech processing connects audio and text in both directions. Automatic speech recognition (ASR) converts a waveform into a transcript; text-to-speech (TTS) synthesizes a waveform from text. Both are now dominated by Transformer-family models operating on log-mel spectrograms.

This page walks through the feature pipeline, the three decoder families used in ASR (CTC, RNN-Transducer, attention encoder-decoder), Word Error Rate, the TTS stack, and runnable examples. Part of Natural Language Processing.

1. ASR Pipeline

Raw audio is resampled (16 kHz is standard for speech) and converted to a log-mel spectrogram: short-time Fourier transform over ~25 ms windows every 10 ms, mapped onto 80 (or 128) mel-scaled filter banks, then log-compressed. The mel scale approximates human pitch perception and makes the features compact.

An acoustic encoder — typically a Conformer (convolution + self-attention) or a plain Transformer after a convolutional downsampler — turns the frame sequence into contextual representations. A decoder then maps those to tokens. Self-supervised pretraining on unlabelled audio (wav2vec 2.0, HuBERT) or weakly supervised training on very large transcribed corpora (Whisper) is what made modern ASR robust to accents and noise.

2. CTC, RNN-T, and Attention Decoders

CTC (Connectionist Temporal Classification) emits a label or a blank for every frame and sums over all alignments that collapse to the target. It is non-autoregressive and fast, but assumes output tokens are conditionally independent given the audio, so it benefits from an external language model at decode time.

RNN-Transducer adds a prediction network over previously emitted tokens and a joint network, removing the independence assumption while staying frame-synchronous. That makes it the usual choice for streaming on-device ASR.

Attention encoder-decoder models (Whisper is the best-known) decode autoregressively with cross-attention over the whole utterance. Accuracy is high and one model can do multilingual transcription, translation-to-English and timestamps via special task tokens, but it needs the full window (30 s for Whisper) and can hallucinate text on silence or music. Hybrid CTC/attention training combines the strengths.

3. Runnable ASR Example and WER

# pip install transformers torch librosa
import torch, librosa
from transformers import pipeline

device = 0 if torch.cuda.is_available() else -1
asr = pipeline("automatic-speech-recognition",
               model="openai/whisper-small",
               chunk_length_s=30,          # sliding 30 s windows for long audio
               device=device)

audio, sr = librosa.load("meeting.wav", sr=16_000)   # Whisper expects 16 kHz mono
result = asr(audio, return_timestamps=True,
             generate_kwargs={"language": "en", "task": "transcribe"})
print(result["text"])
for chunk in result["chunks"]:
    print(chunk["timestamp"], chunk["text"])

# Word Error Rate via edit distance
def wer(ref: str, hyp: str) -> float:
    r, h = ref.split(), hyp.split()
    d = [[0] * (len(h) + 1) for _ in range(len(r) + 1)]
    for i in range(len(r) + 1): d[i][0] = i
    for j in range(len(h) + 1): d[0][j] = j
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            cost = 0 if r[i-1] == h[j-1] else 1
            d[i][j] = min(d[i-1][j] + 1, d[i][j-1] + 1, d[i-1][j-1] + cost)
    return d[-1][-1] / max(len(r), 1)

print(wer("turn the lights off", "turn lights of"))   # 0.5 (1 deletion + 1 substitution)

WER = (S + D + I) / N: substitutions, deletions and insertions from the minimum edit alignment, divided by the number of reference words. Normalize case, punctuation and numbers identically on both sides before scoring — most "WER regressions" in practice are normalization mismatches. Character Error Rate (CER) is preferred for languages without whitespace word boundaries.

4. Text-to-Speech

The text frontend expands numbers, abbreviations and dates and converts graphemes to phonemes (G2P), since spelling is a poor guide to pronunciation. The acoustic model predicts a mel spectrogram: Tacotron 2 does this autoregressively with attention; FastSpeech 2 is non-autoregressive, using explicit duration, pitch and energy predictors for speed and controllability. A neural vocoder (WaveNet historically; HiFi-GAN and similar GAN vocoders today) converts the mel spectrogram to a waveform.

End-to-end models such as VITS train both stages jointly. A newer family treats audio as discrete tokens from a neural codec and generates them with a language model, enabling voice cloning from a short prompt — which is also why consent and watermarking matter for deployment.

# pip install transformers torch soundfile
from transformers import pipeline
import soundfile as sf

tts = pipeline("text-to-speech", model="facebook/mms-tts-eng")   # VITS-based
out = tts("Speech synthesis turns text into a waveform.")
sf.write("out.wav", out["audio"].squeeze(), out["sampling_rate"])

5. Choosing an Approach

6. Common Pitfalls

• Sample-rate mismatch — feeding 44.1 kHz audio to a 16 kHz model silently degrades accuracy. Resample explicitly.
• Hallucination on silence — autoregressive ASR may invent text for non-speech. Run voice activity detection first.
• Normalization drift — "10" vs "ten" counts as an error unless both sides use the same text normalizer.
• Long-audio boundaries — words cut at chunk edges get dropped or duplicated; use overlapping windows and merge on timestamps.
• Privacy — voice is biometric data; plan retention, redaction and consent for both recordings and cloned voices.

7. Feature Arithmetic, Worked Through

At 16 kHz, a 25 ms analysis window covers 400 samples and a 10 ms hop advances 160 samples, so one second of audio becomes about 100 frames. The FFT is usually 512 points (the next power of two above 400), producing 257 frequency bins that 80 triangular mel filters compress into 80 numbers per frame. One second of speech is therefore an 80 × 100 matrix of log energies. Whisper pads or trims every input to 30 seconds, giving 3,000 frames; two convolution layers, the second with stride 2, downsample this to 1,500 encoder positions, roughly one per 20 ms. A typical speaking rate of 2 to 3 words per second means the decoder emits far fewer tokens than there are frames, which is why CTC needs the blank symbol and why encoders downsample aggressively (Conformer models commonly reach a 40 ms frame rate).

These numbers matter in practice. Telephone audio sampled at 8 kHz contains nothing above 4 kHz, so upsampling it to 16 kHz for a wideband model leaves the upper mel bands empty; models trained or fine-tuned on narrowband audio do better on call-centre data. Clipping, heavy compression codecs and far-field reverberation all change the spectrogram in ways that augmentation (speed perturbation, SpecAugment masking of time and frequency bands, added noise and room impulse responses) is designed to cover during training.

8. Measuring Quality and Speed

A worked WER example. Reference: "turn the kitchen lights off now" (6 words). Hypothesis: "turn kitchen light off right now". The minimum alignment has one deletion ("the"), one substitution ("lights" → "light") and one insertion ("right"), so WER = (1 + 1 + 1) / 6 = 0.50. WER can exceed 1.0 when the hypothesis has many insertions, and it treats every word equally: getting a drug name or an account number wrong costs the same as dropping "the". For applications that act on transcripts, add a task metric such as entity or keyword accuracy, and report results per accent, microphone type and noise condition, since averages hide groups that perform much worse.

9. Streaming and Production Architecture

Live captioning, voice assistants and contact-centre agents need words within a few hundred milliseconds. Streaming models restrict how far into the future the encoder may look (chunked or limited right-context attention) and emit partial hypotheses that may be revised as more audio arrives; the interface should show them as provisional. Endpointing decides when the speaker has finished, using VAD silence duration and sometimes a model that predicts end-of-turn from the words so far; too eager and it cuts people off, too slow and the assistant feels sluggish. Final transcripts then pass through punctuation and inverse text normalisation ("twenty five dollars" to "$25").

Batch transcription has different priorities: throughput and cost. Split long recordings at silences found by VAD, batch segments on the GPU, and merge using word timestamps; for Whisper-style models, faster inference engines such as CTranslate2-based implementations and quantisation reduce cost substantially. Add speaker diarization (pyannote-style pipelines are common) and align speaker turns to word timestamps. For specialised vocabulary, contextual biasing with a list of expected phrases is often cheaper than fine-tuning, and fine-tuning on a few hours of in-domain audio typically helps more than switching to a larger general model.

10. Use Cases and Responsible Deployment

Common applications include meeting and lecture transcription, captioning for accessibility, voice search and assistants, call analytics feeding summarisation and classification, clinical documentation, speech translation with machine translation, and TTS for screen readers, navigation and audiobook narration. Recordings may require all-party consent depending on jurisdiction, and voice prints can fall under biometric privacy laws, so make retention, redaction of personal data and opt-outs part of the design. For voice cloning, obtain documented consent from the speaker, restrict who can create voices, and consider watermarking synthetic audio; detection of synthetic speech is an active research area but not reliable enough to be the only safeguard.

Further reading: Graves et al., "Connectionist Temporal Classification" (ICML 2006); Graves, "Sequence Transduction with Recurrent Neural Networks" (2012); Gulati et al., "Conformer" (2020); Baevski et al., "wav2vec 2.0" (2020); Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision" (Whisper, 2022, arXiv:2212.04356); Park et al., "SpecAugment" (2019); Shen et al., "Tacotron 2" (2018); Kong, Kim & Bae, "HiFi-GAN" (2020); Kim, Kong & Son, "VITS" (2021).

11. Related Topics

• Natural Language Processing (NLP) — parent section
• Encoder-Decoder (T5, BART) — Whisper uses the same seq2seq structure
• Self-Attention — core of Conformer and Transformer acoustic encoders
• Machine Translation — speech translation chains ASR and MT
• Vision Transformers (ViT) — patch-based Transformers on 2D inputs such as spectrograms