Skip to content
GenAI Learn/Diffusion Models & Multimodal Generation
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Audio, Speech & Where Multimodal Is Headed

5 min read

You'll learn to

  • -Explain why speech models read a spectrogram rather than the raw waveform
  • -Use the prism analogy to describe the Discrete Fourier Transform, and read its output on three small signals
  • -Describe how speech recognition and speech generation reuse ideas from earlier chapters
  • -Compare a pipeline of specialist models with a native multimodal model

Text and images are not the only things a generative system handles. This closing chapter covers the one piece you need to make sense of speech models, such as Whisper-style transcription and modern text-to-speech, and previews where multimodal generation is heading: video, and models that handle every modality natively instead of stitching specialists together.

How Machines "Hear": From Waveform to Spectrogram

Sound reaches a computer as a waveform: a long list of numbers, each saying how far the microphone's membrane was pushed at that instant. For speech, that is typically 16,000 numbers per second. A neural network reading them directly would struggle, because what makes an "ah" different from an "ee" is not any one number but which frequencies are present. So every speech model first turns the waveform into a spectrogram: a picture showing which frequencies are present, over time.

An Analogy: A Prism and a Stereo Display

A prism splits white light into its colors, showing how much of each is mixed in. The Discrete Fourier Transform (DFT) does the same for sound: it splits a short stretch of signal into frequencies and reports how much of each is present. Stack those readings side by side over time and you get the spectrogram, much like the bouncing bars on a stereo's display.

  • -White light is the raw waveform.
  • -The prism is the DFT.
  • -The colors are the frequencies.
  • -How bright each color is tells you how strongly that frequency is present, which is the magnitude.
  • -The stereo display, bars moving over time, is the spectrogram.

The DFT, Read on Three Tiny Signals

The idea is to ask, for each frequency bin k, how much of a wave with exactly k full cycles fits inside the window. Multiply the signal against a cosine and a sine of that frequency, add up the results, and take the size. A big total means the signal lines up well with that wave. The code below uses a window of 8 samples and three signals, and the comments show what to expect.

Discrete Fourier Transform (magnitude)
real[k] = Σₙ signal[n]·cos(2πkn/N), imag[k] = −Σₙ signal[n]·sin(2πkn/N), |X[k]| = √(real[k]² + imag[k]²)
N is the number of samples in the window. Bin k measures how many full cycles of a wave fit in the window. Real systems use the FFT, a fast algorithm that computes the exact same result; the definition itself needs nothing but sums of sines and cosines.
The DFT magnitude spectrum, computed directly from its definition

The flat signal has all its energy in bin 0, the steady level. The pure tone lights up only bin 2, because exactly two cycles fit the window. The chord shows two peaks, a strong one at 1 cycle and a weaker one at 2. That reading of "which frequencies, how strongly" is what a spectrogram column contains. The FFT is simply a much faster way to get the same numbers: for a 1,024-sample window, about 10,000 steps instead of over a million.

Speech Models Built on Top: ASR and TTS

Whisper-style automatic speech recognition (ASR) feeds a spectrogram into an encoder-decoder Transformer: structurally the same architecture family from the Transformers module, with a spectrogram instead of token embeddings as the encoder's input. Text-to-speech (TTS) runs in the other direction. A model generates a spectrogram conditioned on text, and a second model called a vocoder turns that spectrogram back into an audible waveform, often using a diffusion process like the one earlier in this module, generating audio frames instead of image pixels.

Video Generation and Where Multimodal Is Headed

Video generation systems such as Sora extend diffusion by one dimension. Instead of denoising a 2D grid of pixels, they denoise spacetime patches, small chunks of video spanning both space and time, using a Transformer-based denoiser instead of the U-Net architecture of classic image diffusion. The forward-noise and reverse-denoise process underneath is the same.

Two Ways to Build a Multimodal System
Separate models for each modality (an ASR model, an LLM, a TTS model, a diffusion model), glued together - easier to build and swap pieces, but errors compound across the pipeline and latency stacks up
Pipeline of specialists
One model trained to understand and generate across modalities directly (the GPT-4o/Gemini direction) - harder to train, but avoids the information loss and latency of converting between modalities at every pipeline boundary
Native multimodal model

AI Lab's Level 72 (Mel-Style Spectrograms) builds the DFT magnitude spectrum in numpy, vectorized across every frequency bin at once with a single matrix multiply instead of the nested loop above.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Implement it from scratch: Level 72 (Mel-Style Spectrograms via a from-scratch DFT) in AI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!