Inside Voice: analysis by synthesis with a differentiable vocal tract

A voice you cannot see

Sing a short phrase into Inside Voice and a physical model of the vocal tract sings it back. On screen, a tongue rises and falls, the jaw opens, the lips round and spread, and the soft palate lifts or drops to open the way to the nose. None of that motion was measured. It comes from a model whose controls were adjusted until the sound it makes resembles yours. The animation is the model's estimate of how you made the sounds in your recording; its synthesized sound lets you hear that estimate.

A listener gets one pressure wave at the lips. Fant's acoustic theory treats that wave as a source signal shaped by the resonances and losses of the tract, and Stevens built much of acoustic phonetics on the same decomposition⁠[1, 2][1] Acoustic Theory of Speech Production: With Calculations based on X-Ray Studies of Russian Articulations. Gunnar Fant, 1960.[2] Acoustic Phonetics. Kenneth N. Stevens, 1999.. The recording does not arrive with the two parts separated, and it says nothing directly about where the tongue is. A pitch trace or a spectrogram shows a note drifting or energy moving between frequency bands, but not which articulator moved, or whether a change came from the vocal folds or from the tract above them. Singing stretches both parts of the model⁠[3, 4][3] Principles of Voice Production. Ingo R. Titze, 1994.[4] The Science of the Singing Voice. Johan Sundberg, 1989.. At a high sung pitch the harmonics lie hundreds of hertz apart, so they sample the tract's response at only a few frequencies, and a resonance can sit between two of them.

The method behind the animation is called analysis-by-synthesis, and the name dates to 1959. This post follows it from the first computer spectrum matching to gradient-based fitting, then describes a faster differentiable waveguide tract and a differentiable version of the much richer CHIMERA airway, and reports one result we did not expect, about the velum.

Make a sound, then compare

Morris Halle and Kenneth Stevens named the idea at a 1959 seminar on speech compression and processing in Bedford, Massachusetts, as a model of how speech might be recognized: a listener, or a machine, generates comparison signals internally and searches for the one that matches what it hears⁠[5, 6][5] Analysis by Synthesis. M. Halle and K. N. Stevens, 1959.[6] Speech Recognition: A Model and a Program for Research. M. Halle and K. N. Stevens, 1962..

Two years later Bell, Fujisaki, Heinz, Stevens and House, at MIT's Research Laboratory of Electronics, turned it into a measurement procedure⁠[7][7] Reduction of Speech Spectra by Analysis-by-Synthesis Techniques. C. G. Bell, H. Fujisaki, J. M. Heinz, K. N. Stevens, and A. S. House, 1961.. Their analyzer, a program on MIT's TX-0 computer, held a model that could produce spectra from a small set of parameters. For each measured spectrum of a vowel or consonant it generated candidates, compared them with the measurement, and adjusted the parameters until the two agreed. The output of the analysis was the parameter set; ours is a set of continuous trajectories for a voice source and an airway, found by the same loop of proposing, synthesizing, comparing and revising.

In modern notation, let $y$ be the recording, $S$ the synthesizer, and $\theta$ its time-varying controls. The inverse problem is

where $\Phi$ extracts acoustic features, $\mathcal{L}$ measures their mismatch, and $R$ encodes preferences such as smooth motion or staying near an initial estimate. Most of this post is about $S$, but $\Phi$ and $R$ matter as much. $\Phi$ defines what counts as a match: a log-mel spectrogram ignores waveform phase and will accept a smeared consonant. $R$ decides which of several equally good matches you get.

The synthesizer can be anything that turns controls into sound. The case we care about is the one where the controls are physical, so that a fitted $\theta$ reads as a lip aperture or a tongue position over time. Such a fit is a statement about anatomy, and the question is how much of it the sound actually constrains. Figure 1 lets you try it on a small tube model.

Tubes and travelling waves

The physical models in this story share Fant's source–filter picture⁠[1, 2][1] Acoustic Theory of Speech Production: With Calculations based on X-Ray Studies of Russian Articulations. Gunnar Fant, 1960.[2] Acoustic Phonetics. Kenneth N. Stevens, 1999.. In a voiced sound, a quasi-periodic glottal source excites the airway, and the airway's resonances shape the spectrum that leaves the lips. An area function gives the cross-sectional area of the airway at each point from glottis to lips. Change the area function and the resonances move; narrow it far enough and the constriction adds turbulence or closes altogether.

Kelly and Lochbaum's speech synthesizer cut the tract into a chain of short cylinders⁠[8][8] Speech Synthesis. J. L. Kelly and C. C. Lochbaum, 1962.. A pressure wave crosses each cylinder unchanged. Where two cylinders meet, the change in area sends part of the wave back and lets the rest through, and the split depends only on the two areas. With the left-minus-right convention used by the VocalTrax code we compare against later, the reflection coefficient at junction $i$ is

The constant $\epsilon$ is a small numerical stabilizer. A narrowing gives a positive coefficient and a widening a negative one. Losses, the glottal and lip boundaries, and radiation complete the filter.

The tube model sang before it was published. Kelly and Lochbaum presented it at the International Congress on Acoustics in Copenhagen in 1962, but a year earlier at Bell Labs they had programmed its voice for “Daisy Bell” on an IBM 704, using vowel shapes from Fant's book, with Max Mathews programming the accompaniment⁠[9, 10][9] Singing Kelly-Lochbaum Vocal Tract. Julius O. Smith III, online book.[10] “Daisy Bell (Bicycle Built for Two)”—Max Mathews, John L. Kelly, Jr., and Carol Lochbaum (1961). Cary O’Dell, Library of Congress National Recording Registry.. Julius Smith calls it most likely the first digital physical-modelling synthesis by any method. Arthur C. Clarke heard the demonstration, and it became the song HAL sings in 2001: A Space Odyssey.

Digital waveguides, which Julius Smith worked out at Stanford's CCRMA in the mid-1980s, run travelling-wave physics of this kind as digital filters⁠[11, 12][11] Music Applications of Digital Waveguides. Julius O. Smith III, 1987.[12] Waveguide Filter Tutorial. Julius O. Smith III, 1987.. A waveguide keeps the two travelling waves in delay lines and does arithmetic only where something happens, at junctions and boundaries. His 1992 Computer Music Journal article placed the approach within physical sound synthesis generally, from strings to bores to tubes⁠[13][13] Physical Modeling Using Digital Waveguides. Julius O. Smith III, 1992.. Välimäki and Karjalainen later replaced the cylinders with conical sections and used fractional delays, so that section lengths need not be whole samples⁠[14][14] Improving the Kelly-Lochbaum vocal tract model using conical tube sections and fractional delay filtering techniques. Vesa Välimäki and Matti Karjalainen, 1994..

Johan Liljencrants's 1985 thesis at KTH in Stockholm turned the Kelly–Lochbaum line into a fuller speech synthesizer⁠[15][15] Speech Synthesis with a Reflection-Type Line Analog. Johan Liljencrants, 1985.. He added the losses of a real tract (viscosity, heat conduction, vibration of the walls and radiation at the lips), built the glottis into the line through its kinetic pressure drop, and attached parallel branches for the nose, which produce the pole–zero pairs of nasal sounds. He also worked out rules for changing the section areas over time without adding energy that a real tract would not gain. Later tube models cite this work: Story's TubeTalker for the wave-reflection algorithm it runs⁠[16][16] Phrase-level speech simulation with an airway modulation model of speech production. Brad H. Story, 2013., and Perry Cook's 1990 dissertation for richer physical tracts, including extra tubes for the space below the tongue⁠[17][17] Identification of Control Parameters in an Articulatory Vocal Tract Model, with Applications to the Synthesis of Singing. Perry R. Cook, 1990..

A singer made of tubes

Perry Cook, one of us, then an electrical engineering graduate student at Stanford's CCRMA, turned the waveguide tract into a singer. At the 1989 International Computer Music Conference in Columbus, Ohio, he presented singing synthesis with a physically parameterized model of the human vocal tract⁠[18][18] Synthesis of the Singing Voice Using a Physically Parameterized Model of the Human Vocal Tract. Perry R. Cook, 1989.. The system he described in Computer Music Journal in 1993 had two parts: SPASM, a real-time graphical controller for the tract, and Singer, the companion software synthesizer⁠[19][19] SPASM, a Real-Time Vocal Tract Physical Model Controller; and Singer, the Companion Software Synthesis System. Perry R. Cook, 1993.. The tract was a Kelly–Lochbaum waveguide with eight oral sections and a six-section nasal branch, joined at a three-way junction, and it ran in real time on the DSP chip of a NeXT computer. In Cook's demonstrations the synthetic singer has a name, “Shiela” (his spelling), and his page of singing-synthesis examples lists a 1991 “Shiela and Daisy Guy Duet” next to the original Bell Labs “Daisy”⁠[20][20] Singing voice synthesis examples. Perry R. Cook, web page.. His 1996 survey traces the history of singing synthesis up to that point⁠[21][21] Singing Voice Synthesis: History, Current Work, and Future Directions. Perry R. Cook, 1996..

The part of Cook's work closest to ours is his 1990 dissertation, Identification of Control Parameters in an Articulatory Vocal Tract Model, with Applications to the Synthesis of Singing⁠[17][17] Identification of Control Parameters in an Articulatory Vocal Tract Model, with Applications to the Synthesis of Singing. Perry R. Cook, 1990.. It runs the tube model backwards: given a recording, which tract shapes and glottal settings would make the model reproduce it, and how do the shapes change over time? One route converts linear prediction coefficients, by the step-down recursion, into reflection coefficients and section radii. The other, FAST (a “Formant-driven Articulatory Speech Tracker”), moves the articulators by gradient descent on spectral features: the least-squares movement that yields the desired formant pattern is applied to the articulators, and the resulting shape is compared with a library of shapes. The glottal waveform comes from inverse filtering, and the identified parameters include section radii, velum opening, pitch, pitch deviation and noise. Gradient steps on a physical voice model were therefore in use by 1990. What has changed since is that automatic differentiation carries the gradient through the whole synthesizer to a loss on the spectrogram. Inside Voice asks Cook's question that way, of a larger airway.

Neil Thapen released Pink Trombone in 2017, a vocal tract you play with a mouse in a web browser⁠[22][22] Pink Trombone. Neil Thapen, 2017.. It belongs to the same family: a 44-section Kelly–Lochbaum waveguide with a 28-section nose joined at section 17 by a three-port junction, and reflection coefficients of 0.75 at the glottis and −0.85 at the lips. Its source credits Julius Smith's waveguide work and Story's research rather than SPASM, so it is better described as a sibling of Cook's singer than a descendant. It matters here for a practical reason: both gradient-based projects below started from its code.

Explaining a sound without a throat

Analysis-by-synthesis does not require a physical model, and some of its most successful forms use none. The alternative is to model the sound instead of the instrument, and CCRMA was developing it in the same years as the waveguide tract. PARSHL, described by Julius Smith and Xavier Serra in 1987, tracks the peaks of a short-time spectrum from frame to frame and resynthesizes them as sinusoids, including for sounds that are not harmonic⁠[23][23] PARSHL: An Analysis/Synthesis Program for Non-Harmonic Sounds Based on a Sinusoidal Representation. Julius O. Smith III and Xavier Serra, 1987.. Serra's 1989 dissertation added what sinusoids miss: a sound is split into a deterministic part, the tracked partials, and a stochastic residual modelled as noise⁠[24][24] A System for Sound Analysis/Transformation/Synthesis Based on a Deterministic Plus Stochastic Decomposition. Xavier Serra, 1989.. Written up with Julius Smith in Computer Music Journal in 1990, this became Spectral Modeling Synthesis⁠[25][25] Spectral Modeling Synthesis: A Sound Analysis/Synthesis System Based on a Deterministic Plus Stochastic Decomposition. Xavier Serra and Julius O. Smith III, 1990..

A sinusoids-plus-noise model fits a voice closely because it has a free parameter for nearly every partial. Serra's 1997 chapter on such models contrasts them with physical models of the vocal tract⁠[26][26] Musical Sound Modeling with Sinusoids plus Noise. Xavier Serra, 1997.. The signal route also reached singing synthesis: Bonada and Serra's 2007 system combines spectral models with samples of real performances⁠[27][27] Synthesis of the Singing Voice by Performance Sampling and Spectral Models. Jordi Bonada and Xavier Serra, 2007..

Randal Leistikow, one of us, pursued the signal-model side with explicit Bayesian generative models. With Harvey Thornburg, Julius Smith and Jonathan Berger, he wrote down what the peaks of a short-time Fourier transform should look like given a set of notes, including how the harmonics of different notes overlap, and then inverted those models. One paper identifies closely spaced chords from the peaks of a single frame⁠[28][28] Bayesian Identification of Closely-Spaced Chords from Single-Frame STFT Peaks. Randal J. Leistikow, Harvey D. Thornburg, Julius O. Smith III, and Jonathan Berger, 2004.. Another computes a pitch posterior from spectral peaks, exactly or approximately by Markov chain Monte Carlo⁠[29][29] A New Probabilistic Spectral Pitch Estimator: Exact and MCMC-approximate Strategies. Harvey D. Thornburg and Randal J. Leistikow, 2004.. Others extract sinusoidal parameters with an iterative filterbank⁠[30][30] An Iterative Filterbank Approach for Extracting Sinusoidal Parameters from Quasi-Harmonic Sounds. Harvey D. Thornburg and Randal J. Leistikow, 2003., and melody and onsets from framewise peak data⁠[31][31] Melody Extraction and Musical Onset Detection via Probabilistic Models of Framewise STFT Peak Data. Harvey D. Thornburg, Randal J. Leistikow, and Jonathan Berger, 2007.. His 2006 dissertation models musical expectation with maximum-entropy stochastic grammars⁠[32][32] Bayesian Modeling of Musical Expectations via Maximum Entropy Stochastic Grammars. Randal J. Leistikow, 2006..

These systems answer musical questions: which notes are sounding, what the pitch is, where an onset falls. They answer them well because the generative model is built from the same quantities the question asks about. Signal models have ambiguities too (overlapping partials, octave errors), but they are ambiguities about the sound. A physical model adds one about the body: many tract shapes give the same spectrum.

Many throats, one sound

That ambiguity was demonstrated long before anyone fitted a tract with gradients. Atal, Chang, Mathews and Tukey, in 1978, used a computer-sorting technique on large sets of model tract shapes and found families of different shapes that map to the same acoustic description⁠[33][33] Inversion of articulatory-to-acoustic transformation in the vocal tract by a computer-sorting technique. B. S. Atal, J. J. Chang, M. V. Mathews, and J. W. Tukey, 1978.. Maeda's 1989 analysis of X-ray data showed speakers exploiting this: coordinated changes in different articulators preserve an acoustic result, a pattern known as compensatory articulation⁠[34][34] Compensatory articulation in speech: analysis of x-ray data with an articulatory model. Shinji Maeda, 1989.. Schroeter and Sondhi's 1994 review separated ambiguity in tract geometry from ambiguity introduced by the source, and examined continuity constraints that narrow the possibilities⁠[35][35] Techniques for estimating vocal-tract shapes from the speech signal. Juergen Schroeter and Man Mohan Sondhi, 1994..

The 1988 Distinctive Region Model of Mrayati, Carré and Guérin builds this ambiguity in⁠[36][36] Distinctive regions and modes: a new theory of speech production. M. Mrayati, R. Carré, and B. Guérin, 1988.. It starts from Fant and Pauli's 1974 sensitivity functions, which describe how much each resonance shifts when the area changes at each point along the tract⁠[37][37] Spatial characteristics of vocal tract resonance modes. Gunnar Fant and S. Pauli, 1974., and divides a uniform tube, closed at the glottis and open at the lips, into eight regions bounded by the zero crossings of the sensitivity functions of F1, F2 and F3. Inside one region, a change of area moves the three formants in a fixed combination of directions. The theory also names a compensation: when a front region and its corresponding back region change area together, in the same direction, the formants barely move, so two different tubes give one set of formants, as in Figure 1. Krstulović later used the model as the forward model of an inversion from linear prediction⁠[38][38] LPC-based inversion of the DRM articulatory model. Sacha Krstulović, 1999., and Cook's 1998 paper on singing synthesis discussed it beside Liljencrants's rules for reshaping a tube⁠[39][39] Toward the Perfect Audio Morph? Singing Voice Synthesis and Processing. Perry R. Cook, 1998..

The articulatory model sets how much ambiguity you start with. Mermelstein's 1973 model derives tract shape from variables for the jaw, tongue, lips and other structures; it specifies geometry, not acoustics⁠[40][40] Articulatory model for the study of speech production. P. Mermelstein, 1973.. Story's TubeTalker simulates phrase-level speech by modulating an airway over time and computes the sound with a wave-reflection algorithm, a relative of the Kelly–Lochbaum tube⁠[16][16] Phrase-level speech simulation with an airway modulation model of speech production. Brad H. Story, 2013.. Birkholz's VocalTractLab models articulatory targets and how they overlap in consonant–vowel coarticulation⁠[41][41] Modeling Consonant-Vowel Coarticulation for Articulatory Speech Synthesis. Peter Birkholz, 2013., and simulates the sound by turning its branched tube into a lumped-element transmission-line circuit⁠[42][42] Effects of the Piriform Fossae, Transvelar Acoustic Coupling, and Laryngeal Wall Vibration on the Naturalness of Articulatory Speech Synthesis. Peter Birkholz and Susanne Drechsel, 2021.. More anatomical controls also mean more redundant ways to produce one spectrum.

The source adds its own share. Rosenberg resynthesized natural vowels with simplified glottal pulses and measured how the pulse shape changed their quality⁠[43][43] Effect of Glottal Pulse Shape on the Quality of Natural Vowels. A. E. Rosenberg, 1971.. Fant, Liljencrants and Lin's LF model uses four parameters⁠[44][44] A four-parameter model of glottal flow. Gunnar Fant, Johan Liljencrants, and Q. Lin, 1985.; its first stage, which the paper calls the L-model, was a three-parameter model of the flow derivative that Liljencrants had introduced. Both describe the flow waveform (LF, strictly, its derivative), not the folds, so a fitted source parameter is a property of the excitation model. A tilt in the spectrum can come from the source or from the tract, and the fit has to assign it to one of them.

Gradients

Before automatic differentiation, inversion relied on codebook lookup, dynamic programming over codebooks, or hand-derived and finite-difference gradients⁠[35][35] Techniques for estimating vocal-tract shapes from the speech signal. Juergen Schroeter and Man Mohan Sondhi, 1994.; Cook's FAST combined hand-derived least-squares steps on formants with a library of shapes. DDSP, by Engel, Hantrakul, Gu and Roberts in 2020, showed that classical signal-processing modules such as oscillators, filtered noise and reverberation can be written in an automatic-differentiation framework, so that a loss on the output audio sends gradients back to every control⁠[45][45] DDSP: Differentiable Digital Signal Processing. Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts, 2020.. DDSP trained neural encoders that predict controls for any input; the fits in this post instead optimize the controls of one recording directly. Hayes and colleagues' 2024 review covers what followed, including sound matching and its optimization difficulties⁠[46][46] A review of differentiable digital signal processing for music and speech synthesis. Ben Hayes, Jordie Shier, György Fazekas, Andrew McPherson, and Charalampos Saitis, 2024..

Südholt, Cámara, Xu and Reiss brought this to the vocal tract at DAFx 2023⁠[47][47] Vocal Tract Area Estimation by Gradient Descent. David Südholt, Mateo Cámara, Zhiyuan Xu, and Joshua D. Reiss, 2023.. They wrote the closed-form frequency-domain transfer function of Pink Trombone's 44-section Kelly–Lochbaum tract, without the nose, and estimated tract shapes and source parameters for vowels by inverse filtering and gradient descent through that response and the articulatory-to-area mapping. They fit one frame at a time, starting each frame of a changing sound from the previous frame's estimate, and synthesize audio only after the fit.

VocalTrax, by Mo, Cherep, Singh, Langford and Maes, presented at the NeurIPS 2024 Audio Imagination Workshop, fits end to end instead⁠[48][48] Articulatory Synthesis of Speech and Diverse Vocal Sounds via Optimization. Luke Mo, Manuel Cherep, Nikhil Singh, Quinn Langford, and Pattie Maes, 2024.. It ports Pink Trombone's recurrent waveguide to JAX, with 44 sections updated twice per audio sample inside a jax.lax.scan, and fits speech, singing and other vocal sounds. Pitch comes from the CREPE tracker beforehand⁠[49][49] CREPE: A Convolutional Representation for Pitch Estimation. Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello, 2018., and the remaining controls are optimized against a log-mel spectral loss. In the default oral model we use for comparison, five framewise controls describe tongue diameter, tongue position, throat constriction, lip constriction and glottal tenseness.

The bottleneck is the recurrence. Every output sample depends on the wave state left by the one before it, so the forward pass is a loop over time: two junction updates per sample is 88,200 sequential steps for each second of audio at 44.1 kHz. Backpropagation walks the same loop in reverse. A GPU can parallelize across the 44 sections, but not across time, and time is the long axis. Our convolutional version replaces the recursion with an approximation that is parallel in time.

Convolutional VocalTrax

We kept everything in VocalTrax except the tract renderer: the same controls, LF glottal source, CREPE pitch, aspiration noise, log-mel loss, initialization and optimizer. The recurrent waveguide is replaced by a frozen-block convolution that rests on two assumptions common in speech acoustics: the tract geometry changes little within a few milliseconds, and losses in the tract make its impulse response die away within tens of milliseconds.

Freeze the tract at excitation time

For fixed reflection coefficients, the waveguide junctions can be eliminated in the frequency domain, the move Südholt and colleagues made for static vowels. We derived the response of the waveguide exactly as the VocalTrax version we compared against executes it: two half-sample updates per audio sample, the source entering on the first⁠** Our baseline is a fork of VocalTrax (kaushik327/vocaltrax, commit 6c79060) with its optional extras (nasal coupling, frication, tilt, radiation) switched off. In this fork the source enters only on the first half step. The public release feeds the same sample on both half steps, and so does the simplified tube in Figure 4. The same argument applies to both: with an even number of segments the response is a single evaluation, and only its gain differs., and the output taken as a quarter of the sum of the two waves arriving at the lips. Converting that half-rate response back to the audio rate gives two terms: the response at each frequency, and an alias of it shifted by half the half-step rate. With an even number of segments, every path from the glottis to the lips takes an even number of half steps, so the alias is the same response as the original. The two terms then merge, and with 44 segments the whole response comes from a single evaluation.

For a time-varying vocal tract shape we make two assumptions. The geometry is held constant within blocks of $H=256$ samples (5.8 ms at 44.1 kHz), with the reflection coefficients interpolated at each block's midpoint. And the impulse response $h_b$ of block $b$ is truncated to its first $L=4096$ samples (93 ms), taken from an inverse FFT of the response on a grid of 8192 frequencies. With source $u$, the output is

computed with zero-padded FFTs and overlap-add, so the convolution is linear rather than circular. Gradients with respect to the controls pass through the FFTs and the 43-step recursion along the tract. All blocks and frequencies are processed in parallel, and the length of the gradient path no longer depends on the duration of the clip.

With the geometry fixed, the waveguide is linear and time-invariant, and the convolution reproduces VocalTrax exactly except for the truncation. For a lossy tract the truncation is negligible: in the recurrent model, at most $1.2\times10^{-11}$ of the energy of an output sample's sensitivity to the source lies more than 100 ms in the past, for neutral, constricted and fitted tracts alike. Computed in double precision, the two impulse responses agree to a relative error of $6\times10^{-10}$; in single precision the waveforms agree to $8.1\times10^{-6}$, the level of round-off.

The approximation enters only when the geometry moves. In the convolution, the tail of each block keeps the filter of the block that excited it; in the waveguide, waves already in the tract propagate through the current geometry. We measured this on a noise-excited tract whose diameters vary sinusoidally across control frames by 5% and 20% of their mean. With 256-sample blocks the relative waveform error is 1.0% and 4.0%. It grows with the speed of the motion and with the block length; 64-sample blocks bring it down to 0.8% and 3.0% but take twice as long to compute.

Recognition and transfer

The evaluation fits 600 AudioMNIST recordings: 12 held-out speakers, ten digits and five repetitions of each, between 0.42 and 1.0 s long⁠[50][50] AudioMNIST: Exploring Explainable Artificial Intelligence for audio analysis on a simple benchmark. Sören Becker, Johanna Vielhaben, Marcel Ackermann, Klaus-Robert Müller, Sebastian Lapuschkin, and Wojciech Samek, 2023.. Each recording is fitted independently, with a protocol shared by both models: AdamW at a learning rate of 0.01 for 1000 steps, projection of the controls back into range after every update, Savitzky–Golay smoothing every 200 steps, a log-mel MSE loss, and the same aspiration noise for both models at every step.

A low spectral loss does not make a digit intelligible, so we also ask a classifier. Following the VocalTrax evaluation, a frozen wav2vec 2.0 base model⁠[51][51] wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, 2020. with a linear head is trained only on 18,000 real recordings from 36 other speakers and then tested on the reconstructions. We average three head seeds and give 95% intervals from a paired bootstrap over speakers.

The recurrent reconstructions are recognized at chance (10.0%), ours at 14.5%, an improvement of 4.5 points (interval 2.2 to 6.6). On the log-mel objective our fits are 11.3% worse (interval 10.1% to 12.4%), while the pitch error drops from 46.0 to 40.1 cents. The absolute rates are low, and for every synthetic condition the classifier's predictions concentrate on a few digits, so these numbers measure relative rather than practical intelligibility. The original VocalTrax study reported 73.5% for real recordings and 20.0% for its reconstructions; we could not reproduce those figures, but every condition here is scored by the same classifiers.

The transfer row asks whether our fitted controls mean the same thing to the original model, rendered with the same pitch and noise. The waveform changes by 5.7% (median relative ℓ2 distance) and the log-mel loss rises by 0.032 (interval 0.024 to 0.041), yet recognition stays at 13.8%, 3.8 points above the recurrent model's own fits. Both models share one source artifact that probably lowers these absolute rates: unvoiced frames use a 1 Hz pitch placeholder, which puts about 45% of the output energy below 60 Hz, against 3.8% in the recordings. The study also uses one dataset and one optimization seed.

Less tract motion, better recognition

Better recognition at a higher log-mel loss appears related to how much the fitted tract moves. Between consecutive control frames, the four tract controls fitted by our model change about half as much as those fitted by VocalTrax (ratios 0.46 to 0.52), while tenseness changes 0.80 as much. The two renderers also disagree more when the reflection coefficients change faster: for the same controls, the difference between their outputs correlates with reflection-coefficient speed (Spearman ρ = 0.45). Fast tract motion lowers the loss less in the frozen-block model than in the recurrent one.

The effect differs from digit to digit. “Six” and “eight” lose the most motion (ratios 0.38 and 0.39) and show the largest gaps in loss, while the fully voiced “one” and “nine” show almost none. Our reading is that the frozen blocks act as a temporal smoothness prior. They give up some spectral detail, but they discourage jittery trajectories, which the recurrent model may use to lower the log-mel loss without making the digit easier to recognize. In the notation of the objective above, part of $R$ comes from the renderer rather than from an explicit penalty.

Speed

All timings use one NVIDIA A100 at batch size one, on inputs from 1 to 60 s made by repeating AudioMNIST recordings. The tract forward pass is 119 to 130 times faster than in VocalTrax and the backward pass 37 to 50 times faster, so forward and backward together run 47 to 63 times faster: 0.023 s against 1.111 s for one second of audio, and 1.092 s against 68.06 s for sixty. Both grow linearly with duration.

A full optimizer step is only about three times faster (0.50 s against 1.45 s for one second of audio; 28.98 s against 85.19 s for sixty), and complete 1000-step fits of three AudioMNIST recordings took 3.8 to 5.8 minutes instead of 12 to 18. Most of the remaining time goes to the LF source, which both models share and which computes the pitch phase with a loop over samples. Subtracting the source's time gives an estimated step speed-up of 17 to 25 times. Because pitch is fixed during fitting, that phase could be computed once per clip.

Removing the recursion also removes long products of Jacobians from the gradient, the usual source of vanishing or exploding gradients in recurrences. In VocalTrax's damped tract they were not a problem: with a waveform loss, the gradient norm divided by the square root of the number of frames stayed constant from 0.5 to 60 s, consistent with the tract's short memory. The gain is in the time and depth of the gradient computation, not in a smoother loss. At initialization, our model showed much larger local curvature along a probed direction.

The advantage belongs to the GPU and to single or long signals. On a CPU the recurrent model needs only 0.15 s per step for a one-second clip, and with large batches on a GPU it reaches 514 clip updates per second at one second of audio, against 73 for ours, which also needs more memory (379 against 145 MiB per fit).

Differentiable CHIMERA

With a fast differentiable renderer in hand, we turned to a much larger instrument. Jameson Hodge's CHIMERA is an MIT-licensed physical-modelling voice synthesizer, released in 2026, that simulates an airway with lungs, vocal folds or a syrinx, a branched tract, lips and nose⁠[52][52] CHIMERA. Jameson Hodge, 2026.. Rather than a waveguide, its engine is a staggered-grid finite-difference time-domain (FDTD) transmission-line network: pressures live on nodes, volume velocities on the branches between them, time advances by leapfrog steps, and kinetic losses are nonlinear. Its architecture notes say it chose this over Kelly–Lochbaum because one framework then handles variable section lengths, steady aerodynamics such as pressure build-up behind a closure and the burst that follows, nonlinear losses, yielding walls and arbitrary branching.

The human airway in CHIMERA runs from the lungs through the trachea and a glottal port into the pharynx, where it splits. One path goes through the oral tract to the lips. The other goes through a velum port into the nasal cavity, which has a maxillary sinus attached, and out through the nostrils. Seventeen articulators (jaw; tongue body x and y; tongue tip x, y and curl; tongue root; lip aperture, protrusion and width; labiodental; velum; larynx; epilarynx; ventricular; lateral; dorsum) shape a two-dimensional midsagittal contour, and a power law per region turns distances on that contour into areas. At the glottis, a parametric pulse sets the open area of a port, and the flow through it comes out of the network.

Our aim was to fit this instrument to singing with the frozen-block renderer. We split the model into three maps: from articulator controls to airway geometry, from geometry to an acoustic response, and from excitation through that response to sound. The first map is exact but piecewise, the second is linearized, and the third is where the frozen-block approximation enters. The Inside Voice app runs an approximation of this geometry, not the exact version described here, and renders it with the frozen-block waveguide described above, with a nose added.

Geometry and airway acoustics

We rewrote CHIMERA's static articulatory geometry in differentiable operations. Its 17 instantaneous controls determine oral and pharyngeal areas, tract lengths and the area of the velum port, and the contact and closure rules are part of the map. Against the original equations with hard contacts, on 20,000 synthetic oracle poses for each of three geometries, the maximum float64 area errors are $5.95\times10^{-13}$, $3.79\times10^{-13}$ and $2.65\times10^{-13}$ cm². The map keeps CHIMERA's discrete topology choices and piecewise derivatives, so it is not smooth everywhere.

For a fixed geometry, CHIMERA's acoustics can be linearized into a network of inductances, resistances and capacitances set by the areas, lengths, losses and radiation. The network is a tree, so eliminating it branch by branch from the leaves gives the complex transfer function from an injected source to the radiated flow, with the nasal branch included. The CHIMERA repository already contains a linearized frequency-domain solver of the same network, which it uses as a test oracle and for its formant display; ours is written to be differentiated and to run for every block. With frequency warping disabled, our solver matches the original formulation to a maximum relative transfer-magnitude error below $10^{-9}$. An optional warp accounts for the discrete frequency response of CHIMERA's leapfrog scheme.

So the fit sees only CHIMERA's linear, small-signal behaviour: no nonlinear losses, no steady airflow, no history dependence, which are CHIMERA's reasons for using FDTD.

Rendering a time-varying vocal tract shape

Rendering uses the frozen-block principle from the VocalTrax work: compute each block's complex response, take a finite impulse response, convolve the excitation and overlap-add. Windowed, overlapping source blocks smooth the transitions between responses, and a ringing tail still keeps the geometry of the block that excited it (the same tail error as in Figure 4).

The voiced source is a Rosenberg-type pulse⁠[43][43] Effect of Glottal Pulse Shape on the Quality of Natural Vowels. A. E. Rosenberg, 1971.. Unlike the live engine, which sets glottal area and lets flow emerge from the network, the linear fit injects a flow pulse, so source–tract interaction is lost. Aspiration, frication and release bursts add excitation a periodic source cannot supply. Frication and bursts enter as pressure sources at the locations of constrictions and releases, so their spectra depend on the airway on either side of the injection point.

Relaxing contacts so gradients can pass

Hard contact rules contain operations such as setting a negative gap to zero. The rule encodes a real physical distinction, but once a seal forms it leaves the optimizer with no gradient for opening it again. In the research variant we relax such rules. The positive part of a signed clearance $x$, for example, becomes

which approaches $\max(0,x)$ as the width $s_\tau$ shrinks. Smooth sealing gates soften selected closure decisions in the same way. Fitting starts with a broad relaxation and anneals it toward the hard geometry, and the final evaluation uses hard contacts and closures.

The relaxation is selective. Ray selection, topology changes and other piecewise decisions remain hard, and deep inside a contact a finite derivative can still be too weak to point anywhere useful. Smoothing helps the optimizer move through the model, but it cannot add acoustic evidence the recording does not contain.

What the sound cannot decide

We fit articulatory and source trajectories against a multiresolution log-mel and MFCC objective, with preferences for smooth motion and for staying near the initialization.

Replacing the app's approximate geometry with the exact one did not consistently lower the loss on 20 sung clips, and did not change the most striking feature of the fits: the velum. The fitted velum port stayed open on about 99% of voiced frames, measured as the median over clips of each clip's share of open frames, with an opening threshold of $2\times10^{-4}$ cm². With a nose available, the optimizer used it, or at least never found a reason to close it. The fits behind the app therefore add a soft velum penalty to the preference term of the objective: on voiced frames it penalizes an open port while the oral tract is open, and a closed one while it is sealed (see Limitations).

Our reading is that this objective cannot see the velum well enough to decide it. Initialization and plain non-identifiability are also candidates, and we have not separated them. An open port is not implausible in trained singers: nasal and oral airflow measurements on 17 professional operatic singers showed nasal airflow at least at some pitches in a majority of them⁠[53][53] Velum Behavior in Professional Classic Operatic Singing. Peer Birch, Bodil Gümoes, Hanne Stavad, Svend Prytz, Eva Björkner, and Johan Sundberg, 2002..

Cook's thesis asked which settings of his tube model produced a voice. With gradients and a full airway we can also ask which settings the sound leaves undecided, and for sung vowels heard through a log-mel and MFCC objective, the opening to the nose is one of them.

In the app

After a fit, the app shows a few cards about your phrase. Some come from the sound itself, such as your pitch range and vibrato; others, such as how much your jaw and tongue moved, are read from the fitted model. Any comparison on a card is rough: it depends on your phone and your room as much as on your voice, so it is not a score of how good a singer you are. The cards are not medical or vocal-health advice, and your recording is processed to give you the result and is not kept.

Limitations

The time-varying vocal tract shape in Inside Voice is one explanation of your recording, not a picture of your throat. Very different shapes of the tongue, lips and soft palate can make almost the same sound, and speech scientists showed this long before anyone fitted a vocal tract with gradients⁠[33, 34][33] Inversion of articulatory-to-acoustic transformation in the vocal tract by a computer-sorting technique. B. S. Atal, J. J. Chang, M. V. Mathews, and J. W. Tukey, 1978.[34] Compensatory articulation in speech: analysis of x-ray data with an articulatory model. Shinji Maeda, 1989.. When several shapes fit equally well, the model picks a smooth one close to where it started. That choice is useful for watching how a phrase moves, but it may not be what your body did.

The soft palate is the clearest example. The sound the model compares against says little about it, so the fit penalizes an open port unless the mouth is sealed. Only a direct measurement, such as airflow through the nose or an MRI scan taken while you sing, could say whether yours was open. The lips, likewise, sometimes stay slightly apart where a closure would be expected.

Use Try it to explore a short phrase, compare the reconstruction with the recording, and look at which changes the model needed to explain the sound.

Yongyi Zang, Debasish Ray Mohapatra, Christodoulos Benetatos, Randal Leistikow, and Perry R. Cook. "Inside Voice: analysis by synthesis with a differentiable vocal tract". Smule Labs, 2026-11-12. https://labs.smule.com/blog/inside-voice/

@misc{zang2026insidevoice,
  author = {Zang, Yongyi and Mohapatra, Debasish Ray and Benetatos, Christodoulos and Leistikow, Randal and Cook, Perry R.},
  title = {Inside Voice: analysis by synthesis with a differentiable vocal tract},
  year = {2026},
  url = {https://labs.smule.com/blog/inside-voice/}
}