Sound analysis
Let’s start by building basic sound analysis methods for whatever comes next. To make the visuals react to different parts of the audio, we split the track into multiple stems.
See Sound visualization.
The audio used throughout this site is compressed to MP3 and trimmed for demo purposes. All audio is by Yaporigami (Yu Miyashita), licensed under CC BY-NC-SA 4.0.
Amplitude and spectrum
open full size ↗Each stem has two live views: a running RMS amplitude trace over the last 15 seconds, and the frequency content of the current frame as 128 Mel-spaced bands, both updating at 60 fps. The frequency label in the corner shows the active range the analysis detected for that stem.
We use a Python script to precompute this data before the browser ever sees it, so every frame’s visual state comes from a fixed row of numbers — the same input always renders the same output, with no runtime signal processing and no timing drift between audio and visuals during video export.
How the analysis works. A full-file STFT at 2048-point resolution builds a global energy profile across all frequencies. Spectral roll-off at 5 % and 85 % of cumulative energy marks where the track actually lives — ignoring sub-bass rumble below the content and empty headroom above it. A Mel-scale filterbank then maps 128 bands logarithmically across that active range, concentrating resolution where the musical content is densest. The script processes left and right channels separately and writes one row of numbers per frame at 60 fps: L and R RMS amplitude followed by 128 Mel band magnitudes for each channel. The output is stored as raw little-endian 32-bit floats — 258 values per frame, 1032 bytes per frame — which the browser loads directly into a typed array without any parsing overhead.
The browser fetches that file once and advances through it frame by frame in step with the audio, skipping each stem’s silent lead-in by scanning for the first frame whose amplitude clears a small threshold and cueing both audio and display to that point.
Linear vs. log scaling. The Mel band values are stored as linear amplitude — each number is proportional to energy in that frequency band. Human perception is not linear — loudness is logarithmic: a band at 1 % of peak energy is audible, not negligible. Before display, the browser maps each value to decibels: \( v_\text{dB} = 20 \log_{10}(\max(v, 10^{-5})) \), then rescales \([-80,\, 0]\) dB to \([0, 1]\). The floor at \(-80\) dB is below any content that matters; the ceiling at \(0\) dB is the global peak of each stem (since the data is normalised to its own maximum). The result is a spectrum that fills the display range without clipping.
Spectrograms
Each strip shows a one-minute excerpt — the densest section of each stem, selected by finding the sixty-second window with the highest cumulative amplitude. The horizontal axis is time — ten frames collapsed into one pixel column. The vertical axis runs from the stem’s lowest active frequency at the bottom to its highest at the top. Pixel brightness is the peak Mel-band amplitude within that 10-frame window, with a square-root curve applied so quieter detail stays visible. L and R channels are shown separately.