|Beethoven, "Moonlight" Sonata:
||Daft Punk, Superheroes:
I was able to start this project with a very nice audio analysis tool, courtesy of The Echo Nest. This tool takes an MP3 file, breaks it up into little segments, and gives pitch, loudness, and high-level timbral descriptions of each one of those segments. My program maps a subset of this audio data onto a visual scale (see "Mapping" below), and creates video playback of the song. There are some fun interactive features in the real program, but here I just show pictures and video of the visualizer's basic operation.
I am visualizing three pieces of data: time, pitch, and timbre.
Time is mapped to the horizontal axis, just as you'd expect. The piece has been broken up into a series of irregularly-sized segments (that are usually about 0.1 to 0.3 seconds long), and those are the vertical slices that you see scrolling past.
Each one of those vertical slices is chopped up into twelve pitches along the vertical axis; these correspond to the twelve semitones in an octave (the chromatic scale). Along the left side of the screen, you can see that each of these chunks is labelled with the corresponding note on a scale. Note that all pitches are compressed into one octave, so notes that are an octave apart appear as the same pitch on the vertical axis. Another result of this is that, series of notes that "run off" the top of the visual space loop back around to the bottom of the space, and vice versa.
Each vertical slice is colored with an RGB value corresponding to the timbre of the sound in that segment. Timbre is what makes, say, a saxophone sound different than a trumpet. The hope here is that the color will capture enough of the variability so that this statement is true: The sounds that sound similar will appear similar in color. You can see from the demos that this works remarkably well sometimes, and is a little confusing at other times.
The image scrolls leftwards as the song plays.
The pitches are labelled on the left-hand side.
In the center of the screen is a white vertical line, which represents the current point in the piece. What appears to the left has already played, and what appears to the right is still to come. Under the white line, the pitches are just outlined boxes (instead of filled, like the rest) to aid you in picking out the notes being played. You only see a few seconds of the piece at a time.
Pictures of songs
The video shows playback for short segments of a small selection of songs. You can use the chapter menu in the controls to skip to the piece you're most interested in seeing. Or, you can just play all the way through!
- Chords as groups of horizontal bars. For example, the beginning of the "Moonlight" Sonatalooks like a piece in C-sharp minor (C-sharp is the second row from bottom), which it is:
- Changes in timbre/color. As the sound of the instruments changes, the color changes to match. A great example of this is in the second clip of Daft Punk's Crescendolls, after the first fade-out:
- Happy faces in Nirvana's Come As You Are:
Feel free to post comments on this project on my blog. Or, if you prefer, .