Paul Lemare, who blogs about music and technology, breaks down the Instant Mix feature on Google Music, which is similar to Apple’s Genius feature.
Evaluating playlists is hard. However, there is something that we can do that is fairly easy to give us an idea of how well a playlisting engine works compared to others. I call it the WTF test. It is really quite simple. You generate a playlist, and just count the number of head-scratchers in the list. If you look at a song in a playlist and say to yourself ‘How the heck did this song get in this playlist’ you bump the counter for the playlist. The higher the WTF count the worse the playlist. As a first order quality metric, I really like the WTF Test. It is easy to apply, and focuses on a critical aspect of playlist quality. If a playlist is filled with jarring transitions, leaving the listener with iPod whiplash as they are jerked through songs of vastly different styles, it is a bad playlist.
By this metric, Instant Mix is not so good at the moment. Much more inferior to iTunes and Echo Nest. Obviously, these features are only as good as the music in your personal collection. As someone with close to 20,000 songs (as compared to the 7,800 in Lemare’s collection) one metric that is important for playlist building aside from the WTF count is the list’s eclecticism. iTunes fails spectacularly when building Genius playlists because when you pick a song the same four or five artists are consistently being recycled into the playlist. It gets annoying really quickly.
Like all things Google, the more it learns your habits, the more it can tailor its results. I wonder how different this feature will become the longer you use it and the longer they refine the algorithms.