Over Christmas I sat under a blanket coding up a new prototype involving HTML5, Javascript, classical music and wintery landscapes. It’s called Orchestrated Text.

Annotating audio

Before the Christmas break I’d been messing about with some code for a personal project, and it was working out really well. I was working through Christmas and, since my active projects paused for the break, I wanted a new idea/prototype to work on to show everyone when we regrouped in the New Year.

Orchestrated Text

The concept: to take the brilliant content found in the liner notes of many classical music CDs, and create a digital product that deepens the experience of streaming music.

Users click on a piece to play it, and read along as text appears on screen detailing what’s happening in the music, what it depicts, and what the composer was inspired by.

View the working prototype here.

Audio and Text without Video

I’d been looking for a way to make things happen on a screen along with music, but do it in-browser and not have to faff around making videos. I started to do javascript animations with text, with music running in the background of the page. This worked fine when developing locally, but as soon as I put it on a server the animations would load instantly while the music buffered and everything would be out of sync.

So I started hunting around the web for alternatives and found this. It’s a nice thing inspired by Portal 2, using a song by The National that’s hidden in the game.

In a nutshell, the page loads the song, and each piece of text has a Time and an Animation Instruction. When the audio player reaches that time, the browser renders the text in the way specified by the Animation Instruction.

So I grabbed the code and delved into it and started to re-write it for my own ends.

Classical Music & Stories

Tony with his neck axe.

Earlier in December I’d been working on a pitch for a classical music label that utilised the amazing content you get in the liner notes of classical CDs and vinyl. Personally, I wouldn’t have become as attached to some pieces of classical music without the stories told in these booklets. They are stories of culture, ideas, inspiration, and references.

My Dad has a huge collection of classical CDs, and I asked him what the best liner notes were. He gave me a copy of Vivaldi’s The Four Seasons. It’s a book with a CD in it, released by Deutsche Grammophon as part of their enlightening annotated books called ‘La Gran Musica’.

For each piece on the cd there’s a section in the book. Within each section are paragraphs of descriptive text telling you about the instruments and the orchestra and the type of things they’re playing. The imagery of it. The best part is that these paragraphs have times on them, so you know to listen out for what the first violin is doing at 02:13. It’s perfect for the classical music beginner, interesting for the enthusiast, and validating for the expert.

Movement

So it’s Christmas and there’s a certain kind of peace to the weekdays. Time stops moving in its usual way for some people, but for everyone there’s one thing the same – it’s winter. So I picked the first movement of ‘Winter’ from The Four Seasons to work with, to create a digital version of the booklet that helps people see further into the music.

Planning timings and animations

Over the course of the day I did these things:

  • Import the mp3 of the track into a wave editor so I could get precise timings of when things should come in
  • Write a script to accompany the audio based on the content of the book my Dad gave me
  • Split the script into sections according to timings
  • Add the script into the code
  • Add the timings into the code
  • Run through the piece of music about thirty times, watching how each line animates and making adjustments to its style and time of appearance.

Wrap It Up

We’re calling the project Orchestrated Text, and you can view it here. Clear your schedule for three minutes, get some headphones on, have a look, and let me know what you think.

The design is very basic at the moment. It’s all about the music and words for now. So it won’t look quite right on your mobile, but on tablet and bigger screens it works fine.

I’d like to do the same for more pieces. I’m thinking Chopin’s Raindrop Prelude, Beethoven’s Pastoral Symphony (2nd Movement) or Carnival of the Animals by Saint-Saens. Pieces rich with imagery that can be teased out for people to see. What kind of people? That’s what I hope to find out from this day on. I’d like to hear what kids think of it, what adults who don’t like classical music think of it. The same for adults who do like classical music, or really really do.

But, first and foremost, I hope that someone somewhere hears a familiar piece of music and learns something new about it that inspires them to listen further.

Happy Wintertime!

Comments for this entry

Emma

In todays digital music world it’s easy to feel disconnected from music, This is such a lovely simple way of deepening the experience and taking the time out – Good job!

[...] Birkin, von dem die Idee stammt, schreibt dazu in einem Blogpost: “Over Christmas I sat under a blanket coding up a new prototype involving HTML5, Javascript, [...]

This was amazing! So clean and simple, but so powerful. I can’t wait to see more pieces — I like the idea of the Carnival of the Animals. Peter and the Wolf is another possibility… maybe too obvious. Or the Moldau by Smetana.

Leave your comment