ze's blog :: zefrank.com
Ads Via The Deck



Via BuzzFeed


« for dinner: | Main | another idea »

June 2, 2008

a thesis idea

maybe this has been done - if not, it might be a fun media/interactive project/thesis for someone still in school? thinking about the distance between the written word and audio.

Fun project: develop a soundtrack to a book. Use eye tracking (would probably have to be a big book) to trigger the pace and positioning of an ambient score while the observer reads. Could try and create sound in emotional categories - assign them based on the artists experience but allow the observer to point out what are perceived mismatches.


Bookmark and Share
Comments (11)

I did something a little like this for an experimental type class.

We developed different graphemes (characters) for the english phonemes (sounds of letters, think ch, é, th, etc).

Check the site: www.livefire.org/exp

It's explained on the site, so go give it a look. Beware: the full program will bog down your computer. It's really CPU intensive.

Posted by: Mike Garrett at June 2, 2008 9:42 AM

I did something a little like this for an experimental type class.

We developed different graphemes (characters) for the english phonemes (sounds of letters, think ch, é, th, etc).

Check the site: www.livefire.org/exp

It's explained on the site, so go give it a look. Beware: the full program will bog down your computer. It's really CPU intensive.

Posted by: Mike Garrett at June 2, 2008 9:44 AM

That's a pretty good idea. I agree, too, in that ambient music would probably be the best choice (as anything "heavier" might be distracting to the reader). If you've ever listened to music by Patrick O'Hearn, certain songs of his just sound like they're making an environment for a story to be told.

Posted by: Josh at June 2, 2008 10:53 AM

I liike it! if you do the simple thing and make playback speed vary with reading rate, text with uneven vocabulary would go all warbly and drunken when sillly words like "antidisestablishmentarianism" are used.

Posted by: Richard Walker at June 2, 2008 10:56 AM

whoa. I, no joke, had this exact idea last night around 1am. I was working on a short story after drinking far too much coffee and was thinking of ways to link up an abrasive noise score/soundtrack to the eyes reading the story. I figured eye tracking would be too hard and that, given that the reader was reading an electronic version, i could track the position of viewable words on the screen and estimate at least what paragraph they might be on and play appropriate tracks.

weird, weird, weird coincidence - weirder because I haven't been to this site in months (though i used to be a theshow regular) and just happened to visit on a whim :)

Posted by: Nate at June 2, 2008 4:30 PM

I really like this idea.

Only have a problem with the emotional linkeage, as in: Does the pace of reading actually give away things like an emotional state? I am not expirienced enough with this, so I need to look it up, but on the first guess I would rather link the soundtrack to biophysical reactions, such as pulse and temperature or biochemical reactions, such as hormones, since they might be more accurate different giveaways, especially when it comes to emotions. As far as I know, emotional influence can be noticed in eye-tracking as the movement of the I stops, however it might be difficult to categorize.
I only know, that the speed of reading depends on the text and your familiarity with the words written (as we all know, we don't read out most of the words) and that you can fix on a word, if you don't know it and read each single letter.

And then the regression, if you don't understand a text at first glance might pose a problem in the flow of the music. What happens to the soundtrack, does it skip back, if you jump back a few lines?

Well, guess I would wanna be overly accurate with this... :/

Posted by: Liella at June 2, 2008 8:24 PM

Audio books do already to it to some extent and a few years ago I read of folks forging ahead with soundtracks to go with books. But they do not mix in my head that way. One eats the other even ambient. I would be very surprised to find any soundtrack that would not make a sit and read harder.

I would be interested in how a subject listening and reading did on a reading comprehension test afterward. I think there are some studies already about this though (with white noise and such). But book structure could be altered to reflect the new paragraphing that has developed due to the net and that in itself has a very lyrical page geography at times.

Posted by: boo at June 3, 2008 12:45 AM

It's an interesting idea, but if it were done, I would hate it to ever eventually become a big thing. Part of the art of writing, I think, is getting the desired response and evoking certain feelings by manipulating the language. That takes real skill.

Posted by: Floipoid at June 4, 2008 10:06 PM

Seems like a rudimentary verion of this could be done in HTML without the limiting tech of the eye-tracking (I know it's out there [the eye-tracking tech I mean--I remember reading about how ad companies were using it to learn about customer reading patterns, or check how someone glances over a sales display to learn about preferences]), but chances are it's not economically feasible for this kind of project. but with the HTML idea (or maybe Flash?) if you had a story in a webpage window, you could track readers' progress as they scroll down the side, and could potentially trigger different musical phrases/motifs at different points of the story. Graphics and interactivity obviously could enter in here too--if the reader did disagree with the choices, maybe certain words could be clickable, altering keys or modes or textures or mixes, etc.

Could be cool, but unless this was really integrated into the story, it could come off as gimmicky (like much of the typography play in prose and poetry has been--even now with the explosion of new fonts available to us all, most everyone prefers the basics [obviously fonts are a visual linkage with words, not a aural one, but I think the concept is similar enough that it raises good questions about the enterprise--is there something about the act of reading that makes trying to involve the other senses in that act a distraction or an intrusion? Or have we all been to trained to perform the act of reading a certain overly-rigid way, and fusing other experiences with it will help widen our horizons? ])

I've thought about putting together soundtracks for books or stories before (something that would sync up like "The Wall" and "The Wizard of Oz"), but something just never felt right about it. One of the beautiful things about reading is that the experience is only lightly guided--you get to do a lot of creation (acting, scene, music, etc.) while you follow the author's blueprint. and often in a very non-linear time-limited way: you can go back over a passage multiple times as you think of better ways for a character to deliver a line of dialogue, you can bring in complementary backgrounds and sounds (and even scents!) when a writer paints an evocative, picturesque scene, you can even score your own soundtrack to twist with every new emotion--and change it all on the fly or even multiple times, or remix it..whatever you wish. Literature is the least imposing art.

Posted by: Christian at June 5, 2008 12:12 AM

A simpler prototype would be have people to contribute itunes playlists where each song corresponds to a chapter of a book.

Posted by: Steve at June 6, 2008 10:47 AM

It might be practical to make something for a tool like the Kindle, where instead of eye tracking, you touch the screen wherever you're reading and an ambient piece mixes into the previous. You'd have to use looping and fading, which can certainly be done with relatively small mp3 files in a sort of mix-and-match way. I've composed pieces for live performance in just this way, and the files would fit on any of those tiny USB drives. If I had access to eye tracking technology, I'd turn in my thesis this week.

Posted by: Michael Andrew at June 6, 2008 10:48 PM

Post a comment




Remember Me?