Audio & Educational Technology

This has been a very enjoyable semester.  I feel like I’ve covered a lot of ground.  With Dr. Pennycook’s guidance, I’ve done a survey of the world of generative music–and even though I feel as if I’ve learned much, I also feel like I’ve just scratched the surface.   I think I’ve come to the place where the next step is one that takes the theories and knowledge provided by the people I’ve studied so far–namely Cope (and Hoffsteader), Clarke, (just a bit) of Lerdahl, and of course Dr. Pennycook, and venture into the exciting world of generative music technology through Max/MSP.

In addition to taking an experimental performance interfaces course, I will concurrently be doing a project for Dr. Liu’s multimedia class, where a team of us will be designing a micro-simulation using the Unity 3d game engine. Here’s a few questions I came up with to begin this exploration…

• How do we apply the principles of telling stories to designing with Max/MSP?
• How does Max/MSP work exactly?
• How do we build end-user control in generative music?
• How does Max/MSP take in end-user data from Unity?
• How will Max/MSP output music through Unity?
• How do we build sound design in Unity?
• How does Unity record and handle end-user data?
• What coding will need to be done through Unity to allow these systems to interact?
• In short, how will these 2 work and play together?

And yes, the reading will continue and I plan to take a sneak peak at these systems between now & the end of January…So, there’s more to come…

So, back to the question that I had a few days ago…“So, how can generative music be designed in a way that “makes sense” or even provides a “meaningful” experience for the learner?” Perhaps a way to begin an approach to this question is to see what it would take to generate ’stories’ for the learner that support their experience–that help to immerse them in a way that provides a stories that are appropriate to their learning context–and stories that are triggered by what the learners do…Will this attention to story-driven generative music provide for a “meaningful” learning experience?  I think it’s likely that this type of immersion can have an effect on the learner.  I also think this is a much deeper question, and one that will take more time and research to fully explore, as I think it takes us into the realm where music, cognition, and psychology intersect…

As is always the case, meetings with Dr. Pennycook are both interesting and informative…I had the chance today to follow-up on something he encouraged me to think about–the idea of ‘musical stories’.  Prior to this, I had several conceptions of what the idea of musical ’stories’ may might mean–and didn’t know which corresponded to the idea being discussed.

I had been used to thinking of a ’story’ as a sort of ‘package’–that is, a story as narrative having a beginning, a development, and an ending.  I’ve been used to conceiving of stories as being larger entities–passages that communicate to the listener a mood, emotional quality as told through music, and something that’s done over a somewhat developed passage of music.

However, I got the idea that there might be more to the ’story’ than that.  So, I took a couple of questions that I had to Dr. Pennycook–was a story a complete melody, or set of phrases? (I was still thinking structurally)  If so, how would the generative design of musical stories work?  A minimalist (cellular) approach to composition tells stories in a somewhat different way.  So, I wondered–to what extent should we rely on melodic-based approaches or minimalist approaches to composition in generative music?

Dr. Pennycook shed some light on both of these questions–a ’story’ is simply a “sequence of recognizable musical metaphors”.

Musical metaphors have been used throughout the history of music composition, and were ‘understood’ from a cultural perspective.  As I understand it, metaphors reflected what people expected to hear from the music of the day (a phenomenon which I think still continues even now).

Innovative composers played around with musical expectations, and in many cases defied them or even broke people’s expectations completely.  The example that comes to mind comes from a story that I heard from my music comp tutor (Dr. Chandler) about the first performance of Stravinsky’s ballet, the Rite of Spring.  Stravinsky’s composition was so groundbreaking & unexpected, people literally came unglued from their seats–in fact it caused a riot. (see the NPR interview)

So, back to stories–minimalist composition is somewhat different in that instead of metaphors in that this approach picks familiar “idioms”, then develops the h*** out of them. In any case, the idea of a story–which I’ve learned has historical roots in film composition, takes the intended emotional affect the music is to evoke and combines it with a topic.  It takes both these elements (affect+topic) to make up a story.

Therefore, it’s possible for generative music to be composed using a collection of algorithms that are topical in nature.  As Dr. Pennycook previously posited & wanted me to reflect on: “can we build interesting musical hierarchies using [David Cope's] ATN-like structures?” Well, yes–and in fact that’s how it’s done.  (an ATN, according to Cope is a transition network, based on Chomskian theories of transformational grammar that represents musical structure through a form of ‘generative grammar’.  This structure is comprised of interconnected “states”.  Although we don’t have to follow Chomskian grammar in generative music per-se, the idea of interconnected states is key, and is reflected when composing with Max/MSP.  I think the idea of this type of network is related to E. Clarke’s description of Connectionist Modeling, which according to Clark is a, ‘network of interlinked nodes with connections that can have variable values that represent strength and weight’.

Where’s the link?  Let me begin by revisiting the idea of musical style.  Cope has concretely established that it’s possible to reproduce musical style.  Pandora has demonstrated that it can cue into listener style.  What about someone who is playing the role of a learner?  Can we take the notion of ’style’ and apply it to learning? Can the notion of generative music be connected to the notion of choices that a learner makes?  What if we could identify key ‘decision points’ that identify with the idea of ‘progress’ through a learning event–and let the learner ‘create’ the sonic environment in which they learn?  What if we take this one step further–and build the music and audio in such a way that as the learner progresses through the learning experience they are given subtle cues and shifts in what they hear that impact their perception of the learning experience?

How can we begin to address these questions–and what can we use as a technical basis for this examination?

Lessons from Spore & Bloom

One important facet that perhaps sheds light on this question is our examination of Eno’s audio system in Spore & Bloom.  Spore is successful at providing the user with basic “control” over what they hear in key “building” stages of the game.  In a sense, the user–without musical knowledge–can create music.

The same is true in Bloom.  Bloom lets the listener create their own musical experience through the use of “moods” to provide an ambient musical bed, while provide different scalar modes that let them play patterns by touching the ipod or iphone’s screen.  The program then generates music based on these patterns–in much the same way that Spore generates music in the game.  In this sense, the user “perceives” they are controlling what they here–which is true–the patterns a user plays is incorporated by Bloom which continues to generates music with similar patterns.  However, one important facet here is that Bloom is providing a musical framework–a structure in which this happens.  When you “play” Bloom, it’s nearly impossible to create dissonance that doesn’t resolve, since everything is related harmonically.  In a sense, user choice drives what they hear–and what they hear makes “sense”.

So, how can generative music be designed in a way that “makes sense” or even provides a “meaningful” experience for the learner?

…So with all this attention to technologies that ‘revolve’ around or have otherwise ‘attempted’ to incorporate some form of style-based automated composition into a technology product, what have I learned?  And, how does this relate to educational technology?  These are two questions that I started with–hoping that these questions would lead to deeper questions of how to pursue a path that could lead to a project–and ultimately help uncover research questions that could guide me through a dissertation.

Again, where does this leave me?  What have I learned about style-based auto-comp. as has been produced in industry?  The products produced are promising in that:

• There are several products that have recently come to market that incorporate generative music–that is music that changes dynamically over time.  Examples of these include:  iphone apps such as Bloom!, composition tools such as Noatikl, and generative audio in game environments such as Spore.
• Although these generative music applications are interesting, to what extent do they relate to style?  This is a more complicated answer.  I’ve read a good chunk of David Cope’s first book, ‘Computers and Musical Style’, and I’ve also gotten through the first part of another book of his, ‘Virtual Music…Computer Synthesis of Musical Style’.  According to Cope, musical style has a structure that be related to linguistics.  Cope’s SPEAC system (a hierarchical analysis consisting of a: statement, preparation,extension, antecedent, & consequent) gives the ability to analyze (and with his system) generate music in a way that can adapt to musical “styles”.  Along with Cope’s notion of augmented transition networks (ATN’s), which are related to–and to some degree dependent upon Chomsky’s notion of Transformational Grammar, Cope has been successful in designing a system (EMI : Experiments in Musical Intelligence) that can reproduce musical styles to the likes of the greats–Bach, Chopin, Rachmaninoff, and the like…

Musical Style…

“Styles” in Cope’s view are also described through attention to musical ’signatures’, ‘tension-resolution’ logic, ’semantic meshing’ (a series of nested musical contexts that range from local notes & measures to phrases, periods & large-scale sections), and final musical ‘recombination’ (which theoretically allows for recombination of discrete passages of music that can be recombined in ways that are still stylistically “viable”, according to Cope).

But these issues of style uncover other questions:  Cope’s work focuses on reproducing ’styles’ of different composers.  How can these systems support generation of music to match the style of someone interacting with a system–can someone listening to music participate in the construction of what they hear?  Well, Pandora makes an attempt at doing this.  Pandora doesn’t generate music, but it does try to give the listener what they want to hear.  Here’s my theory on this–which is based on connectionist modeling concepts described in Eric Clarke’s book, “Ways of Listening”, and described in the first chapter of the book–Perception, Ecology, and Music.

• Here, Clarke provides a background for connectionist modeling from McLelland’s: Parallel Distributed Processing (Rumelhart and McLelland 1986) and describes a theoretical experiment that describes the ability of a ‘network’ to learn (based on programming & guided input) the musical preferences of listeners (who are listening to discrete melodies) and be theoretically be able to predict and classify new melodies using the preferences on which the model is based.  (I’m oversimplifying Clarke’s example, sorry Clarke).  However, he describes this process of predictability as hinging on assignment of “connection weights between ‘input units’ (basic definition rules/conditions–in this case intervals between notes in each melody) and ‘output units’ (’like or dislike’ of the listener)
• My theory is that Pandora is based on exactly this system of analysis of ‘connection weights’ that use listener ‘voting’ to determine how well Pandora ‘understands’ listener preference–e.g. “style” and is therefore able to predict with a degree of probability whether the listener will like what comes up next.  If this is true–then can we say this way of matching listener ‘preference’ is related to the idea of musical ’style’?

Next question:

How do these ideas of generative music as it relates to style tie back to design of educational technology?

I believe that a discussion about musical “style” would be incomplete without at least a mention of Pandora.  Pandora comes to us as a result of the Music Genome Project, by Tim Westergren.   Pandora comes to us as a result of musical analysis of “tens of thousands” of artists.  As Tim describes it, they analyzed,

“the musical qualities of each song one attribute at a time. This work continues each and every day as we endeavor to include all the great new stuff coming out of studios, clubs and garages around the world.”

Tim goes to describe how their team attempted to describe music according to unique “attributes” that they call “genes”.

“We ended up assembling literally hundreds of musical attributes or “genes” into a very large Music Genome. Taken together these genes capture the unique and magical musical identity of a song – everything from melody, harmony and rhythm, to instrumentation, orchestration, arrangement, lyrics, and of course the rich world of singing and vocal harmony.”

Professor Pennycook & I were discussing the mechanism they likely used to do this, which almost certainly involves meta-data rather than musical pattern-matching.   Pandora lets users create stations based on their favorite songs or artists, and will deliver songs/artists that match the attributes of the artist/song used to create the station.  You can also vote in favor of the selection played or against it.  In this way, you can “train” Pandora to match the songs that you think the station should play.  Pandora will actually send you an email reminding you to do this periodically.  One quesiton we had is whether our votes influenced the way the database tags the music.  One thing that we found curious was why in some cases when we created a station, the first song that plays is often not by the artist for which the station was named.  This happened with Dr. Pennycook with Mahler and with me when creating a station for Beethoven.

In any case, Pandora has clearly been designed around a taxonomy of musical attributes that are, at least, populated by its creators.  My question is–does this taxonomy also imply a “folksonomy” (re: Thomas Vanderwal)?

Does this represent “generative music”?  No–but perhaps it’s another step in a direction to consider what it means to adapt to listener “style”.  I use Pandora regularly, and think it’s tres cool.  Frankly, I’ve heard some people describe it as a ‘killer app’‘.  This is debatable, but I do know that Pandora & their “musical genome project” is significant in that it’s the closest app that I’ve found that tries to give me my tunes, my way.

Well, I took the next step in my exploration of experimental performance interfaces & Pd and decided to install Pd for Windows & tried the Wiimote.  I spent some time looking for a PC Wii to midi driver and eventually found one that worked called Glovepie.  I also installed midiYoke as a midi “patch bay” driver that links everything together.  I then used Pd to receive & process the midi data from the Wiimote.

In the process, I took a script for an oscillator that came bundled with Pd & modified it to accept midi channel data, using a piece of the test script that also comes bundled with Pd.   I then had to use & modify a script in Glovepie to have a different starting note value & made sure that Pd used midi yoke (channel 2) as an input device.   I’m only succeeding in making the wiimote increment & decrement pitches in a *very* rough way.  The process is akin, I think, to learning how to steer a tank.   However, it does seem to work.

I captured a short video with my phone so that you can see it for yourself.  this time I exported the movie as a .mov & posted it  @ http://www.adaptablearts.com/media/jason-wii-pd.mov .

Here’s the process that I used:

I recently attended a workshop on iphone app development.  In light of that presentation and our research of generative music in Spore, I thought this new iphone app developed by Brian Eno called “Bloom” was interesting.  I downloaded the app for the iTouch (I’m on Verizon, so I don’t use the iPhone).

http://createdigitalmusic.com/2008/10/08/hands-on-with-bloom-new-generative-iphone-app-by-eno-and-chilvers/

Here’s a Youtube clip that demos the app

The music the app produced was interesting–amazingly it sounded a *lot* like the stuff in Spore..  The app lets you either set a random setting to play music based on some basic choices, or you can interact by tapping the screen to make music.  Tapping the bottom of the screen generates lower pitches, while tapping higher generates higher pitches.

Some points:

• Bloom has a 4 octave range
• There are 9 different “moods” that can be selected to provide a different textural experience
• Each “mood” changes the tonic of the scale that plays.  One “mood” may have a range from C2 – C6, while another mood ranges from G2 – G6, etc.
• When you touch the iTouch, it generates pitches that then repeats a pattern using a delay.  There’s a setting called “evolve when idle” that I think takes a pattern & develops it so that something plays when you’re not.
• For example, the mood “Neroli” is in the key of G, and I’m pretty sure in a mixolydian mode.  The lower areas of the touch pad isn’t as responsive to changes in pitch as the middle & upper registers.

Listen to a sample of me playing the iTouch with the  “Neroli” mood!

• You’ll hear ambient background stuff that Bloom generates.  Also, the application was set to randomize the “mood” during playback (not sure what this does), and the delay was set a pretty short interval
• The first part of the clip is me trying to play a scale from the bottom up.  You can hear me playing the same notes even as I’m moving up the touchpad.  I actually moved from I-V-I for the first octave before moving up the scale.
• The later part of the clip is me “playing” the app.  Funny thing, last weekend I was in the middle of my music composition lesson when my tutor & I got into a jam session with Bloom.  He was on the guitar & we were just doing some ambient stuff.  I’ve *never* jammed with an ipod before.  That was a first.

Something arguably even more interesting is info on an iphone app called RjDj : http://www.rjdj.me .  It uses Pd and the iphone accelerometer & mic to sample audio.  Since I don’t have an iPhone, I can’t try this myself, but more info can be found here on this blog:
http://createdigitalmusic.com/2008/09/02/iphonetouch-roundup-control-art-snow-patrol-visualizers-recording-one-for-india/ .