Audio & Educational Technology

This has been a very enjoyable semester.  I feel like I’ve covered a lot of ground.  With Dr. Pennycook’s guidance, I’ve done a survey of the world of generative music–and even though I feel as if I’ve learned much, I also feel like I’ve just scratched the surface.   I think I’ve come to the place where the next step is one that takes the theories and knowledge provided by the people I’ve studied so far–namely Cope (and Hoffsteader), Clarke, (just a bit) of Lerdahl, and of course Dr. Pennycook, and venture into the exciting world of generative music technology through Max/MSP.

In addition to taking an experimental performance interfaces course, I will concurrently be doing a project for Dr. Liu’s multimedia class, where a team of us will be designing a micro-simulation using the Unity 3d game engine. Here’s a few questions I came up with to begin this exploration…

• How do we apply the principles of telling stories to designing with Max/MSP?
• How does Max/MSP work exactly?
• How do we build end-user control in generative music?
• How does Max/MSP take in end-user data from Unity?
• How will Max/MSP output music through Unity?
• How do we build sound design in Unity?
• How does Unity record and handle end-user data?
• What coding will need to be done through Unity to allow these systems to interact?
• In short, how will these 2 work and play together?

And yes, the reading will continue and I plan to take a sneak peak at these systems between now & the end of January…So, there’s more to come…

So, back to the question that I had a few days ago…“So, how can generative music be designed in a way that “makes sense” or even provides a “meaningful” experience for the learner?” Perhaps a way to begin an approach to this question is to see what it would take to generate ’stories’ for the learner that support their experience–that help to immerse them in a way that provides a stories that are appropriate to their learning context–and stories that are triggered by what the learners do…Will this attention to story-driven generative music provide for a “meaningful” learning experience?  I think it’s likely that this type of immersion can have an effect on the learner.  I also think this is a much deeper question, and one that will take more time and research to fully explore, as I think it takes us into the realm where music, cognition, and psychology intersect…

As is always the case, meetings with Dr. Pennycook are both interesting and informative…I had the chance today to follow-up on something he encouraged me to think about–the idea of ‘musical stories’.  Prior to this, I had several conceptions of what the idea of musical ’stories’ may might mean–and didn’t know which corresponded to the idea being discussed.

I had been used to thinking of a ’story’ as a sort of ‘package’–that is, a story as narrative having a beginning, a development, and an ending.  I’ve been used to conceiving of stories as being larger entities–passages that communicate to the listener a mood, emotional quality as told through music, and something that’s done over a somewhat developed passage of music.

However, I got the idea that there might be more to the ’story’ than that.  So, I took a couple of questions that I had to Dr. Pennycook–was a story a complete melody, or set of phrases? (I was still thinking structurally)  If so, how would the generative design of musical stories work?  A minimalist (cellular) approach to composition tells stories in a somewhat different way.  So, I wondered–to what extent should we rely on melodic-based approaches or minimalist approaches to composition in generative music?

Dr. Pennycook shed some light on both of these questions–a ’story’ is simply a “sequence of recognizable musical metaphors”.

Musical metaphors have been used throughout the history of music composition, and were ‘understood’ from a cultural perspective.  As I understand it, metaphors reflected what people expected to hear from the music of the day (a phenomenon which I think still continues even now).

Innovative composers played around with musical expectations, and in many cases defied them or even broke people’s expectations completely.  The example that comes to mind comes from a story that I heard from my music comp tutor (Dr. Chandler) about the first performance of Stravinsky’s ballet, the Rite of Spring.  Stravinsky’s composition was so groundbreaking & unexpected, people literally came unglued from their seats–in fact it caused a riot. (see the NPR interview)

So, back to stories–minimalist composition is somewhat different in that instead of metaphors in that this approach picks familiar “idioms”, then develops the h*** out of them. In any case, the idea of a story–which I’ve learned has historical roots in film composition, takes the intended emotional affect the music is to evoke and combines it with a topic.  It takes both these elements (affect+topic) to make up a story.

Therefore, it’s possible for generative music to be composed using a collection of algorithms that are topical in nature.  As Dr. Pennycook previously posited & wanted me to reflect on: “can we build interesting musical hierarchies using [David Cope's] ATN-like structures?” Well, yes–and in fact that’s how it’s done.  (an ATN, according to Cope is a transition network, based on Chomskian theories of transformational grammar that represents musical structure through a form of ‘generative grammar’.  This structure is comprised of interconnected “states”.  Although we don’t have to follow Chomskian grammar in generative music per-se, the idea of interconnected states is key, and is reflected when composing with Max/MSP.  I think the idea of this type of network is related to E. Clarke’s description of Connectionist Modeling, which according to Clark is a, ‘network of interlinked nodes with connections that can have variable values that represent strength and weight’.

Where’s the link?  Let me begin by revisiting the idea of musical style.  Cope has concretely established that it’s possible to reproduce musical style.  Pandora has demonstrated that it can cue into listener style.  What about someone who is playing the role of a learner?  Can we take the notion of ’style’ and apply it to learning? Can the notion of generative music be connected to the notion of choices that a learner makes?  What if we could identify key ‘decision points’ that identify with the idea of ‘progress’ through a learning event–and let the learner ‘create’ the sonic environment in which they learn?  What if we take this one step further–and build the music and audio in such a way that as the learner progresses through the learning experience they are given subtle cues and shifts in what they hear that impact their perception of the learning experience?

How can we begin to address these questions–and what can we use as a technical basis for this examination?

Lessons from Spore & Bloom

One important facet that perhaps sheds light on this question is our examination of Eno’s audio system in Spore & Bloom.  Spore is successful at providing the user with basic “control” over what they hear in key “building” stages of the game.  In a sense, the user–without musical knowledge–can create music.

The same is true in Bloom.  Bloom lets the listener create their own musical experience through the use of “moods” to provide an ambient musical bed, while provide different scalar modes that let them play patterns by touching the ipod or iphone’s screen.  The program then generates music based on these patterns–in much the same way that Spore generates music in the game.  In this sense, the user “perceives” they are controlling what they here–which is true–the patterns a user plays is incorporated by Bloom which continues to generates music with similar patterns.  However, one important facet here is that Bloom is providing a musical framework–a structure in which this happens.  When you “play” Bloom, it’s nearly impossible to create dissonance that doesn’t resolve, since everything is related harmonically.  In a sense, user choice drives what they hear–and what they hear makes “sense”.

So, how can generative music be designed in a way that “makes sense” or even provides a “meaningful” experience for the learner?

…So with all this attention to technologies that ‘revolve’ around or have otherwise ‘attempted’ to incorporate some form of style-based automated composition into a technology product, what have I learned?  And, how does this relate to educational technology?  These are two questions that I started with–hoping that these questions would lead to deeper questions of how to pursue a path that could lead to a project–and ultimately help uncover research questions that could guide me through a dissertation.

Again, where does this leave me?  What have I learned about style-based auto-comp. as has been produced in industry?  The products produced are promising in that:

• There are several products that have recently come to market that incorporate generative music–that is music that changes dynamically over time.  Examples of these include:  iphone apps such as Bloom!, composition tools such as Noatikl, and generative audio in game environments such as Spore.
• Although these generative music applications are interesting, to what extent do they relate to style?  This is a more complicated answer.  I’ve read a good chunk of David Cope’s first book, ‘Computers and Musical Style’, and I’ve also gotten through the first part of another book of his, ‘Virtual Music…Computer Synthesis of Musical Style’.  According to Cope, musical style has a structure that be related to linguistics.  Cope’s SPEAC system (a hierarchical analysis consisting of a: statement, preparation,extension, antecedent, & consequent) gives the ability to analyze (and with his system) generate music in a way that can adapt to musical “styles”.  Along with Cope’s notion of augmented transition networks (ATN’s), which are related to–and to some degree dependent upon Chomsky’s notion of Transformational Grammar, Cope has been successful in designing a system (EMI : Experiments in Musical Intelligence) that can reproduce musical styles to the likes of the greats–Bach, Chopin, Rachmaninoff, and the like…

Musical Style…

“Styles” in Cope’s view are also described through attention to musical ’signatures’, ‘tension-resolution’ logic, ’semantic meshing’ (a series of nested musical contexts that range from local notes & measures to phrases, periods & large-scale sections), and final musical ‘recombination’ (which theoretically allows for recombination of discrete passages of music that can be recombined in ways that are still stylistically “viable”, according to Cope).

But these issues of style uncover other questions:  Cope’s work focuses on reproducing ’styles’ of different composers.  How can these systems support generation of music to match the style of someone interacting with a system–can someone listening to music participate in the construction of what they hear?  Well, Pandora makes an attempt at doing this.  Pandora doesn’t generate music, but it does try to give the listener what they want to hear.  Here’s my theory on this–which is based on connectionist modeling concepts described in Eric Clarke’s book, “Ways of Listening”, and described in the first chapter of the book–Perception, Ecology, and Music.

• Here, Clarke provides a background for connectionist modeling from McLelland’s: Parallel Distributed Processing (Rumelhart and McLelland 1986) and describes a theoretical experiment that describes the ability of a ‘network’ to learn (based on programming & guided input) the musical preferences of listeners (who are listening to discrete melodies) and be theoretically be able to predict and classify new melodies using the preferences on which the model is based.  (I’m oversimplifying Clarke’s example, sorry Clarke).  However, he describes this process of predictability as hinging on assignment of “connection weights between ‘input units’ (basic definition rules/conditions–in this case intervals between notes in each melody) and ‘output units’ (’like or dislike’ of the listener)
• My theory is that Pandora is based on exactly this system of analysis of ‘connection weights’ that use listener ‘voting’ to determine how well Pandora ‘understands’ listener preference–e.g. “style” and is therefore able to predict with a degree of probability whether the listener will like what comes up next.  If this is true–then can we say this way of matching listener ‘preference’ is related to the idea of musical ’style’?

Next question:

How do these ideas of generative music as it relates to style tie back to design of educational technology?

I recently attended a workshop on iphone app development.  In light of that presentation and our research of generative music in Spore, I thought this new iphone app developed by Brian Eno called “Bloom” was interesting.  I downloaded the app for the iTouch (I’m on Verizon, so I don’t use the iPhone).

http://createdigitalmusic.com/2008/10/08/hands-on-with-bloom-new-generative-iphone-app-by-eno-and-chilvers/

Here’s a Youtube clip that demos the app

The music the app produced was interesting–amazingly it sounded a *lot* like the stuff in Spore..  The app lets you either set a random setting to play music based on some basic choices, or you can interact by tapping the screen to make music.  Tapping the bottom of the screen generates lower pitches, while tapping higher generates higher pitches.

Some points:

• Bloom has a 4 octave range
• There are 9 different “moods” that can be selected to provide a different textural experience
• Each “mood” changes the tonic of the scale that plays.  One “mood” may have a range from C2 – C6, while another mood ranges from G2 – G6, etc.
• When you touch the iTouch, it generates pitches that then repeats a pattern using a delay.  There’s a setting called “evolve when idle” that I think takes a pattern & develops it so that something plays when you’re not.
• For example, the mood “Neroli” is in the key of G, and I’m pretty sure in a mixolydian mode.  The lower areas of the touch pad isn’t as responsive to changes in pitch as the middle & upper registers.

Listen to a sample of me playing the iTouch with the  “Neroli” mood!

• You’ll hear ambient background stuff that Bloom generates.  Also, the application was set to randomize the “mood” during playback (not sure what this does), and the delay was set a pretty short interval
• The first part of the clip is me trying to play a scale from the bottom up.  You can hear me playing the same notes even as I’m moving up the touchpad.  I actually moved from I-V-I for the first octave before moving up the scale.
• The later part of the clip is me “playing” the app.  Funny thing, last weekend I was in the middle of my music composition lesson when my tutor & I got into a jam session with Bloom.  He was on the guitar & we were just doing some ambient stuff.  I’ve *never* jammed with an ipod before.  That was a first.

Something arguably even more interesting is info on an iphone app called RjDj : http://www.rjdj.me .  It uses Pd and the iphone accelerometer & mic to sample audio.  Since I don’t have an iPhone, I can’t try this myself, but more info can be found here on this blog:
http://createdigitalmusic.com/2008/09/02/iphonetouch-roundup-control-art-snow-patrol-visualizers-recording-one-for-india/ .

First, let me say that the music design behind Spore is as impressive as the game itself. For the sake of educational research for my class, I recorded the audio that played during each of the three processes.

Why not record video? Unfortunately, although Spore allows recorded video, it only does this at very specific stages of the game, such as ‘testing’ a creature once created, and screencapture software cannot sufficiently capture the video, since the video doesn’t play in a ‘window’ on the screen.

However, because I was able to use Cubase to record the audio, I was able to easily compare the audio on 3 different tracks–and in one instance I played a selection of audio from all 3 tracks simultaneously and was pleasantly surprised to hear overlapping harmony in all 3 tracks, even as the music played was subtly different.

A few broad points to start, based on my initial testing:

1. Spore plays ‘dynamic’ music at key points in the game that include: the main menu, creature selection from Sporepedia, creature editor, randomizing creatures, etc.

2. Spore appears to play the same music during: the animation sequences that open the cellular & creature stages, and as background loops during the cellular organism stage.

3. The experience of ‘dynamic music’ can be described in each stage in the following ways:

a. Each key stage (i.e. main menu, creature selector, etc.) tends to have a distinct musical ’style’.
b. If I use Cope’s definition of musical style as including:

“the identifiable characteristics of a composer’s music which are recognizably similar from one work to another. These include, but are not limited to, pitch and duration…timbre, dynamics and nuance. “

then I would characterize the styles represented in each key stage in Spore as having a distinct and recognizable: tempo, density (instrumentation), and a trend towards similar harmonies & rhythms (these can be congruent in successive plays, are sometimes transposed, and sometimes shift between instrumental parts–again, these seem to me to be ‘tendencies’, not hard/fast rules).

c.  Upon successive playing of the audio from each ’stage’, and as a result of the same menu choices & user interactions, it appears that “style” can be found in listening to the repetitive variations of musical ‘fragments’.  How these fragments are generated (e.g. single pitches, phrases, loops, etc.) are a mystery to me.

Well, I’ve spent the last week playing Spore repeatedly in an attempt to analyze the procedural music generated by Spore.  The game is organized according to several different stages–life as a cell, life as a creature, life in a tribe, etc.  You start the game as a cellular organism that has to eat to survive–or be eaten & perish.

Along the way, you get to ‘customize’ your organism, ‘mate’ with others from the same ’species’, and eventually ‘evolve’.  From a musical point of view, there are several key ‘hitpoints’ worth noting.  My analysis of Spore necessitated playing the game long enough to be able to restart a new creation at the ‘creature’ level, and figure out a sequence of duplicable steps to test the results of the audio as initiated by my choices in the menus.

Since there are an enormous amount of combinations of choices that one can make (especially at the creature stage), I decided to fix a single path of interactivity from the very first menu selection.  Here’s a summary of the overall process that I followed several times in succession:

1.  Choose an empty planet; select a new ‘creature’ stage to begin.

2.  Use the creature selector to choose a creature from Sporepedia (the online gallery).

3.  Select the ‘Krinkut Maxis’ creature because of its flexibility and attributes.

4.  Select the ‘easy’ difficulty setting.

5.  Choose the default planet name that appears (this is randomly generated by Spore).

6.  Spore now randomizes the creatures that I will encounter in the game.

7.  The game environment appears.  Follow the beginning ‘entry pop-up’s’ that appear.

8.  Feed the creature by letting it eat fruit from the closest tree.

9.  Head in a Northeasterly direction & interact with the first set of creatures encountered.

10.  “Sing” to the creatures to “Impress” them.  Earn DNA points to spend on enhancements.

11.  Head back to the “home nest” to mate.  This brings up the creature editor. (the idea is that you can ‘evolve’ if you have enough DNA points to spend on an enhancement.

12.  Don’t make any changes to the creature, and exit the creature editor.

13.  The creature will automatically get the ability to ‘charm’ at a higher level.  Practice this ability by following the prompt.

14.  Exit the game.