Audio & Educational Technology

As is always the case, meetings with Dr. Pennycook are both interesting and informative…I had the chance today to follow-up on something he encouraged me to think about–the idea of ‘musical stories’.  Prior to this, I had several conceptions of what the idea of musical ’stories’ may might mean–and didn’t know which corresponded to the idea being discussed.

I had been used to thinking of a ’story’ as a sort of ‘package’–that is, a story as narrative having a beginning, a development, and an ending.  I’ve been used to conceiving of stories as being larger entities–passages that communicate to the listener a mood, emotional quality as told through music, and something that’s done over a somewhat developed passage of music.

However, I got the idea that there might be more to the ’story’ than that.  So, I took a couple of questions that I had to Dr. Pennycook–was a story a complete melody, or set of phrases? (I was still thinking structurally)  If so, how would the generative design of musical stories work?  A minimalist (cellular) approach to composition tells stories in a somewhat different way.  So, I wondered–to what extent should we rely on melodic-based approaches or minimalist approaches to composition in generative music?

Dr. Pennycook shed some light on both of these questions–a ’story’ is simply a “sequence of recognizable musical metaphors”.

Musical metaphors have been used throughout the history of music composition, and were ‘understood’ from a cultural perspective.  As I understand it, metaphors reflected what people expected to hear from the music of the day (a phenomenon which I think still continues even now).

Innovative composers played around with musical expectations, and in many cases defied them or even broke people’s expectations completely.  The example that comes to mind comes from a story that I heard from my music comp tutor (Dr. Chandler) about the first performance of Stravinsky’s ballet, the Rite of Spring.  Stravinsky’s composition was so groundbreaking & unexpected, people literally came unglued from their seats–in fact it caused a riot. (see the NPR interview)

So, back to stories–minimalist composition is somewhat different in that instead of metaphors in that this approach picks familiar “idioms”, then develops the h*** out of them. In any case, the idea of a story–which I’ve learned has historical roots in film composition, takes the intended emotional affect the music is to evoke and combines it with a topic.  It takes both these elements (affect+topic) to make up a story.

Therefore, it’s possible for generative music to be composed using a collection of algorithms that are topical in nature.  As Dr. Pennycook previously posited & wanted me to reflect on: “can we build interesting musical hierarchies using [David Cope's] ATN-like structures?” Well, yes–and in fact that’s how it’s done.  (an ATN, according to Cope is a transition network, based on Chomskian theories of transformational grammar that represents musical structure through a form of ‘generative grammar’.  This structure is comprised of interconnected “states”.  Although we don’t have to follow Chomskian grammar in generative music per-se, the idea of interconnected states is key, and is reflected when composing with Max/MSP.  I think the idea of this type of network is related to E. Clarke’s description of Connectionist Modeling, which according to Clark is a, ‘network of interlinked nodes with connections that can have variable values that represent strength and weight’.

Where’s the link?  Let me begin by revisiting the idea of musical style.  Cope has concretely established that it’s possible to reproduce musical style.  Pandora has demonstrated that it can cue into listener style.  What about someone who is playing the role of a learner?  Can we take the notion of ’style’ and apply it to learning? Can the notion of generative music be connected to the notion of choices that a learner makes?  What if we could identify key ‘decision points’ that identify with the idea of ‘progress’ through a learning event–and let the learner ‘create’ the sonic environment in which they learn?  What if we take this one step further–and build the music and audio in such a way that as the learner progresses through the learning experience they are given subtle cues and shifts in what they hear that impact their perception of the learning experience?

How can we begin to address these questions–and what can we use as a technical basis for this examination?

Lessons from Spore & Bloom

One important facet that perhaps sheds light on this question is our examination of Eno’s audio system in Spore & Bloom.  Spore is successful at providing the user with basic “control” over what they hear in key “building” stages of the game.  In a sense, the user–without musical knowledge–can create music.

The same is true in Bloom.  Bloom lets the listener create their own musical experience through the use of “moods” to provide an ambient musical bed, while provide different scalar modes that let them play patterns by touching the ipod or iphone’s screen.  The program then generates music based on these patterns–in much the same way that Spore generates music in the game.  In this sense, the user “perceives” they are controlling what they here–which is true–the patterns a user plays is incorporated by Bloom which continues to generates music with similar patterns.  However, one important facet here is that Bloom is providing a musical framework–a structure in which this happens.  When you “play” Bloom, it’s nearly impossible to create dissonance that doesn’t resolve, since everything is related harmonically.  In a sense, user choice drives what they hear–and what they hear makes “sense”.

So, how can generative music be designed in a way that “makes sense” or even provides a “meaningful” experience for the learner?

…So with all this attention to technologies that ‘revolve’ around or have otherwise ‘attempted’ to incorporate some form of style-based automated composition into a technology product, what have I learned?  And, how does this relate to educational technology?  These are two questions that I started with–hoping that these questions would lead to deeper questions of how to pursue a path that could lead to a project–and ultimately help uncover research questions that could guide me through a dissertation.

Again, where does this leave me?  What have I learned about style-based auto-comp. as has been produced in industry?  The products produced are promising in that:

• There are several products that have recently come to market that incorporate generative music–that is music that changes dynamically over time.  Examples of these include:  iphone apps such as Bloom!, composition tools such as Noatikl, and generative audio in game environments such as Spore.
• Although these generative music applications are interesting, to what extent do they relate to style?  This is a more complicated answer.  I’ve read a good chunk of David Cope’s first book, ‘Computers and Musical Style’, and I’ve also gotten through the first part of another book of his, ‘Virtual Music…Computer Synthesis of Musical Style’.  According to Cope, musical style has a structure that be related to linguistics.  Cope’s SPEAC system (a hierarchical analysis consisting of a: statement, preparation,extension, antecedent, & consequent) gives the ability to analyze (and with his system) generate music in a way that can adapt to musical “styles”.  Along with Cope’s notion of augmented transition networks (ATN’s), which are related to–and to some degree dependent upon Chomsky’s notion of Transformational Grammar, Cope has been successful in designing a system (EMI : Experiments in Musical Intelligence) that can reproduce musical styles to the likes of the greats–Bach, Chopin, Rachmaninoff, and the like…

Musical Style…

“Styles” in Cope’s view are also described through attention to musical ’signatures’, ‘tension-resolution’ logic, ’semantic meshing’ (a series of nested musical contexts that range from local notes & measures to phrases, periods & large-scale sections), and final musical ‘recombination’ (which theoretically allows for recombination of discrete passages of music that can be recombined in ways that are still stylistically “viable”, according to Cope).

But these issues of style uncover other questions:  Cope’s work focuses on reproducing ’styles’ of different composers.  How can these systems support generation of music to match the style of someone interacting with a system–can someone listening to music participate in the construction of what they hear?  Well, Pandora makes an attempt at doing this.  Pandora doesn’t generate music, but it does try to give the listener what they want to hear.  Here’s my theory on this–which is based on connectionist modeling concepts described in Eric Clarke’s book, “Ways of Listening”, and described in the first chapter of the book–Perception, Ecology, and Music.

• Here, Clarke provides a background for connectionist modeling from McLelland’s: Parallel Distributed Processing (Rumelhart and McLelland 1986) and describes a theoretical experiment that describes the ability of a ‘network’ to learn (based on programming & guided input) the musical preferences of listeners (who are listening to discrete melodies) and be theoretically be able to predict and classify new melodies using the preferences on which the model is based.  (I’m oversimplifying Clarke’s example, sorry Clarke).  However, he describes this process of predictability as hinging on assignment of “connection weights between ‘input units’ (basic definition rules/conditions–in this case intervals between notes in each melody) and ‘output units’ (’like or dislike’ of the listener)
• My theory is that Pandora is based on exactly this system of analysis of ‘connection weights’ that use listener ‘voting’ to determine how well Pandora ‘understands’ listener preference–e.g. “style” and is therefore able to predict with a degree of probability whether the listener will like what comes up next.  If this is true–then can we say this way of matching listener ‘preference’ is related to the idea of musical ’style’?

Next question:

How do these ideas of generative music as it relates to style tie back to design of educational technology?

First, let me say that the music design behind Spore is as impressive as the game itself. For the sake of educational research for my class, I recorded the audio that played during each of the three processes.

Why not record video? Unfortunately, although Spore allows recorded video, it only does this at very specific stages of the game, such as ‘testing’ a creature once created, and screencapture software cannot sufficiently capture the video, since the video doesn’t play in a ‘window’ on the screen.

However, because I was able to use Cubase to record the audio, I was able to easily compare the audio on 3 different tracks–and in one instance I played a selection of audio from all 3 tracks simultaneously and was pleasantly surprised to hear overlapping harmony in all 3 tracks, even as the music played was subtly different.

A few broad points to start, based on my initial testing:

1. Spore plays ‘dynamic’ music at key points in the game that include: the main menu, creature selection from Sporepedia, creature editor, randomizing creatures, etc.

2. Spore appears to play the same music during: the animation sequences that open the cellular & creature stages, and as background loops during the cellular organism stage.

3. The experience of ‘dynamic music’ can be described in each stage in the following ways:

a. Each key stage (i.e. main menu, creature selector, etc.) tends to have a distinct musical ’style’.
b. If I use Cope’s definition of musical style as including:

“the identifiable characteristics of a composer’s music which are recognizably similar from one work to another. These include, but are not limited to, pitch and duration…timbre, dynamics and nuance. “

then I would characterize the styles represented in each key stage in Spore as having a distinct and recognizable: tempo, density (instrumentation), and a trend towards similar harmonies & rhythms (these can be congruent in successive plays, are sometimes transposed, and sometimes shift between instrumental parts–again, these seem to me to be ‘tendencies’, not hard/fast rules).

c.  Upon successive playing of the audio from each ’stage’, and as a result of the same menu choices & user interactions, it appears that “style” can be found in listening to the repetitive variations of musical ‘fragments’.  How these fragments are generated (e.g. single pitches, phrases, loops, etc.) are a mystery to me.

After digging into Cope’s book a bit, I find the musical examples that describe how patterns of musical style & structure relate to their grammatical ‘counterparts’–and how these elements could be used to generate music.  (It also reinforced for me the usefulness in becoming more proficient in my understanding of theoretical & compositional grammar).

The process that Cope presents with phrasal structure & Automated Transition Networks is quite fascinating.  I’m curious if any companies have operationalized the idea.  I also wonder how far an approach like Cope’s can be taken to adapt to composition of complex works with several instrumental parts. I can imagine how automated generation would apply nicely to environments needing, several shorter segments (e.g. a computer game)…

My research survey in style-based automated composition is happening as a result of an independent study supervised by Professor Bruce Pennycook at the University of Texas @ Austin.  Professor Pennycook directed my attention to the work of David cope, who has done much of the grounding for this field.  I’m starting my review of David Cope’s work by reading “Computers and Musical Style”, published in 1991.

In reading about musical style representations, I suppose I’m not surprised to learn that there are overlaps between linguistic patterns in the way that we construct language to the way in which music can thought of as language.  To help illustrate this connection, Cope compares the diagrammatic parsing of sentences in a way that’s analogous to a diagrammatic parsing of music.

Cope also formulates a definition of ‘musical style’ for his book in a way that relates style with characteristics of music & musical structures.  He talks about style as,

“the identifiable characteristics of a composer’s music which are recognizably similar from one work to another.  These include, but are not limited to, pitch and duration…timbre, dynamics and nuance. “

Cope goes on to discuss other elements that have a “grammar” which constitute style when appear repeatedly across multiple works.  These include: melody, harmony, & counterpoint, as well as their connections to introductions, motives, transitions, modulations, & cadences.

I look forward to digging deeper in hopes that I can catch a glimpse of how these elements fit together & can be structurally represented as style, which as I believe Cope has gone on to demonstrate, can then be represented through the language of programming.