Audio & Educational Technology

This has been a very enjoyable semester.  I feel like I’ve covered a lot of ground.  With Dr. Pennycook’s guidance, I’ve done a survey of the world of generative music–and even though I feel as if I’ve learned much, I also feel like I’ve just scratched the surface.   I think I’ve come to the place where the next step is one that takes the theories and knowledge provided by the people I’ve studied so far–namely Cope (and Hoffsteader), Clarke, (just a bit) of Lerdahl, and of course Dr. Pennycook, and venture into the exciting world of generative music technology through Max/MSP.

In addition to taking an experimental performance interfaces course, I will concurrently be doing a project for Dr. Liu’s multimedia class, where a team of us will be designing a micro-simulation using the Unity 3d game engine. Here’s a few questions I came up with to begin this exploration…

• How do we apply the principles of telling stories to designing with Max/MSP?
• How does Max/MSP work exactly?
• How do we build end-user control in generative music?
• How does Max/MSP take in end-user data from Unity?
• How will Max/MSP output music through Unity?
• How do we build sound design in Unity?
• How does Unity record and handle end-user data?
• What coding will need to be done through Unity to allow these systems to interact?
• In short, how will these 2 work and play together?

And yes, the reading will continue and I plan to take a sneak peak at these systems between now & the end of January…So, there’s more to come…

Well, I took the next step in my exploration of experimental performance interfaces & Pd and decided to install Pd for Windows & tried the Wiimote.  I spent some time looking for a PC Wii to midi driver and eventually found one that worked called Glovepie.  I also installed midiYoke as a midi “patch bay” driver that links everything together.  I then used Pd to receive & process the midi data from the Wiimote.

In the process, I took a script for an oscillator that came bundled with Pd & modified it to accept midi channel data, using a piece of the test script that also comes bundled with Pd.   I then had to use & modify a script in Glovepie to have a different starting note value & made sure that Pd used midi yoke (channel 2) as an input device.   I’m only succeeding in making the wiimote increment & decrement pitches in a *very* rough way.  The process is akin, I think, to learning how to steer a tank.   However, it does seem to work.

I captured a short video with my phone so that you can see it for yourself.  this time I exported the movie as a .mov & posted it  @ http://www.adaptablearts.com/media/jason-wii-pd.mov .

Here’s the process that I used:

I recently attended a workshop on iphone app development.  In light of that presentation and our research of generative music in Spore, I thought this new iphone app developed by Brian Eno called “Bloom” was interesting.  I downloaded the app for the iTouch (I’m on Verizon, so I don’t use the iPhone).

http://createdigitalmusic.com/2008/10/08/hands-on-with-bloom-new-generative-iphone-app-by-eno-and-chilvers/

Here’s a Youtube clip that demos the app

The music the app produced was interesting–amazingly it sounded a *lot* like the stuff in Spore..  The app lets you either set a random setting to play music based on some basic choices, or you can interact by tapping the screen to make music.  Tapping the bottom of the screen generates lower pitches, while tapping higher generates higher pitches.

Some points:

• Bloom has a 4 octave range
• There are 9 different “moods” that can be selected to provide a different textural experience
• Each “mood” changes the tonic of the scale that plays.  One “mood” may have a range from C2 – C6, while another mood ranges from G2 – G6, etc.
• When you touch the iTouch, it generates pitches that then repeats a pattern using a delay.  There’s a setting called “evolve when idle” that I think takes a pattern & develops it so that something plays when you’re not.
• For example, the mood “Neroli” is in the key of G, and I’m pretty sure in a mixolydian mode.  The lower areas of the touch pad isn’t as responsive to changes in pitch as the middle & upper registers.

Listen to a sample of me playing the iTouch with the  “Neroli” mood!

• You’ll hear ambient background stuff that Bloom generates.  Also, the application was set to randomize the “mood” during playback (not sure what this does), and the delay was set a pretty short interval
• The first part of the clip is me trying to play a scale from the bottom up.  You can hear me playing the same notes even as I’m moving up the touchpad.  I actually moved from I-V-I for the first octave before moving up the scale.
• The later part of the clip is me “playing” the app.  Funny thing, last weekend I was in the middle of my music composition lesson when my tutor & I got into a jam session with Bloom.  He was on the guitar & we were just doing some ambient stuff.  I’ve *never* jammed with an ipod before.  That was a first.

Something arguably even more interesting is info on an iphone app called RjDj : http://www.rjdj.me .  It uses Pd and the iphone accelerometer & mic to sample audio.  Since I don’t have an iPhone, I can’t try this myself, but more info can be found here on this blog:
http://createdigitalmusic.com/2008/09/02/iphonetouch-roundup-control-art-snow-patrol-visualizers-recording-one-for-india/ .

I decided to explore the options given to us by Noatikl, by Intermorphic, for generative music first (since I’m waiting for my copy of Spore to arrive).  (By the way, in case you were wondering “Noatikl” is pronounced: “No-ti-cul”, emphasis on the first syllable.) What I found interesting about this software was it’s apparent ability to serve as both a stand-alone application as well as a VSTi (Virtual Studio Technology instrument) in a Digital Audio Workstation (DAW).  Noatikl apparently allows for Cubase, Garage Band, or Logic users to load Noatikl as a VSTi.   Although VSTi integration is standard for virtual instruments, what’s interesting about Noatikl is how it works as a VSTi.  When you instantiate Noatikl, you then have the ability to set the midi input of your tracks to come from Noatikl, using Cubase’s input transformer to filter for the correct channel.  There’s a brief tutorial on this integration @ http://www.vimeo.com/kantudok/.

Noatikl apparently also works as a “hyperinstrument” that can receive text/mouse input as well as midi cc events.  Noatikl claims to harmonize its voices based on midi data.  See: http://www.intermorphic.com/tools/noatikl/doc/noatiklUG_scripting_2.html for more of an explanation.

Although I have Cubase 4 running on Windows, my initial attempt at installing Noatikl was problematic.  Cubase doesn’t recognize the Noatikl VSTi, nor does the stand-alone version recognize incoming midi data.  I’ve reported the problem to Intermorphic, who has responded in a timely fashion, so I’m hoping to get the technical issues resolved when I can return to the problem.

Although Pd is certainly occupied my recent interest, there are other tools that support generative music that are also on my list.  These include:

Intermorphic’s Noatikl: the original progenetor of this software was Koan–which was pioneered by Brian Eno and written by SSEYO in 1996.  Intermorphic eventually acquired the rights to Koan.  Noatikl is promising since it works as a VSTi.

Intermorphic’s Mixtikl:  although this software is not yet released, it promises to offer support for mobile devices–including the iPhone.

Cycling 74’s Max/MSP: Max/MSP is a commercial analog of Pd, and has a long history in computer music creation.

Musigenesis: A fun-looking, shareware application that lets people of non-musical background create music.

Nodal: This promising music creation tool generates music in midi form, and is designed for the Apple.  The screenshots display a ‘circuit-diagram’ view of musical “events” and the “connection between events”.  This software appears to also let multiple people play music simultaneously.

The more I learn about Pd, the more I learn about its flexibility–and of course, the more that I also learn about its technical requirements.  Although Pd is Windows compatible, its preferred OS is linux.  I did manage to get Pd to run in a virtual install of Ubuntu on a WinXP machine–but of course Ubuntu did not recognize the audio hardware of the machine I was using.  My next step is to try the same process on a much faster machine with a more industry-standard audio device.  Since it’s an EMU audio card, I might have some luck.  The goal is to get a completely working version of Pd running in a way that supports looking at the work that’s been done with the tool–and to eventually learn the scripting that makes Pd go.

It would seem that Pure Data (Pd) has been used to build the dynamic soundtrack behind the ’soon-to-be-smash-hit’ Spore.  Brian Eno led this effort (as he led the generative music movement in the mid-90’s with Koan) and designed a tool that he calls, “The Shuffler” in Pd.  Apparently, this technology ensures that the player will never hear the same composition twice. In fact, we take a look at this Gamespy article, we can see a photo of the Pd script used in the game!

According to an MTV blog posting & video demo, as the player ‘builds’ parts of their environment (in the demo they show the building of a spaceship), the user’s choices drive the music they hear.  The team is calling this kind of music “procedural music”.  The presenter even references how user choices influence the use of counterpoint.  Take a look at a demo on the MTV blog to hear more.

Spore comes out next week–so of course my research will compel me to check it out.

Much has been done in the field of generative music.  The early work done with Pd in the mid-90’s continues to this day, and is one pillar of the technology systems that support generative music and algorithmic composition.

Although the technology that drives this next example is unclear, one example of a site that demonstrates algorithmic composition is Wolfram Tones, which takes computational algorithms (from Mathematica) and combines them with music theory to produce an interface that lets people “make” music in a variety of styles.  It would appear that Wolfram takes visual patterns as represented by “cellular automation”, flips the pattern on its side and represents the relationship between cells through musical ideas such as pitch.  This is, of course a gross-oversimplification, so please consult Wolfram for the details.

One interesting thing about Wolfram is that users can use the tool to create their own midi–which can either be saved on the site, emailed as a midi file–and even sent directly to your cell phone as ringtone.  How’s that for sharp?

Another interesting feature is that Wolfram builds in compositional controls that include not only pitch-mapping, instrumentation, meter & tempo, but also a “generator” that allows users to change the computational “rules” used to derive music from the pattern seen.

Of course, the next best feature of the software is that it’s free.

I will begin by examining style-based automated music composition, and how the ability to create this form of music can be applied to designing immersive educational technology.  I will be examining technologies such as Pd (Pure Data) and Max/MSP with the hope of understanding enough of the mechanics to assess their transportability to virtual environments.