Chapter 7: Sonification and Big Data

Topics: Data sonification, mapValue() and mapScale(), Kepler, Python strings, music from text, Guido d’Arezzo, nested loops, file input/output, Python while loop, big data, biosignal sonification, defining functions, image sonification, Python images, visual soundscapes.

Sonification allows us to capture and better experience phenomena that are outside our sensory range by mapping values into sound structures that we can perceive by listening to them. Data for sonification may come from any measurable vibration or fluctuation, such as planetary orbits, magnitudes of earthquakes, positions of branches on a tree, lengths of words in this chapter, and so on. More information is provided in the reference textbook.

Here is code from this chapter:

• Sonifying planetary data
• Making music from text
• Recreating Guido d’Arezzo’s “Word Music” (ca. 1000)
• Sonifying biosignals
• Sonifying images

---

Sonifying planetary data

In 1619 Johannes Kepler wrote his “Harmonices Mundi (Harmonies of the World)” book (Kepler, 1619).

This code sample (Ch. 7, p. 196) sonifies one aspects of the celestial organization of planets. In particular, it converts the orbital velocities of the planets to musical notes.

# harmonicesMundi.py
#
# Sonify mean planetary velocities in the solar system.
#

from music import *

# create a list of planet mean orbital velocities
# Mercury, Venus, Earth, Mars, Ceres, Jupiter,
#  Saturn, Uranus, Neptune, Pluto
planetVelocities = [47.89, 35.03, 29.79, 24.13, 17.882, 13.06,
                     9.64, 6.81, 5.43, 4.74]

# get minimum and maximum velocities:
minVelocity = min(planetVelocities)
maxVelocity = max(planetVelocities)

# calculate pitches
planetPitches   = []    # holds list of sonified velocities
planetDurations = []    # holds list of durations
for velocity in planetVelocities:
   # map a velocity to pitch and save it
   pitch = mapScale(velocity, minVelocity, maxVelocity, C1, C6,
                    CHROMATIC_SCALE)
   planetPitches.append( pitch )
   planetDurations.append( EN )   # for now, keep duration fixed

# create the planet melodies
melody1 = Phrase(0.0)      # starts at beginning
melody2 = Phrase(10.0)     # starts 10 beats into the piece
melody3 = Phrase(20.0)     # starts 20 beats into the piece

# create melody 1 (theme)
melody1.addNoteList(planetPitches, planetDurations)

# melody 2 starts 10 beats into the piece and
# is elongated by a factor of 2
melody2 = melody1.copy()
melody2.setStartTime(10.0)
Mod.elongate(melody2, 2.0)

# melody 3 starts 20 beats into the piece and
# is elongated by a factor of 4
melody3 = melody1.copy()
melody3.setStartTime(20.0)
Mod.elongate(melody3, 4.0)

# repeat melodies appropriate times, so they will end together
Mod.repeat(melody1, 8)
Mod.repeat(melody2, 3)

# create parts with different instruments and add melodies
part1 = Part("Eighth Notes", PIANO, 0)
part2 = Part("Quarter Notes", FLUTE, 1)
part3 = Part("Half Notes", TRUMPET, 3)
part1.addPhrase(melody1)
part2.addPhrase(melody2)
part3.addPhrase(melody3)

# finally, create, view, and write the score
score = Score("Celestial Canon")
score.addPart(part1)
score.addPart(part2)
score.addPart(part3)
View.sketch(score)
Play.midi(score)
Write.midi(score, "harmonicesMundi.mid")

It plays this sound:

---

Making music from text

This code sample (Ch. 7, p. 202) demonstrates how to generate music from text. This program converts the values of ASCII characters to MIDI pitches. For variety, note durations are randomized; other note properties (volume, etc.) are the same for all notes.

# textMusic.py
#
# It demonstrates one way to generate music from text.
#
# * Note pitch is based on the ASCII value of characters.
# * Note duration is selected from a weighted-probability list
#   (again, based on ASCII value).
# * Note dynamic is selected randomly.
#
# All other music parameters (panoramic, instrument, etc.)
# are kept constant.

from music import *
from random import *

# Define text to sonify.
# Excerpt from Herman Melville's "Moby-Dick", Epilogue (1851)

text = """The drama's done. Why then here does any one step forth? - Because one did survive the wreck. """

##### define the data structure
textMusicScore  = Score("Moby-Dick melody", 130)
textMusicPart   = Part("Moby-Dick melody", GLOCK, 0)
textMusicPhrase = Phrase()

# create durations list (factors correspond to probability)
durations = [HN] + [QN]*4 + [EN]*4 + [SN]*2

##### create musical data
for character in text:  # loop enough times

   value = ord(character)         # convert character to ASCII number

   # map printable ASCII values to a pitch value
   pitch = mapScale(value, 32, 126, C3, C6, PENTATONIC_SCALE, C2)

   # map printable ASCII values to a duration value
   index = mapValue(value, 32, 126, 0, len(durations)-1)
   duration = durations[index]

   print("value", value, "becomes pitch", pitch,)
   print("and duration", duration)

   dynamic = randint(60, 120)    # get a random dynamic

   note = Note(pitch, duration, dynamic)   # create note
   textMusicPhrase.addNote(note)  # and add it to phrase

# now, all characters have been converted to notes

# add ending note (same as last one - only longer)
note = Note(pitch, WN)
textMusicPhrase.addNote(note)

##### combine musical material
textMusicPart.addPhrase(textMusicPhrase)
textMusicScore.addPart(textMusicPart)

##### view score and write it to a MIDI file
View.show(textMusicScore)
Play.midi(textMusicScore)
Write.midi(textMusicScore, "textMusic.mid")

It plays this sound:

---

Recreating Guido d’Arezzo’s “Word Music” (ca. 1000)

One of the oldest known algorithmic music processes is a rule-based algorithm that selects each note based on the letters in a text, credited to Guido d’Arezzo (991 – 1033).

This code sample (Ch. 7, p. 207) is an approximation to d’Arezzo’s algorithm, adapted to text written in ASCII.

# guidoWordMusic.py
#
# Creates a melody from text using the following rules:
#
# 1) Vowels specify pentatonic pitch, 'a' is C4, 'e' is D4,
#    'i' is E4, 'o' is G4, and 'u' is A4.
#
# 2) Consonants are ignored, but contribute to note duration of
#    all vowels within a word (if any).
#

from music import *

# this is the text to be sonified
text = """One of the oldest known algorithmic music processes is a rule-based algorithm that selects each note based on the letters in a text, credited to Guido d'Arezzo."""

text = text.lower()   # convert string to lowercase

# define vowels and corresponding pitches (parallel sequences),
# i.e., first vowel goes with first pitch, and so on.
vowels       = "aeiou"
vowelPitches = [C4, D4, E4, G4, A4]

# define consonants
consonants = "bcdfghjklmnpqrstvwxyz"

# define parallel lists to hold pitches and durations
pitches   = []
durations = []

# factor used to scale durations
durationFactor = 0.1   # higher for longer durations

# separate text into words (using space as delimiter)
words = text.split()

# iterate through every word in the text
for word in words:

   # iterate through every character in this word
   for character in word:

      # is this character a vowel?
      if character in vowels:

         # yes, so find its position in the vowel list
         index = vowels.find(character)

         # and use position to find the corresponding pitch
         pitch = vowelPitches[index]

         # finally, remember this pitch
         pitches.append( pitch )

         # create duration from the word length
         duration = len( word ) * durationFactor

         # and remember it
         durations.append( duration )

# now, pitches and durations have been created

# so, add them to a phrase
melody = Phrase()
melody.addNoteList(pitches, durations)

# view and play melody
View.notation(melody)
Play.midi(melody)

It plays this sound:

---

Sonifying biosignals

Here we explore pre-processing and sonification of data from biological processes. The figure below displays heart data, captured by measuring blood pressure over time.

Sample raw heart data (x-axis is time, y-axis is pressure)

The figure below displays skin conductance, captured by measuring electrical conductivity between two fingers over time.

Sample skin-conductance data (x-axis is time, y-axis is skin conductance)

This code sample (Ch. 7, p. 217) explores one possible sonification of these data. Before running this program, download the complete data file in your PythonMusic folder.

# sonifyBiosignals.py
#
# Sonify skin conductance and heart data to pitch and dynamic.
#
# Sonification design:
#
# * Skin conductance is mapped to pitch (C3 - C6).
# * Heart value is mapped to a pitch variation (0 to 24).
# * Heart value is mapped to dynamic (0 - 127).
#
# NOTE: We quantize pitches to the C Major scale.
#

from music import *

# first let's read in the data
data = open("biosignals.txt", "r")

# read and process every line
skinData  = []     # holds skin data
heartData = []     # holds heart data
for line in data:

   time, skin, heart = line.split()  # extract the three values
   skin = float(skin)               # convert from string to float
   heart = float(heart)             # convert from string to float
   skinData.append(skin)            # keep the skin data
   heartData.append(heart)          # keep the heart data

# now, heartData contains all the heart values

data.close()    # done, so let's close the file

##### define the data structure
biomusicScore  = Score("Biosignal sonification", 150)
biomusicPart   = Part(PIANO, 0)
biomusicPhrase = Phrase()

# let's find the range extremes
heartMinValue = min(heartData)
heartMaxValue = max(heartData)
skinMinValue  = min(skinData)
skinMaxValue  = max(skinData)

# let's sonify the data
i = 0;   # point to first value in data
while i < len(heartData):   # while there are more values, loop

   # map skin-conductance to pitch
   pitch = mapScale(skinData[i], skinMinValue, skinMaxValue, C3, C6,
                    MAJOR_SCALE, C4)
   # map heart data to a variation of pitch
   pitchVariation = mapScale(heartData[i], heartMinValue,
                             heartMaxValue, 0, 24, MAJOR_SCALE, C4)

   # also map heart data to dynamic
   dynamic = mapValue(heartData[i], heartMinValue, heartMaxValue,
                      0, 127)

   # finally, combine pitch, pitch variation, and dynamic into note
   note = Note(pitch + pitchVariation, TN, dynamic)

   # add it to the melody so far
   biomusicPhrase.addNote(note)

   # point to next value in heart and skin data
   i = i + 1

# now, biomusicPhrase contains all the sonified values

##### combine musical material
biomusicPart.addPhrase(biomusicPhrase)
biomusicScore.addPart(biomusicPart)

##### view score and write it to a MIDI file
View.sketch(biomusicScore)
Write.midi(biomusicScore, "sonifyBiosignals.mid")
Play.midi(biomusicScore)

It plays this sound:

---

Sonifying images

This code sample (Ch. 7, p. 231) demonstrates how to sonify (generate music from) images. It sonifies the following image:

Loutraki Sunset (320 x 213 pixels)

Before running this program, download this image in your PythonMusic folder.

Here is the code:

# sonifyImage.py
#
# Demonstrates how to create a soundscape from an image.
# It also demonstrates how to use functions.
# It loads a jpg image and scans it from left to right.
# Pixels are mapped to notes using these rules:
#
# + left-to-right column position is mapped to time,
# + luminosity (pixel brightness) is mapped to pitch within a scale,
# + redness (pixel R value) is mapped to duration, and
# + blueness (pixel B value) is mapped to volume.
#

from music import *
from image import *
from random import *

##### define data structure
soundscapeScore = Score("Loutraki Soundscape", 60)
soundscapePart  = Part(PIANO, 0)

##### define  musical parameters
scale = MIXOLYDIAN_SCALE

minPitch = 0        # MIDI pitch (0-127)
maxPitch = 127

minDuration = 0.8   # duration (1.0 is QN)
maxDuration = 6.0

minVolume = 0       # MIDI velocity (0-127)
maxVolume = 127

# start time is randomly displaced by one of these
# durations (for variety)
timeDisplacement = [DEN, EN, SN, TN]

##### read in image (origin (0, 0) is at top left)
image = Image("soundscapeLoutrakiSunset.jpg")

# specify image pixel rows to sonify - this depends on the image!
pixelRows = [0, 53, 106, 159, 212]
width = image.getWidth()     # get number of columns in image
height = image.getHeight()   # get number of rows in image

##### define function to sonify one pixel
# Returns a note from sonifying the RGB values of 'pixel'.
def sonifyPixel(pixel):

   red, green, blue, alpha = pixel  # get pixel RGBA value

   luminosity = (red + green + blue) / 3   # calculate brightness

   # map luminosity to pitch (the brighter the pixel, the higher
   # the pitch) using specified scale
   pitch = mapScale(luminosity, 0, 255, minPitch, maxPitch, scale)

   # map red value to duration (the redder the pixel, the longer
   # the note)
   duration = mapValue(red, 0, 255, minDuration, maxDuration)

   # map blue value to dynamic (the bluer the pixel, the louder
   # the note)
   dynamic = mapValue(blue, 0, 255, minVolume, maxVolume)

   # create note and return it to caller
   note = Note(pitch, duration, dynamic)

   # done sonifying this pixel, so return result
   return note

##### create musical data

# sonify image pixels
for row in pixelRows:   # iterate through selected rows

   for col in range(width):  # iterate through all pixels on this row

      # get pixel at current coordinates (col and row)
      pixel = image.getPixel(col, row)

      # sonify this pixel (we get a note)
      note = sonifyPixel(pixel)

      # wrap note in a phrase to give it a start time
      # (Phrases have start time, Notes do not)

      # use column value as note start time (e.g., 0.0, 1.0, and so on)
      startTime = float(col)   # phrase start time is a float

      # add some random displacement for variety
      startTime = startTime + choice( timeDisplacement )

      phrase = Phrase(startTime)   # create phrase with given start time
      phrase.addNote(note)         # and put the note in it

      # put result in part
      soundscapePart.addPhrase(phrase)

   # now, all pixels on this row have been sonified

# now, all pixelRows have been sonified, and soundscapePart
# contains all notes

##### combine musical material
soundscapeScore.addPart(soundscapePart)

##### view score and write it to an audio and MIDI files
Play.midi(soundscapeScore)
Write.midi(soundscapeScore, "soundscapeLoutrakiSunset.mid")

It plays this sound:

---

More examples of image sonification

“Daintree Drones” is another example of image sonification to generate a circular piece – by Kenneth Hanson.

This is from a Computing in the Arts student exhibit, entitled “Visual Soundscapes“, which was funded in part by the National Science Foundation (DUE #1044861).

Other examples from the exhibit are available here.