COMP2720 • 2008
Assignment 2
Due week 13 (Tuesday 21 October, 18:00 EDST)
Draft — elaborations and
clarifications may follow
This assignment is worth 15% of your total course mark. It will be
marked out of 15 as indicated below.
The estimated time we expect you to spend on this assignment is
around 15 hours in total (1 hour per mark).
A Synaesthetic Étude: Music in Colours
Objectives
This assignment is a (very) modest attempt at creating a piece of
synthetic art, which combines music and visual imagery. You are given
a few files containing sequences of (pitch,amplitude,duration) data which
can be performed by the JES function playNote().
These files are single track extractions from midi files,
which means they are not complete music tracks, but rather mangled pieces of such.
Your task is to write a Python program which will extract the information about
pitch, duration and amplitude of every sound from a given sequence file, and
to map these data into the visual objects, characterized by colour, shape and
size. You will create a picture object and store it in a separate file for a group
of notes in the given sequence. This way, a melody represented by a note
sequence will be performed in colour, as a motion picture made of frames
which are represented by the above pictures.
The Linux version of JES which we use in the labs, does not allow you
to save the movie in one of the standard movie formats (though, JES
on Mac and Windows does allow this), but if named and organized
correctly, these image files can be assembled into a movie and played by JES.
Prior musical knowledge assumed: none
Background
Syneasthesia is a rare (perhaps not too rare as the recent article
in NewScientist, 9 August 2008, p. 17 indicates —
added on 05.09.2008)
psychological phenomenon which consists in
sensing an impression with organs different from those by which
this impression is normally stimulated. For example, to see colours when
hearing sound, and vice versa — colour-hearing. Several composers are
known to have possessed such colour-hearing abilities
— Richard Wagner, Nikolai Rimsky-Korsakov, Olivier Messiaen, and most
particlularly the Russian composer Alexander Scriabin. In the scores of his 1910 symphonic work
"Prometheus, or the Poem of Fire", Scriabin included the part called luce
(pronounced loõ-che — Italian for "light"; compare the English word
lucent) which was to be performed on a then non-existed colour-tonal instrument.
Scriabin associated every musical note with a particular colour. In a simplified
form the colour notes are represented on this picture:
Your task in this assignment is to "perform" a melody provided as a sequence of
notes by considering groups of successive notes from the sequence, and using the
above notes →colours correspondance, generate shapes on a
canvas and save the resulting picture as a frame of a movie. When these picture files are
properly named and stored, JES can be used to play them as a single movie
(on Linux; on Mac and Window a movie file, in the QuickTime or AVI, respectively, can
be actually created). The details of the process are provided below, in
the tasks section.
Note sequences and midi files
Normally, the music visualization is performed on audio files (wav,
mp3, aiff etc) when
the music characteristics like volume, balance (for stereo sounds), melody, rate and
mood are extracted, often with the help of sophisticated techniques like spectral
(using the Fast Fourier Transform, FFT) and other analyses. Usually, some of these
analyses (especially mood and melody) are faked: visual effects are achieved
using pattern generated engines which operate more or less randomly.
This difficulty is not surprising since genuine visualization of music
is possible only through its adequate description which exists as musical
scores written by music's composers (even scores are not enough —
that's why performers are artists too, and every good performer is co-creator
of the performed music). Such adequate description of digital music does
indeed exist. Just like an image can be described in a vector graphics format,
a music piece can be faithfully represented in a special digital format called
MIDI, which is more or less a music score in binary form.
The MIDI format involves one or several tracks,
each being a collection of data chunks —
an uninterrupted data stream. Each chunk carries information about a music event
— the time stamp (when the event takes place), the actual note (pitch), its volume
and the character of the event (note-on, note-off, and some other to help imitate
various basic instruments). Since normally note events overlap, it is more convenient
to include multiple tracks in one MIDI file (tacks are stored
in a MIDI file one after another), though any sequence
of overlapping note events can be represented by a single track.
For this assignment, several existing classical music pieces were converted from
the original MIDI files into so-called comma-separated
values (.csv) files which are essentially one-track
extractions from the original MIDIs. The reason we cannot
use the MIDI files is that JES cannot read them (though,
it may change soon, since JES is created on the top of Java which does have
MIDI capabilities — the Java sound.midi
package is already used in JES to handle sounds). Therefore, if we want to use
"real" melodies in our colour-music experiments, we must compromise and
simplify the MIDI-melodies so JES can play and make
further use of them.
Below under Note (capital 'N')
we mean the full set of data from one line in the sequence file —
the musical note (AKA 'key') and octave, duration and volume. The key (one of the
values C, C#, D, D#, E, F, F#, G, G#, A, A#, B) and octave
(-1,0,1,…8,9) are derived from the value of pitch (0…127) using the same
MIDI table,
which is used in the Lab 4. Remember, that the last octave is shortened, it has
only eight keys (C,…,G).
1. The note–colour dictionary
Begin with creating a dictionary of (note,colour)
pairs using the colour scheme in this picture. Simply, create a picture object in
JES by loading the picture file, open it in
the Media Tools, read the (R,G,B)-values of all twelve "colour-notes", and then use them
to create colour constants. (Name them like colourC,
colourDsharp, etc.) Introduce the dictionary variable and
fill it with the appropriate (key,value) entries.
2. Mapping sounds to shapes
Next, start reading the music sequence file line by line, parsing it and
determining the pitch, duration and amplitude (volume) of every sound. Use the
MIDInotes.csv file from the lab
4 to map every pitch to the (note,octave) pair (reversed
to what you will have done in the lab 4 exercise 2). You will have to separate the note and octave values for later use.
Once the (note,octave,duration,volume) values are determined, the Note can be
be transformed into an image. You can use one and the same shape for every note,
a simple one, like a circle or a square, which JES can generated easily. But
depending on the pitch (note and octave) the colour and size of generating shapes
will be different. To set the colour, you will use the note-colour dictionary
created earlier, while the size will be determined by the note octave. The low
sounds from the Octave number -1 (pitch 0…11) should be the largest,
the sounds from higher Octaves should progressively get smaller, with
the sounds from the Octave 9 should be the smallest.
The position of shapes on the canvas will be chosen randomly.
3. "The melody"
The effect of successively played notes will be achieved by retaining the shapes
from the previous sounds on the next frame. Every frame will contain
up to seven shapes — one from
the current note and others from the previously played notes. To express attenuation
of played notes, the present shapes on the current frame will fade their original colour
proportionally to their "age" and "life span".
The note life is determined by the duration value: you will have to establish
the note with a shortest duration durmin in the
sequence, and the note with the longest duration durmax.
Then, you will break the interval
[durmin,…,durmax]
into seven equal sub-intervals. Thus, every note in the
sequence will have a duration from one of the sub-intervals. The sub-interval
number will determine the note life span — the number of frames on which the
note will be present. The note luminance will be the highest on the first frame
(when the note is "youngest"). With every consecutive frame the note luminance
will decrease according to its life span, eg, the note with the duration
from the fifth duration sub-intervals will "live" for five frames, loosing 20%
of its original luminance every consecutive frame, and disappearing after
five frames. The longest sounding notes will live for seven frames.
Some notes in a sequence ma have zero duration (they fake silent pauses).
These notes will not be represented by a shape, the frame corresponding to such
note event, will be created as usual (on such frames, the present shapes will
be "aged" shapes which where already generated on the preceding frames.
Thus, every "shape-note" will undergo
a fading transition from the original full colour to the pure white in up to seven
steps. When the last note in the sequence is played, the "colour-music" will
last several (up to six) more frames until the shape created by the last played note
dissolves in the canvas white background.
4. The movie
The frames will have to be successively numbered and saved in a special
directory named as the sequence file (without .csv suffix,
eg, the sequence from satie_gymnopedie-1.csv
will generate frames stored in the directory satie_gymnopedie-1).
The movie can be played by creating a movie abject and adding frames to it from the
directory which contains the frames.
5. Advanced effects
The successively generated shape must not obscure (completely overlap) one another.
This would require putting some constraint on the position where a newly created shape
is placed. Try modify the random algorithm of selecting the new shape position
which takes into account the already present shapes so that the new shape fits (if
possible) in between the old shapes, and does not completely covers some of them.
More details about program tasks and
the structure of luce.py
program are given in the assignment elaboration
on tasks and hints below (to follow…).
Below is some music sequences which you can use in this assignment work:
You can play midi-files using QuickTime or similar
software, and you can play the sequence csv-files using the JES
function playSequence.py
Python function by loading it in JES. Compare how crude and mangled the original
midi-sound becomes in the sequence.
The sequence files were created from the original midi-files by
Ben Swift to whom
I am very thankful.
The midi-file are used here (by the virtue of their Artistic License)
courtesy of Classical Piano Midi Pages.
Submission
You will have to submit one program file in this assignment and one sequence
note csv file, one of those provided in
The music sequences or
one of your own (it can be fun to write them,
even if you do have formal musical training):
- The Python/JES program, named
luce.py, that contains
all manipulation functions you implemented. This program
should be based on the program skeleton
luce.py.
(Added
on 1 October 2008: Since you will need the dictionary
file MIDInotes.csv, you can define
luce() function with three parameters —
the directory name, the sequence file name and the dictionary file.
Alternatively, you can define luce() with two
arguments (as in the original template) but include
setMediaPath() command into you program and
then open MIDInotes.csv file. Needless to say,
you will also need to submit this file alongside with the other two).
Make sure that you fill in the preamble at the beginning of the
luce.py program with your details!
- One
csv file containing the note sequence
which luce.py will "perform in colour". When we will
run your program for marking, we can use a different csv
file. So it is important to make your luce.py be able
to work with any correctly formatted sequence file. One constraint,
though — it sould not be too long (200-300 lines/notes is enough).
Submission protocol
To submit you work, you will have to use the command-line interface
(ie, to open the Konsole and execute the submission
command by typing and hitting the return button.)
Log into your DCS account (ground floor computer rooms), and execute the following steps:
|
- Start the
Terminal (aka Konsole) application.
- cd into your assignment folder, something like this
(replace u1234567 with your Uni ID):
cd ~/u1234567/comp2720/ass2
- Now you can use the
submit command to submit your
assignment files:
submit comp2720 ass2 luce.py
submit comp2720 ass2 mymusic.csv
submit comp2720 ass2 MIDInotes.csv (optional)
Note that submit will only accept these three file names
(case sensitive, so you must use lowercase!).
- Note that you can submit a file many times, we will always keep the
last two submissions.
- Finally, you can check your assignment submission on
https://cs.anu.edu.au/streams/
(click on View Marks button).
|
If you work on a different machine (your home computer or on one of
the InfoCommons computers) and need to log in to your DCS account or
transfer file to your DCS account (using the network, not a USB memory stick),
you can achieve this by opening the Terminal or shell
application and then use the following commands:
All files must be submitted by
Tuesday 21 October, 18:00 DLST
Extensions
Students will only be granted an extension on the submission deadline
in exceptional circumstances. Work and sporting commitments are normally
NOT sufficient grounds. If you think you have grounds for an
extension, you should notify the course coordinator as soon as possible
and provide written evidence in support of your case (e.g. medical
certificate). The course coordinator will then decide whether to grant
an extension and inform you as soon as practical.
Late Penalties
Penalties for late submissions are as follows.
| How late | less than 6 hours | 6 to 24 hours |
24 to 48 hours | 48 to 72 hours |
72 to 96 hours | more than 96 hours |
| Penalty from 15 marks |
-0.5 | -1 | -2 | -4 |
-8 | -15 (forget it!) |
Late penalties apply even if you submit just one file after
the deadline.
Plagiarism
You should read the chapter in the Department of Computer Science
Student Handbook that discusses assessment (Chapter 6, pages
17-25), particularly the sections headed Misconduct in
examinations (which also applies to assignments and other forms
of assessment) and Guidelines for assignments.
We will compare all submissions electronically, with any that are
suspiciously similar being investigated by the lecturer. If such
similarities cannot be satisfactorily explained, appropriate action
(see the penalties mentioned in the handbook) will be taken.
That's how a movie with the "colour sound track" may look like:
The movie was generated for about one third of initial sound track
of Modest Mussorsgsky's Pictures at the
Exhibition, Promenade–1 (to be honest, I have duplicated every
frame several times to get approximately the same duration for the movie as the
original sound track, and I also included the original MIDI sound track into the movie;
the Assignment does not ask you to do the tall this —
one movie frame per one note from the corresponding sequence file
and no sound).
Now, I have been planning to provide additional elaborations and hints for this
assignment long time ago, but as I looked at the assignment paper, it was clear to
me that the original task descriptions were clear enough and detailed enough already.
As I said at the beginning, the assignment 2 in terms of computational complexity
is simpler (and the code should be shorter) than the assignment 1. But the assignment
2 is more advanced, meaning that you are expected to have better knowledge of Python
programming techniques and overall programming skills. If this is the case, the
effective ("awesome") list manipulation facilities of Python make the assignment 2
tasks easy to accomplish. Instead of going through every task and giving you hints
how at could be tackled, I would ask you a few questions:
Have you followed the top-down programming approach
while working on this assignment? Formulate the high level operations which
the luce() function must execute (like open the sequence file,
read the lines, determine the duration intervals, read the
MIDInotes.csv file and build the (pitch->(key,octave))
dictionary, read every line in the sequence file: parse every line and
determine the note pitch and duration, convert the pitch into the key and octave,
and calculate the colour primaries and size of the corresponding colour shape,
set the value of the shape's life span and its age, randomly generate
the positions of the note on the canvas frame, put all those values into
the shape-list, add the shape to the existing list of shapes;
don't forget to fade the older shapes colour and age them, draw all the
shapes from the shape list on the canvas, name the frame file appropriately
and write the canvas into that file,… go on to the next note from the
sequence). That's it (I told you this was going to be easy-:)!
Once you identified an operation
which the luce() function will use, define the corresponding
function — give it a name and arguments, determine whether it should return
a value, and if so, return an arbitrary value of the right type:
def getShapeSize(duration, durIntervals):
return 1
By this, you've
created a function stab sufficient for the main function to execute, but the
actual implementation of the function (its body) can be worked out later.
Did you separate the data which you need to describe and modify
the coloured shapes which are displayed on the movie frames? Most of the data
manipulations involve numbers, lists, perhaps tuples, but not the JES image objects.
No need
to drag them along while you are doing all that fading, ageing, etc.
How do you describe the colour shape object which represents a Note on the
movie frame — using a list with the elements of colour, size, duration (life span),
age, horizontal and vertical position? If so, this is a reasonable approach. But if you
used classes (we shall talk about classes and Object-Oriented programming in Python
at the very end of the course), then it is also OK.
How do you describe the whole collection of colour shapes which you
draw on the current movie frame? Using list of shapes? If your shape is
encoded as list (see previous item), then "current shapes" are represented
as a list of lists. Don't be scared — we dealt with nested lists already:
look up the lectures slides, check the textbook.
You can process the shapes list (or what data structure you use to describe shapes)
when you need to fade or age the shape elements using the standard list looping
technique. But in Python you can use functional programming techniques to
efficiently process lists in just one line of code. Let's say you have a list of
integers and you want to weed out all elements which are divisible by three. Here is
how you can do it (I use randomly generated integers):
from random import randint
mylist = []
for i in range(50):
mylist.append(randint(1,100))
#mylist now has 50 randomly generated integers each between 1 and 100
#define the function to determine whether the number is divisible by 3
def not_divisible(x): return x % 3 != 0
#now I remove all element divisible by 3 elements by using the filter function
mylist = filter(not_divisible,mylist)
That's all! If you know how to define functions without naming them (so called
lambda notations, we shall talk about this in lectures soon), this
code can be made even shorter. Apart from filter()
another very useful function for doing this sort of things is
map(). Read about them (using the resources links from the
course web page), and make use of them. All this is not necessary though —
the good old loops will do as well.
You may ask how to define, say, the fading function or the ageing
function to use them with the above approach? But... this is what assignments
are for — stretching your brain a little, isn't! Good luck, don't snap it!
The assignment marks will be awarded as follows:
| 0. For creating the program structure, including the movie object,
reading the value of colour notes and including them into the program |
2 point |
| 1. For reading the sequence file and determining
the duration intervals |
1.5 point |
| 2. For mapping note pitch to the octave and key, and for
mapping the octave to sizes and keys to colours |
2.5 points |
3. For creating shapes matching the note pitch and
randomly
placing them on the frame without going "overbaord" |
2.5 points |
| 4. For implementing the aging (fading note-shapes) effect |
1.5 point |
| 5. For numbering, naming and writing to frame files |
1.5 point |
| 6. For implementing advanced shapes non-overlapping effect |
3.5 points |
Last modified: 17/10/2008, 21:05
