Week 7: sound & music

Dr Charles Martin

Semester 2, 2020

admin

assignment 2 marks released

assignment 3 is due Tuesday week 9

on the horizon: week 9 MP storyboard lab

sound

why should you, the code artist, care about sound?

sound demands attention

sound communicates emotion

enhances realism & immersion

provides feedback during interaction

types of sound

musical

abstract

skeuomorphic/sfx/diagetic

sonic interaction design: key questions

  1. what sound to make?

  2. when to start & stop?

  3. how high (or low)?

  4. how loud?

sound lingo

pitch: high or low (measured in Hz) ranging from 40Hz (low bass rumble) to 20kHz (high-pitched buzz)

amplitude: AKA volume or loudness, usually ranging from 0.0 (silence) to 1.0 (full volume)

duration: long or short, measured in seconds (it’s just time!)

timbre: often called the “colour” or “quality” of a sound (e.g. piano & saxophone can play the same pitch, but have different timbre)

what/when/how high/how loud

timbre (i.e. what)

in computer music (and in p5.sound), there are two ways to make a sound:

  • synthesis (mathematically calculating the signal)

  • sampling (playback of recorded sound)

Synthesis

Image: Wikipedia

Sampling

Image: Tascam

control (i.e. when)

Lots of options:

direct control through an input device (e.g. keyboard)

time-based

scene-based (e.g. connected to events in the sketch)

random

starting & stopping

pitch (i.e. how high/low)

different pitches give us music

pitch can also communicate other things e.g. size, mood

amplitude (i.e. how loud)

slow changes in loudness—build tension, think verse vs chorus

quicker changes in loudness—accents in music

difference between loudest & softest sound is called the dynamic range—make use of it!

interaction

connect sound input to action on the screen

connect keyboard/mouse to sounds

connect camera/accelerometer to sounds

can we think about this as a new musical instrument?

sound interaction examples

Touch Pianist

Magenta.js Demos

Repeater Orchestra (1.0)

ANU Laptop Ensemble

Get started making music at ableton.com

let's make some sounds!

synthesis in p5!

sampling in p5!

interactive music in p5!

sound is different to graphics

when we say frame rate, what are we talking about?

what’s the p5 draw() loop frame rate? how would we find out?

what’s the normal sound frame rate?

44100Hz–96000Hz

44100 vs 60: that’s a huge difference (735x!)

making sound in p5

p5 has a sound library: it’s called p5.sound

note: there’s no draw() loop for sound to directly program frames of sound (it would be too slow)

instead, p5.sound provides a bunch of sound “building blocks” (which are just objects!) which you need to create and modify.

You need to make sure the p5.sound library is loaded in your index.html

building blocks

sources produce sound: e.g., playing a sound file, “synthesising” a sine wave

processors process sounds: e.g., bass boost, reverb

sources: synthesis

p5.sound provides a p5.Oscillator which creates a basic sound signal—here’s a simple example sketch

let osc;

function setup() {
  osc = new p5.Oscillator();
  osc.setType("sine"); // or "triangle", "sawtooth", "square"
  osc.start();
}

function draw() {
  osc.freq(1000*mouseY/height);
  osc.amp(mouseX/width);
}

basic synthesis parameters

// assign p5.Oscillator object to variable
osc = new p5.Oscillator();

remember the key sonic interaction design questions:

  1. what sound to make? use the .setType() method
  2. when to start & stop? start with .start() method
  3. how high/low? use the .freq() method
  4. how loud? use the .amp() method

look at the reference for the full list of properties/methods

sources: sampling

loadSound() returns a p5.SoundFile object useful for playing pre-recorded sounds (e.g. mp3s, wav files, etc.)

let shimmer;
function preload() {
  shimmer = loadSound("assets/shimmer.wav");
}

function setup() {
  shimmer.setVolume(0.5);
  shimmer.play();
}

basic sampling parameters

// assign p5.SoundFile object to variable
shimmer = loadSound("assets/shimmer.wav");
  1. what sound to make? depends on the sound file
  2. when to start & stop? start with the .play() method, stop when file ends (maybe)
  3. how high/low? use the .rate() method
  4. how loud? use the .setVolume() method

note the subtle differences: the p5.SoundFile object (sampling) has different methods to the p5.Oscillator object (synthesis)

the preload() function

we’ve talked about setup() and draw()

but in some of the examples (again, especially the sound/image ones) there’s a preload() function…

let’s check the reference

necessary because of the way the web browser works

talk

where should shimmer.play() go?

function preload(){
  // here?
}

function setup(){
  // here?
}

function draw(){
  // here?
}

function mousePressed(){
  // here?
}

the rest of the p5.sound library

the reference has the full list, but some highlights:

so now I know all about sound?

nope, sorry! we could have a whole course on computer music, but you know how to make some basic sounds in p5

let’s dig a bit deeper and look at making some interactive music.

Let’s look at a bit of synth theory…

Amplitude Envelope

An enveloped sound

  • Amplitude is the “volume” of our note.
  • Envelope is the chunk of time for our note to exist in.
  • We can change the amplitude over the envelope to give a note a sonic “shape”.
  • In synth lingo, an EG (envelope generator) makes envelopes.

ADSR Envelope

The ADSR Envelope

  • The adsr shape is often used for pitched sounds.
  • ADSR: attack, decay, sustain, release

Additive Synthesis

Adding two sine waves together

  • Take multiple oscillators and add them together!
  • Need lots of oscillators to make complex sound.

Subtractive Synthesis

Applying a low pass filter to a square wave

FM synthesis

An frequency modulated sound

  • “frequency modulation”
  • Use one oscillator to control the frequency of another.
  • Cool sounds with few oscillators (see Yamaha DX7)

Let's make some interactive music

We need to connect some events in time to an interactive input…

further reading/watching

look at the sound examples on the p5.js examples page

https://freesound.org/ for CC-licenced sound files

Facebook: how to enhance mobile interactions with sound design

Some of the Best Music Pens on CodePen (not in p5.js, but still cool and useful for inspiration)

questions?